AWS comes clean about recent Sydney outage

Public Cloud provider said both primary and backup power failed

Amazon Web Services has highlighted the issues behind its power-related outage in its Sydney availability zone on Sunday night.

At 4pm Sydney time, the company reported the power issue at its Sydney region datacentres delivering its EC2 and S3 services.

The blackout lead to disruption for Sydney citizens and AWS clients as the outage took out major websites such as Foxtel Play, Channel Nine, Domain and Domino’s Pizza.

Partners including Comunet, Bulletproof, RXP Services and Strut Digital were also affected as many worked through the night with clients to work through business-critical challenges.

In a recent blog post, AWS explained how every instance is served by the main utility power and a backup generator diesel rotary uninterruptible power supply (DRUPS), as two independent power delivery sources.

AWS said if either source provides power, the instance will maintain availability as the DRUPS as the secondary source, stores power and starts up if the main utility power is compromised.

However, during the severe weather, the instances that lost power lost access to both primary and secondary powers and consequently, the backup generator could not start up.

AWS described the power failure as an ‘unusually long voltage sag’, as opposed to ‘a complete outage’ and said that the unexpected nature of the voltage sag caused the set of breakers responsible for isolating the DRUPS from utility power, fail to open fast enough.

“Normally, these breakers would assure that the DRUPS reserve power is used to support the datacenter load during the transition to generator power. Instead, the DRUPS system’s energy reserve quickly drained into the degraded power grid,” the company explained.

“The rapid, unexpected loss of power from DRUPS resulted in DRUPS shutting down, meaning the generators which had started up could not be engaged and connected to the datacenter racks. DRUPS shutting down this rapidly and in this fashion is unusual and required some inspection.”

In remediation, AWS said it will add additional beakers to assure a quicker break to connections to degraded utility power to allow the generators to activate before the UPS systems are depleted.

The company added that it will also make fixing the ‘latent bug’ that disabled the automatic recovery systems in customer instances, a priority.

AWS said more than 80 per cent of the impacted customer instances and volumes were online and operational by 1 am PDT after power was restored at 11:46 am PDT.

According to Comunet chief executive, Mark Ogden, 100 of his clients in total were affected and issues across all clients, bar one, were resolved in three hours.

However, this was not the case for all. Strut Digital chief executive, Zack Levy, told ARN that his engineers were still restoring services at 3:30 am.

“We apologise for any inconvenience this event caused. We know how critical our services are to our customers’ businesses. We are never satisfied with operational performance that is anything less than perfect, and we will do everything we can to learn from this event and use it to drive improvement across our services,” AWS said.

AWS channel partners recently told ARN the interruption has proven that business should consider reviewing their architecture model and strategy before considering jumping on the Cloud bandwagon.


Join the PC World newsletter!

Error: Please check your email address.

Tags domainZack LevyWestpacAWSStrut DigitalATMcommonwealth bankrxp services

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.
Holly Morgan
Show Comments

Essentials

Lexar® JumpDrive® S57 USB 3.0 flash drive

Learn more >

Microsoft L5V-00027 Sculpt Ergonomic Keyboard Desktop

Learn more >

Mobile

Lexar® JumpDrive® S45 USB 3.0 flash drive 

Learn more >

Exec

Lexar® Professional 1800x microSDHC™/microSDXC™ UHS-II cards 

Learn more >

Lexar® JumpDrive® C20c USB Type-C flash drive 

Learn more >

Audio-Technica ATH-ANC70 Noise Cancelling Headphones

Learn more >

HD Pan/Tilt Wi-Fi Camera with Night Vision NC450

Learn more >

Budget

Back To Business Guide

Click for more ›

Most Popular Reviews

Latest News Articles

Resources

PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?