AWS comes clean about recent Sydney outage

Public Cloud provider said both primary and backup power failed

Amazon Web Services has highlighted the issues behind its power-related outage in its Sydney availability zone on Sunday night.

At 4pm Sydney time, the company reported the power issue at its Sydney region datacentres delivering its EC2 and S3 services.

The blackout lead to disruption for Sydney citizens and AWS clients as the outage took out major websites such as Foxtel Play, Channel Nine, Domain and Domino’s Pizza.

Partners including Comunet, Bulletproof, RXP Services and Strut Digital were also affected as many worked through the night with clients to work through business-critical challenges.

In a recent blog post, AWS explained how every instance is served by the main utility power and a backup generator diesel rotary uninterruptible power supply (DRUPS), as two independent power delivery sources.

AWS said if either source provides power, the instance will maintain availability as the DRUPS as the secondary source, stores power and starts up if the main utility power is compromised.

However, during the severe weather, the instances that lost power lost access to both primary and secondary powers and consequently, the backup generator could not start up.

AWS described the power failure as an ‘unusually long voltage sag’, as opposed to ‘a complete outage’ and said that the unexpected nature of the voltage sag caused the set of breakers responsible for isolating the DRUPS from utility power, fail to open fast enough.

“Normally, these breakers would assure that the DRUPS reserve power is used to support the datacenter load during the transition to generator power. Instead, the DRUPS system’s energy reserve quickly drained into the degraded power grid,” the company explained.

“The rapid, unexpected loss of power from DRUPS resulted in DRUPS shutting down, meaning the generators which had started up could not be engaged and connected to the datacenter racks. DRUPS shutting down this rapidly and in this fashion is unusual and required some inspection.”

In remediation, AWS said it will add additional beakers to assure a quicker break to connections to degraded utility power to allow the generators to activate before the UPS systems are depleted.

The company added that it will also make fixing the ‘latent bug’ that disabled the automatic recovery systems in customer instances, a priority.

AWS said more than 80 per cent of the impacted customer instances and volumes were online and operational by 1 am PDT after power was restored at 11:46 am PDT.

According to Comunet chief executive, Mark Ogden, 100 of his clients in total were affected and issues across all clients, bar one, were resolved in three hours.

However, this was not the case for all. Strut Digital chief executive, Zack Levy, told ARN that his engineers were still restoring services at 3:30 am.

“We apologise for any inconvenience this event caused. We know how critical our services are to our customers’ businesses. We are never satisfied with operational performance that is anything less than perfect, and we will do everything we can to learn from this event and use it to drive improvement across our services,” AWS said.

AWS channel partners recently told ARN the interruption has proven that business should consider reviewing their architecture model and strategy before considering jumping on the Cloud bandwagon.


Join the PC World newsletter!

Error: Please check your email address.

Tags domainZack LevyWestpacAWSStrut DigitalATMcommonwealth bankrxp services

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.
Holly Morgan
Show Comments

Cool Tech

Crucial Ballistix Elite 32GB Kit (4 x 8GB) DDR4-3000 UDIMM

Learn more >

Gadgets & Things

Lexar® Professional 1000x microSDHC™/microSDXC™ UHS-II cards

Learn more >

Family Friendly

Lexar® JumpDrive® S57 USB 3.0 flash drive 

Learn more >

Stocking Stuffer

Plox Star Wars Death Star Levitating Bluetooth Speaker

Learn more >

Christmas Gift Guide

Click for more ›

Most Popular Reviews

Latest News Articles

Resources

GGG Evaluation Team

Kathy Cassidy

STYLISTIC Q702

First impression on unpacking the Q702 test unit was the solid feel and clean, minimalist styling.

Anthony Grifoni

STYLISTIC Q572

For work use, Microsoft Word and Excel programs pre-installed on the device are adequate for preparing short documents.

Steph Mundell

LIFEBOOK UH574

The Fujitsu LifeBook UH574 allowed for great mobility without being obnoxiously heavy or clunky. Its twelve hours of battery life did not disappoint.

Andrew Mitsi

STYLISTIC Q702

The screen was particularly good. It is bright and visible from most angles, however heat is an issue, particularly around the Windows button on the front, and on the back where the battery housing is located.

Simon Harriott

STYLISTIC Q702

My first impression after unboxing the Q702 is that it is a nice looking unit. Styling is somewhat minimalist but very effective. The tablet part, once detached, has a nice weight, and no buttons or switches are located in awkward or intrusive positions.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?