Northrop Grumman takes blame for Va. IT services outage

Service contractor supports state's call for independent probe

Northrop Grumman today apologized for an outage that began last Wednesday and caused 26 Virginia state agencies to lose their Web services, some for more than a week.

"Northrop Grumman deeply regrets the disruption and inconvenience this has caused state agencies and Virginia citizens. Over the last week, we have deployed significant resources and our staff has worked tirelessly in unison with VITA and our partners to thoroughly verify and restore access to data and restore normal operations to the agencies," Linda Mills, president of Northrop Grumman Information Systems, said in a statement.

The Virginia Information Technologies Agency (VITA) outsources the management of its data centers to Northrop Grumman through a 10-year, $2.4 billion contract that it signed in 2005.

The outage affected 13% of the Commonwealth's file servers.

Northrop Grumman did not respond to a request for additional details surrounding the outage.

VITA's contract with Northrop Grumman has been criticized in the past for a number of project delays, cost overruns and performance problems that included other service outages.

After an audit by Virginia's Joint Legislative Audit and Review Commission last year, VITA's contract with Northrop Grumman was modified, resulting in more stringent performance requirements and greater accountability. The contract, however, also boosted payments to Northrop Grumman by $105 million over nine years.

Earlier this year, after a long history of problems within VITA, Virginia Gov. Bob McDonnell placed the agency's CIO under his direct authority, appointing former Virginia lawmaker Sam Nixon to the position. Nixon, who co-authored the legislation establishing VITA in 2003, was charged with the task of fixing the IT problems.

Gov. McDonnell has called for an outside investigation of the latest incident.

On Aug. 25, the failure of a storage area network (SAN) caused Web site outages at 26 of Virginia's 83 state agencies. VITA originally said 27 agencies had lost connectivity, but later admitted it had included Northrop Grumman's systems in that count.

As of Tuesday, all but three agency sites had been restored, leaving the commonwealth's Department of Motor Vehicles (DMV), the Department of Taxation and the state Board of Elections without services.

Yesterday, a VITA spokesman said the state's tax department was back online, but the DMV was still unable to process drivers' licenses at its customer centers. The State Board of Elections (SBE) has is now partially operational.

At the root of the outage was an EMC DMX-3 storage array -- the vendor's flagship product, according to Virginia Secretary of Technology Jim Duffey.

The EMC SAN malfunction has been blamed on a memory card failure. A backup SAN that was supposed to act as a fail-over system then also experienced problems, according to published reports.

Yesterday, Nixon said in published accounts that the failure wasn't so much the memory card as it was the system in place to back it up.

In its apology, Northrop Grumman admitted that its technology partnership with the Commonwealth of Virginia's government has "experienced its share of obstacles," but went on to say that problems of this sort are not unusual with large technology transformation programs.

"With this modernized system, Virginia and her citizens should find themselves years ahead of other states with the service provided by its IT infrastructure," the company said.

Northrop Grumman went on to say the hardware problems have been fixed, and it has begun tracking down the factors that contributed to the initial problem.

"We support Governor McDonnell's call for an independent review of the incident and we will reimburse the Commonwealth for the reasonable costs of an assessment as it is an essential and responsible measure that will strengthen our ability to protect against future issues," the company said.

The company also said it is trying to learn from the problems it experienced, and intends to improve its own internal processes so it can respond more quickly in restoring state agency operations.

Lucas Mearian covers storage, disaster recovery and business continuity, financial services infrastructure and health care IT for Computerworld. Follow Lucas on Twitter at @lucasmearian or subscribe to Lucas's RSS feed . His e-mail address is lmearian@computerworld.com .

Read more about storage in Computerworld's Storage Topic Center.

Join the PC World newsletter!

Error: Please check your email address.

Tags Northrop GrummanGovernment use of ITstorageIT in Governmentstorage hardwaregovernment

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Lucas Mearian

Computerworld (US)
Show Comments

Essentials

Lexar® JumpDrive® S57 USB 3.0 flash drive

Learn more >

Microsoft L5V-00027 Sculpt Ergonomic Keyboard Desktop

Learn more >

Mobile

Lexar® JumpDrive® S45 USB 3.0 flash drive 

Learn more >

Exec

Lexar® JumpDrive® C20c USB Type-C flash drive 

Learn more >

HD Pan/Tilt Wi-Fi Camera with Night Vision NC450

Learn more >

Lexar® Professional 1800x microSDHC™/microSDXC™ UHS-II cards 

Learn more >

Audio-Technica ATH-ANC70 Noise Cancelling Headphones

Learn more >

Budget

Back To Business Guide

Click for more ›

Most Popular Reviews

Latest News Articles

Resources

PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?