White House CIO's first 40 days included 'worst day' ever

Colangelo started his job the same day as President Obama, found IT assets 'in pretty bad shape'

PHOENIX - Here are two facts about Brook Colangelo 's job as the CIO of the Executive Office of President.

On taking his new position on inauguration day, he and his staff put in 80 hour weeks, "if not more." And in his first 40 days on the job, the White House email system was down 23% of the time.

Colangelo began his job on Jan. 20, 2009, the same day that President Barack Obama started his.

On that first day, Colangelo walked to the White House, found his office with some difficulty, and then "delivered the first presidential Blackberry," as well as handhelds to all the top administration officials. "It was just a mind blowing experience," he said.

But Colangelo quickly realized that the White House's IT assets "were in pretty bad shape."

Over 82% of the White House's technology had reached end of life. Desktops, for instance, still had floppy disk drives, including the one Colangelo delivered to Rahm Emanuel, Obama's then chief of staff and now Mayor of Chicago.

The White House CIO office had one data center, said Colangelo. "We had no redundancy," he said, before attendees of Computerworld Premier 100 IT Leaders Conference here.

The problems became apparent on Jan 26, six days after the administration was sworn in.

"Our email servers went down for 21 hours," said Colangelo. "In my professional career, there has not been a worst day since or ever."

At 5:30 a.m. on the morning of that already long outage, Colangelo was called over the West Wing to brief chief of staff Emanuel about it.

"I was walking (to the meeting) with some other leadership. It was pitch black and I haven't gone home, and then the most amazing thing happened," said Colangelo. "As my two feet hit the door of the West Wing, my Blackberry started to buzz. I normally hate that feeling, but I got to tell you it was the best feeling I ever felt."

The email was back up.

By the time Colangelo reached Emanuel's office, the meeting with the chief of staff was no longer needed.

Colangelo said the White House technology situation called for a massive review of technology, people and processes to determine the situation.

The White House faced three or four more outages in the next 30 days or so.

"When it was all said and done, in the first 40 days of the administration we were down 23% of the time," said Colangelo.

The White House CIO provides all unclassified services to nearly a dozen business units, including, the Office of Management and Budget and the Office of the U.S. Trade Representative.

Some of Colangelo's previous jobs included working as the CIO of the Democratic Convention Committee, and IT project manager for the American Red Cross Hurricane Recovery Program.

As part of the process to upgrade White House IT operations, Colangelo held town halls with their customers, some whom were angry. "They had floppy drives -- I knew what they were going to say," he said.

The White House needed to replace much of the technology, including the email systems and storage area networks.

As part of the effort, the White House created new positions, including the "GOALIE," which is the Government Operations and Lead for Inspection and Execution. GOALIE employees worked 12 hour shifts to give 24 hour coverage of the data center and manage any crisis. Previously, the data center was only staffed from 9-to-5.

A year ago, in Feb. 2011, the White House suffered a nine-hour outage when its Sonet networking ring was cut, interrupting email and Internet access.

"I had to fax updates to the president while on the road - not a great thing to do," said Colangelo. "But it proved to be a very beneficial thing."

Colangelo said the outage allowed him to focus attention on the need for a disaster recovery data center. With a second data center accessible, "none of these issues would have happened," he said.

The White House now has a data recovery data center for its unclassified systems, which includes redundant email servers.

The White House IT office also worked to give mobile capability to staff.

The tech staff developed a Web-based portal to allow staff to access email and other services from home, "in a secure and records managed way," said Colangelo.

The mobility effort paid dividends when Washington was shut down for nearly a week in Feb. 2010 by two back-to-back blizzards that dumped several feet of snow. The White House had 60% of its staff online the entire week, said Colangelo.

The White House has since rolled out access to tablets for those employees who bring their own, and has rolled out support for more smartphones. It also rebuilt WhiteHouse.gov.

"Our modernization program was very successful," said Colangelo. It included increasing Internet speeds over 300%, as well as reducing the number of assets at their end of life by over 50%.

Along with an absence of new technology and redundant systems, the White House operations also lacked automation tools for many record keeping tasks and processes.

Colangelo was also open to new ideas.

An intern in the Council of Economic Advisors, who had a computer science background but was working on projects unrelated to his training, sparked the effort. When asked to update a spreadsheet or a memo, he automated it using macros or some other type of technology to make it move faster.

The intern convinced Colangelo that there was a great need for automation.

This led to new initiative. There wasn't money in the IT budget to hire professional software engineers. Instead, they created a program to hire computer science savvy interns to work on the project.

The interns met with the business customers, and it led to new approaches "for small and really painful problems for customers," said Colangelo.

In eight weeks, 40 applications were built, including a parking application that had previously been done in Excel, and a printer dashboard to tell them when a printer was running out of toner. The program also led to the development of survey tools, memo generators, and an online tool that replaced paper-based goods and services processing.

"I'm proud of what we've done here," said Colangelo. Among the things he believed helped him to make progress was to "never let a bad day go to waste."

"Every time we had a bad day, we were able to articulate our vision for the IT systems," said Colangelo, and to show leadership why they really wanted certain things, such as a disaster recovery center to prevent email outages.

Patrick Thibodeau covers SaaS and enterprise applications, outsourcing, government IT policies, data centers and IT workforce issues for Computerworld. Follow Patrick on Twitter at @DCgov , or subscribe to Patrick's RSS feed . His e-mail address is pthibodeau@computerworld.com .

Read more about it in government in Computerworld's IT in Government Topic Center.

Join the PC World newsletter!

Error: Please check your email address.

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Patrick Thibodeau

Computerworld (US)
Show Comments

Most Popular Reviews

Latest News Articles

Resources

PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?