White House CIO's first 40 days included 'worst day' ever

Colangelo started his job the same day as President Obama, found IT assets 'in pretty bad shape'

PHOENIX - Here are two facts about Brook Colangelo 's job as the CIO of the Executive Office of President.

On taking his new position on inauguration day, he and his staff put in 80 hour weeks, "if not more." And in his first 40 days on the job, the White House email system was down 23% of the time.

Colangelo began his job on Jan. 20, 2009, the same day that President Barack Obama started his.

On that first day, Colangelo walked to the White House, found his office with some difficulty, and then "delivered the first presidential Blackberry," as well as handhelds to all the top administration officials. "It was just a mind blowing experience," he said.

But Colangelo quickly realized that the White House's IT assets "were in pretty bad shape."

Over 82% of the White House's technology had reached end of life. Desktops, for instance, still had floppy disk drives, including the one Colangelo delivered to Rahm Emanuel, Obama's then chief of staff and now Mayor of Chicago.

The White House CIO office had one data center, said Colangelo. "We had no redundancy," he said, before attendees of Computerworld Premier 100 IT Leaders Conference here.

The problems became apparent on Jan 26, six days after the administration was sworn in.

"Our email servers went down for 21 hours," said Colangelo. "In my professional career, there has not been a worst day since or ever."

At 5:30 a.m. on the morning of that already long outage, Colangelo was called over the West Wing to brief chief of staff Emanuel about it.

"I was walking (to the meeting) with some other leadership. It was pitch black and I haven't gone home, and then the most amazing thing happened," said Colangelo. "As my two feet hit the door of the West Wing, my Blackberry started to buzz. I normally hate that feeling, but I got to tell you it was the best feeling I ever felt."

The email was back up.

By the time Colangelo reached Emanuel's office, the meeting with the chief of staff was no longer needed.

Colangelo said the White House technology situation called for a massive review of technology, people and processes to determine the situation.

The White House faced three or four more outages in the next 30 days or so.

"When it was all said and done, in the first 40 days of the administration we were down 23% of the time," said Colangelo.

The White House CIO provides all unclassified services to nearly a dozen business units, including, the Office of Management and Budget and the Office of the U.S. Trade Representative.

Some of Colangelo's previous jobs included working as the CIO of the Democratic Convention Committee, and IT project manager for the American Red Cross Hurricane Recovery Program.

As part of the process to upgrade White House IT operations, Colangelo held town halls with their customers, some whom were angry. "They had floppy drives -- I knew what they were going to say," he said.

The White House needed to replace much of the technology, including the email systems and storage area networks.

As part of the effort, the White House created new positions, including the "GOALIE," which is the Government Operations and Lead for Inspection and Execution. GOALIE employees worked 12 hour shifts to give 24 hour coverage of the data center and manage any crisis. Previously, the data center was only staffed from 9-to-5.

A year ago, in Feb. 2011, the White House suffered a nine-hour outage when its Sonet networking ring was cut, interrupting email and Internet access.

"I had to fax updates to the president while on the road - not a great thing to do," said Colangelo. "But it proved to be a very beneficial thing."

Colangelo said the outage allowed him to focus attention on the need for a disaster recovery data center. With a second data center accessible, "none of these issues would have happened," he said.

The White House now has a data recovery data center for its unclassified systems, which includes redundant email servers.

The White House IT office also worked to give mobile capability to staff.

The tech staff developed a Web-based portal to allow staff to access email and other services from home, "in a secure and records managed way," said Colangelo.

The mobility effort paid dividends when Washington was shut down for nearly a week in Feb. 2010 by two back-to-back blizzards that dumped several feet of snow. The White House had 60% of its staff online the entire week, said Colangelo.

The White House has since rolled out access to tablets for those employees who bring their own, and has rolled out support for more smartphones. It also rebuilt WhiteHouse.gov.

"Our modernization program was very successful," said Colangelo. It included increasing Internet speeds over 300%, as well as reducing the number of assets at their end of life by over 50%.

Along with an absence of new technology and redundant systems, the White House operations also lacked automation tools for many record keeping tasks and processes.

Colangelo was also open to new ideas.

An intern in the Council of Economic Advisors, who had a computer science background but was working on projects unrelated to his training, sparked the effort. When asked to update a spreadsheet or a memo, he automated it using macros or some other type of technology to make it move faster.

The intern convinced Colangelo that there was a great need for automation.

This led to new initiative. There wasn't money in the IT budget to hire professional software engineers. Instead, they created a program to hire computer science savvy interns to work on the project.

The interns met with the business customers, and it led to new approaches "for small and really painful problems for customers," said Colangelo.

In eight weeks, 40 applications were built, including a parking application that had previously been done in Excel, and a printer dashboard to tell them when a printer was running out of toner. The program also led to the development of survey tools, memo generators, and an online tool that replaced paper-based goods and services processing.

"I'm proud of what we've done here," said Colangelo. Among the things he believed helped him to make progress was to "never let a bad day go to waste."

"Every time we had a bad day, we were able to articulate our vision for the IT systems," said Colangelo, and to show leadership why they really wanted certain things, such as a disaster recovery center to prevent email outages.

Patrick Thibodeau covers SaaS and enterprise applications, outsourcing, government IT policies, data centers and IT workforce issues for Computerworld. Follow Patrick on Twitter at @DCgov , or subscribe to Patrick's RSS feed . His e-mail address is pthibodeau@computerworld.com .

Read more about it in government in Computerworld's IT in Government Topic Center.

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Patrick Thibodeau

Computerworld (US)
Show Comments

Cool Tech

Toys for Boys

Family Friendly

Stocking Stuffer

SmartLens - Clip on Phone Camera Lens Set of 3

Learn more >

Christmas Gift Guide

Click for more ›

Brand Post

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Aysha Strobbe

Microsoft Office 365/HP Spectre x360

Microsoft Office continues to make a student’s life that little bit easier by offering reliable, easy to use, time-saving functionality, while continuing to develop new features that further enhance what is already a formidable collection of applications

Michael Hargreaves

Microsoft Office 365/Dell XPS 15 2-in-1

I’d recommend a Dell XPS 15 2-in-1 and the new Windows 10 to anyone who needs to get serious work done (before you kick back on your couch with your favourite Netflix show.)

Maryellen Rose George

Brother PT-P750W

It’s useful for office tasks as well as pragmatic labelling of equipment and storage – just don’t get too excited and label everything in sight!

Cathy Giles

Brother MFC-L8900CDW

The Brother MFC-L8900CDW is an absolute stand out. I struggle to fault it.

Luke Hill

MSI GT75 TITAN

I need power and lots of it. As a Front End Web developer anything less just won’t cut it which is why the MSI GT75 is an outstanding laptop for me. It’s a sleek and futuristic looking, high quality, beast that has a touch of sci-fi flare about it.

Emily Tyson

MSI GE63 Raider

If you’re looking to invest in your next work horse laptop for work or home use, you can’t go wrong with the MSI GE63.

Featured Content

Product Launch Showcase

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?