Google corporate IT builds before buying

Google manages fleet of internal employee computers with home built and open source software

Bucking the corporate practice of buying instead of building internally, Google's corporate IT department will typically build management software itself, or adopt an open source software package, before investigating the feasibility of purchasing proprietary software.

"In the long run, it is cheaper to build and not buy," said Justin McWilliams, a software engineer in Google's corporate engineering department, which provisions and manages computers and other technology for Google employees. McWilliams shared some of the company's practices at the O'Reilly Open Source Conference (OSCON), being held this week in Portland, Oregon. "We typically don't default to buying a commercial offering. We think about building it from scratch first, or look to the open source world," he said.

Google uses a number of home-built or modified open source programs for IT management, including software for full disk encryption (FDE), remote computer management, compliance management, virtual private networks (VPN), video teleconferencing, and for single sign on (SSO).

Over the past few decades, IT departments at large organizations have learned to purchase commercial, off-the-shelf software to manage their infrastructure, typically because it is less expensive than writing and maintaining the software in-house. Due to a number of factors, however, this approach does not work well at Google, McWilliams explained.

"Even when we buy we still have to build on top of what we bought in order to be effective within Google. We want all of our systems to communicate with one another. Otherwise, we'd just have all these silos of data," McWilliams said. The cost of employing engineers to write and maintain code is still more cost-effective than maintaining costly support contracts with IT management software providers, McWilliams said.

One key reason behind this build-first philosophy is that Google is a rapidly growing company. The company currently has over 32,000 employees, almost twice as many as it did in 2008. Because of this rapid growth, the company's IT staff, which is not growing at the same pace, has to keep scalability in mind when setting up operations. "We have to find other ways to scale. We try to scale by building [in] automation and self-service, as opposed to just throwing more people at the problem," McWilliams said. Typically, use of commercial software can not scale at such a dramatic rates, at least not in an economically feasible way.

Like most organizations, machine management has been a challenge for Google. Google engineers get their choice of operating systems on their work machines, either Apple's OS X, Google's own Chrome, one of several distributions of Linux, or Microsoft Windows. The Apple machines in particular have been hard to manage, given the limited tools available from Apple and third parties for enterprises. By McWilliams estimate, Google has one of the largest corporate IT deployments of Macs in the world, with over 30,000 units now in use. "That creates a lot of challenges for us," McWilliams said.

To push patches and software updates to the Macs, Google initially used Puppet, an open source configuration management tool. The organization quickly ran into scaling problems, however. It looked at commercial solutions, though most charge about $100 per machine per year. Additional Web servers, file servers and load balancers would also have to be deployed. "It would have cost us several million a year for the infrastructure and licensing," McWilliams said.

The company finally found an answer for its Mac support issues in open source software called Munki, which was developed by an engineer at Walt Disney Animation Studios. McWilliams' team had deployed Munki on Google App Engine, which meant they did not have to manage any additional physical servers to run the software. "We have days where we are pushing out over six terabytes of traffic, or thousands and thousands of updates," he said.

Encrypting Macintosh disks was another task that Google tackled without the use of commercial software. OS X Lion 10.7 offered built-in FDE with a program called FileVault 2, but it had some issues for corporate users. For instance, the software doesn't force users to encrypt the disks, nor does it offer an escrow repository for storing keys, other than one provided by Apple itself. So Google developed its own software in-house, called Cauliflower Vest (an anagram of the phrase "FileVault Escrow"), which provides a companywide escrow service.

"When a Googler forgets the password, an admin can fetch the recovery key, unlock the hard drive and reset the password," McWilliams said. As with Munki, Cauliflower Vest runs on the Google App Engine hosted service.

Another piece of open source software the corporate engineering uses is OpenVPN, which the company chose because "we could extend it and adopt it to our environment," McWilliams said. The company modified the software so that when employees log in, they can access internal Web sites without re-authenticating on those sites, thanks to a certificate VPN provided to the browser upon log-in. "With other proprietary software, we probably wouldn't have been able to do that," McWilliams said.

Google also has what is probably the world's largest civilian video conferencing network, which was built on top of Google+ Hangouts. Even Google's telephony software is open source: The company uses FreeSwitch for telephone routing for its call centers.

Interestingly, Google's approach to scalability is not limited to IT. According to McWilliams, the company also runs California's largest private fleet of buses, shuttles that covey employees to and from work. Each bus is equipped with WiFi, so "Googlers can remain productive while they are commuting," he said.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments

Father’s Day Gift Guide

Brand Post

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Luke Hill

MSI GT75 TITAN

I need power and lots of it. As a Front End Web developer anything less just won’t cut it which is why the MSI GT75 is an outstanding laptop for me. It’s a sleek and futuristic looking, high quality, beast that has a touch of sci-fi flare about it.

Emily Tyson

MSI GE63 Raider

If you’re looking to invest in your next work horse laptop for work or home use, you can’t go wrong with the MSI GE63.

Laura Johnston

MSI GS65 Stealth Thin

If you can afford the price tag, it is well worth the money. It out performs any other laptop I have tried for gaming, and the transportable design and incredible display also make it ideal for work.

Andrew Teoh

Brother MFC-L9570CDW Multifunction Printer

Touch screen visibility and operation was great and easy to navigate. Each menu and sub-menu was in an understandable order and category

Louise Coady

Brother MFC-L9570CDW Multifunction Printer

The printer was convenient, produced clear and vibrant images and was very easy to use

Edwina Hargreaves

WD My Cloud Home

I would recommend this device for families and small businesses who want one safe place to store all their important digital content and a way to easily share it with friends, family, business partners, or customers.

Featured Content

Product Launch Showcase

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?