Google corporate IT builds before buying

Google manages fleet of internal employee computers with home built and open source software

Bucking the corporate practice of buying instead of building internally, Google's corporate IT department will typically build management software itself, or adopt an open source software package, before investigating the feasibility of purchasing proprietary software.

"In the long run, it is cheaper to build and not buy," said Justin McWilliams, a software engineer in Google's corporate engineering department, which provisions and manages computers and other technology for Google employees. McWilliams shared some of the company's practices at the O'Reilly Open Source Conference (OSCON), being held this week in Portland, Oregon. "We typically don't default to buying a commercial offering. We think about building it from scratch first, or look to the open source world," he said.

Google uses a number of home-built or modified open source programs for IT management, including software for full disk encryption (FDE), remote computer management, compliance management, virtual private networks (VPN), video teleconferencing, and for single sign on (SSO).

Over the past few decades, IT departments at large organizations have learned to purchase commercial, off-the-shelf software to manage their infrastructure, typically because it is less expensive than writing and maintaining the software in-house. Due to a number of factors, however, this approach does not work well at Google, McWilliams explained.

"Even when we buy we still have to build on top of what we bought in order to be effective within Google. We want all of our systems to communicate with one another. Otherwise, we'd just have all these silos of data," McWilliams said. The cost of employing engineers to write and maintain code is still more cost-effective than maintaining costly support contracts with IT management software providers, McWilliams said.

One key reason behind this build-first philosophy is that Google is a rapidly growing company. The company currently has over 32,000 employees, almost twice as many as it did in 2008. Because of this rapid growth, the company's IT staff, which is not growing at the same pace, has to keep scalability in mind when setting up operations. "We have to find other ways to scale. We try to scale by building [in] automation and self-service, as opposed to just throwing more people at the problem," McWilliams said. Typically, use of commercial software can not scale at such a dramatic rates, at least not in an economically feasible way.

Like most organizations, machine management has been a challenge for Google. Google engineers get their choice of operating systems on their work machines, either Apple's OS X, Google's own Chrome, one of several distributions of Linux, or Microsoft Windows. The Apple machines in particular have been hard to manage, given the limited tools available from Apple and third parties for enterprises. By McWilliams estimate, Google has one of the largest corporate IT deployments of Macs in the world, with over 30,000 units now in use. "That creates a lot of challenges for us," McWilliams said.

To push patches and software updates to the Macs, Google initially used Puppet, an open source configuration management tool. The organization quickly ran into scaling problems, however. It looked at commercial solutions, though most charge about $100 per machine per year. Additional Web servers, file servers and load balancers would also have to be deployed. "It would have cost us several million a year for the infrastructure and licensing," McWilliams said.

The company finally found an answer for its Mac support issues in open source software called Munki, which was developed by an engineer at Walt Disney Animation Studios. McWilliams' team had deployed Munki on Google App Engine, which meant they did not have to manage any additional physical servers to run the software. "We have days where we are pushing out over six terabytes of traffic, or thousands and thousands of updates," he said.

Encrypting Macintosh disks was another task that Google tackled without the use of commercial software. OS X Lion 10.7 offered built-in FDE with a program called FileVault 2, but it had some issues for corporate users. For instance, the software doesn't force users to encrypt the disks, nor does it offer an escrow repository for storing keys, other than one provided by Apple itself. So Google developed its own software in-house, called Cauliflower Vest (an anagram of the phrase "FileVault Escrow"), which provides a companywide escrow service.

"When a Googler forgets the password, an admin can fetch the recovery key, unlock the hard drive and reset the password," McWilliams said. As with Munki, Cauliflower Vest runs on the Google App Engine hosted service.

Another piece of open source software the corporate engineering uses is OpenVPN, which the company chose because "we could extend it and adopt it to our environment," McWilliams said. The company modified the software so that when employees log in, they can access internal Web sites without re-authenticating on those sites, thanks to a certificate VPN provided to the browser upon log-in. "With other proprietary software, we probably wouldn't have been able to do that," McWilliams said.

Google also has what is probably the world's largest civilian video conferencing network, which was built on top of Google+ Hangouts. Even Google's telephony software is open source: The company uses FreeSwitch for telephone routing for its call centers.

Interestingly, Google's approach to scalability is not limited to IT. According to McWilliams, the company also runs California's largest private fleet of buses, shuttles that covey employees to and from work. Each bus is equipped with WiFi, so "Googlers can remain productive while they are commuting," he said.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Join the newsletter!

Error: Please check your email address.
Rocket to Success - Your 10 Tips for Smarter ERP System Selection
Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments

Cool Tech

Breitling Superocean Heritage Chronographe 44

Learn more >

SanDisk MicroSDXC™ for Nintendo® Switch™

Learn more >

Toys for Boys

Family Friendly

Panasonic 4K UHD Blu-Ray Player and Full HD Recorder with Netflix - UBT1GL-K

Learn more >

Stocking Stuffer

Razer DeathAdder Expert Ergonomic Gaming Mouse

Learn more >

Christmas Gift Guide

Click for more ›

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Edwina Hargreaves

WD My Cloud Home

I would recommend this device for families and small businesses who want one safe place to store all their important digital content and a way to easily share it with friends, family, business partners, or customers.

Walid Mikhael

Brother QL-820NWB Professional Label Printer

It’s easy to set up, it’s compact and quiet when printing and to top if off, the print quality is excellent. This is hands down the best printer I’ve used for printing labels.

Ben Ramsden

Sharp PN-40TC1 Huddle Board

Brainstorming, innovation, problem solving, and negotiation have all become much more productive and valuable if people can easily collaborate in real time with minimal friction.

Sarah Ieroianni

Brother QL-820NWB Professional Label Printer

The print quality also does not disappoint, it’s clear, bold, doesn’t smudge and the text is perfectly sized.

Ratchada Dunn

Sharp PN-40TC1 Huddle Board

The Huddle Board’s built in program; Sharp Touch Viewing software allows us to easily manipulate and edit our documents (jpegs and PDFs) all at the same time on the dashboard.

George Khoury

Sharp PN-40TC1 Huddle Board

The biggest perks for me would be that it comes with easy to use and comprehensive programs that make the collaboration process a whole lot more intuitive and organic

Featured Content

Product Launch Showcase

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?