Google's datacentres grow too fast for normal networks, so it builds its own

Google has been developing networks for 10 years and is now sharing some details

Google has been building its own software-defined datacentre networks for 10 years because traditional gear can't handle the scale of what are essentially warehouse-sized computers.

The company hasn't said much before about that homegrown infrastructure, but one of its networking chiefs provided some details on Wednesday at Open Network Summit and in a blog post.

The current network design, which powers all of Google's datacentres, has a maximum capacity of 1.13 petabits per second. That's more than 100 times as much as the first datacentre network Google developed 10 years ago. The network is a hierarchical design with three tiers of switches, but they all use the same commodity chips. And it's not controlled by standard protocols but by software that treats all the switches as one.

Networking is critical in Google's data centers, where tasks are distributed across pools of computing and storage, said Amin Vahdat, Google Fellow and networking technical lead. The network is what lets Google make the best use of all those components. But the need for network capacity in the company's datacentre has grown so fast that conventional routers and switches can't keep up.

"The amount of bandwidth that we have to deliver to our servers is outpacing even Moore's Law," Vahdat said. Over the past six years, it's grown by a factor of 50. In addition to keeping up with computing power, the networks will need ever higher performance to take advantage of fast storage technologies using flash and non-volatile memory, he said.

Back when Google was using traditional gear from vendors, the size of the network was defined by the biggest router the company could buy. And when a bigger one came along, the network had to be rebuilt, Vahdat said. Finally, that didn't work.

"We could not buy, for any price, a datacentre network that would meet the requirements of our distributed systems," Vahdat said. Managing 1,000 individual network boxes made Google's operations more complex, and replacing a whole datacentre's network was too disruptive.

So the company started building its own networks using generic hardware, centrally controlled by software. It used a so-called Clos topology, a mesh architecture with multiple paths between devices, and equipment built with merchant silicon, the kinds of chips that generic white-box vendors use. The software stack that controls it is Google's own but works through the open-source OpenFlow protocol.

Google started with a project called Firehose 1.0, which it couldn't implement in production but learned from, Vahdat said. At the time, there were no good protocols with multiple paths between destinations and no good open-source networking stacks at first, so Google developed its own. The company is now using a fifth-generation homegrown network, called Jupiter, with 40-Gigabit Ethernet connections and a hierarchy of top-of-rack, aggregation and spine switches.

The design lets Google upgrade its networks without disrupting a datacentre's operation, Vahdat said. "I have to be constantly refreshing my infrastructure, upgrading the network, having the old live with the new."

Google is now opening up the network technology it took a decade to develop so other developers can use it.

"What we're really hoping for is that the next great service can leverage this infrastructure and the networking that goes along with it, without having to invent it," Vahdat said.

Stephen Lawson covers mobile, storage and networking technologies for The IDG News Service. Follow Stephen on Twitter at @sdlawsonmedia. Stephen's e-mail address is stephen_lawson@idg.com

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags internetGoogleNetworkingsearch engines

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Stephen Lawson

IDG News Service
Show Comments

Cool Tech

Bang and Olufsen Beosound Stage - Dolby Atmos Soundbar

Learn more >

Toys for Boys

Nakamichi Delta 100 3-Way Hi Fi Speaker System

Learn more >

Sony WF-1000XM3 Wireless Noise Cancelling Headphones

Learn more >

ASUS ROG, ACRONYM partner for Special Edition Zephyrus G14

Learn more >

Family Friendly

Philips Sonicare Diamond Clean 9000 Toothbrush

Learn more >

Mario Kart Live: Home Circuit for Nintendo Switch

Learn more >

Stocking Stuffer

Teac 7 inch Swivel Screen Portable DVD Player

Learn more >

SunnyBunny Snowflakes 20 LED Solar Powered Fairy String

Learn more >

Christmas Gift Guide

Click for more ›

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Tom Pope

Dynabook Portégé X30L-G

Ultimately this laptop has achieved everything I would hope for in a laptop for work, while fitting that into a form factor and weight that is remarkable.

Tom Sellers

MSI P65

This smart laptop was enjoyable to use and great to work on – creating content was super simple.

Lolita Wang

MSI GT76

It really doesn’t get more “gaming laptop” than this.

Jack Jeffries

MSI GS75

As the Maserati or BMW of laptops, it would fit perfectly in the hands of a professional needing firepower under the hood, sophistication and class on the surface, and gaming prowess (sports mode if you will) in between.

Taylor Carr

MSI PS63

The MSI PS63 is an amazing laptop and I would definitely consider buying one in the future.

Christopher Low

Brother RJ-4230B

This small mobile printer is exactly what I need for invoicing and other jobs such as sending fellow tradesman details or step-by-step instructions that I can easily print off from my phone or the Web.

Featured Content

Product Launch Showcase

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?