Facebook designing network fabric to meet massive performance needs

The company is developing new technologies to feed its insatiable need for data bandwidth

With more than a billion monthly active users, it's easy to imagine that most of the data travelling over Facebook's networks is delivering photos, status updates and "likes" to its end users, but that's far from the case.

The social network moves about 1,000 times as much data between the servers inside its data centers as it does from its servers out to end users, company executives said Wednesday. They talked about the challenges that this creates for Facebook and the network technologies it's developing to overcome them.

"Our traffic going from machine to machine far exceeds the traffic going from the machines out to our end users," said Jay Parikh, vice president of Infrastructure Engineering at Facebook, in an on-stage interview at the GigaOm Structure conference in San Francisco.

That's because of all the processing work Facebook does on the back end to figure out what information it needs to send to end users. The systems analyze data, rank results, and perform a myriad of other tasks to generate the pages Facebook delivers to users' smartphones and Web browsers.

The problem, as Parikh described it, is that Facebook is using network equipment and protocols that were designed for a different era: for the ISPs (Internet service providers) whose chief role was to provide connectivity for end users, rather than companies like Facebook that are delivering applications and content at massive scale.

One result is data bottlenecks. Facebook sees "flash floods, choke points, all types of scenarios," said Najam Ahmad, the director responsible for Facebook's network operations, who also talked at the Structure conference

"One of our services handles 2.5 billion operations per second; that's a lot of packets being sent over the wire," he said.

Another issue is that the network technologies aren't smart or flexible enough for the types of applications Facebook is running. "There's really no way to communicate between an application and the network today. The app just puts the packet on the [network] and hopes it gets to the other end," Ahmad said.

"We want to come up with technologies that give the app a better feel for what the network is doing -- where it has capacity, where there are problems ... where you can find a better path."

It's addressing the network challenges in a couple of ways. One is by designing new switching hardware through the Open Compute Project, a multi-company effort to build new types of equipment for Internet-scale data centers. OCP's initial goal for networks, announced in May, is to design a top-of-rack switch that's not tied to any one operating system.

Aside from that, Parikh said Wednesday, Facebook is developing a network "fabric" that is, in effect, the company's own take on software-defined networking. SDN refers to a set of technologies that, among other things, could shift control in networks out of specialized switches and routers and into software that can run on a variety of standard hardware.

For Facebook, the new fabric should mean lower operating costs and the ability to deliver new services more quickly. For end users, it should mean better services that operate more quickly, Parikh said.

Networks traditionally use a three-layer hierarchy that was designed primarily to pull data from storage systems, up through servers and out to end users. Facebook wants a "flatter" architecture that's better suited to moving data from machine to machine, Ahmad said.

"If you can build a fabric where every cabinet, every rack, is at the same level, where the connectivity between any two racks is uniform, then you have a better chance of managing the apps, and the construct becomes more logical than the physical, hierarchical-based systems that we have today," he said.

The software layer will allow Facebook to add more functionality to the network more quickly. Today, innovation in Facebook's applications is happening much more quickly than innovation in the network, Ahmad said. "There's a huge mismatch there. How can we make the network development happen much faster? That's what SDN brings."

Like Google, Facebook is also working to improve its connectivity outside the data center, by investing in the massive fiber-optic cables that carry Internet traffic between countries. It's invested in the Asia Pacific Gateway undersea cable and in a "fiber loop" that connects a data center in Sweden with other parts of Europe, Parikh said.

More than four out of five Facebook users are outside the U.S., and operating its own network can reduce costs for Facebook and allow it to increase network capacity in markets much more quickly when demand there increases. "We're talking about bringing up capacity in seconds or minutes rather than weeks or months it takes today," Ahmad said.

Facebook isn't the only online company developing new technologies to support the unique types of services it delivers. Google is well known for designing its own server hardware, to make its data centers more energy-efficient, and it developed the MapReduce software for analyzing massive data sets that eventually formed the basis for Hadoop.

James Niccolai covers data centers and general technology news for IDG News Service. Follow James on Twitter at @jniccolai. James's e-mail address is james_niccolai@idg.com

Join the PC World newsletter!

Error: Please check your email address.

Tags Internet-based applications and servicesNetworkingrouterssocial networkinginternetswitchesFacebooknetworking hardwaremanagementTraffic managementGoogle

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

James Niccolai

IDG News Service
Show Comments

Most Popular Reviews

Latest News Articles


PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?