Facebook tackles Hadoop Achilles' Heel

The social networking giant stores over 100 PB of data in Hadoop clusters

Running what they believe is the world's largest Hadoop-based collection of data, Facebook engineers have developed a way to circumnavigate a core weakness of the data analysis platform, that of relying on only a single name server to coordinate all operations.

Facebook engineer Andrew Ryan discussed the work-around at the Hadoop Summit, being held this week in San Jose, California. He also posted a summary of his talk on Facebook.

Facebook has what it believes is the world's largest collection of data on the Hadoop Distributed File System (HDFS), over 100 PBs worth, spread out over 100 different clusters across its data centers.

While increasingly popular for large-scale data analysis tasks, Hadoop has what is known in engineering terms as a single point of failure. While a Hadoop deployment may across hundreds or thousands of servers, the entire operation depends on a single server, called the namenode, to coordinate all the traffic among the data nodes. Should that single namenode stop operating, then the data nodes could not communicate and, in effect, the whole system would cease to function.

Facebook has estimated that resolving this weakness would cut the downtime in its data warehouse by almost half.

In order to solve this problem, Facebook created software, called Avatarnode, that can switch to a backup namenode should the primary fail for some reason. In this setup, each data node routinely sends updates to both the primary and backup namenodes. Should the primary namenode stop functioning, then the backup namenode would take over operations. The software, named after the James Cameron film "Avatar," relies on the Hadoop Zookeeper configuration management tool.

The company offers Avatarnode as open source, in the hopes that Hadoop administrators could benefit from its use. Facebook released the software in 2010, and has been pressed into production duty at the company.

"The Avatarnode is running our most demanding production workloads inside of Facebook today, and will continue to lead to substantial improvements in reliability and administration of HDFS clusters," Ryan wrote. "Moving forward, we're working to improve Avatarnode further and integrate it with a general high-availability framework that will permit unattended, automated, and safe failover."

Facebook is not alone in trying to tackle this problem with Hadoop. Both MapR and Cloudera's distributions of Hadoop also come with redundant namenode capabilities as well.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Join the newsletter!

Error: Please check your email address.
Rocket to Success - Your 10 Tips for Smarter ERP System Selection
Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments


James Cook University - Master of Data Science Online Course

Learn more >


Sansai 6-Outlet Power Board + 4-Port USB Charging Station

Learn more >



Back To Business Guide

Click for more ›

Brand Post

Most Popular Reviews

Latest Articles


PCW Evaluation Team

Andrew Teoh

Brother MFC-L9570CDW Multifunction Printer

Touch screen visibility and operation was great and easy to navigate. Each menu and sub-menu was in an understandable order and category

Louise Coady

Brother MFC-L9570CDW Multifunction Printer

The printer was convenient, produced clear and vibrant images and was very easy to use

Edwina Hargreaves

WD My Cloud Home

I would recommend this device for families and small businesses who want one safe place to store all their important digital content and a way to easily share it with friends, family, business partners, or customers.

Walid Mikhael

Brother QL-820NWB Professional Label Printer

It’s easy to set up, it’s compact and quiet when printing and to top if off, the print quality is excellent. This is hands down the best printer I’ve used for printing labels.

Ben Ramsden

Sharp PN-40TC1 Huddle Board

Brainstorming, innovation, problem solving, and negotiation have all become much more productive and valuable if people can easily collaborate in real time with minimal friction.

Sarah Ieroianni

Brother QL-820NWB Professional Label Printer

The print quality also does not disappoint, it’s clear, bold, doesn’t smudge and the text is perfectly sized.

Featured Content

Product Launch Showcase

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?