Facebook heat maps pinpoint trouble spots

A Facebook engineer developed heat-map technology to quickly identify server, rack or cluster failures

Faced with the challenge of overseeing the health of large caching systems, a Facebook engineer developed heat-map software to quickly pinpoint problems in the social network's data centers.

The visualization monitoring tool, called Claspin, uses the heat map format to portray the working status of Facebook's servers.

"As Facebook grew both in size and complexity, it became more and more difficult to figure out which piece was broken when something went wrong," wrote Sean Lynch, an engineer with Facebook's cache performance team, in a blog entry explaining how he developed the technology.

The idea of using heat maps in overseeing data center operations is an emerging one. At least one Oracle engineer has investigated ways of using heat maps to quickly convey potential problems in the data center.

Whenever the popular social networking service experiences technical difficulties, the cache performance group must make sure that the caching mechanisms are not the problem, or part of the problem. A heat map could be an efficient way of representing operational status of a large number of components. Each component is represented as a cell on a large matrix, and the color of the cell represents the health of the component. A green cell may represent a node that is operating within acceptable bounds, while a red cell may represent one not operating correctly.

Facebook uses two major cache systems. One uses Memcache, and the other relies on a caching graph database called TAO.

Both of these systems produce copious performance metrics -- on various latency, request rate, and error rate statistics. According to Lynch, the caching team was already using a generic heat map to monitor performance. The software, however, could not easily fit the visual data into a single screen. The colors the heat map software used to represent different values offered little intuitive indication of whether a server was performing adequately. And the software didn't interpret the source data in a way that could immediately indicate whether an individual host was running within acceptable bounds.

Lynch designed Claspin, named after a protein that monitors for DNA damage in cells, so that each cluster of servers would get its own heat map, ordered by the rack number within a data center. So problems at the rack level or at the cluster level would become readily apparent by simply viewing the heat map.

"On a 30-inch screen we could easily fit 10,000 hosts at the same time, with 30 or more stats contributing to their color, updated in real time--usually in a matter of seconds or minutes," Lynch said. The code to parse and compile the operational metrics was written in JavaScript, and the heat maps were rendered using the SVG format.

With this heat map, a black box indicates an individual host is down. A green block means the host is performing adequately and a red box means that some aspect of the operation, such as a large number of timeouts, is beyond acceptable levels. In addition to providing a visual glimpse into operations, Claspin allows the users to drill down to specific metrics, by running the mouse pointer over a specific host.

Thus far, the tool has been a success within Facebook, Lynch noted. Additional engineering groups have also started using the heat maps to watch over their own servers. Also, more servers seem to be operational due to the engineers being able to consult Claspin.

"When I first deployed Claspin, the view above had a lot more red in it. By making it easier for more people to spot server issues quickly, Claspin has allowed us to catch more 'yellows' and prevent more 'reds,'" Lynch wrote. "I suppose there's no better validation of one's choice of statistics and thresholds than to have things start out red and then turn green as the service improves."

Facebook is considering releasing Claspin as open source, like it has done with numerous other internally developed tools, though releasing it as a stand-alone application might prove difficult given how deeply connected it is with other Facebook-specific infrastructure, according to the company.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments

Cool Tech

Toys for Boys

Family Friendly

Stocking Stuffer

SmartLens - Clip on Phone Camera Lens Set of 3

Learn more >

Christmas Gift Guide

Click for more ›

Brand Post

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Aysha Strobbe

Microsoft Office 365/HP Spectre x360

Microsoft Office continues to make a student’s life that little bit easier by offering reliable, easy to use, time-saving functionality, while continuing to develop new features that further enhance what is already a formidable collection of applications

Michael Hargreaves

Microsoft Office 365/Dell XPS 15 2-in-1

I’d recommend a Dell XPS 15 2-in-1 and the new Windows 10 to anyone who needs to get serious work done (before you kick back on your couch with your favourite Netflix show.)

Maryellen Rose George

Brother PT-P750W

It’s useful for office tasks as well as pragmatic labelling of equipment and storage – just don’t get too excited and label everything in sight!

Cathy Giles

Brother MFC-L8900CDW

The Brother MFC-L8900CDW is an absolute stand out. I struggle to fault it.

Luke Hill

MSI GT75 TITAN

I need power and lots of it. As a Front End Web developer anything less just won’t cut it which is why the MSI GT75 is an outstanding laptop for me. It’s a sleek and futuristic looking, high quality, beast that has a touch of sci-fi flare about it.

Emily Tyson

MSI GE63 Raider

If you’re looking to invest in your next work horse laptop for work or home use, you can’t go wrong with the MSI GE63.

Featured Content

Product Launch Showcase

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?