Facebook rolls out storage system to handle photo stores

Homegrown system, Facebook Haystack, built to handle multiplying files of user photos

Needing to better deal with 50 billion files worth of photos, engineers at Facebook are installing a new photo storage system they say is 50% faster than traditional systems.

The storage system, dubbed Haystack, has been under development in-house for the past couple of years and Facebook has been rolling it out in limited test versions to parts of the network for the past few months. The company expects to use Haystack to store all Facebook photos by next week, according to Bobby Johnson, director of engineering at Facebook.

And Jonathan Heiliger, vice president of technical operations, told Computerworld today that based on their tests, Haystack is more than 50% faster than traditional photo storage systems.

"In terms of cost, if it's twice as efficient, we can have 50% less hardware," said Johnson. "With 50 billion files on disk, the cost adds up. It's essentially give us some [financial] headroom."

Johnson and Heiliger said they began building the new storage system to better handle the growing number of photos Facebook has to store. Many of their 175 million and 200 million users share photos of everything from their pets to vacations, weddings to days at the beach. That means that users are posting and calling up their own photos, as well as their friends' and family members' photos, as well. Keeping those trains running efficiently and on time was a growing challenge.

Johnson noted that Facebook deals with 15 billion photos - not including all of the replications. Their user data grows by 500GB a day. And they have 50 million requests per second to their backend servers.

A spokesman for Facebook said more specifics about the system will be released in a few weeks.

Johnson, though, said the new system is so much faster than the previous one because of changes made to its setup. Haystack is tailored for small files that don't change very often, instead of for a small number of large files that are changing all the time. Traditional file directories also need file names and a lot of resource cost goes to just finding the files. With the new system, however, he said they don't have to deal with directory structures or file names. They use ID numbers instead of names. That mapping is very small.

Johnson said that so far the rollout of the new system has gone very smoothly.

Five-year-old Facebook's user base passed one-time leader MySpace last year, according to a recent report.

Facebook, once thought of as the up-and-coming social network, had almost 222 million unique visitors last month, while MySpace came in at 125 million, according to online researcher comScore Inc. That's a dramatic change since the Facebook-MySpace race for unique visitors was a near dead heat in April 2008.

The company is closing in on a big milestone -- 200 million users, executives said today.

Join the PC World newsletter!

Error: Please check your email address.

Tags storageonline storageFacebook

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Sharon Gaudin

Computerworld
Show Comments

Most Popular Reviews

Latest News Articles

Resources

PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?