Facebook goes open source with its workhorse embedded data store

Facebook's RocksDB is an embeddable, persistent key-value store designed for fast reads and writes

Facebook's open source RocksDB can serve as a quick cache that sits between a backend database and user-facing application.

Facebook's open source RocksDB can serve as a quick cache that sits between a backend database and user-facing application.

Continuing its practice of sharing internally developed software, Facebook has released as open source RocksDB, the embedded data store the company developed to serve content to its 1.2 billion users.

The company has posted the code for the database on Github, in hopes that others, both in industry and the academic community, will refine the software. With Facebook's emphasis on scalability, RocksDB may be of interest to other Internet services and enterprises that are building high-traffic apps for customers and employees.

Related: Inside the social network Tim Campos lifts the lid on ICT at Facebook and how CIOs can provide the greatest differentiator for the enterprise.

In its newfound liberation, RocksDB joins other software that Facebook has released as open source. Facebook has posted the source code this year for the HipHop PHP virtual machine, the Presto query engine, the Flashcache caching software, and the Corona Hadoop scheduler, all of which were developed in-house.

RocksDB is not a full database of either a SQL or NoSQL variety. It has no indexing capabilities nor can it parse SQL queries. The software is a persistent key-value store designed to quickly serve material to users, according to a blog post by Dhruba Borthakur, an engineer on Facebook's database engineering team. It can also write or delete material to a database, but offers no advanced parsing capabilities.

Written in C++ as a library, RocksDB can be embedded into other applications, particularly those that need access to large sets of data with very low latency, such as a spam detection application or a search engine.

RocksDB is actually a fork of Google's LevelDB, a simple non-SQL data store for reading, writing and deleting data. Facebook, however, found that LevelDB did not perform well with data sets that could not fit into the server's working memory, so engineers modified Google's open-source code.

Facebook also modified LevelDB so that it can be run across many processor cores of a server. Because of this work, it can support extremely fast I/O: Facebook tests showed that the data store can perform 10 times faster for random writes, as well as 30 percent faster for random reads over LevelDB.

Borthakur offered a few details of how Facebook uses RocksDB in production. In one configuration, the data store is run in front of 10 solid-state drives, striped to support a million reads and writes a second. The software now manages over a petabyte of data that it regularly serves to Facebook's users.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags open sourcedatabasesFacebooksoftwareapplications

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments

Brand Post

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Luke Hill

MSI GT75 TITAN

I need power and lots of it. As a Front End Web developer anything less just won’t cut it which is why the MSI GT75 is an outstanding laptop for me. It’s a sleek and futuristic looking, high quality, beast that has a touch of sci-fi flare about it.

Emily Tyson

MSI GE63 Raider

If you’re looking to invest in your next work horse laptop for work or home use, you can’t go wrong with the MSI GE63.

Laura Johnston

MSI GS65 Stealth Thin

If you can afford the price tag, it is well worth the money. It out performs any other laptop I have tried for gaming, and the transportable design and incredible display also make it ideal for work.

Andrew Teoh

Brother MFC-L9570CDW Multifunction Printer

Touch screen visibility and operation was great and easy to navigate. Each menu and sub-menu was in an understandable order and category

Louise Coady

Brother MFC-L9570CDW Multifunction Printer

The printer was convenient, produced clear and vibrant images and was very easy to use

Edwina Hargreaves

WD My Cloud Home

I would recommend this device for families and small businesses who want one safe place to store all their important digital content and a way to easily share it with friends, family, business partners, or customers.

Featured Content

Product Launch Showcase

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?