Twitter revamps search engine

It's now much faster and scalable

Twitter has overhauled the back-end infrastructure of its search engine, boosting its speed and capacity to index posts, process queries and deliver results, while making the system more stable and better suited for the addition of new features.

Twitter transferred its search engine to the new platform in recent weeks, after working on the new back-end system for about six months, according to the company.

Twitter's search engine ran on a system based on MySQL from Summize, a company Twitter acquired in mid-2008, but scaling up the system had become difficult.

The engineering team in charge of the project decided to do an extreme makeover of the search engine using a different technology: the open-source, text search engine Lucene, which is written in Java.

Twitter modified some aspects of Lucene, including its garbage collection, query termination, posting lists, and data structures and algorithms, and was left with an inverted, index-based search engine that scales much more and has better performance.

Twitter fields 12,000 search queries per second -- or more than 1 billion per day -- and "tweets" become part of its search index less than 10 seconds after they are posted.

"We estimate that we're only using about 5 percent of the available backend resources, which means we have a lot of headroom. Our new indexer could also index roughly 50 times more Tweets per second than we currently get," Twitter official Michael Busch wrote in a blog post.

Twitter will contribute back to the Lucene project the modifications and improvements it made to the code.

Although Twitter makes available its index of "tweets" to external search engines like Google and Microsoft's Bing, its internal search engine is a key component of its microblogging service.

In addition to being the preferred vehicle among private citizens, public figures and companies for broadcasting short status updates, Twitter has become an increasingly valued repository of real-time data, tapped for following news, trends and collective musings.

To maximize the value of this "tweet" repository, the company must have a search engine that is fast, comprehensive and scalable, and its massive revamping of its search technology shows the company recognizes the importance of its internal search capabilities.

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags social mediainternettwittersearch enginesInternet-based applications and services

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Juan Carlos Perez

IDG News Service
Show Comments

Cool Tech

Toys for Boys

Family Friendly

Stocking Stuffer

SmartLens - Clip on Phone Camera Lens Set of 3

Learn more >

Christmas Gift Guide

Click for more ›

Brand Post

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Aysha Strobbe

Microsoft Office 365/HP Spectre x360

Microsoft Office continues to make a student’s life that little bit easier by offering reliable, easy to use, time-saving functionality, while continuing to develop new features that further enhance what is already a formidable collection of applications

Michael Hargreaves

Microsoft Office 365/Dell XPS 15 2-in-1

I’d recommend a Dell XPS 15 2-in-1 and the new Windows 10 to anyone who needs to get serious work done (before you kick back on your couch with your favourite Netflix show.)

Maryellen Rose George

Brother PT-P750W

It’s useful for office tasks as well as pragmatic labelling of equipment and storage – just don’t get too excited and label everything in sight!

Cathy Giles

Brother MFC-L8900CDW

The Brother MFC-L8900CDW is an absolute stand out. I struggle to fault it.

Luke Hill

MSI GT75 TITAN

I need power and lots of it. As a Front End Web developer anything less just won’t cut it which is why the MSI GT75 is an outstanding laptop for me. It’s a sleek and futuristic looking, high quality, beast that has a touch of sci-fi flare about it.

Emily Tyson

MSI GE63 Raider

If you’re looking to invest in your next work horse laptop for work or home use, you can’t go wrong with the MSI GE63.

Featured Content

Product Launch Showcase

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?