Yahoo Hadoop spin-off could propel software's growth

Yahoo's engineering unit would become its own company if Yahoo pursues that option, according to the Wall Street Journal

Yahoo's reported plan to spin off its Hadoop engineering unit into a separate company, should it happen, could spur more competition in the already-growing field of providing support for this data processing framework, observers said.

On Wednesday, the Wall Street Journal reported that Yahoo is mulling the idea of setting up its Hadoop development team as a new standalone company, one that would continue to develop the software and offer Hadoop-related consulting services.

"This is another good sign that the Hadoop space continues to grow," said Justin Borgman, co-founder and CEO of Hadapt, which provides tools for bridging Hadoop with data warehouses. "This will put the pressure on other Hadoop companies. The more competition the better."

Hadoop is "the biggest movement in enterprise software in years," Benchmark Capital partner Rob Bearden told the Wall Street Journal. Forrester Research senior analyst James Kobielus has estimated that the market for Hadoop products and services could grow to US$1 billion per year.

Hadoop can process data that resides across multiple servers, making it suitable for analyzing larger amounts of information than the typical data warehouse can handle.

Yahoo has been instrumental in developing the Hadoop data processing platform, and it devotes considerable engineering resources to the project. Yahoo uses the technology for the data processing and analysis needed to personalize content and advertisements for users. Other giant Internet services, such as eBay, Facebook and Twitter, also use the technology.

"Hadoop is the platform we run the company on. It's at the core of what we do," said Todd Papaioannou, Yahoo's vice president of cloud computing, in an interview with IDG last month. Yahoo did not immediately respond to a request for comment for this story.

Doug Cutting, the creator of Hadoop, joined Yahoo in 2006 to help the company develop the technology. He is now with Cloudera, which distributes a version of Hadoop. The Apache Foundation oversees Hadoop, which is an open-source project, though Yahoo contributes a sizeable portion of the new code for the project.

Designating its Hadoop team as its own entity makes sense for Yahoo, Hadapt's Borgman noted. Yahoo is primarily an Internet media company, and the market for Hadoop would be enterprises, a market Yahoo does not currently cater to.

"Maybe they feel that by spinning it out in a different company, it will have a better opportunity to pursue a different line of business," Borgman said.

Should the spin-off take place, the new entity would be one of a growing number of companies supporting Hadoop. Cloudera offers its own distribution of Hadoop. DataStax offers a version paired with its Cassandra data store. IBM has packaged Hadoop into its analytics offerings, and even used the framework as part of its Watson Jeopardy-playing supercomputer.

Kobielus predicted that the spin-off entity would not create its own distribution of Hadoop, given that Yahoo discontinued its own distribution earlier this year. Instead the new company would focus on consulting, professional services and systems integration for enterprises.

The possible spin-off could also trigger further investment in the technology, Kobielus said. Venture capitalists might fund more Hadoop-based startups, which traditional enterprise data warehouse (EDW) vendors might snap up for themselves.

"Forrester strongly expects that many of the startups already in the Hadoop market will be acquired by established EDW and analytics vendors who need to bootstrap their efforts in this new arena," Kobielus said.

Companies such as Oracle, Teradata, Microsoft and Hewlett-Packard could be taking a hard look at Hadoop-focused startups such as Cloudera, Karmasphere, Datameer, Hadapt and HStreaming, Kobielus wrote.

Despite all this interest, work still needs to be done to bring Hadoop to enterprise IT departments, which may not have the engineering expertise to implement the technology to run the software in-house, Kobielus noted. It has no reference framework providing the core specifications, APIs (application programming Interfaces) or code base. It still lacks some basic features, such as real-time analysis. Also, there are not yet enough modeling tools that could translate the data integration logic in Hadoop components such as MapReduce and Pig into terms that can be understood by business intelligence analysts.

Despite these limitations, Hadoop is "extremely close" to be being ready for the enterprise, Borgman said. "We're on the cusp of a really interesting new world of big data analytics."

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags business issuesbusiness intelligencesoftwareapplicationsrestructuringdata miningYahoo!Mergers / acquisitions

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments

Cool Tech

Toys for Boys

Family Friendly

Stocking Stuffer

SmartLens - Clip on Phone Camera Lens Set of 3

Learn more >

Christmas Gift Guide

Click for more ›

Brand Post

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Aysha Strobbe

Microsoft Office 365/HP Spectre x360

Microsoft Office continues to make a student’s life that little bit easier by offering reliable, easy to use, time-saving functionality, while continuing to develop new features that further enhance what is already a formidable collection of applications

Michael Hargreaves

Microsoft Office 365/Dell XPS 15 2-in-1

I’d recommend a Dell XPS 15 2-in-1 and the new Windows 10 to anyone who needs to get serious work done (before you kick back on your couch with your favourite Netflix show.)

Maryellen Rose George

Brother PT-P750W

It’s useful for office tasks as well as pragmatic labelling of equipment and storage – just don’t get too excited and label everything in sight!

Cathy Giles

Brother MFC-L8900CDW

The Brother MFC-L8900CDW is an absolute stand out. I struggle to fault it.

Luke Hill

MSI GT75 TITAN

I need power and lots of it. As a Front End Web developer anything less just won’t cut it which is why the MSI GT75 is an outstanding laptop for me. It’s a sleek and futuristic looking, high quality, beast that has a touch of sci-fi flare about it.

Emily Tyson

MSI GE63 Raider

If you’re looking to invest in your next work horse laptop for work or home use, you can’t go wrong with the MSI GE63.

Featured Content

Product Launch Showcase

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?