Cloudera moves Hadoop beyond MapReduce

Cloudera CDH increases the ways in which data can be analyzed on Hadoop

With the latest update to its Apache Hadoop distribution, Cloudera has provided the possibility of using data processing algorithms beyond the customary MapReduce, the company announced Tuesday.

Version 4 of Cloudera's Distribution including Apache Hadoop (CDH) also comes with a number of resiliency improvements that should allow organizations to "run more critical workloads on the system," said Charles Zedlewski, Cloudera vice president of products.

CDH4 expands the number of computational processes that can be executed under Hadoop, Zedlewski explained. Typically, Hadoop will use MapReduce, which breaks a data analysis task up across multiple nodes, and then collects the results as the nodes complete their portions of the job.

CDH4 introduces a new feature called coprocessors, which allows software programs to be embedded with the data itself. The programs are executed when certain conditions are met, such as when the average of a set of numbers hits a predefined threshold. The idea is similar to database triggers and stored procedures. The programs reside with the data, which is spread across multiple servers.

Coprocessors allow for more flexibility than a MapReduce operation. "We can now do more real-time or continuous operation on data in motion," Zedlewski said. "This allows you to push data-intensive operations into the data layer and parallelize the workload there."

CDH4 also allows users to implement their own data analysis frameworks apart from MapReduce. "You no longer have to shoehorn all your user workloads into one paradigm," Zedlewski said. "MapReduce is a very linear process, but sometimes things need to work on an iterative process."

One example of a program that could work on CDH4 is the Apache Hama, a bulk synchronous parallel computing framework that can be used for scientific calculations. Hama "can work on the same data as MapReduce. It can borrow the same CPU and memory that the MapReduce jobs use," Zedlewski said.

CDH4 comes with a number of other features as well, all of them adapted from the latest versions of the open-source components that make up the Hadoop platform, such as the HDFS file system and the HBase database system.

The new distribution tackles one of Hadoop's fundamental weaknesses, namely the file system's reliance on a single namenode to direct all traffic. A namenode keeps track of where all the data in a Hadoop cluster resides. Having only one namenode for a cluster is considered a weakness. Should that namenode stop working correctly, the entire system will be unusable. This version of CDH beats that problem by including the ability to set up a backup namenode that would automatically spring into use should the primary namenode fail.

CDH4 provides the ability to run different versions of Hadoop in a single system. This should ease the process of upgrading to a new version of Hadoop. Every node does not need to be updated at the same time. CDH4 also comes with a number of security updates. For instance, HBase now can permit access to tables and columns based on designated users and groups. This helps secure sensitive information and also permits service providers to run Hadoop for multiple clients.

The company's management software for CDH4, called Cloudera Manager, has been updated as well. The new edition, version 4, can be used to manage multiple clusters. An administrator can set up different clusters for testing and production, or offer separate clusters of different users in an enterprise. The software includes new visualization tools that use heat maps to provide a visual clue of where trouble spots may reside. This version is the first to have an API (application programming interface), so all functionality can be accessed from other programs, such as IBM's Tivoli integrated service management software.

In addition to these product releases, Cloudera also announced that it has signed with 250 partners, who offer complementary products or services based on Cloudera's version of Hadoop.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is

Join the newsletter!

Error: Please check your email address.
Rocket to Success - Your 10 Tips for Smarter ERP System Selection
Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments





Back To Business Guide

Click for more ›

Brand Post

Most Popular Reviews

Latest Articles


PCW Evaluation Team

Louise Coady

Brother MFC-L9570CDW Multifunction Printer

The printer was convenient, produced clear and vibrant images and was very easy to use

Edwina Hargreaves

WD My Cloud Home

I would recommend this device for families and small businesses who want one safe place to store all their important digital content and a way to easily share it with friends, family, business partners, or customers.

Walid Mikhael

Brother QL-820NWB Professional Label Printer

It’s easy to set up, it’s compact and quiet when printing and to top if off, the print quality is excellent. This is hands down the best printer I’ve used for printing labels.

Ben Ramsden

Sharp PN-40TC1 Huddle Board

Brainstorming, innovation, problem solving, and negotiation have all become much more productive and valuable if people can easily collaborate in real time with minimal friction.

Sarah Ieroianni

Brother QL-820NWB Professional Label Printer

The print quality also does not disappoint, it’s clear, bold, doesn’t smudge and the text is perfectly sized.

Ratchada Dunn

Sharp PN-40TC1 Huddle Board

The Huddle Board’s built in program; Sharp Touch Viewing software allows us to easily manipulate and edit our documents (jpegs and PDFs) all at the same time on the dashboard.

Featured Content

Product Launch Showcase

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?