Amazon Elastic MapReduce
Based on Hadoop, MapReduce equips users with potent distributed data-processing tools
- Doesn't take long to get the hang of
- Currently available in the US region only
You'll want to be familiar with the Apache Hadoop framework before you jump into Elastic MapReduce. It doesn't take long to get the hang of it, though. Most developers can have a MapReduce application running within a few hours.
Have you got a few hundred gigabytes of data that need processing? Perhaps a dump of radio telescope data that could use some combing through by a squad of processors running Fourier transforms? Or maybe you're convinced some statistical analysis will reveal a pattern hidden in several years of stock market information? Unfortunately, you don't happen to have a grid of distributed processors to run your application, much less the time to construct a parallel processing infrastructure.
Well, cheer up: Amazon has added Elastic MapReduce to its growing list of cloud-based Web services. Currently in beta, Elastic MapReduce uses Amazon's Elastic Compute Cloud (EC2) and Simple Storage Service (S3) to implement a virtualized distributed processing system based on Apache Hadoop.
Hadoop's internal architecture is the MapReduce framework. The mechanics of MapReduce are well documented in a paper by J. Dean and S. Ghemawat [PDF], and a full treatment is beyond the scope of this article. Instead, I'll illustrate by example.
Suppose you have a set of 10 words and you want to count the number of times those words appear in a collection of e-books. Your input data is a set of key/value pairs, the value being a line of text from one of the books and the key being the concatenation of the book's name and the line's number. This set might comprise a few megabytes big -- or gigabytes. MapReduce doesn't much care about size.
You write a routine that reads this input, a pair at a time, and produces another key/value pair as output. The output key is a word (from the original set of 10) and the associated value is the number of times that word appears in the line. (Zero values are not emitted.) This routine is the map part of map/reduce. Its output is referred to as the intermediate key/value pairs.
The intermediate key/value pairs are fed to another function (another "step" in the parlance of MapReduce). For this step, you write a routine that iterates through the intermediate data, sums up the values, and returns a single pair whose key is the word and whose value is the grand total. You don't have to worry about grouping the results of like keys (i.e., gathering all the intermediate key/values for a given word), because Hadoop does that grouping for you in the background.
Join the PC World newsletter!
Most Popular Reviews
- 1 Google Pixel XL full, in-depth smartphone review: Phones just got smarter
- 2 Sony Xperia XZ review: turbo-charged last-gen phone
- 3 Sony X9300D and X8500D UHD 4K TV review
- 4 Hisense Series 7 ULED 4K UHD TV review
- 5 Moto X Force review: Leading features from a mid-range phone
Latest News Articles
- Largest DDoS attack ever delivered by botnet of hijacked IoT devices
- Hackers have a treasure trove of data with the Yahoo breach
- Judge paves the way for British hacker's extradition to US
- FBI faces lawsuit because it's stayed mum on iPhone 5c hack
- Banks find big innovation payoff in hackathons
GGG Evaluation Team
First impression on unpacking the Q702 test unit was the solid feel and clean, minimalist styling.
For work use, Microsoft Word and Excel programs pre-installed on the device are adequate for preparing short documents.
The Fujitsu LifeBook UH574 allowed for great mobility without being obnoxiously heavy or clunky. Its twelve hours of battery life did not disappoint.
The screen was particularly good. It is bright and visible from most angles, however heat is an issue, particularly around the Windows button on the front, and on the back where the battery housing is located.
My first impression after unboxing the Q702 is that it is a nice looking unit. Styling is somewhat minimalist but very effective. The tablet part, once detached, has a nice weight, and no buttons or switches are located in awkward or intrusive positions.
- Google Pixel XL full, in-depth smartphone review: The new best Android phone
- Japan Robot, gadget and car expo slideshow
- Panasonic DX900U UHD 4K smart TV review: Best all-round TV ever?
- What's the difference between an Intel Core i3, i5 and i7?
- Laser vs. inkjet printers: which is better?
- CCSenior Siebel DeveloperACT
- FTEmbedded Software EngineerSA
- CCContract IT Assistant (System Backup Operation) 161014/ITA/523Asia
- CCMultiple Defence Opportunities - NV2ACT
- FTSenior .Net Software EngineerVIC
- FTSenior Murex DeveloperVIC
- CCHuman Sciences Professional - DefenceNSW
- TPSenior Finance Systems AnalystVIC
- CCDigital ProducerNSW
- FTFrontend DeveloperNSW
- FTSolution Architect - Application/IntegrationVIC
- FTCapacity PlannerNSW
- CCFull Stack Developer - Be a part of the innovation programVIC
- CCSenior Java DeveloperNSW
- CCHuman Sciences Professional - DefenceVIC
- TPTraining LeadVIC
- FTJava Developer - Canberra RoleNSW
- CCHelpdesk/Desktop supportVIC
- CCProject Manager - DigitisationQLD
- TPService Desk AnalystVIC
- CCData Centre EngineerNSW
- CCSAP GRC ConsultantACT
- CCKofax DeveloperQLD
- CCeCommerce Project ManagerNSW
- CCService Now DeveloperVIC