Amazon Elastic MapReduce
Based on Hadoop, MapReduce equips users with potent distributed data-processing tools
- Doesn't take long to get the hang of
- Currently available in the US region only
You'll want to be familiar with the Apache Hadoop framework before you jump into Elastic MapReduce. It doesn't take long to get the hang of it, though. Most developers can have a MapReduce application running within a few hours.
Have you got a few hundred gigabytes of data that need processing? Perhaps a dump of radio telescope data that could use some combing through by a squad of processors running Fourier transforms? Or maybe you're convinced some statistical analysis will reveal a pattern hidden in several years of stock market information? Unfortunately, you don't happen to have a grid of distributed processors to run your application, much less the time to construct a parallel processing infrastructure.
Well, cheer up: Amazon has added Elastic MapReduce to its growing list of cloud-based Web services. Currently in beta, Elastic MapReduce uses Amazon's Elastic Compute Cloud (EC2) and Simple Storage Service (S3) to implement a virtualized distributed processing system based on Apache Hadoop.
Hadoop's internal architecture is the MapReduce framework. The mechanics of MapReduce are well documented in a paper by J. Dean and S. Ghemawat [PDF], and a full treatment is beyond the scope of this article. Instead, I'll illustrate by example.
Suppose you have a set of 10 words and you want to count the number of times those words appear in a collection of e-books. Your input data is a set of key/value pairs, the value being a line of text from one of the books and the key being the concatenation of the book's name and the line's number. This set might comprise a few megabytes big -- or gigabytes. MapReduce doesn't much care about size.
You write a routine that reads this input, a pair at a time, and produces another key/value pair as output. The output key is a word (from the original set of 10) and the associated value is the number of times that word appears in the line. (Zero values are not emitted.) This routine is the map part of map/reduce. Its output is referred to as the intermediate key/value pairs.
The intermediate key/value pairs are fed to another function (another "step" in the parlance of MapReduce). For this step, you write a routine that iterates through the intermediate data, sums up the values, and returns a single pair whose key is the word and whose value is the grand total. You don't have to worry about grouping the results of like keys (i.e., gathering all the intermediate key/values for a given word), because Hadoop does that grouping for you in the background.
Join the PC World newsletter!
Most Popular Reviews
- 1 Huawei P10 smartphone review
- 2 Huawei P10 Plus phone: Full, in-depth review
- 3 Motorola Moto G5 smartphone review
- 4 Oppo A57 phone: full, in-depth review
- 5 Moto G5 Plus phone: full, in-depth review
Latest News Articles
- Microsoft shows the power of its Pen with a new Whiteboard app and other upgrades
- Wanawiki is the WannaCry fix that might save affected PCs—if you work fast
- Microsoft redesigns OneNote UI to make it more universally accessible
- Google's Standalone VR and VPS address the clutter and clumsiness of virtual reality
- The WannaCry ransomware might have a link to North Korea
PCW Evaluation Team
A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.
I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.
As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.
I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.
Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!
For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.
- Asus launches laptops to start Computex 2017
- LG 2017 OLED TV range full review: W7 Signature Wallpaper, G7, E7 and C7 UHD TVs
- Huawei P10 Plus phone: Full, in-depth review
- What's the difference between an Intel Core i3, i5 and i7?
- Laser vs. inkjet printers: which is better?
- FTSocial Media Executive / Specialist (Facebook) - online gamblingNSW
- FTState IT Systems EngineerTAS
- FTPrincipal Project ManagerVIC
- FTJunior - Mid Level Technical Customer SupportQLD
- TPAEM DeveloperNSW
- FTJunior-Mid Level Implementation CoordinatorQLD
- FTFrontend DeveloperNSW
- CCSenior Project Manger - Global MarketsVIC
- CCSolution ArchitectQLD
- FTLead Drupal DeveloperQLD
- FTAutomation Tester | C# & Selenium | Leading SaaS FintechNSW
- FTImplementation Engineer - Cisco UCSWA
- FTSenior Android DeveloperNSW
- TPEOI - Developers and Tester (APS)ACT
- FTCyber Security Technical WriterNSW
- CCBusiness Analyst - SalesforceVIC
- FTLearning & Development Consultant, Wealth ManagementNSW
- FTNetwork SpecialistNSW
- FTNV1 Cleared Software Engineer (Mid level) - Defence Projects - North Ryde areaNSW
- FTSenior Business AnalystNSW
- CCAppian Developer x 2VIC
- FTFrontend DeveloperNSW
- FTEnterprise Architect - Network and TelecommunicationsNSW
- FTNetwork EngineerWA
- TPChange ManagerVIC