Amazon Elastic MapReduce
Based on Hadoop, MapReduce equips users with potent distributed data-processing tools
- Doesn't take long to get the hang of
- Currently available in the US region only
You'll want to be familiar with the Apache Hadoop framework before you jump into Elastic MapReduce. It doesn't take long to get the hang of it, though. Most developers can have a MapReduce application running within a few hours.
Have you got a few hundred gigabytes of data that need processing? Perhaps a dump of radio telescope data that could use some combing through by a squad of processors running Fourier transforms? Or maybe you're convinced some statistical analysis will reveal a pattern hidden in several years of stock market information? Unfortunately, you don't happen to have a grid of distributed processors to run your application, much less the time to construct a parallel processing infrastructure.
Well, cheer up: Amazon has added Elastic MapReduce to its growing list of cloud-based Web services. Currently in beta, Elastic MapReduce uses Amazon's Elastic Compute Cloud (EC2) and Simple Storage Service (S3) to implement a virtualized distributed processing system based on Apache Hadoop.
Hadoop's internal architecture is the MapReduce framework. The mechanics of MapReduce are well documented in a paper by J. Dean and S. Ghemawat [PDF], and a full treatment is beyond the scope of this article. Instead, I'll illustrate by example.
Suppose you have a set of 10 words and you want to count the number of times those words appear in a collection of e-books. Your input data is a set of key/value pairs, the value being a line of text from one of the books and the key being the concatenation of the book's name and the line's number. This set might comprise a few megabytes big -- or gigabytes. MapReduce doesn't much care about size.
You write a routine that reads this input, a pair at a time, and produces another key/value pair as output. The output key is a word (from the original set of 10) and the associated value is the number of times that word appears in the line. (Zero values are not emitted.) This routine is the map part of map/reduce. Its output is referred to as the intermediate key/value pairs.
The intermediate key/value pairs are fed to another function (another "step" in the parlance of MapReduce). For this step, you write a routine that iterates through the intermediate data, sums up the values, and returns a single pair whose key is the word and whose value is the grand total. You don't have to worry about grouping the results of like keys (i.e., gathering all the intermediate key/values for a given word), because Hadoop does that grouping for you in the background.
Join the PC World newsletter!
Microsoft L5V-00027 Sculpt Ergonomic Keyboard Desktop
Epson WorkForce ET-4550
Everki ContemPRO Roll Top Laptop Backpack
Samsung portable 1TB T3 drive
UE Boom 2 Bluetooth speaker
Lexar® JumpDrive® S57 USB 3.0 flash drive
Smart LED Bulb LB130
Epson EcoTank Expression ET-2500
Lexar® JumpDrive® S45 USB 3.0 flash drive
3SIXT Ultra HD Sports Action Camera
Belkin MIXIT Metallic Lightning to USB Cable
Acer Swift 7
Logitech G403 Prodigy mouse
Lexar® Portable SSD
Epson WorkForce DS-360W
Google Daydream VR headset
Huawei Mate 9
Garmin Fenix Chronos smartwatch
HP Pavilion x360 13”
Dell Inspiron 5000 series 2-in-1
HD Pan/Tilt Wi-Fi Camera with Night Vision NC450
Lexar® JumpDrive® C20c USB Type-C flash drive
Blade 28 backpack by Arc’teryx
Surface Pro 4
Dell XPS 13 laptop
Audio-Technica ATH-ANC70 Noise Cancelling Headphones
Lexar® Professional 1800x microSDHC™/microSDXC™ UHS-II cards
Most Popular Reviews
- 1 Huawei Mate 9 full in-depth smartphone review
- 2 ZTE Axon 7 review: Is ZTE dumping old stock on Australia?
- 3 Oppo R9s smartphone full review
- 4 Huawei Nova Plus smartphone review
- 5 Google Pixel XL full, in-depth smartphone review: Phones just got smarter
Latest News Articles
- FCC rolls back net neutrality ISP transparency rules
- Israeli soldiers hit in cyberespionage campaign using Android malware
- Researcher develops ransomware attack that targets water supply
- Analysts peer into Microsoft's rumored Windows 10 Cloud
- AT&T, IBM, Nokia join to make IoT systems safer
PCW Evaluation Team
I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.
Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!
For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.
First impression on unpacking the Q702 test unit was the solid feel and clean, minimalist styling.
For work use, Microsoft Word and Excel programs pre-installed on the device are adequate for preparing short documents.
- How to quit Pokemon Go (or to start enjoying it again)
- Huawei Mate 9 full in-depth smartphone review
- Time to ditch Foxtel and the iQ3: How to replace Foxtel packages with cheaper alternatives
- What's the difference between an Intel Core i3, i5 and i7?
- Laser vs. inkjet printers: which is better?
- CCDesktop Engineer l WollongongNSW
- CCAgile CoachNSW
- TPSCCM SpecialistVIC
- CCIT Procurement OfficerNSW
- FTMid-Level Software Engineer x 2 - Adelaide Based (PV, NV2 or NV1 required)VIC
- CCSQL Server BI DeveloperQLD
- CCUnix AdministratorNSW
- TPIteration ManagerNSW
- CCSolution DesignerVIC
- CCUnix AdministratorNSW
- FTSenior Business AnalystSA
- CCFinance Analyst/ Project SpecialistVIC
- TPMobile DeveloperWA
- TPSenior Business AnalystQLD
- CCProject Manager - Adelaide basedNSW
- CCDevops Consultant - 12 month contractVIC
- FTMid-Level Software Engineer x 2 - Positive Vetting, NV2 or NV1 required!!SA
- FTDeveloper/ ProgrammerSA
- CCTest AnalystWA
- FTDeveloper - XML & JavaVIC
- FTJava Developer/IntegratorACT
- FTInfrastructure Security Compliance OfficerNSW
- CCIT Solutions ArchitectQLD
- CCSenior Mobile Application DeveloperNSW
- CCDesktop Engineer l WollongongNSW