Amazon launches workflow orchestration service

The AWS Data Pipeline can stream big data analysis jobs
  • (IDG News Service)
  • — 29 November, 2012 23:10

Users of Amazon Web Services will soon be able to orchestrate workflows across different AWS services and their own internal resources, using a new orchestration engine called the AWS Data Pipeline.

Amazon Chief Technology Officer Werner Vogels introduced the technology at the company's Re:Invent conference, held this week in Las Vegas. The service is now available in limited beta preview, though Vogels did not say when it would be commercially available, nor what the price would be.

The service can "automate the movement and processing of any amount of data using data-driven workflows and built-in dependency checking," according to a blog post AWS issued that further explained the technology.

Amazon designed the service to automate the process of parsing large sets of data. For example, one pipeline can move log data from an AWS EC2 (Elastic Cloud Compute) instance to the AWS S3 (Simple Storage Service) once a day, and then, once a week, evoke an analysis job on the data on an AWS Elastic MapReduce cluster.

To set up a workflow pipeline, the user identifies some data sources and describes the steps that AWS should take to process the data. The user would also identify the destination for the processed data as well as a schedule for when the pipeline should be executed. Preconditions can also be established that the service will check before executing a job, such as checking if a file that is needed for the operation exists.

Pipelines can run across EC2, Elastic MapReduce clusters, and the user's own hardware. Pipelines can be set up in the AWS Management Console or by writing a script.

This is not the first workflow engine on AWS. The company also launched the Amazon Simple Workflow in February. However, AWS Data Pipeline is more focused on executing data-driven jobs.

The AWS Data Pipeline is one of a number of announcements Amazon made at the conference. The company also unveiled a data warehouse service and an auto-discovery service to ease the management of its ElastiCache. It has also cut the prices of some of its storage services and created two new EC2 instance types, for high-memory usage and large data usage.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Topics: applications, amazon.com, software, Managed Services, internet, cloud computing, Software as a service, Data management
Comments are now closed.

Latest News Articles

Most Popular Articles

Follow Us

GGG Evaluation Team

Kathy Cassidy

STYLISTIC Q702

First impression on unpacking the Q702 test unit was the solid feel and clean, minimalist styling.

Anthony Grifoni

STYLISTIC Q572

For work use, Microsoft Word and Excel programs pre-installed on the device are adequate for preparing short documents.

Steph Mundell

LIFEBOOK UH574

The Fujitsu LifeBook UH574 allowed for great mobility without being obnoxiously heavy or clunky. Its twelve hours of battery life did not disappoint.

Andrew Mitsi

STYLISTIC Q702

The screen was particularly good. It is bright and visible from most angles, however heat is an issue, particularly around the Windows button on the front, and on the back where the battery housing is located.

Simon Harriott

STYLISTIC Q702

My first impression after unboxing the Q702 is that it is a nice looking unit. Styling is somewhat minimalist but very effective. The tablet part, once detached, has a nice weight, and no buttons or switches are located in awkward or intrusive positions.

Resources

Best Deals on GoodGearGuide

Compare & Save

Deals powered by WhistleOut
Use WhistleOut's technology to compare:
Mobile phone plans & deals
Mobile phone models
Mobile phone carriers
Broadband plans & deals
Broadband providers
Deals powered by WhistleOut
WhistleOut

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?