Google service analyzes live streaming data

Google Cloud Dataflow can analyze both streaming and batched data with the same programming models

Google's Dataflow can analyze data as it crosses the wire

Google's Dataflow can analyze data as it crosses the wire

Taking what many see as the next step in big data analysis, Google is previewing a service called Google Cloud Dataflow that analyzes live data, potentially giving users the ability to view trends and be alerted to events as they happen.

"There's an enormous amount of data being created, and so you need a way to ingest that in a more intelligent way," said Brian Goldfarb, Google Cloud Platform head of marketing. With big data, "the program models are different. The technologies are different. It requires developers to learn a lot and manage a lot to make it happen."

"It is a fully managed service that lets you create data pipelines for ingesting, transforming and analyzing arbitrary amounts of data in both batch or streaming mode, using the same programming model," Goldfarb said.

Google Cloud Dataflow is designed so the user can focus on devising proper analysis, without worrying about setting up and maintaining the underlying data piping and processing infrastructure.

It could be used for live sentiment analysis, for instance, where an organization estimates the popular sentiment around a product by scanning social networks such as Twitter. It could also be used as a security tool to watch activity logs for unusual activity.

"There are a bunch of different business applications in which it could apply. In a lot of data-centric verticals, like retail or oil and gas, a technology like this could open the door to getting analytics," Goldfarb said.

It could also be used an alternative to commercial ETL (extract, transform and load) programs, widely used to prepare data for analysis by business intelligence software.

Google Cloud Dataflow is based on technologies that the company built internally for its own use, following up on work it did on the MapReduce programming model, which is used in Apache Hadoop.

Live data stream analysis appears to be the next logical step in big data analysis, a field pioneered by Hadoop. Hadoop provides a way to analyze massive amounts of unstructured data spread across multiple servers. Originally, Hadoop used MapReduce as the platform to write programs that analyze the data.

MapReduce's limitation is that it can only analyze data in batch mode, which means all the data must be collected before it can be analyzed. A number of new software programs have been developed to get around the limitation of batch processing, such as Twitter Storm and Apache Spark, which are both available as open source and can run on Hadoop.

Google's own approach to live data analysis uses a number of technologies built by the company, notably Flume and MillWheel. Flume aggregates large amounts of data and MillWheel provides a platform for low-latency data processing.

The service provides a software development kit that can be used to build complex pipelines and analysis. Like MapReduce, Cloud Dataflow will initially use the Java programming language. In the future, other languages may be supported.

The pipelines can ingest data from external sources and use them for a variety of things. The service provides a library to prepare and reformat data for further analysis, and users can write their own transformations.

The treated dataset can be queried against using Google's BigQuery service. Or the user can write modules to examine the data as it crosses the wire, to look for aberrant behavior or trends in real-time.

Google announced Cloud Dataflow at the company's Google I/O user conference in San Francisco. A small number of Google customers are testing it and the company plans to open it up as a public preview later this year.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags cloud computinginternetGooglesoftwareapplicationsdevelopment platformsdata miningInfrastructure services

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments

Cool Tech

Toys for Boys

Family Friendly

Stocking Stuffer

SmartLens - Clip on Phone Camera Lens Set of 3

Learn more >

Christmas Gift Guide

Click for more ›

Brand Post

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Aysha Strobbe

Microsoft Office 365/HP Spectre x360

Microsoft Office continues to make a student’s life that little bit easier by offering reliable, easy to use, time-saving functionality, while continuing to develop new features that further enhance what is already a formidable collection of applications

Michael Hargreaves

Microsoft Office 365/Dell XPS 15 2-in-1

I’d recommend a Dell XPS 15 2-in-1 and the new Windows 10 to anyone who needs to get serious work done (before you kick back on your couch with your favourite Netflix show.)

Maryellen Rose George

Brother PT-P750W

It’s useful for office tasks as well as pragmatic labelling of equipment and storage – just don’t get too excited and label everything in sight!

Cathy Giles

Brother MFC-L8900CDW

The Brother MFC-L8900CDW is an absolute stand out. I struggle to fault it.

Luke Hill

MSI GT75 TITAN

I need power and lots of it. As a Front End Web developer anything less just won’t cut it which is why the MSI GT75 is an outstanding laptop for me. It’s a sleek and futuristic looking, high quality, beast that has a touch of sci-fi flare about it.

Emily Tyson

MSI GE63 Raider

If you’re looking to invest in your next work horse laptop for work or home use, you can’t go wrong with the MSI GE63.

Featured Content

Product Launch Showcase

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?