Google promises a Hadoop or Spark cluster in 90 seconds with Cloud Dataproc

Minute-by-minute billing is another key piece of this new managed service

Getting insights out of big data is typically neither quick nor easy, but Google is aiming to change all that with a new, managed service for Hadoop and Spark.

Cloud Dataproc, which the search giant launched into open beta on Wednesday, is a new piece of its big data portfolio that's designed to help companies create clusters quickly, manage them easily and turn them off when they're not needed.

Enterprises often struggle with getting the most out of rapidly evolving big data technology, said Holger Mueller, a vice president and principal analyst with Constellation Research.

"It's often not easy for the average enterprise to install and operate," he said. When two open source products need to be combined, "things can get even more complex."

An easy way to implement and operate Hadoop and Spark clusters could be of significant value for enterprises, he added. For Google, meanwhile, Cloud Dataproc will ultimately mean more load, utilization and customers, which creates better economies of scale, Mueller noted.

Cloud Dataproc offers a number of advantages over both traditional, on-premises products and competing cloud services, Google said.

Whereas creating Spark and Hadoop clusters on-premises or through Infrastructure-as-a-Service (IaaS) providers can take anywhere from five to 30 minutes, for instance, Cloud Dataproc clusters take 90 seconds or less on average to start, and the same amount of time to scale or shut down. That, in turn, can mean users have more time to spend working with their data.

"When you do self-managed deployment, either on-premises or in the cloud, you're effectively paying in your own time for your clusters," said Greg DeMichillie, director of product management for the Google Cloud Platform. "What Cloud Dataproc allows you to do is shorten the window of time between when you ask a question and when you get insight."

Pricing is 1 cent per virtual CPU in each cluster per hour, and Cloud Dataproc clusters can include pre-emptible instances that have still lower compute prices, thereby reducing costs further. Whereas many providers round up usage to the nearest hour, Cloud Dataproc uses minute-by-minute billing and a 10-minute-minimum billing period.

Cloud Dataproc also offers built-in integration with Google Cloud Platform services such as BigQuery, Cloud Storage, Cloud Bigtable, Cloud Logging and Cloud Monitoring. Companies can use it to extract, transform and load terabytes of raw log data directly into BigQuery for business reporting, for example.

Because the service is managed, companies can use Spark and Hadoop clusters without the assistance of an administrator or special software, Google said. Rather, they can interact with clusters and Spark or Hadoop jobs through the Google Developers Console, the Google Cloud SDK or the Cloud Dataproc REST API; when they're done with a cluster, they can turn it off and avoid spending money needlessly.

The current implementation of Cloud Dataproc features clusters based on Spark 1.5 and Hadoop 2.7.1.

Join the PC World newsletter!

Error: Please check your email address.

Tags Google

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Katherine Noyes

IDG News Service
Show Comments

Most Popular Reviews

Latest News Articles

Resources

PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?