Databricks takes on Google data streaming analysis with Spark

Databricks Cloud will provide Spark-based streaming analysis as a service

Taking on Google, Databricks plans to offer its own cloud service for analyzing live data streams, one based on the Apache Spark software.

Databricks Cloud is designed to provide a platform for analyzing streaming data, much like the Google DataFlow service announced last week.

Like Google DataFlow, Databricks Cloud promises to offer a single programming model that cuts across different approaches to data analysis, including support for batch programming and live data streaming. And like Google DataFlow, Databricks Cloud will first be offered in preview mode, with full commercial support due by the end of the year.

The two services are aimed to different markets, according to Ion Stoica, CEO of Databricks.

"Google DataFlow is really targeted to developers. We also have higher-level interfaces for data scientists and data engineers," Stoica said.

Databricks also guarantees application portability. Because the entire stack is based on open source software, users can move their workloads to other Apache Spark installations should they need to, Stoica said. "You can take your application and run it in another cloud," Stoica said.

Such a service could be used by enterprises for tasks such as churn analysis, which can determine why a customer stops using a product, or for fraud detection, where a malicious activity can be spotted while it is still taking place.

The University of California, Berkeley's AMP (Algorithms, Machines and People) Lab originally developed Spark as a unified processing engine, one able to provide a platform for a variety of data analysis tasks, including interactive queries, steaming data analysis, machine learning and graph computation.

A number of developers behind Spark went on to form Databricks. The software itself, designed to run on a cluster of servers, is now managed as an open source project under the guidance of the Apache Software Foundation.

Offering Spark as a service eliminates the arduous task for setting up and maintaining an in-house implementation of Spark, Stoica noted.

"Clusters are hard to set up and maintain. To build a data pipeline, you need to stitch together multiple tools, and the tools are still hard to use. So extracting value out of the data is still a struggle," Stoica said.

Initially, Databricks Cloud will be run on Amazon Web Services, though eventually it will also run on other cloud providers such as Google.

In addition to the Spark platform itself, Databricks will provide a set of built-in applications that can do common data analysis tasks. Users can build their own workflows, or issue queries and interact with the data directly. Output can be piped to a dashboard or a report.

Databricks is not the only company making use of Spark's capabilities. ClearStory offers an analytics software package based on Spark that allows organizations to aggregate dozens of unstructured data sources for analysis, far more than can be easily done through traditional business intelligence tools, said ClearStory CEO Sharmila Mulligan.

Databricks also announced Monday that it has received US$33 million in series B funding led by venture capital firm, New Enterprise Associates, with follow-on investment from Andreessen Horowitz.

Join the PC World newsletter!

Error: Please check your email address.

Tags open sourceapplicationsDatabricksdata miningManaged Servicessoftwarecloud computinginternet

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Marc Ferranti

IDG News Service
Show Comments

Most Popular Reviews

Latest News Articles

Resources

PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?