Spark update brings R support and machine learning chops

Spark users can run R jobs across data stored on multiple servers

The Spark data processing platform now includes support for the R programming language

The Spark data processing platform now includes support for the R programming language

One of the most popular big data processing platforms, Spark, now supports one of the premier statistical programming languages, R, which could pave the way for easier big data statistical analysis.

"R is the lingua franca of data scientists and its adoption has exploded in the last two years," wrote Patrick Wendell, one of the chief contributors to Spark, in an e-mail. Wendell is also a cofounder and software engineer at Databricks, which offers a commercial cloud-based version of Spark for enterprises.

The new version "will let R users work directly on large datasets, scaling to hundreds or thousands of machines, well beyond the limits of a stand-alone R program," Wendell wrote.

The newly updated Spark, version 1.4, also includes production-ready machine learning capabilities and a more comprehensive set of visual debugging tools.

With more than 2 million users worldwide, R is one of the most widely used programming languages specifically designed for statistical computing and predictive analytics.

An open source project, R was designed to work only on a single computer, which limits the size of the analysis jobs that can be readily executed. There have been a few efforts to make large R jobs run on clusters of computers for larger jobs, such as Hewlett-Packard's Distributed R package.

The newly updated Spark provides another boost for running R in parallel. In the past year, the Spark data processing platform, an open source project overseen by the Apache Software Foundation, has grown in popularity, as many organizations have used the technology for analyzing data stored across a cluster of computers.

Companies such as Autodesk, eBay, NASA, Opentable and Yahoo have all used Spark to make sense of large collections of data. About 17 percent of 3,000 Java professionals noted that they were running Spark in their operations, according to a December 2014 survey conducted by Java tool provider TypeSafe.

Spark 1.4 comes with SparkR, which is an API (application programming interface) that allows programs to submit R-based analysis jobs to Spark to execute. The data to be analyzed can come from a variety of sources, including Hadoop Hive-based data warehouses, the Hadoop File System, the Apache Parquet columnar store, or a JSON (JavaScript Object Notation)-formatted data feed.

"Because SparkR uses Spark's parallel engine underneath, operations take advantage of multiple cores or multiple machines, and can scale to data sizes much larger than stand-alone R programs," noted Wendell, in a blog post announcing the release.

The new release also comes with a production-ready machine learning pipeline, first introduced as an alpha feature in Spark 1.2. Machine learning is the programmatic approach for computers to infer new information through the use of preset rules and copious amounts of data. The new machine learning pipeline comes with a set of commonly used algorithms for preparing and transforming the data. Emerging from alpha status means that the developers safely can use the API without worrying that it will change in future editions of Spark.

The new release comes just in time for the Spark Summit user conference, held next week in San Francisco.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Join the newsletter!

Or
Error: Please check your email address.
Rocket to Success - Your 10 Tips for Smarter ERP System Selection

Tags softwareapplicationsdata miningDatabricks

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments

Essentials

James Cook University - Master of Data Science Online Course

Learn more >

Mobile

Sansai 6-Outlet Power Board + 4-Port USB Charging Station

Learn more >

Exec

Budget

Back To Business Guide

Click for more ›

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Louise Coady

Brother MFC-L9570CDW Multifunction Printer

The printer was convenient, produced clear and vibrant images and was very easy to use

Edwina Hargreaves

WD My Cloud Home

I would recommend this device for families and small businesses who want one safe place to store all their important digital content and a way to easily share it with friends, family, business partners, or customers.

Walid Mikhael

Brother QL-820NWB Professional Label Printer

It’s easy to set up, it’s compact and quiet when printing and to top if off, the print quality is excellent. This is hands down the best printer I’ve used for printing labels.

Ben Ramsden

Sharp PN-40TC1 Huddle Board

Brainstorming, innovation, problem solving, and negotiation have all become much more productive and valuable if people can easily collaborate in real time with minimal friction.

Sarah Ieroianni

Brother QL-820NWB Professional Label Printer

The print quality also does not disappoint, it’s clear, bold, doesn’t smudge and the text is perfectly sized.

Ratchada Dunn

Sharp PN-40TC1 Huddle Board

The Huddle Board’s built in program; Sharp Touch Viewing software allows us to easily manipulate and edit our documents (jpegs and PDFs) all at the same time on the dashboard.

Featured Content

Product Launch Showcase

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?