IBM speeds up data analysis with new algorithm

The algorithm could help to more quickly predict weather and financial risks

IBM researchers have developed a new algorithm that could in minutes analyze terabytes' worth of raw data to more quickly predict weather and electricity usage, the company said Thursday.

The mathematical algorithm, developed by IBM's laboratories in Zurich, can sort, correlate and analyze millions of random data sets, a task that could otherwise take days for supercomputers to process, said Costas Bekas, a researcher at IBM.

The algorithm is just under a thousand lines of code and will be instrumental in establishing usage patterns or trends based on data gathered from sources such as sensors or smart meters, he said. The algorithm could be used to analyze a growing mass of data measuring electricity usage trends as well as air or water pollution levels. The algorithm could also break down data from global financial markets and assess individual and collective exposure to risk, Bekas said.

"We are interested in measuring the quality of data," Bekas said. Efficient analysis of large data sets requires new mathematical techniques that reduce computational complexity, Bekas said.

The algorithm combines models of data calibration and statistical analysis that can assess measurement models and hidden relationships between data sets. IBM has been working on the research for two years, Bekas said.

The algorithm can also reduce the cost burden on companies by analyzing data in a more energy-efficient way, Bekas said. The lab used a Blue Gene/P Solution system at the Forschungszentrum Julich research center in Germany to validate 9TBs of data in less than 20 minutes. Analyzing the same amount of data without the algorithm would have taken a day with the supercomputer operating at peak speeds, which would have added up to higher electricity bills, Bekas said.

According to Top500.org, the Blue Gene/P is the fourth-fastest supercomputer in the world as of last November, with 294,912 IBM Power processing cores that can provide peak performance of up to 1 petaflop.

The traditional approach to data analysis is to take multiple data sets and look at them individually, said Eleni Pratsini, manager of mathematical and computational sciences at the IBM research labs. However, the algorithm compares data sets against each other, which could help enterprises point toward larger trends in particular areas, such as risk reduction in financial portfolios.

Enterprises will want faster ways of generating business intelligence as masses of data flood servers with the expansion of computing to new devices, he said.

Now that the algorithm has been proven to work scientifically, the research lab is collaborating with IBM's Global Services unit to use it for specific services, Pratsini said. Ultimately, the algorithm could make its way to IBM applications such as the SPSS statistical analysis software, but the company didn't provide a specific time frame for that.

Join the newsletter!

Error: Please check your email address.
Rocket to Success - Your 10 Tips for Smarter ERP System Selection

Tags IBManalyticsalgorithmsdata analysis

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Agam Shah

IDG News Service
Show Comments

Cool Tech

Breitling Superocean Heritage Chronographe 44

Learn more >

SanDisk MicroSDXC™ for Nintendo® Switch™

Learn more >

Toys for Boys

Family Friendly

Panasonic 4K UHD Blu-Ray Player and Full HD Recorder with Netflix - UBT1GL-K

Learn more >

Stocking Stuffer

Razer DeathAdder Expert Ergonomic Gaming Mouse

Learn more >

Christmas Gift Guide

Click for more ›

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Edwina Hargreaves

WD My Cloud Home

I would recommend this device for families and small businesses who want one safe place to store all their important digital content and a way to easily share it with friends, family, business partners, or customers.

Walid Mikhael

Brother QL-820NWB Professional Label Printer

It’s easy to set up, it’s compact and quiet when printing and to top if off, the print quality is excellent. This is hands down the best printer I’ve used for printing labels.

Ben Ramsden

Sharp PN-40TC1 Huddle Board

Brainstorming, innovation, problem solving, and negotiation have all become much more productive and valuable if people can easily collaborate in real time with minimal friction.

Sarah Ieroianni

Brother QL-820NWB Professional Label Printer

The print quality also does not disappoint, it’s clear, bold, doesn’t smudge and the text is perfectly sized.

Ratchada Dunn

Sharp PN-40TC1 Huddle Board

The Huddle Board’s built in program; Sharp Touch Viewing software allows us to easily manipulate and edit our documents (jpegs and PDFs) all at the same time on the dashboard.

George Khoury

Sharp PN-40TC1 Huddle Board

The biggest perks for me would be that it comes with easy to use and comprehensive programs that make the collaboration process a whole lot more intuitive and organic

Featured Content

Product Launch Showcase

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?