Move over, Linpack: Supercomputers get new performance test

The new test should be a better measure of performance but could lead to disagreement over who has the world's fastest system

Jack Dongarra, creator of the Linpack supercomputing benchmark test

Jack Dongarra, creator of the Linpack supercomputing benchmark test

The developer of the most widely used test for ranking the performance of supercomputers has said his metric is out of date and proposed a new test that will be introduced starting in November.

Jack Dongarra, distinguished professor of computer science at the University of Tennessee, said the Linpack test he developed in the 1970s, which has been the basis for the Top500 list of the world's fastest computers for the past 20 years, is no longer the most useful benchmark for how well a system can perform.

The new metric, he said, could change the way vendors design their supercomputers and will provide customers with a better measure of the performance they can expect for the types of real-world applications they'll be running.

The Top500 list is published twice a year, in June and November, and is closely watched as vendors and nations seek bragging rights for who has the fastest system. The current leader is the Tianhe-2, developed by China's National University of Defense Technology.

Linpack has been used to rank the systems since the first Top500 list was published in 1993, but it's no longer an indicator of real application performance, Dongarra said.

"Linpack measures the speed and efficiency of linear equation calculations," according to a statement Wednesday announcing the new benchmark, called the High Performance Conjugate Gradient (HPCG). "Over time, applications requiring more complex computations have become more common. These calculations require high bandwidth and low latency, and access data using irregular patterns. Linpack is unable to measure these more complex calculations."

HPCG is needed, Dongarra said in a telephone interview, in part because computer vendors optimize their systems to rank highly on the Top500 list. If that list is based on an out of date test, it encourages vendors to architect their systems in a way that's not optimal for today's applications.

"We don't want to build a machine that does well on this 'fake' problem. We want to build a machine that does well for a larger set of applications," said Dongarra, who developed the new test with a colleague, Michael Heroux, of Sandia National Laboratories in Albuquerque.

Because of the way the new test is being introduced, however, it could potentially spark disagreements over who really has the world's fastest supercomputer. That's because HPCG will be introduced gradually over time, and it could be years before it becomes the primary method for ranking the Top500.

"One of the nice things about Linpack is that there's one number, so it's very clear what we mean by the fastest computer. This will in fact generate two numbers," Dongarra said.

He plans to maintain the Linpack test alongside HPCG in part for the valuable trending information that Linpack provides, he said. But it will also continue to be used because it could take years before a significant number of supercomputers are tested against the new benchmark.

"I expect in November we'll just have a few entries based on this new benchmark. Populating the list with 500 entries is going to take some time, so I'd guess over the next five years we'd have a chance of seeing that list fully populated," he said.

Starting in November, "we're going to have a list of the Top500, and then we're going to have a second column, and that second column will be the new benchmark," Dongarra said.

"It may ultimately lead to a list that is based on this new benchmark, but certainly not right away," he said.

The dueling benchmarks could potentially lead to different supercomputing centers, which covet positions on the Top500, claiming leadership based on both the old and the new tests. That could make it hard to say definitively who has the fastest supercomputer, though it seems the Top500 will consider Linpack to be the primary ranking metric at least for now.

The new test could lead to some "big changes" in which systems show the greatest performance potential, Dongarra said. The HPCG benchmark stresses architectural features which might not be easy for systems that perform well on the Linpack test to optimize for, he said.

"I think individuals will have to then evaluate what number makes sense for their particular mix of problems. And over time I would hope that the new [benchmark] would carry more weight."

HPCG was developed partly at the behest of the U.S. Department of Energy, Dongarra said. "They're looking towards exascale now, and the concern is that if you build an exascale computer that will do this Linpack test well, it may not do well at other problems. So that's one of the issues here."

The University of Tennessee conducts joint projects with the DOE and Dongarra said he's familiar with their application requirements. But he said the new test will be a good indicator of how computers will run other types of applications as well, such as those used for oil and gas exploration or weather modeling.

"One of the problems with the Linpack is that is stresses only one component, that being the floating point potential of the computer," he said. It doesn't stress areas like system latency and memory hierarchy, and the new test will be able to expose weaknesses of systems as they relate to those areas.

Dongarra plans to distribute the software for the new test to computer vendors in the next few months, giving them a chance to begin optimizing their systems and to propose changes to HPCG before it's introduced formally at the SC13 supercomputing conference in Denver this November, where the next Top500 list will be announced.

James Niccolai covers data centers and general technology news for IDG News Service. Follow James on Twitter at @jniccolai. James's e-mail address is james_niccolai@idg.com

Join the PC World newsletter!

Error: Please check your email address.

Tags supercomputersHigh performancehardware systems

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

James Niccolai

IDG News Service
Show Comments

Essentials

Lexar® JumpDrive® S57 USB 3.0 flash drive

Learn more >

Microsoft L5V-00027 Sculpt Ergonomic Keyboard Desktop

Learn more >

Mobile

Lexar® JumpDrive® S45 USB 3.0 flash drive 

Learn more >

Exec

Lexar® JumpDrive® C20c USB Type-C flash drive 

Learn more >

Audio-Technica ATH-ANC70 Noise Cancelling Headphones

Learn more >

HD Pan/Tilt Wi-Fi Camera with Night Vision NC450

Learn more >

Lexar® Professional 1800x microSDHC™/microSDXC™ UHS-II cards 

Learn more >

Budget

Back To Business Guide

Click for more ›

Most Popular Reviews

Latest News Articles

Resources

PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?