Futuremark says Nvidia didn't cheat, but broke rules

Futuremark Corp., the developer of the benchmarking software 3DMark03, has retracted accusations of cheating it leveled against graphics chip vendor Nvidia Corp. over modified software drivers on Nvidia products. However, it remains opposed to Nvidia's techniques and is not reinstating the original benchmark results.

Futuremark has changed its description, but not its opinion, of Nvidia's decision to use application-specific optimizations in its drivers that detected the presence of specific 3DMark03 tests and altered the performance of its graphics chip to achieve a higher benchmark result. Benchmarks are software tests used to simulate application behavior on a processor or device. They are designed to be used by buyers of hardware to compare the performance of different products.

On May 23, Futuremark published a seven-page report detailing eight specific examples of where it believed Nvidia unfairly modified drivers after detecting certain types of shaders in 3DMark03 tests. Some shaders -- code that denotes how the surface of an object should appear -- in 3DMark03 were replaced in favor of ones in Nvidia's drivers that rendered the image more quickly, but at the expense of overall image quality, Futuremark said in that report.

"Please note, that the cheating described here is totally different from optimization. Optimizing the driver code to increase efficiency is a technique often used to enhance game performance and carries greater legitimacy, since the rendered image is exactly what the (game) developer intended," Futuremark said in its report.

However, the rhetoric has cooled down, at least on Futuremark's part.

Futuremark, in Espoo, Finland, released a statement on its Web site Monday that read, "Futuremark now has a deeper understanding of the situation and Nvidia's optimization strategy. In the light of this, Futuremark now states that Nvidia's driver design is an application specific optimization and not a cheat."

However, application-specific optimizations are still forbidden under Futuremark's rules, said Tero Sarkkinen, executive vice president of sales and marketing for Futuremark. The benchmarking company put out its most recent statement because it believes the industry will not benefit from an extended "soap opera" between itself and Nvidia, he said.

Nvidia stands by its drivers, which have not been changed and are still available for download on its Web site, said Derek Perez, an Nvidia spokesman. The Santa Clara, California, company still believes it did nothing wrong and will continue to use optimizations, despite Futuremark's objections, Perez said.

"They made a bunch of rules without understanding the intricacies of the architecture," he said. "There are application-specific optimizations in every CPU and every GPU."

Rival ATI Technologies Inc. was quite willing to continue the mudslinging. "It's a shame that a large company with a lot of money and access to expensive lawyers can beat a small company into submission," said Chris Evendon, director, public relations for ATI.

Along with its report on May 23, Futuremark released a new build of 3DMark03 with slightly different code that did not affect the workload required by the tests. That release lowered the performance of Nvidia's GeForceFX 5900 Ultra processor by 24 percent and is the most accurate representation of processor performance, Sarkkinen said.

The issue stems from the different architectures used by Nvidia and ATI in developing their graphics processors, said Jon Peddie, president of Jon Peddie Research, in Tiburon, California.

Nvidia has both a 16-bit and a 32-bit architecture in its GeForce FX chips, while ATI uses a 24-bit architecture for its Radeon products, he said. 3DMark03 was written for the latest version of Microsoft Corp.'s DirectX 9.0 graphics technology, which also uses a 24-bit architecture, he said.

Generally speaking, a 24-bit architecture will render a image faster than a 32-bit architecture, but the 32-bit architecture will produce sharper images on applications that can take advantage of that sharper image. In this situation, Nvidia replaced the 24-bit shaders with its own 12-bit shader that rendered the image more quickly, but at the expense of image quality, Peddie said.

This was immediately decried by Futuremark and hundreds of gamers as cheating, since Nvidia changed the outcome of the image. Perez denied Tuesday that the optimization resulted in a poorer image.

Futuremark also mentioned in its report that ATI used optimization in its drivers, but ATI defended that move last week by pointing out that ATI only changed the order in which the 3DMark03 instructions flowed through its processor, without changing the instructions themselves.

However, because ATI's driver detected the 3DMark03 test, it was also prohibited under Futuremark's rules, Sarkkinen said. ATI has already said it will remove this optimization from its drivers and will release a new driver as part of its regular release schedule within about three weeks, Evendon said.

The end result of this saga has further blurred the line between cheating and optimizing. Futuremark has nothing against general optimizations that improve gameplay, but the use of optimizations that detect the presence of benchmarking software and adjust the performance accordingly will mean Futuremark cannot guarantee the reliability of its results, Sarkkinen said.

Any user turning to benchmarks for support in making buying decisions will make the best choice after consulting numerous sources and not relying on any one result, Peddie said.

"You cannot use one benchmark that produces one score to evaluate multiple architectures. It's impossible, misleading, and lazy," he said.

Join the newsletter!

Error: Please check your email address.
Rocket to Success - Your 10 Tips for Smarter ERP System Selection
Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Tom Krazit

IDG News Service
Show Comments

Cool Tech

Breitling Superocean Heritage Chronographe 44

Learn more >

SanDisk MicroSDXC™ for Nintendo® Switch™

Learn more >

Toys for Boys

Family Friendly

Panasonic 4K UHD Blu-Ray Player and Full HD Recorder with Netflix - UBT1GL-K

Learn more >

Stocking Stuffer

Razer DeathAdder Expert Ergonomic Gaming Mouse

Learn more >

Christmas Gift Guide

Click for more ›

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Walid Mikhael

Brother QL-820NWB Professional Label Printer

It’s easy to set up, it’s compact and quiet when printing and to top if off, the print quality is excellent. This is hands down the best printer I’ve used for printing labels.

Ben Ramsden

Sharp PN-40TC1 Huddle Board

Brainstorming, innovation, problem solving, and negotiation have all become much more productive and valuable if people can easily collaborate in real time with minimal friction.

Sarah Ieroianni

Brother QL-820NWB Professional Label Printer

The print quality also does not disappoint, it’s clear, bold, doesn’t smudge and the text is perfectly sized.

Ratchada Dunn

Sharp PN-40TC1 Huddle Board

The Huddle Board’s built in program; Sharp Touch Viewing software allows us to easily manipulate and edit our documents (jpegs and PDFs) all at the same time on the dashboard.

George Khoury

Sharp PN-40TC1 Huddle Board

The biggest perks for me would be that it comes with easy to use and comprehensive programs that make the collaboration process a whole lot more intuitive and organic

David Coyle

Brother PocketJet PJ-773 A4 Portable Thermal Printer

I rate the printer as a 5 out of 5 stars as it has been able to fit seamlessly into my busy and mobile lifestyle.

Featured Content

Product Launch Showcase

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?