Benchmark sparks
Benchmark sparks
“Figures don’t lie, but liars figure,” they used to say back in my engineering days. You can be scrupulously careful about quoting numbers accurately and still be blowing smoke at people if you are choosy about which numbers you choose to report. I've been thinking about this problem recently because it always comes to mind in the lead-up to a US election.
Dear Linux Pro Reader,
"Figures don't lie, but liars figure," they used to say back in my engineering days. You can be scrupulously careful about quoting numbers accurately and still be blowing smoke at people if you are choosy about which numbers you choose to report. I've been thinking about this problem recently because it always comes to mind in the lead-up to a US election. At this writing, two candidates are both claiming they are ahead in the race and quoting different polls and surveys. To each, the other's polls and surveys don't even exist – their world does not have room for information that conflicts with their story.
This same phenomenon was also in the High Tech news this month, when AMD posted a video on YouTube criticizing Intel's use of the SYSmark benchmarking tool for measuring PC performance. SYSmark shows the latest Intel processors outperforming AMD equivalents by a large margin. In the video, AMD spokesmen John Hampton and Tony Salinas argued that the alternative PCMark 8 benchmark shows a much more competitive race between the Intel and AMD chips, and they demonstrate some practical use cases that show the PCMark 8 is more accurate than anything Intel is using.
They even quote a 2010 Federal Trade Commission (FTC) ruling that forced Intel to include some fine print with their SYSmark citations stating, "Software and workloads in performance tests may have been optimized only on Intel microprocessors."
The problem, according to Hampton and Salinas, is that SYSmark places the emphasis on the CPU only, whereas PCMark 8 looks at the CPU, GPU, and video subcomponents together, which is more like how the computer actually behaves.
I try not to take sides in vendor disputes, but in this case, since the FTC has taken sides, it does seem very likely that AMD has a point. If SYSmark is not an accurate benchmark for comparing real-world PC system performance, why does Intel keep using it? Because marketing departments of major corporations don't exactly put the emphasis accuracy.
Hampton invokes another recent news story when he mentions "the recent debacle over the emissions standards provided by a major auto maker …" This apparent reference to the Volkswagen emissions scandal highlights another case where customers were misled with numbers. Of course, as Intel would readily point out, Volkswagen is actually providing inaccurate information, whereas Intel is reporting the results of the SYSmark benchmarks very accurately. Is using the wrong facts as bad as giving the wrong answer? If it leads people to incorrect conclusions, the effect is the same.
We get so many numbers thrown at us every day that we really can't chase down all of them. Still, if you're going to go to the trouble of buying a computer that you will have to work with every day for at least a couple of years, it really might be worth taking a closer look at the numbers. The next time I read a triumphant claim of success that references a computational benchmark, I'm going to Google the benchmark and find out what it is really testing.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
![Learn More](https://www.linux-magazine.com/var/linux_magazin/storage/images/media/linux-magazine-eng-us/images/misc/learn-more/834592-1-eng-US/Learn-More_medium.png)
News
-
AlmaLinux Now Supports Raspberry Pi 5
If you're looking to create with the Raspberry Pi 5 and want to use AlmaLinux as your OS, you're in luck because it's now possible.
-
Kubuntu Focus Releases New Iterations of Ir14 and Ir16 Laptops
If you're a fan of the Kubuntu Focus laptops or have been waiting for the right time to purchase one, that time might be now.
-
NixOS 24.05 Is Ready for Prime Time
The latest release of NixOS (Uakari) has arrived and offers its usual reproducible, declarative, and reliable goodness.
-
Linux Lite 7.0 Officially Released
Based on Ubuntu 24.04 and kernel 6.8, Linux Lite version 7 now offers more options than ever.
-
KaOS Linux 2024.05 Adds Bcachfs Support and More
With updates all around, KaOS Linux now includes support for the bcachefs file system.
-
TUXEDO Computers Unveils New Iteration of the Stellaris Laptop Line
The Stellaris Slim 15 is the 6th generation and includes either an AMD or Intel CPU
-
KDE Releases Plasma 6.0.5
The latest release of the Plasma desktop has arrived with several improvements and the usual bug fixes.
-
Gnome OS Adopting systemd-sysupdate
Gnome OS is about to undergo a major under-the-hood change that promises enhanced security.
-
Endless OS 6 Now Available
After more than a year since the last update, the latest release of Endless OS is now available for general usage.
-
Fedora Asahi 40 Remix Available for Macs with Apple Silicon
If you've been anticipating KDE's Plasma 6 for your Apple Silicon-powered Mac, then you're in luck.