SPEC CPU Benchmarks

[Mario F.]

Is anyone familiar with SPEC and their CPU2006 benchmarks?

SPEC is a non profit organization that includes hardware and service vendors (From Acer, Intel, IBM, to Red Hat or VMWare) and that aims at producing standard performance evaluation tests, aka benchmarks.

CPU2006 is a package of proposed standardized cpu benchmarks aiming at testing:

CPU integer and floating-point precision performance
The system memory architecture
And the user compilers

I'm especially interested in understanding the general feel the community has for these tests; Whether SPEC has come up with a good methodology and whether -- for those that may have dabbled deep within any of the individual benchmarks source code -- if there is a sense of a reliable benchmark strategy.

I may soon have access to the CPU2006 set of tests. The objective will be to help come up with a subset of that package that can be integrated into a CPU Test Suite workflow for a tech news website. But I'd like for opinions beforehand. Most especially on the quality of these tests, in case anyone has any knowledge of them.

As a parallel discussion, I'd very much appreciate your thoughts on compiler performance benchmarking.

[Mario F.]

Nada?

[brewbuck]

With everything going multi-core (even phones now) this test suite looks already obsolete as it doesn't seem to contain any multi-threaded tests.

[Mario F.]

They've in fact always had it, I believe. But refined it in v1.1.

[Salem]

Q5. What does SPEC CPU2006 measure?
SPEC CPU2006 focuses on compute intensive performance, which means these benchmarks emphasize the performance of:

• the computer processor (CPU),
• the memory architecture, and
• the compilers.

Things not measured, but could be a factor in some other cases.

• the performance of the disk(s), including choice of file system
• the performance of graphics hardware, and choice of graphics library
• the choice of OS, and the extent it can be tuned to your application
• the choice of programming language, and the available optimisation choices
• the skill of the programmer, in choosing the best algorithms and data structures.

If your application is one of the reference applications, or is a closely related to one of them, then it's probably a useful thing.

For all hand held devices, and most portable devices in general, another important metric is power consumption. Measuring the fractions of a second between two CPUs running flat without any regard to power consumption is about as relevant as drag racing is to helping you choose your next car, where fuel economy is an important consideration.

[Mario F.]

Oh yes, they will be running other benchmarks. This is just one portion of the full benchmark setup.

>> the performance of graphics hardware, and choice of graphics library

This is a very interesting point to make. Thanks for that. It's just not the hardware, but indeed what graphics libraries are being used. The focus at this point is on their intended CPU test suit, though. But off the record we often see the media running benchmarks on the graphics hardware alone, paying very little attention to the actual libraries being used. With hardware manufacturers known differences, it pays to understand how a GPU will perform under different libraries.

>> the choice of programming language, and the available optimisation choices

This is sort of covered by the CPU2006 suits as I understand it so far (not much I can do until I actually take a look at it. So, why I'm asking if anyone has already). The suits covers C, C++, Perl and Fortran. The actual benchmarks (and the reason they are provided in source code format) are meant to be compiled by the system under test and any flags can, and should be, changed and dully documented.

>> the skill of the programmer, in choosing the best algorithms and data structures.

Indeed. And this is my major concern. Not that I have any reason to suspect this consortium of well known companies haven't come up with reasonable code. But what anyone who has worked with this suit and took a look at the source code of the tests thinks of it.

>> If your application is one of the reference applications, or is a closely related to one of them, then it's probably a useful thing.

Absolutely. This media outlet shares a dislike for synthetic benchmarks. But we if loose that definition enough, looking at a series of CPU benchmarks on Maya or Final Cut pro is as a synthetic benchmark to me as they come. Whereas, a CPU test on Visual C++ or gcc is a real measure of performance for me, for instance. They understand this and are exactly trying to come with with a test suit that covers a broad spectrum of real-world applications.