Is anyone familiar with SPEC and their CPU2006 benchmarks?
SPEC is a non profit organization that includes hardware and service vendors (From Acer, Intel, IBM, to Red Hat or VMWare) and that aims at producing standard performance evaluation tests, aka benchmarks.
CPU2006 is a package of proposed standardized cpu benchmarks aiming at testing:
CPU integer and floating-point precision performance
The system memory architecture
And the user compilers
I'm especially interested in understanding the general feel the community has for these tests; Whether SPEC has come up with a good methodology and whether -- for those that may have dabbled deep within any of the individual benchmarks source code -- if there is a sense of a reliable benchmark strategy.
I may soon have access to the CPU2006 set of tests. The objective will be to help come up with a subset of that package that can be integrated into a CPU Test Suite workflow for a tech news website. But I'd like for opinions beforehand. Most especially on the quality of these tests, in case anyone has any knowledge of them.
As a parallel discussion, I'd very much appreciate your thoughts on compiler performance benchmarking.
