SPEC CPU Benchmarks

This is a discussion on SPEC CPU Benchmarks within the Tech Board forums, part of the Community Boards category; Is anyone familiar with SPEC and their CPU2006 benchmarks? SPEC is a non profit organization that includes hardware and service ...

  1. #1
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Portugal
    Posts
    7,439

    SPEC CPU Benchmarks

    Is anyone familiar with SPEC and their CPU2006 benchmarks?

    SPEC is a non profit organization that includes hardware and service vendors (From Acer, Intel, IBM, to Red Hat or VMWare) and that aims at producing standard performance evaluation tests, aka benchmarks.

    CPU2006 is a package of proposed standardized cpu benchmarks aiming at testing:

    CPU integer and floating-point precision performance
    The system memory architecture
    And the user compilers

    I'm especially interested in understanding the general feel the community has for these tests; Whether SPEC has come up with a good methodology and whether -- for those that may have dabbled deep within any of the individual benchmarks source code -- if there is a sense of a reliable benchmark strategy.

    I may soon have access to the CPU2006 set of tests. The objective will be to help come up with a subset of that package that can be integrated into a CPU Test Suite workflow for a tech news website. But I'd like for opinions beforehand. Most especially on the quality of these tests, in case anyone has any knowledge of them.

    As a parallel discussion, I'd very much appreciate your thoughts on compiler performance benchmarking.
    Last edited by Mario F.; 10-05-2011 at 04:40 AM.
    The programmer’s wife tells him: “Run to the store and pick up a loaf of bread. If they have eggs, get a dozen.”
    The programmer comes home with 12 loaves of bread.


    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  2. #2
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Portugal
    Posts
    7,439
    Nada?
    The programmer’s wife tells him: “Run to the store and pick up a loaf of bread. If they have eggs, get a dozen.”
    The programmer comes home with 12 loaves of bread.


    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  3. #3
    Captain Crash brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,239
    With everything going multi-core (even phones now) this test suite looks already obsolete as it doesn't seem to contain any multi-threaded tests.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  4. #4
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Portugal
    Posts
    7,439
    They've in fact always had it, I believe. But refined it in v1.1.
    The programmer’s wife tells him: “Run to the store and pick up a loaf of bread. If they have eggs, get a dozen.”
    The programmer comes home with 12 loaves of bread.


    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  5. #5
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,505
    Q5. What does SPEC CPU2006 measure?
    SPEC CPU2006 focuses on compute intensive performance, which means these benchmarks emphasize the performance of:
    • the computer processor (CPU),
    • the memory architecture, and
    • the compilers.
    Things not measured, but could be a factor in some other cases.
    • the performance of the disk(s), including choice of file system
    • the performance of graphics hardware, and choice of graphics library
    • the choice of OS, and the extent it can be tuned to your application
    • the choice of programming language, and the available optimisation choices
    • the skill of the programmer, in choosing the best algorithms and data structures.

    If your application is one of the reference applications, or is a closely related to one of them, then it's probably a useful thing.

    For all hand held devices, and most portable devices in general, another important metric is power consumption. Measuring the fractions of a second between two CPUs running flat without any regard to power consumption is about as relevant as drag racing is to helping you choose your next car, where fuel economy is an important consideration.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  6. #6
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Portugal
    Posts
    7,439
    Oh yes, they will be running other benchmarks. This is just one portion of the full benchmark setup.

    >> the performance of graphics hardware, and choice of graphics library

    This is a very interesting point to make. Thanks for that. It's just not the hardware, but indeed what graphics libraries are being used. The focus at this point is on their intended CPU test suit, though. But off the record we often see the media running benchmarks on the graphics hardware alone, paying very little attention to the actual libraries being used. With hardware manufacturers known differences, it pays to understand how a GPU will perform under different libraries.

    >> the choice of programming language, and the available optimisation choices

    This is sort of covered by the CPU2006 suits as I understand it so far (not much I can do until I actually take a look at it. So, why I'm asking if anyone has already). The suits covers C, C++, Perl and Fortran. The actual benchmarks (and the reason they are provided in source code format) are meant to be compiled by the system under test and any flags can, and should be, changed and dully documented.

    >> the skill of the programmer, in choosing the best algorithms and data structures.

    Indeed. And this is my major concern. Not that I have any reason to suspect this consortium of well known companies haven't come up with reasonable code. But what anyone who has worked with this suit and took a look at the source code of the tests thinks of it.

    >> If your application is one of the reference applications, or is a closely related to one of them, then it's probably a useful thing.

    Absolutely. This media outlet shares a dislike for synthetic benchmarks. But we if loose that definition enough, looking at a series of CPU benchmarks on Maya or Final Cut pro is as a synthetic benchmark to me as they come. Whereas, a CPU test on Visual C++ or gcc is a real measure of performance for me, for instance. They understand this and are exactly trying to come with with a test suit that covers a broad spectrum of real-world applications.
    The programmer’s wife tells him: “Run to the store and pick up a loaf of bread. If they have eggs, get a dozen.”
    The programmer comes home with 12 loaves of bread.


    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Problems compiling when using spec file.
    By DaMunky89 in forum C Programming
    Replies: 2
    Last Post: 01-22-2011, 09:55 AM
  2. Not understanding CPU C programming benchmarks...
    By Kleid-0 in forum A Brief History of Cprogramming.com
    Replies: 4
    Last Post: 07-22-2005, 03:49 AM
  3. Borland C++ 5.02 and the "left" spec.
    By johnnyd in forum C++ Programming
    Replies: 1
    Last Post: 03-08-2003, 09:29 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21