Standard Performance Evaluation Corporation

Metrics

This article is a side bar to History & Reasoning Behind SPEC95.
Published September, 1995; see disclaimer.

Even within the area of compute-intensive performance, there are several distinct areas/categories that must be considered and chosen:

Integer versus floating point

These are two distinct types of computation as reflected in different software data types or functional units on a processor. SPEC recognized this by creating SPEC CINT95 and CFP95 as separate suites.

Conservative versus aggressive compilation

To achieve portability to a wide variety of platforms, SPEC provides the benchmarks as source code. But how should this source code be compiled? Some people test every possible compiler option to squeak the last bit of performance out of a compiler. Others simply use the standard options provided by the compiler vendor, and yet others may use a combination of both. While SPEC recognizes it cannot capture everyone's model of usage in a single metric, SPEC has defined a required baseline measure (with more conservative rules on how the applications are to be compiled) and an optional non- baseline measure (with more aggressive rules governing compilation). This provides a required conservative point of reference plus an optional, more tuned reference point. Speed versus throughput: People, especially those with multi-threaded environments, are often not only concerned with how fast they can get a single task done but also the rate at which they can get multiple tasks done (throughput). With this in mind as well, the SPEC95 suites provide a comparative measure for speed of a single compute-intensive task and for multiple tasks (SPECrate). SPEC has provided a reference point for each of these. The combinations of each of these choices leads to eight possible metrics in the SPEC95 suites:

    SPEED   THROUGHPUT
  Aggressive
    SPECint95 SPECint_rate95
    SPECfp95  SPECfp_rate95
  Conservative
    SPECint_base95  SPECint_rate_base95
    SPECfp_base95 SPECfp_rate_base95

These metrics provide the user with the ability to choose the criteria that best matches his need. (Note that this does require the users to be aware of the environment with which they are concerned.)