SPECgpc Project Group Rules

The Graphics Performance Characterization Project Group Rules

Version 3.1

Last Updated: 05/23/2018

Overview
1. Rules Inheritance
  1. All rules declared in "The Graphics and Workstation Performance Group (SPEC/GWPG): Rules For Project Groups" document (known hereinafter as the GWPG Project Groups Ruleset) shall apply, unless specifically overruled by a rule in this document.
  2. Rules declared in this document shall apply in addition to the rules declared in the GWPG Project Groups Ruleset.
2. General Philosophy
  1. The Graphics Performance Characterization Project of SPEC/GWPG (henceforth abbreviated as SPECgpc^SM) believes the user community will benefit from an objective series of tests, which can serve as common reference and be considered as part of an evaluation process.
  2. The SPECgpc seeks to develop benchmarks for generating accurate graphics performance measures in an open, accessible and well-publicized manner.
  3. The SPECgpc wishes to contribute to the coherence of the field of graphics performance measurement and evaluation so that vendors will be better able to present well-defined performance measures; and customers will be better able to compare and evaluate vendors' products and environments.
  4. The SPECgpc will provide formal beta software to members and final software releases to the public in a timely fashion.
  5. Hardware and software used to run the SPECgpc benchmarks must provide a suitable environment for running typical graphics programs.
  6. SPECgpc reserves the right to adapt its benchmarks as it deems necessary to preserve its goal of fair and useful benchmarking (e.g. remove benchmark, modify benchmark code or data, etc). If a change is made to the suite, SPECgpc will notify the appropriate parties (i.e. SPECgpc members and users of the benchmark) and SPECgpc will re-designate the metrics (e.g. changing the metric from SNX-01 composite to SNX-02 composite). In the case that a benchmark is removed in whole or in part, SPECgpc reserves the right to republish in summary form "adapted" results for previously published systems, converted to the new metric. In the case of other changes, such a republication may necessitate re-testing and may require support from the original test sponsor.
3. Overview of Optimizations
  1. SPECgpc is aware of the importance of optimizations in producing the best system performance. SPECgpc is also aware that it is sometimes hard to draw an exact line between legitimate optimizations that happen to benefit SPECgpc benchmarks and optimizations that specifically target SPECgpc benchmarks. However, with the list below, SPECgpc wants to increase awareness of implementers and end-users to issues of unwanted benchmark-specific optimizations that would be incompatible with SPECgpc's goal of fair benchmarking.
  2. To ensure that results are relevant to end-users, SPECgpc expects that the hardware and software implementations used for running SPECgpc benchmarks adhere to a set of general rules for optimizations.
4. General Rules for Optimization
  1. Optimizations must generate correct images. Correct images are those deemed by the SPECgpc committee to be sufficiently adherent to the respective graphics API specification for the targeted end-user community.
  2. Optimizations must not have an adverse effect on system stability. A published SPEC GPC result carries an implicit claim that the performance methods employed are more than just "prototype" or "experimental" or "research" methods. It is a claim that there is a certain level of maturity and general applicability in its methods.
  3. Optimizations affecting Viewperf must also improve performance for at least one commonly-available application.
  4. It is not permitted to detect Viewperf in order to invoke an optimization.
  5. The Graphics API stream from the Viewperf binary must not be modified.
  6. Graphics Subsystem implementations must fully process Viewperf's Graphics and related API streams affecting the frame buffer and Graphics state.
  7. Optimizations must be generally available and supported by the providing vendor.
  8. In the case where it appears the guidelines in this document have not been followed, SPECgpc may investigate such a claim and request that the optimization in question (e.g. one using SPECgpc benchmark-specific pattern matching) be removed and the results resubmitted. Or, SPECgpc may request that the vendor correct the deficiency (e.g. make the optimization more general purpose or correct problems with image generation) before submitting results based on the optimization.

Benchmarks
1. Benchmark Definition
  1. Benchmark components are defined as
    1. code sets (e.g. SPECviewperf®),
    2. designated executables where included in the downloadable package
    3. run rules
    4. launch framework/GUI and associated benchmark definition file
    5. Viewsets
2. Benchmark Acceptance
  1. New or modified benchmark components require a 2/3-majority vote of the SPECgpc electorate to be accepted for publication.
  2. A minimum 3-week review period is required for new or significantly modified benchmark components.
  3. At the end of the review period a vote will be called to approve the proposed changes.
  4. An amendment to a benchmark component during the review period must be unanimously accepted. If not, the review period shall be restarted.
  5. It is the option of any future SPECviewperf viewset author(s) to require passing of selected conformance tests prior to submission of results for that viewset.
3. Benchmark Code Versioning
  1. Benchmark code is defined as the set of source code required to build and run a benchmark executable (e.g. SPECviewperf).
  2. SPECviewperf Benchmark code uses the following version coding: M.m.p (e.g. 12.0.1) M is the major release number, m is the minor release number and p is the patch level.
    1. The major release number is only incremented when large amounts of code are changed and the scripting language is dramatically changed as a result -- backward compatibility is highly unlikely when moving scripts or data sets between major releases (e.g. running v2 scripts on a v3 executable would almost certainly fail).
    2. The minor release number is bumped if some small set of code is replaced or removed - but the standard, unchanged scripts and data sets, as a whole, must run on the new version (but perhaps with different performance).
    3. Patch releases can contain additions of new properties and additions of new attributes to existing properties, but cannot change or remove any existing properties, attributes or functionality. These are typically used for bug fixes, small enhancements and so forth.
4. SPECviewperf Viewset Versioning
  1. The version of a SPECviewperf viewset should be incremented if:
    1. changes to SPECviewperf affect the performance of the viewset,
    2. or changes to the viewset script affect performance,
    3. or if the viewset data changes,
    4. or if rule changes affect the acceptance criteria.
  2. New results for the previous version of a viewset will no longer be published.
5. SPECviewperf Benchmark Release
  1. On the release date of a new benchmark, it replaces the previous benchmark on the public website. Submissions for the previous benchmark will no longer be accepted.
1. Benchmark Run Rules
  1. Benchmark Run Rules
    1. The system under test must perform all of the respective graphics API's functionality and other work requested by the benchmark.
    2. The systems under test must be OpenGL Conformant for the pixel format or visual used by the benchmark.
    3. Settings for environment variables, registry variables and hints must not disable compliant behavior.
    4. No interaction is allowed with the system under test during the benchmark, unless required by the benchmark.
    5. The system under test cannot skip frames during the benchmark run.
    6. It is not permissible to change the system configuration during the running of a given benchmark. For example, one cannot power off the system, make some changes, then power back on and run the rest of the benchmark.
    7. Screen grabs for SPECviewperf will be full window size.
    8. The color depth used must be at least 24 bits (true color), with at least 8 bits of red, 8 bits of green and 8 bits of blue.
    9. If a depth buffer is requested, it must have at least 24 bits of resolution.
    10. The display raster resolution must be at least 1920 pixels by 1080 pixels and must fully display all of the benchmark tests being submitted. Higher desktop resolutions may be used, and while the operating system may create the appearance of a higher raster resolution, the actual raster resolution remains at 1900 x 1060 and will be noted in the results file and log files. An optional feature to adjust the raster resolution to the native resolution of the display (less a ten-pixel border) may be selected, but not all viewsets will support this method of scaling. Therefore, native resolution runs shall not be accepted for publication or otherwise be considered official until such a point that all viewsets can be scaled to higher native display resolutions.
    11. Screen resolution must be large enough to run the individual tests at their requested window size, with no reduction or clipping of test window.
    12. Results to be made public must be generated by the official benchmark which may not be changed, except to correct errors in reporting by the benchmark, and these changes shall be disclosed voluntarily upon submission.
    13. Screen DPI scaling must be set to 100% on Microsoft Windows, and no UI or other window elements, including the task bar, may occlude the viewport rendering context.
    14. Tests may be run with or without a desktop/window manager, but must be run on some native windowing system.
    15. Virtualized configurations, defined as any operating system configuration running on a Hyper Visor or virtualization layer of any kind, must include the word "virtualized" in the comment field of the config.txt file. This information must be populated before the execution of the benchmark to ensure the results reflect this attribute.
    16. Virtualized configurations as defined above must also include a declaration of the transport layer name and version used by the virtualization software in parenthesis after the graphics accelerator listed in the Graphics Accelerator field of the Graphics Hardware Configuration section of the result file.
    17. Virtualized configurations as defined above must also include a declaration of the Hyper Visor name and version used by the virtualization software in parenthesis after the model listed in the Model field of the System Hardware Configuration section of the result file.
    18. All configurations shall include a link (URL) to a publicly-accessible device driver package used to generate the submission.
1. Submission and Review Rules
  1. Submission Content Rules
    1. These rules are specific to SPECgpc and shall apply in addition to the Submission Content Rules in the GWPG Project Groups Ruleset.
    2. A SPECviewperf submission can be for one or more viewsets per configuration.
    3. A SPECviewperf submission must be run on a 64-bit operating system.
    4. The Specviewperf submission must include all directory and files located in the results folder. The results folder is created by the benchmark with the following naming convention: "results_"YYYYMMDD"T"HHMM"_t1"
  2. Submission Process Rules
    1. These rules are specific to SPECgpc and shall apply in addition to the Submission Process Rules in the GWPG Project Groups Ruleset.
    2. The submission file names must contain gpc_v for SPECviewperf, contain all lower case letters and not contain '.' except prior to the zip or tar file extension (e.g. intel_gpc_v_jun10_v0.zip). The file version is denoted prior to the file extension. The initial file version is v0. Resubmitted files must increment the version number.
  3. Review Period Rules
    1. These rules are specific to SPECgpc and shall apply in addition to the Review Period Rules in the GWPG Project Groups Ruleset.
    2. Reviewers will decide if the image quality of the submission is sufficiently adherent to the respective graphics API's specification to satisfy the intended end user's expectations. If a reviewer rejects the quality of an image for a stated reason, the submitter can ask for a vote of the full SPECgpc electorate. In case of a tie the submission is rejected.
    3. System configurations submitted for the SPECgpc benchmark suite must be able to run the corresponding SPECapc application benchmarks if applicable. If this criterion is not met the submission will be rejected.

Adoption

Changes for version 1.1 adopted June 10, 1999
Changes for version 1.2 adopted January 12, 2000
V1.4 changes -- 5.02 (d) added
V1.5 changes -- 4.01.i.2(2), 4.01.i.2(4), 5.02.w
V1.6 changes -- 1.03.c, 5.04.o - Adopted by the SPECopc on January 23, 2003
V1.17 Adopted by the SPECopc on August 13, 2004
V1.18 Adopted by the SPECopc on October 21, 2004
V1.19 Adopted by the SPECopc on April 19, 2005
V1.20 Adopted by the SPECopc on October 20, 2005
V2.00 Adopted by the SPECopc on January 26, 2006
V2.10: Adopted on 13 September 2007 (reflects SPECopc->SPECgpc name change and wider API charter scope)
V2.11: Adopted on 20 April 2010 - Added section II.4 and rule II.4.a.
V2.12: Adopted on 23 June 2010 - Rules updated for SPECviewperf 11
V2.13: Adopted on 16 December 2010 - Optimization rules updated (section I.4)
V3.0: Adopted on 16 October 2013 - Updated for SPECviewperf 12
V3.1 Adopted on 23 May 2018 - Updated for SPECviewperf 13

Standard Performance Evaluation Corporation