Standard Performance Evaluation Corporation

SPEC Open Systems Group 
Policies and Procedures Document
Version 2.21

Recent Revisions

Version V2.21
5 Dec 2019
  • Updated 2.1.j: officer elections to take place at the OSSC meeting occurring during the SPEC annual meeting.
Version V2.20
29 Oct 2018
  • Clarified 2.1.c, and in section 2.1.d, add the obligation to attend all OSSC meetings, effective as of 29 Jan, 2019.
Version V2.19
23 Sep 2016
  • In section 2.8.1.1, clarify that "Only those who submit results participate in the initial 24-hour review".
Version V2.18
8 Oct 2015
  • In section 2.3.7, the terminology is changed from "Required Disclosure" to Required Full Disclosure, in order to emphasize that the disclosure must be complete.
Version V2.17
23 Aug 2013

For previous revision history, see the bottom of the document.

1. General Policies and Principles

The intent of this section is to provide an overview of the general policies and principles of the SPEC organization, as context for the policies of the Open Systems Group (OSG). The controlling documents for governance of SPEC are the Articles of Incorporation and Bylaws, filed with the State of California, and available from the SPEC Administrator.

In the event of disagreement between this document and SPEC's Bylaws, or disagreement between this document and decisions by the SPEC Board of Directors, said Bylaws or decisions shall take precedence over this document.

Note: much of the material in this section of this document resides here due to historical reasons. It is possible that SPEC's Board of Directors may decide to create a SPEC-wide document that will supersede this section at a later time, in which case this section will be removed and replaced with a pointer to the new material.

1.1 SPEC's Purpose

The primary purpose of the Standard Performance Evaluation Corporation (SPEC) is to develop suites of benchmark programs that are effective and fair in comparing the performance of high performance computing systems, and to assure that they are readily available to manufacturers and users of such systems.

1.2 SPEC's Formation

SPEC was formed from the instigation and sponsorship of Electronic Engineering Times (E.E. Times), and by the cooperative development work of Hewlett-Packard Co., Sun Microsystems Inc., Apollo Computer Inc., and MIPS Computer Systems Inc.

1.3 The Corporation

Upon approval by the representatives of the founding companies, their effort to develop a benchmark standardizing activity was converted into a non-profit Corporation of the state of California, on November 14, 1988.

The Corporation has a Board of Directors, President, and a staff to carry out the business of SPEC. The board has established three technical groups:

Each group has a steering committee to manage and supervise the development of applicable benchmarks. Each steering committee may sponsor a number of subcommittees or working groups to facilitate benchmark development. The Board may establish additional technical groups.

1.4 Meetings

The annual meeting of SPEC members will be held in January.

Election of the Board of Directors and members of the steering committee(s) and a financial report by the Board will be a necessary part of that meeting. The Board of Directors will meet at least quarterly.

1.5 The Board

To facilitate continuity and consistency on the Board, SPEC's policy is that the Board of Directors has staggered two-year terms, with half of the Board elected each year. Decisions of the board require agreement of the majority. Elections to the Board will be held at the annual meeting to be held at the beginning of the calendar year. Board positions are held by individuals, not by companies. Currently there are eight (8) elected directors on the board.

Each member of each SPEC technical group (OSG, GWPG, and HPG) may cast one vote per open board seat. E.g., if a company is a member of two groups it may cast two votes per open board seat.

1.5.1 Officers

The board elects a president, who generally serves for a term of one year, with eligibility for re-election, from among candidates proposed by the board members.

The board also elects a secretary and a treasurer of the Corporation, who are not required to be members of the board. The secretary and treasurer serve terms of one year and are eligible for re-election.

1.5.2 Administrative Duties

SPEC has an administrative organization that carries out the publicity, accounting, organizational and other necessary work of the Corporation. The administrative staff reports to the Board, through the president. The budget of the staff is developed by the president and the treasurer and approved by the board.

1.6 Participation levels

Terminology note about SPEC "Membership": SPEC is a California nonprofit mutual benefit corporation. Governing law has certain formal terminology regarding membership (see, for example, California Corporations Code sections 5056 and 7331). SPEC's Bylaws authorize only one class of membership in the corporation. It is understood that one sometimes sees the term "member" used in a looser sense, and indeed the law appears to recognize that the term "member" may be used in a less formal sense (7333). However, this section 1.6 follows the formal usage of the Bylaws, which clearly say that SPEC has only one class of "Membership".

1.6.1 Sustaining Memberships

Sustaining members are dues paying members and accrue all the rights, privileges and responsibilities that full membership entails.  Sustaining memberships are available to computing-related commercial and non-commercial entities as specified in the Bylaws. Any eligible entity that wishes to join SPEC may join provided they pay the initiation fee and annual dues as determined by the Board of Directors. The fees and dues for sustaining membership must not discriminate among prospective members.

Universities and non-profit institutions are eligible to become sustaining members, but at their option may prefer to join as "Non-profit Associates", as described in section 1.6.2.

Sustaining members may become active participants in any working group or subcommittee of the OSG.

Sustaining members are eligible to stand for election to the steering committee seats, to nominate candidates for offices and the Board of Directors, and to champion proposed benchmarks.

Sustaining members have the opportunity to review and comment on all benchmarks developed by the OSG. This can be done through active participation in the subcommittee developing the suite or during the benchmark's final call for comments.

1.6.2 Associates

To encourage participation from universities and other non-profit institutions interested in its work, SPEC maintains a SPEC Associate category that enables non-profit organizations to share in the SPEC process, but not to participate in general member voting.

Associates pay an initiation fee and annual dues as determined by the Board of Directors.

Non-profit Associates have the opportunity to review and comment on all benchmarks developed by the OSG. This can be done through active participation in the subcommittee developing the suite or during the benchmark's final call for comments.

1.6.3 Supporting Contributors

The role of a Supporting Contributor is available by invitation to a commercial organization, to an academic institution, or to an individual. A subcommittee may issue an invitation to an interested party to apply for this role.   The application must be approved by the Open Systems Steering Committee and the Board.  Supporting contributors may participate in only one subcommittee and have no voting rights and pay no dues.  Supporting contributors may help develop new benchmark suites, provide expertise to the subcommittee or help provide other support within the subcommittee.

1.6.4 Table of Membership Rights and Privileges

                            Sustaining   Associate  Supporting
                            Member                  Contributor
                            ----------   ---------  ----------

Dues                        Full         Discount   None

General Member Voting       Yes          No         No

Participate in meetings,    Yes          Yes        Yes (for 1 subcommittee)
concalls, benchmark 
development 

Eligible to run for OSSC    Yes          No         No

Eligible for Subcommittee   Yes          Yes        No
Voting Rights

Number of Subcommittees     Any/All      Any/All    1
                            in Group     in Group

Copies of Benchmarks        All Releases All        1
                            in Group     in Group

On SPEC email aliases       Any/All      Any/All    1
                            in Group     in Group

Access to members' private  Yes          Yes        Limited access con-
  server, or data from it                           trolled by subcommittee 
                                                    Chair, to subcommittee's 
                                                    materials 

Free Results publication    Unlimited    Unlimited  Subcommittee may award 1
on www.spec.org for                                 free - if in support of
Group's benchmarks                                  new benchmark release.
                                                    Otherwise non-member 
                                                    price.

Term Limit                  Until stop   Until stop 1 year.  Subcommittee may 
                            paying dues  paying     renew it annually. 
                                         dues       Subcommittee may terminate 
                                                    at any time.

Results Review              Yes          Yes        Cannot vote on results.  
                                                    May participate in review
                                                    discussion only if allowed 
                                                    by subcommittee.

1.7 Copyrights and Trademarks

The standards and documents authorized by the Corporation may be copyrighted or trademarked, as appropriate.

The purpose of the copyrights and trademarks is to assure the computer industry that programs and documents developed by SPEC are genuine. The intent of protecting by trademarks and copyrights is to promote confidence in the metrics, and prevent misrepresentation of them.

1.8 Openness

Information about the activities and developments of SPEC are intended to be readily available to all parties with legitimate interests in the performance of advanced computer systems. The board is responsible to carry out this policy.

SPEC meetings must be open to representatives of each member company. For guidelines regarding participation by non-members (including press representatives), see section B.6.

1.9 Fair Use of SPEC Benchmark Results

Consistency and fairness are guiding principles for SPEC. To help assure that these principles are met, any organization or individual who makes public use of SPEC benchmark results must do so in accordance with SPEC's Fair Use Policy, as posted at www.spec.org/fairuse.html.

2. Open Systems Group

The Open Systems Group (OSG) was the first SPEC group. This group focuses on benchmarks for computers running open systems environments.

2.1 Open Systems Steering Committee

The Open Systems Steering Committee (OSSC) manages and supervises the development of SPEC's OSG benchmarks. The intent of this section is to provide an overview of the general policies and principles of the Open Systems Steering Committee.

a. Exceptions to policy: For due cause, the OSSC may vote to approve exceptions to this policy or may adopt resolutions that may conflict with policies expressed in this document; in such cases an amendment to this document will be brought forth within a reasonable time period afterwards.

b. Conflict with Bylaws or Board: In the event of disagreement between OSSC resolutions vs. SPEC's Articles of Incorporation and Bylaws or decisions by the SPEC Board of Directors, the OSSC shall yield.

c. Size of OSSC: The Board sets the number of OSSC members. The OSSC may recommend to the Board an increase or decrease in size based on activities and participation.

d. Qualifications: The OSSC is elected from the OSG membership. OSSC members agree to the following conditions and obligations:

  1. Attend all OSSC meetings, including OSSC face-to-face meetings. (Effective as of 29 Jan 2019)
  2. Have the engineering resources to be able to technically review and help direct all OSG benchmarks.
  3. Devote the equivalent of at least one full-time engineer to SPEC OSG activities.
  4. Under the direction of the OSSC Chair, help to promote OSG cross-pollination and shared practices, for example by reviewing run rules, helping with testing, educating about shared methods, and/or serving as primary steering committee liaison to assigned subcommittee(s).
  5. Per the goal of the SPEC organization, commit to developing performance benchmark suites for computer systems and solutions.

e. Elections for OSSC: To maintain continuity and consistency of representation on the OSSC, the OSSC has staggered two-year terms, with half of the OSSC elected each year. Elections to the OSSC are held in conjunction with the SPEC Annual Meeting. Members representing one-fourth (1/4) of the OSG membership, present in person or by proxy, shall constitute a quorum for the purpose of election of OSSC members.

f. Vacancy: If an OSSC member institution resigns or leaves, the Board may appoint another OSG member to fulfill that OSSC member's remaining term, or may choose to leave the seat vacant until the next Annual Meeting, thereby temporarily decreasing the size of the OSSC.

g. Affiliation: Entering into a relationship of affiliation (as defined in the Bylaws, section 2.1) does not alone constitute the creation of a vacancy. (For example, Company B is not a member of the OSSC. It buys all the assets of, and takes on all the responsibilities of, Company A, including Company A's OSSC seat.) If, however, such affiliation would cause or appear to cause a member and its affiliate(s) to have two or more votes on the OSSC, then, as described in the Bylaws, a single voting representative shall be designated and one or more vacancies shall be declared.

h. Voting representative: Each OSSC member institution shall designate a voting representative. If the designated voting representative is absent from an OSSC meeting, another person employed by the member institution (or its affiliates) may temporarily represent the member. In the event of a lack of clarity as to who has the vote, the OSSC Chair may refuse to accept a vote until the designated voting representative appoints a substitute in writing.

i. Proxies: The OSSC does not allow use of proxy voting (that is, the authorization of an individual who is not an employee of a member institution to vote on behalf of that member), because OSSC members are expected to participate, to be actively involved in discussion, and to evaluate the evidence that may be presented. In rare cases, the Chair may accommodate legitimate scheduling difficulties (e.g. a Special Meeting scheduled at a particularly user-hostile hour for a directly interested participant), by requiring use of the Voting Tool rather than taking a vote during the meeting.

j. Officer elections: The OSSC elects a Chair, Vice-Chair, and Secretary. The election is held at the face-to-face OSSC meeting occurring during the SPEC annual meeting. OSSC Officer positions are held by individuals. It is not a requirement that the individuals be employed by an OSSC member company.

k. Election procedures are the same as for subcommittees, with these exceptions: notice of the election shall be sent to all OSG members; nominations are made via the steering committee mailing list, or at the meeting where the election is to be held; in the event of a tie, even after repeated attempts to break the tie, then the presiding officer shall use a randomized method to determine the winner.

l. Vacancy: An election will be held to fill a vacancy if an officer steps down, except for the case where the Chair steps down and a Vice-Chair has already been elected; in this case, the Vice-Chair shall become the Chair for the duration of the term, and an election for a new Vice-Chair must be held to fill the vacancy.

m. OSSC Quorum: In order to have quorum, the OSSC must have greater than 2/3 of its members present. (To calculate this, the table for the "SPEC Consensus Voting Rule" may be referenced in section 2.2.5.) Votes may be taken only if there is a quorum of the current voting members present.

n. Voting / passage of motions: To pass a proposal at the OSSC, the OSSC uses the "SPEC Consensus Voting Rule", which is defined in section 2.2.5.

All OSSC votes shall be open and public votes, except election of officers, which is by secret ballot. Exceptions to the secret ballot may be made if there is only one candidate per office.

It should be noted that the lack of passage of a motion by the OSSC does NOT imply approval of the CONVERSE of the motion.

o. Voting Tool: The SPEC private server offers a "SPEC OSSC Proposal/Voting Tool" which may be used to act on proposals separately from meetings. When this tool is used, it automatically enforces the SPEC Consensus Voting Rule.

p. Advisers: The OSSC can, after notification in writing to the Board of Directors, enlist the aid of advisers and consultants from the industry and academe.

q. Coordination with Board: Certain organizational activities require board participation and approval. For example, the board must approve press releases, setting prices for benchmark licenses, and any major resource reallocations of SPEC administrative staff and Public Relations to assist with the non-technical aspects of releasing a new benchmark suite. This work may be facilitated by a joint committee of the board and the OSSC (e.g. Planning Committee) or proposals may be brought directly to the board by the Subcommittee or Steering Committee Chair.

2.1.1 OSSC Meetings

a. Parliamentary procedure: OSSC meetings are conducted in accordance with general parliamentary procedures under the direction of the chair. Meeting participants may remind the chair of such principles, e.g. by raising a "point of order" from Robert's Rules, but the chair is responsible for interpretation of such principles, which in some cases may specify a greater degree of complexity than is required in order to accomplish SPEC's business. In such interpretations the chair shall use fairness as the primary criterion, and simplicity of process as a secondary criterion. The chair's interpretations may be over-ruled by the OSSC using its usual voting rules.

b. Regular and special meetings: OSSC meetings are of two types: regularly scheduled, or special. Special meetings may be called by the Chair, the Vice-Chair, or any 3 members, to handle urgent issues.

c. Notice of meeting: At least two full business days' notice is required prior to OSSC meetings, unless every voting member agrees to shorter notice. For example, if the meeting is called for Tuesday at 11AM PST, the meeting notice needs to go out by 11AM PST on the previous Friday.

d. Venue: Meetings may be face-to-face or by teleconference.

e. Face-to-face meetings: Face-to-face meetings take place about 4 times per year. Regular face-to-face meetings are considered essential because the steering committee often:

Remote call-in access to face-to-face meetings is typically not provided, as that would defeat much of the purpose of face-to-face meetings. Candidates for OSSC are strongly encouraged to take account of the face-to-face meeting requirements prior to running for office.

f. Meeting protocol: For regularly scheduled meetings, the OSSC has adopted the following meeting protocol:

2.1.2 OSSC Web Pages and Email Subscription

Steering Committee web pages (including OSSC Wiki pages) may be read by all members of the Open Systems Group. Any OSG member may subscribe to the email list for the OSSC.

2.2 OSG Guidelines for Working Groups and Subcommittees

The goal of the OSG is to produce benchmark software to characterize the performance of modern computer systems. To accomplish this work, the OSSC will from time to time charter subgroups of the membership with the responsibility to research and produce relevant benchmarking products.

There are two types of groups that the OSSC may charter: working groups and subcommittees. The requirements and goals for each are generally distinct, though occasionally there may be some overlap. This section attempts to clarify the purpose, organization, and responsibilities of each and to provide guidelines for the conduct of these groups.

2.2.1 The Working Group

As technologies develop, the OSG needs to keep pace and look toward the development of benchmarks to address the new technologies or to update older benchmark suites. To begin this process, the OSSC will sponsor the formation of a "working group". There will be a steering committee vote and the OSSC will appoint a chair. Specific notice will be sent to the Open Systems Group General Membership on the formation of a new working group. Participation in a working group is open to all interested OSG sustaining members, and all OSG non-profit associates. A supporting contributor may be invited to participate in a single working group. A working group takes on the following responsibilities:

Additionally the working group should work towards:

2.2.2 Subcommittee Formation

A subcommittee's role is to utilize the research and recommendations of the working group and produce finished benchmarks and run rules that can be released after completion of the Benchmark Release Process.

Once the working group's report has been delivered to the OSSC, the OSSC must evaluate the report and decide if a subcommittee should be chartered to take up the task of producing a benchmark for release. Alternately, the OSSC may assign the work to an existing subcommittee.

The OSSC provides the charter for the subcommittee, and must approve changes to it. (A working group produces a draft charter, which the OSSC adjusts as it deems appropriate.) The charter must be posted at one of the subcommittee's top-level web pages on SPEC's private server.

The OSSC must also determine if there is sufficient commitment from interested members to complete the benchmark. If sufficient resources cannot be identified, the OSSC may either table the project or direct the working group to continue with the resources available until enough members can actively participate to justify the founding of a subcommittee.

A minimum of three active members is required for the formation of a subcommittee.

Initial Chair The OSSC appoints the Initial Chair for the new subcommittee. The Initial Chair shall:

  1. Request support from the SPEC Office to create mailing lists, subcommittee web pages, and teleconference times.
  2. Ensure that all OSG members are aware of the new subcommittee and its meeting arrangements.
  3. Discuss publicity with the OSG Chair, who will provide guidance as to available methods to encourage participation by potential new SPEC members.

  4. Lead the subcommittee through the following actions, which must be completed by the conclusion of the second meeting of the subcommittee:

    1. Explicitly identify those who wish to be subcommittee voting members. Prospective voting members must commit to the work of benchmark development, as described in section 2.2.4 below.

    2. Consider whether the subcommittee would like to adopt simple majority as a voting method. The initial voting method for a subcommittee is the SPEC Consensus Voting Rule, until such time as it is changed. A proposal to change to simple majority is bound by the voting rule currently in effect (i.e. the Consensus Rule).

    3. Consider whether the subcommittee would like to adopt simple majority as its Quorum. The initial quorum requirement is 2/3 of the identified subcommittee members.

2.2.3 Joining a Subcommittee

Membership in any OSG subcommittee is open to any OSG sustaining member or OSG associate. Subcommittees may also include members who are non-voting supporting contributors.

Terminology note - Notice that it is therefore possible to be a member of a subcommittee without being a "sustaining member" of SPEC. For details of types of membership of SPEC, see section 1.6.

It is not required that one join a subcommittee as a voting member; informal participation by sustaining members and non-profit associates is also welcome. Informal participants may join in activities of the subcommittee, as they are able. This includes making themselves heard at meetings and contributing to the completion of the benchmark. They also have the opportunity to review and comment on benchmarks in development during the benchmark test phase and during the final call for comments.

2.2.4 Subcommittee Voting Rights

To join a subcommittee as a voting member, the joining party must:

  1. Be an OSG sustaining member or OSG associate.
  2. Agree in principle with the goals of the subcommittee as outlined in its Charter.
  3. Identify a main contact.
  4. If the subcommittee has been active for over three (3) months:
    1. Demonstrate its commitment during an initial four week period, meeting the contribution requirements of section 2.2.4
    2. During its initial four weeks, take responsibility for bootstrapping itself in the past efforts of the group. This includes reviewing meeting minutes and the group's email archive and obtaining assistance offline from the meetings to ensure they are up to speed on the issues.
    3. Attend two (2) consecutive meetings in addition to the requirements listed above.

In order to maintain voting rights in a subcommittee, a member must:

Status as a voting member may change:

Monthly reporting and effect on voting rights: Subcommittees are strongly encouraged to require submission of monthly status reports and may use these to help assess whether contribution requirements are being met. If so, the subcommittee must clearly define when reports are due, how they will be assessed, and how voting rights may be affected by them. The following process is recommended, but may be adjusted by a subcommittee. Any adjusted process must retain a mechanism for protest of assessments similar to item 6 below, to ensure protection of the rights of potentially disenfranchised members.

  1. Reports are due on the last day of each month.

  2. Satisfactory contribution reports list tangible (i.e., backed up by a deliverable and/or evidence) contributions made, such as: checked-in/posted code, checked-in/posted bug fixes, documented bug reports, documented code reviews, checked-in/posted porting, testing bug fixes/new versions of benchmarks and providing written test reports, performance analysis reports, benchmark documentation, documentation edits, run rule editing, technical analyses, minute taking, chairing/running/leading/facilitating meetings, SPEC web site content contributions, assisting users with benchmark questions and configuration issues, managing releases, configuration management, results reviews, and participating in working group meetings.

  3. Result submission activities do not qualify as contributions. Activities which do not result in a tangible deliverable to the subcommittee generally do not qualify as contributions. For example, testing bug fixes/new versions of a benchmark under development do not qualify as a contribution unless the results of that testing are reported to the subcommittee.

  4. Prior to the first meeting of a month, the Chair will send an email to assess which contribution reports meet subcommittee requirements. If any are negative (i.e. assessed as inadequate), the Chair will plainly state why. Members may correct deficiencies or file late reports prior to the subcommittee's first meeting of the month.

  5. If two consecutive assessments are negative, a member's voting rights will be suspended as of the beginning of the next meeting after the Chair's email.

  6. In the event of disagreement with the Chair's assessment of contributions, whether positive or negative, any member may protest the assessment, and a proposal may be made to overturn it. For such a proposal to be considered, it must be sent prior to the second meeting following the sending of the assessment, or must be entered at the beginning of said second meeting; otherwise, the Chair's assessment stands. Proposals to overturn will be considered using the subcommittee's usual voting rules, with the proviso that members whose voting rights would be affected by passage of any such proposal(s) at that meeting are nevertheless entitled to vote on all such proposal(s) that are under consideration at that meeting.

    Example: a member's April contributions are ruled inadequate on May 1. There is no protest. The same member's May contributions are ruled inadequate on June 1. The member's voting rights are suspended, unless a proposal is made and passed to overturn the second assessment. It is too late to protest the first one.

2.2.5 Subcommittee Organization and Voting

a. Initial Chair; election timing. Upon subcommittee formation, the OSSC appoints an initial Chair with certain responsibilities as described above. After 3 to 6 months, preferably at a face-to-face meeting, the subcommittee will elect its own officers. Thereafter, subcommittee elections will take place at the SPEC annual face-to-face meeting.

b. Officers. The subcommittee elects a Chair and Vice-Chair. Other officer positions may be created by subcommittees as needed. Timely minutes are required, whether by a formal Secretary position, or by some other clearly identified mechanism.

c. Vacancy. Subcommittees must hold an election to fill a vacancy when an officer steps down, except for the case where the Chair steps down and a Vice-Chair has already been elected; in this case, the Vice-Chair shall become the Chair for the duration of the term, and an election for a new Vice-Chair must be held to fill the vacancy. The election to fill the vacancy should be held quickly as practical; the requirement to announce elections 4 weeks in advance may be waived by unanimous consent of the regular voting members of the subcommittee.

d. Posting. A list of the elected subcommittee officers must be readily available to any SPEC member, either on, or directly linked from, the top-level page for the subcommittee on the SPEC private server.

e. Quorum: The subcommittee should select its quorum to be either 2/3rds of the active members or a simple majority of the active members. It is also required that quorum not drop below 3 members. Votes may be taken only if there is a quorum of the current voting subcommittee members present.

If a subcommittee repeatedly has difficulty meeting its quorum requirement, the subcommittee Chair shall notify the OSSC of such difficulty in a timely fashion. The subcommittee chair shall also notify the OSSC if quorum difficulty is anticipated. The OSSC may wish to prepare for possible dissolution of the subcommittee, or may wish to take preventive measures to improve participation.

f. Voting: A subcommittee should drive for consensus among its participants whenever possible. However to ensure progress, formal member votes will need to taken. An active member may vote 'yes', 'no', 'abstain', or 'pass' when the vote is first called. If the member votes 'pass', the chair will return to that member to finalize their vote as either 'yes', no', or 'abstain' after the remaining members have voted. Votes of 'abstain' count only toward establishing quorum. The chair should see to it that members' reasons for voting 'no' or 'abstain' are recorded in the minutes. Such recording can help with the drive for consensus, as it allows others to respond to the specific concerns of those who do not vote 'yes'.

g. Passage of motions: The subcommittee selects whether passage of motions requires simple majority of the people voting yes' or 'no', or >2/3rds of the people voting 'yes' or 'no', known as the "SPEC Consensus Voting Rule". The advantage of simple majority is that it may allow progress to be made more quickly; the advantage of the consensus voting rule is that it may force deeper exploration of common ground. (The consensus voting rule may also sometimes lead to deadlock.) A proposal to switch voting rules is itself bound by the rule currently in effect. The table that follows details the number of votes required to pass a motion if the SPEC Consensus Voting Rule is in effect.

SPEC Consensus Voting Rule
Voting yes or no
3
4
5
6
7
8
9
10
11
12
13
14
15
Votes required
3
3
4
5
5
6
7
7
8
9
9
10
11

2.2.6 Subcommittee Election Procedures

  1. Elections shall be announced at least 4 weeks in advance, with date and time expressed using US Pacific Time (PDT or PST, according to the season). As mentioned above, elections take place at the annual face-to-face meeting and at certain other occasions as may be required. For such other occasions, face-to-face is preferred, but a teleconference may also be used.
  2. Nominations may be made via email to the subcommittee mailing list.
  3. A candidate may nominate himself or herself, or may second his or her own nomination.
  4. In order to be considered, a nomination must be seconded and the candidate must agree to the nomination.
  5. At the beginning of the meeting where the election is to be held, the Chair shall confirm who are the active/voting members.
  6. At the meeting, an additional opportunity shall be provided to make nominations, to second nominations, and for candidates to accept or decline nominations.
  7. A formal motion shall be made and passed to close nominations.
  8. After the close of nominations, for contested elections, an opportunity shall be provided for short statements by all candidates. Candidates also have the option of stating their credentials via email.
  9. For non-contested elections (i.e. one nominee for the position), a voice vote may be taken. If all positions are uncontested, a voice vote may be taken for the entire slate.
  10. For contested elections, a subcommittee member who is not standing for the position in question shall preside during the voting for that position.

  11. For contested elections, the voting procedure depends on whether the vote is at a face-to-face meeting:

    1. If the election is face-to-face, written (secret) ballot shall be used. The presiding officer shall count the votes and announce the winner, with a second member confirming.
    2. If the election is not face-to-face, then the ballot may be secret on request from any member. In such a case, the presiding officer for the election, and a member of the SPEC Office, shall count the votes and announce the winner.
  12. The winner is the candidate who receives the highest number of votes (i.e. a plurality is sufficient).
  13. In the event of a tie for highest number of votes, a runoff election shall be held to decide among the tied candidates. If necessary, this step shall be repeated. If the tie cannot be broken after 3 attempts at a runoff, then the Chair of the steering committee shall use a randomized method to determine the winner.

2.2.7 Subcommittee Dissolution

The OSSC creates subcommittees, and it is the OSSC that dissolves them.

A subcommittee may recommend dissolution, or the OSSC may act without any formal recommendation from elsewhere.

If a proposal is made to dissolve a subcommittee, the OSSC could accept such a proposal, or it might take other actions, such as adjusting a subcommittee charter, merging it with another subcommittee, or leaving it with an 'inactive' status.

When a subcommittee is dissolved, a formal Dissolution Plan is required, which must address:

Preventive Measures: The dissolution of a subcommittee should not come as a suprise to the OSSC. If there is a risk that a subcommittee may dissolve in the future, the OSG Chair should be well informed by the subcommittee chair as to the risk. Reasons for the risk should be identified - is it simply resources? Is it that interest in the area has fallen? Efforts may be made to recruit new members, a recommendation may be made to adjust the subcommittee charter, and other ideas may be discussed between the subcommittee and OSSC.

2.3 OSG Guidelines for Result Submission and Review

The following sections address the guidelines for submitting benchmark results to the OSG for review and publication on SPEC's website.

Each benchmark suite produced by the OSG includes a set of specific run and reporting rules that the licensee must follow to produce a publishable result. These basic tenets of these rules require:

Furthermore, SPEC expects that any public use of results from this benchmark suite shall be for systems and configurations that are appropriate for public consumption and comparison. Thus, it is also expected that:

SPEC reserves the right to adapt the benchmark codes, workloads, and rules of its benchmarks as deemed necessary to preserve the goal of fair benchmarking. SPEC will notify members and licensees whenever it makes changes to a benchmark or its run and reporting rules.

2.3.1 Results Review

a. Peer Review: Each subcommittee is responsible for establishing a regular process for peer review of results, to help improve consistency in the understanding, application, and interpretation of the run rules. Typically, this is a two-week cycle where any results received by the start of a review cycle are reviewed by the subcommittee members over the following two weeks.

b. Review may or may not be required prior to independent publication, depending on the benchmark run rules. Some benchmarks (for example, SPECjAppServer2004) require that results be submitted to SPEC for review prior to publication. Other benchmarks (for example, SPEC CPU2006) allow licensees to publish rule-compliant results independently, without formal review by SPEC. Independently published results are still expected to comply with the run rules, and SPEC may choose to review such results, as described in section 2.3.7 on Required Full Disclosures.

c. Results remain responsibility of tester: Although SPEC reviews results, and accepts them for publication on its website, the results nevertheless remain the responsibility of the tester, as noted in the disclaimer at http://www.spec.org/spec/disclaimer.html.

d. Primary Reviewer(s): In order to ensure that every result is read carefully, the subcommittee Chair assigns one or more Primary Reviewer(s) for each result. The Primary Reviewer sends a written review to the subcommittee alias. Exception: If the test sponsor is not a member of the SPEC Open Systems Group, then the Primary Reviewer communicates directly with the test sponsor, rather than using the subcommittee alias, because non-members are not allowed to subscribe to member aliases. The correspondence should be copied separately to the subcommittee alias (or sent as "bcc"). The subcommittee Chair should consult with the SPEC Office in order to locate the membership lists, and/or to answer any questions as to whether a result is from a member or non-member.

The Primary Reviewer should not be a vendor for performance-relevant technology used in producing the result. If for some reason this is not practical, then additional reviewers should be assigned, so as to reduce the appearance of conflict of interest. Review is not limited to the Primary Reviewer(s): all subcommittee members are encouraged to review details of all results.

e. Written direction for disposition of results: At the conclusion of the review, written direction must be provided as to what is to be done with the result submissions. Such direction shall be provided on the same day as the conclusion of the review. The disposition of results shall be in writing, leaving a history that is readily visible to both editorial staff and to all subcommittee members. The precise mechanism shall be agreed to between the Editorial staff and the subcommittee (e.g. meeting minutes, separate memo from the Chair.)

f. Results with pending ACTION(s): In some cases, the disposition of results may note the need for corrections, such as documentation of tuning options. Any needed ACTION must clearly state which result submission(s) is (are) affected and what needs to be done. The description of the action must also clearly state who is responsible for declaring it complete. There are four typical choices:

  1. Upon notification by tester: For minor editorial concerns, the tester may be designated as the responsible party to notify that the ACTION is complete. After finishing, the tester notifies the editorial staff and the subcommittee alias as to what has been done to fulfill the action, and requests publication of the affected result(s). Note: allowing the tester himself or herself to declare an action complete should be reserved for very minor updates, such as fixing a simple spelling error; the more typical case is described in the next paragraph.

  2. Upon notification by the primary reviewer: Usually if an editorial ACTION is requested during review, the primary reviewer should be designated as the responsible party to declare whether it has been completed. Once it is done, the primary reviewer notifies the editorial staff and the subcommittee alias that the action is complete, and requests publication of the affected result(s). In the event that the primary reviewer is not available, an elected subcommittee officer may substitute.

  3. Pending email discussion [2-day minimum] and Chair Direction: Some questions may require research by the tester and then opportunity for further comment by the subcommittee. Such discussion may be taken to the subcommittee email alias, provided that a minimum of 2 business days is allowed for such comments. Upon completion of the designated time, the Chair notifies the editorial staff and the subcommittee alias either that the result can be published, or that it will need to be held pending further discussion at the next subcommittee meeting.

  4. Pending discussion at a subcommittee meeting: For issues that affect rule compliance, the result may be held until the subcommittee receives answers to its questions and has the opportunity to discuss the answers at a scheduled meeting of the subcommittee.

Examples: In all the examples that follow, company TS is the Test Sponsor, company PR is the Primary Reviewer, and Rex is the Chair of the subcommittee. The four examples are intended to parallel the 4 paragraphs just above.

  1. Result 523 refers to "Fortan".
    ACTION: TS to fix the typo and TS to notify when complete.
  2. Result 634 says that this OS was available last June, but other results have all said it was last August.
    ACTION: TS to check the date and correct as needed; PR to notify once complete.
  3. Some reviewers question whether result 745 uses tuning that might not be supported, but TS is convinced that the tuning is documented with other supported options.
    ACTION: TS to send a pointer to that documentation. Result 745 will be held during email discussion of TS's answer; if there are no objections within 2 business days after TS's answer, then Rex will direct that the result be published.
  4. Result 856 says that this OS will be available next March. That would fall outside the acceptable 3-month window.
    ACTION: TS to research the date, and results are to be held until discussion at a future subcommittee meeting.

g. Timing of publication: The SPEC Editorial staff will work to publish results that pass review in a timely fashion on SPEC's web site, by the next business day after the above-mentioned results disposition write-up. If a result has an ACTION that needs to be fulfilled, publication will not occur until after written notification that the ACTION is complete. Such notification shall be provided to both the editorial staff and the subcommittee, after which the editorial staff will publish the result by the next business day.

h. Fees for publication: As described at SPEC's website, www.spec.org, any licensee may submit results to SPEC for technical review and, if accepted, publication on SPEC's website. Publication fees are charged for non-members (see the table in section 1.6.4). Technical subcommittees are not expected to track whether such fees have been paid. Rather, the Editorial staff will coordinate with SPEC's billing department to ensure that any required fees are paid prior to publication. Therefore, even if the results disposition write-up indicates that a result should be published, publication of fee-required submissions will be delayed until payment is confirmed. The Editorial staff will publish results that require fees by the next business day after whichever of these occurs last: completion of technical review; completion of any pending ACTIONs; confirmation of payment.

i. Hold at request of sponsor: At the request of the test sponsor, results that have passed review may nevertheless be requested to be held, for example until the date of a product announcement. Such requests should be sent to the subcommittee alias and to the editorial alias, no later than the end of the review period, and should also be noted in the results disposition write-up.

j. Review during holidays: In the event that holidays interfere with review schedules, it is the responsibility of the subcommittee Chair to ensure that results are still carefully reviewed. Doing so may require advance planning, for example, asking that members who are out of town recruit other employees of their companies to stand in their stead. If essential, a subcommittee might extend a review period in order to accommodate holidays, but this is not the preferred solution. Instead, it is recommended that the subcommittee accelerate the review. Note that the publication date is not to be accelerated.

For example, the SPEC Office is typically closed on 25 December. A set of results begins review on 11 December. The subcommittee holds the review of results meeting on the 18th, and sends out a disposition of results memo the same day. The results which pass review are published by the editorial staff on the SPEC Office's first business day after the holiday.

k. Proposals of non-compliance and appeal thereof: If there are open issues for a given submission that have not and cannot be resolved to the satisfaction of the subcommittee, the following process must be followed:

l. Re-runs for major corrections: If major corrections are required for a result, for example those that necessitate a new test run, then the corrected result starts review over as a new submission.

m. Expedited review: Under the direction of its Chair, a subcommittee may allow an "expedited review" of a corrected result, for example, a 2-business-day review of a re-run. An expedited review is typically appropriate when minor changes are made, and only the minimum changes needed to meet SPEC's concerns. It is typically not appropriate when additional changes have been made. When deciding whether to allow an expedited review, the Chair's primary criterion should be whether the result can receive a full and fair review, taking into account the degree of change, the availability of reviewers, and the other workload in the subcommittee. The Chair's determination may be over-ruled by the usual voting rules of the subcommittee.

n. Use of New Optimizations: To assure fairness to the submitter, new and unique optimization techniques that appear in a result disclosure should not be utilized by another member until the original submission has completed its review cycle. The subcommittee may decide to suspend a result in review should this guideline not be followed.

o. SPEC Confidentiality: Members participating in the review of results are expected to be familiar with the Guidelines for Handling SPEC Information, particularly section 7.0 dealing with Results Review (see Appendix B.)

2.3.2 Handling OSG Run Rule Violations in Published Results

The subcommittee may also [vote to] undertake a re-review of a previously published result, if significant questions as to its adherence to the applicable run and reporting rules arise. The OSSC reserves the ability to denote a publish result as non-compliant if it has been found in violation of the run and reporting rules.

The following process has been established to handle instances where run and reporting rule violations are discovered in results that have been published on SPEC's web site.

The subcommittee is responsible for identifying that a result published on the public web site is in violation of their benchmark run and reporting rules. Once a subcommittee identifies such a result, the result may re-enter a review process unless the submitter agrees with the finding of non-compliance.

If the subcommittee's review returns a finding that the result is not in compliance with the benchmark's run and reporting rules, the following should occur.
The results are modified:

  1. To remove the metric values (summary & individual) and replace with "NC" (non-compliant).
  2. Add a header to the disclosure stating that the result did not comply to the rules and the reason(s) why.
  3. The submitter, with the subcommittee's approval of the text, may add additional wording into a remedy section. The remedy section is expected to explain how the vendor has addressed the problem. For example, mentioning that a new result has been submitted on the same platform with the issue resolved is the expected typical usage of this section.

2.3.3 Handling Continued Availability in Published Results

The various SPEC benchmarks' Run and Reporting Rules have clauses that at a certain time after the first publication (on SPEC's web site or otherwise), all components of the System Under Test (SUT) must be available for general customer shipment. The purpose of this requirement is

For the case that this availability status cannot be maintained over a minimum time length, the following rules apply:

Either at the request of the test sponsor or as a result of a subcommittee resolution, the subcommittee can undertake a re-review of a previously published result if a performance-relevant component ceases to be available for 30 days or more within 90 days after initial overall availability (i.e. initial availability of all components of the SUT). If the subcommittee's review returns a finding that this non-availability condition holds for a performance-relevant component, the result is marked "Not Available" (NA) in the following way:

  1. The metric values (summary and individual) are removed and replaced with "NA" (not available).
  2. A header is added to the disclosure stating that the result is for a system that is, at a time specified in the header, not available.
  3. The test sponsor can, with the subcommittee's approval of the text, add additional wording into a remedy section. The remedy section can for example, identify the part that is not available, contain the reason for non-availability, and/or an estimate when all components of the SUT are expected to be available again. If and when the SUT becomes available, the submitter may notify the subcommittee that the product is available. With the Subcommittee's agreement, the NA marking will be removed from the result page, however the page will retain the notice that the page was marked NA along with the history of the NA marking. This restarts the clock for the continuous availability requirement covered in this section.

For results not published by SPEC, the subcommittee can ask the publisher to withdraw the publication.

"Performance-relevant" is defined in the various benchmarks' Run and Reporting Rules' reproducibility clauses.

Replacement by a new similar component with equivalent or better performance is considered acceptable and not a reason for an NA marking.

In cases where the Run Rules allow use of a beta product, this beta product or a later equivalent or better performing regular product (see clause above) must remain available.

Note: Effective for results published on or after July 1, 2002.

2.3.4 Regarding the calculation of a required date for availability

Availability has sometimes been referenced as "within 3 months" or "within 90 days" of first publication. Both of these phrases have sometimes introduced questions as to interpretation. To avoid confusion, benchmark run rules must shift to using "within 3 months", which shall be interpreted as follows:

  1. Determine the date of first publication.
    • If SPEC is the first to publish the result, the date of publication can be found via the records maintained by SPEC's editorial staff as to when results move to www.spec.org.
    • If some other entity publishes the result first, the publication date is considered to be the date according to the time zone in effect at the headquarters of the entity publishing the result.
  2. Increment the month number by 3, modulo 12, to calculate the required month.
  3. Keep the day number the same.
  4. In the event that the day number does not exist in the required month, select the highest day number that does exist.
  5. The product must be available on the calculated date (any time zone).

Examples:

  1. A result is published on April 15. The product must be available on July 15.
  2. A result is published on 30 November 2012. The product must be available on 28 February 2013.
  3. A result is published at an industry conference in London, England (GMT+0), by Company A just after 11:00 AM on January 15. Company A has its corporate headquarters in Christchurch, New Zealand (GMT+13, with Daylight Savings Time in effect) where the date at that moment is January 16. Therefore, the date of publication is considered to be January 16. The result uses software that becomes generally available via download through Company A's electronic store. At the moment that order fulfillment begins, it is still April 16th on part of the planet (e.g. in areas that use GMT-12). The product was available within the required time window.

2.3.5 SUT Availability for Historical Systems

In the interest of providing some perspective on the performance and/or power consumption for older or obsolete systems, SPEC may consider review and publication of historical results.

Such publication may be considered for systems where the general availability date is beyond general availability limits expressed in the benchmark run rules, or for systems which may no longer meet the General Availability guidelines in this policy document.

The hardware and software availability dates published on the result page will normally reflect the original general availability of the platform. In some cases, the technical subcommittee may choose to allow publication using a date that reflects general availability of what the review committee considers to be the "core components", for example (depending on the benchmark and the historical question under consideration), the chipset and processor combination; or (as another example) the primary software under test.

For historical results, this entry must be included in the appropriate Notes section:

   This benchmark result is intended to provide perspective on
   past [power and/or] performance using the historical hardware
   and/or software described on this result page.

   The system as described on this result page was formerly
   generally available.  At the time of this publication, it may
   not be shipping, and/or may not be supported, and/or may fail
   to meet other tests of General Availability described in the
   SPEC OSG Policy document, http://www.spec.org/osg/policy.html

   This measured result may not be representative of the result
   that would be measured were this benchmark run with hardware
   and software available as of the publication date.

In the above, include the phrase in [square brackets] for benchmarks that include a power component.

2.3.6 Corrections to published results

SPEC's Editorial Committee controls both the mechanics of publishing results (e.g. procedures for transferring raw experiment data to SPEC in the required formats) and the updating of already published results. The Editor is responsible for keeping SPEC licensees informed about how to accomplish these tasks, via maintenance of appropriate instruction pages on SPEC's server.

The key points of the Editorial process for correcting results can be summarized as follows:

2.3.7 Required Full Disclosure for Independently Published Results

[Note: This section was formerly titled "Required Disclosure for Independently Published Results". The word "Full" has been added in order to emphasize that the disclosure must be complete.]

Some benchmarks allow independent publication without review by SPEC. When a result is published independently, SPEC may require full disclosure of details and may nevertheless undertake a review, as provided in this section.

If a SPEC licensee publicly discloses a number from a benchmark that is alleged to be a compliant result (for example in a press release, academic paper, magazine article, or public web site), and does not clearly mark the number as an estimate (if estimates are allowed by the benchmark's run rules), any SPEC member may request that a full disclosure of the result be forwarded to SPEC. The disclosure is expected to be complete, including the configuration description. The full disclosure must be made available to the relevant subcommittee no later than 10 working days after the request.

Public Information: A Required Full Disclosure is considered public information as soon as it is provided, including the configuration description.

For example, Company A claims a result of 1000 SPECint_rate2006. A Required Full Disclosure is requested, and supplied. Company B notices that the result was achieved by stringing together 50 chips in single-user mode. Company B is free to use this information in public (e.g. it could compare the Company A machine vs. a Company B machine that scores 999 using only 25 chips in multi-user mode).

Note that nothing in this section shall be taken to indicate that information that would normally be confidential in a SPEC submission would be public for a Required Full Disclosure.

For example, during review of submitted results, a particular subcommittee routinely asks for the name of a designated responsible engineer (DRE) who ran the result. The DRE answers questions for the group. The DRE name is not considered to be public information and is not published. For a Required Full Disclosure, the name of the DRE is still expected to be provided to the subcommittee, but, as usual, is not considered public information.

Review of the result: Any SPEC member may request that a Required Full Disclosure be formally reviewed by the relevant subcommittee, using its normal review processes. At the conclusion of the review, if the tester does not wish to have the result posted on the SPEC result pages, the result will not be posted. Nevertheless, as described above, the details of the disclosure are public information.

SPEC may take action: When public claims are made about SPEC results, by any party (including but not limited to members, licensees, the press, academic researchers), SPEC reserves the right to take action if the full disclosure is not made available, or shows different performance than the tester's claim, has other rule violations, or violates SPEC policies (such as the Fair Use policy).

2.4 OSG Technical Support Guidelines

The members of the OSSC and its subcommittees are responsible for providing adequate resources to handle technical support questions regarding current benchmark suites. It is the goal of technical support to provide timely response to questions regarding the nature and usage of these benchmark suites to help facilitate testing by other SPEC benchmark licensees. The purpose of SPEC's technical support is NOT to provide OS, compiler or other vendor tool support or assistance. At the technical support person's discretion, the user may be referred to a third party, who may charge a fee for such support.

For each released benchmark, the SPEC administrator will maintain an email alias (for example cpu2000support@spec.org). Support aliases should be mentioned with benchmark documentation (perhaps in a section on how to get technical support.)

The committee responsible for that benchmark will establish a quarterly rotation schedule among its active members to serve as technical support person.

In special cases a member may be assigned to provide support for specific subset of technical support questions for an extended period or to handle support for a specific geography. The email alias includes the subcommittee chair, representatives from the company up for that quarter's rotation, members with special duties and OSG chair. The chair's responsibility is to see that questions are answered promptly and provide backup when the vendor on rotation becomes unavailable. The current support representative or chairs may assign a technical support call to another subcommittee member, if they believe vendor specific knowledge may be required to resolve the call.

The SPEC office may be the initial point of contact for some technical support questions (for example, questions sent to a general SPEC alias, rather than a specific benchmark support alias). Depending on expertise, time available, and the complexity of the question, the office will either answer the question, or determine which specific alias to refer the problem to.

2.5 OSG General Member Guidelines

The OSG is open to any entities that have an interest in the use and development of standardized performance benchmarks for computer systems running open systems environments. Please refer to section 1.6 for information regarding the rights that are allowed for various participation levels.

Members are expected to uphold the policies and principles endorsed by SPEC and the Open System Group. This includes policies described in this document and amended by resolutions of the OSSC; SPEC's Articles of Incorporation and Bylaws; policies adopted by the SPEC Board of Directors; and any applicable set of benchmark run and reporting rules.

Members are also expected to remain members in good standing by the prompt remittance of annual dues. Membership privileges will be suspended if dues have not been received by the SPEC administrator by March 31st. Membership privileges will be restored once dues have been paid.

2.6 OSG General Membership Voting

As a sustaining member of SPEC and the OSG, each member has the right to participate in any ballots of the general membership. The OSG membership votes to elect members to the Open Systems Steering Committee (OSSC).

Elections for seats on the OSSC are held at the Annual Meeting in January.  The OSSC has staggered two-year terms, with half of the steering committee elected each year. Any OSG member in good standing may vote.

2.7 Benchmark Development

This section describes important aspects of the SPEC benchmark development processes.

2.7.1 Checklists

SPEC maintains useful checklists for benchmark developers. Such checklists provide additional detail about SPEC's processes beyond that covered in this policy document.

a. Location: Development checklists are accessible from the top page for the Open Systems Steering Committee on the internal SPEC Member website.

b. Subject to change: The checklists are kept in a wiki so that they can be easily modified as conditions change and as additional details are noticed that are worth bringing to the attention of all development teams.

c. Checklist usage required: All subcommittees are required to consider checklists that apply to their development activities.

d. Flexibility: It is understood that some flexibility is appropriate. Development processes may vary depending on the complexity of the benchmark, the target audience, its expertise, and the nature of the code being developed. Therefore, it is normal that some checklist items may be reported as "not applicable".

e. Checklist reports: Certain milestones (such as release) require reports relative to checklists. When reporting:

2.7.2 Permission to Use

No code may be worked on by SPEC until it is clear that SPEC has permission to use that code. No code shall be circulated among SPEC members, nor stored on SPEC servers, unless the permission is clear. If there is doubt about permission, SPEC's legal counsel must be consulted.

Note: use of legal counsel should be coordinated per the OSSC Chair's direction.

The above applies to all code, no matter where it comes from. The following paragraphs discuss some common examples.

2.7.3 Transparency

Subcommittees should strive to make their development process transparent to subcommittee members. Code repositories should be freely readable by all subcommittee members. Draft run rules and draft documentation should be in the code repository, or posted to some other location that is accessible by all subcommittee members. Requests for enhancement and bug reports should be visible to all subcommittee members.

2.7.4 Run and Reporting Rules

Clear Run and Reporting Rules must be developed. The rules should strive to ensure that results are meaningful, comparable, and reproducible, with full disclosure of relevant conditions. The effort required to develop clear run and reporting rules may, in some cases, rival the amount of effort spent on development of benchmark code. Subcommittees are encouraged to bring rules issues to the surface early, so that they can be discussed as fully and as impartially as possible. On the SPEC private server, a template is maintained to assist with run rule development.

2.7.5 Automation

Wherever practical, reproducibility and rule compliance should be improved by automation of benchmarking run and reporting procedures. Benchmark products may use automated mechanisms (for example, shell scripts, programs, and scripting languages) to control how benchmarks are run, to record system conditions, or to generate reports. It is understood that not all rules can be enforced by automatic mechanisms, but wherever it is practical to do so, automated enforcement may eliminate otherwise frustrating sources of variability.

Disclaimer: Testers should note that passing the automated tests is not necessarily sufficient to constitute proof that all rules have been met since, as noted just above, run rule enforcement is automated only where it is practical to do such automation.

Experimental Record: Each benchmark product automatically produces some form of Experimental Record, a well defined artifact which is contemporaneously generated when the system is actually tested. The Experimental Record might include log files, observations of response times, server traces, power observations, and/or throughput data.

2.7.6 Documentation

All SPEC benchmarks must create documentation as part of the benchmark development effort. The documentation must:

The documentation is required to be sufficiently complete so that a user with the appropriate technical knowledge can install and run the benchmark without direct assistance from developers of the benchmark. Testing must be performed to ensure that this requirement is met.

2.7.7 Representativeness vs. Cost of Benchmarking

SPEC benchmarks are expected to appropriately represent some aspect of real-world computing. Often benchmarks are developed by starting with real-life code found in production applications, or by describing usage scenarios that are considered typical of real applications.

It is understood that all benchmarks simplify the real world, in order to be well-defined, repeatable, and tractable.

Benchmarks may also simplify the real world in order to concentrate on a focus area of interest, such the CPU (such as SPEC CPU2006) or the JVM (such as SPECjvm2008). Benchmarks with a concentrated focus are sometimes referred to as "component" benchmarks. Component benchmarks are usually simpler to run, with a lower cost of benchmarking.

Benchmarks that model wider scenarios strive for broader coverage, and are sometimes referred to as "system" benchmarks (such as SPECjAppServer2004). Such benchmarks may provide a more realistic picture of whole-system performance, although with a higher cost of benchmarking.

SPEC considers both types of benchmark products to be useful, although with differing trade-offs between representativeness and cost of benchmarking. During the development effort, and in the published documentation, each benchmark product should strive for clarity: what is the focus area of testing? What is left aside?

2.8 Benchmark Release Process

2.8.1 New Benchmarks

The process to approve the release of new benchmarks (such as SPECweb2005 or SPEC CPU2006) consists of several steps. The subcommittee that developed the benchmark must finalize the benchmark and prepare a beta version that can be recommended for general membership review. When this beta is ready, the subcommittee must send out a final call for comments to the general membership. A member has four weeks from that time to review and comment on the benchmark. The member should send their comments back to the subcommittee and the OSSC.

At the conclusion of the final call for comments period, the subcommittee will take a final vote to recommend that the OSSC approve the benchmark for release.

The subcommittee report to the OSSC must include a report relative to the current release checklist. See the description of checklists above for more information about checklist location, content, and the required level of detail when reporting.

After the OSSC has received the subcommittee's report and recommendations, the OSSC will review them, and must consider any comments from the General Member Review. The OSSC will vote on whether to approve the benchmark for release or send it back to the subcommittee for further work.

For urgent cause, the Board of Directors may direct action inconsistent with a prior vote by the subcommittee or OSSC. Urgent cause may include cases where an approved benchmark either violates the letter or spirit of SPEC's Articles of Incorporation and Bylaws; infringes on copyrights, trademarks, or patents, or on the advice of SPEC's legal counsel.

With the approval of the Board of Directors, the benchmark approved within the OSG becomes an authorized SPEC Benchmark Suite release.

2.8.1.1 Initial results and "blind submission"

When a new benchmark is announced, SPEC may choose to publish initial results on SPEC's website using that benchmark. A subcommittee may choose to use a "blind submission", in order to help gather an initial set of results. This section provides the definition for that procedure.

The decision regarding whether a blind submission will be used shall be included in the announcement of the beta version for general membership review.

If a blind submission is used:

  1. The subcommittee identifies a specific pre-release version of the benchmark (for example, "RC1") that can be used for submittable runs.
  2. After the specific version is made available, a deadline is announced when initial submissions will be due, with a minimum of 2 weeks notice.
  3. It is strongly recommended that the deadline fall during a subcommittee face-to-face meeting, in order to allow the possibility of distribution of results solely by paper.
  4. During a period of 24 hours after the deadline, restrictions apply to the initial review:
    1. SPEC recognizes that blind submission materials are unusually sensitive, and participants are expected to be diligent to control the replication of such information.
    2. Subcommittee member companies who do not submit do not participate in the initial review, and do not see any of its contents during the initial 24-hour period.
    3. Only those who submit results participate in the initial 24-hour review. Each such company shall designate a single individual who shall take responsibility to manage the information and its access.
    4. At the close of the 24-hour initial review period, submitters are given an opportunity to withdraw any results that they wish to withdraw.
    5. Withdrawn results are not kept in SPEC's result database.
    6. If results were submitted on paper, the paper copies of withdrawn results are given back to the submitters.
  5. At the close of the initial 24-hour period, any submissions that have not been withdrawn become visible to the subcommittee as a whole using normal editorial review procedures. At this point, electronic copies are freely available to reviewers.
  6. After the close of the initial 24-hour period, a submitter is allowed to withdraw a result, but this is strongly discouraged; members who participate should do so in good faith as part of their support for the consortium, with submissions that are intended for inclusion in the benchmark announcement.
  7. Intent: The procedures above are intended to reduce some of the risk that members may feel about participating in a new benchmark. "If I submit a score of 100, will my company be ridiculed because someone else submits a score of 250?" In that situation, you can withdraw your result.
2.8.1.2 Publication embargo ("blackout period")

Although the procedures of the previous section are intended to reduce some of the perceived risk of participating in a new benchmark, it is also useful to add to the benefits for doing so. To that end, a subcommittee may choose to use a 30-day publication embargo, known as a "blackout period". This section provides the definition for that procedure.

The decision regarding whether a blackout period will be used shall be included in the announcement of the beta version for general membership review.

If a blackout period is used:

  1. For the first 30 calendar days after announcement of a new benchmark, independent publication of new results is not allowed, even if the benchmark would otherwise allow independent publication.
  2. SPEC also publishes no new results.
  3. On or after announcement day, members are allowed to further publicize the results that SPEC publishes on announcement day, including their own and competitors' results. (Of course, they may not do so prior to announcement day, since that would violate SPEC Confidentiality requirements for unannounced benchmarks.)
  4. If a member improves its score on a system that was part of the intial set, such improvement may not be publicized until after the blackout period concludes.
  5. If a result submitted prior to announcement has editorial or other issues that prevent its publication on announcement day, it may not be published, neither by SPEC nor by the tester, until after completion of the blackout period, even if said issues are resolved earlier.

When SPEC announces a new benchmark, it is SPEC's intent to generate public interest in the benchmark. The blackout period encourages members to participate in the initial result set, providing results that are visible on the announcement day for the benchmark itself.

2.8.2 New Versions of Benchmarks

After a benchmark has been released, updates may be needed. If a new version of the benchmark product is released (for example, a "V1.1"), OSSC approval is required for the release. The subcommittee should prepare:

About Performance Neutrality

The term "performance neutral" is understood to mean that visual inspection and analysis of a change indicates that it should not affect the measured result, and, if it is possible to run with and without the change, then empirical testing has been done to verify said analysis.

Sometimes, a code change may cause measured changes that are difficult to distinguish from inherent run-to-run variation. Benchmark run rules typically define a reproducibility guideline (for example, 1.75% for SPEC CPU2006, 5% for SPECweb2005) to assist with such judgments. That guideline may help to distinguish observed changes from inherent noise.

The subcommittee should not blindly assume that if a change is less than the reproducibility guideline, then it is neutral, because there can sometimes be real, observable, repeatable effects that are smaller than the reproducibility guideline. Such effects may be classified as "small", but should not be classified as "neutral". An accumulation of small changes since the original release of a benchmark, each less than the reproducibility guideline, may cause results to vary by more than the guideline. In such cases, it may be necessary to amend the metric name and declare new results non-comparable to previous results with that benchmark.

2.8.3 Benchmark Updates

Often, SPEC will attempt to act more quickly than a full release cycle, issuing updates or patches via its website. Such updates may be classified in several ways.

2.8.3.1 Classification of updates

  1. Run and reporting rules. Run rules changes may be divided into two categories:
    (a) clarifications and
    (b) substantitive changes.
    A change may be considered to be a clarification if it is a fix for a typographical error, a formatting change, or a change that has no effect on the behavior of users of the suite. All other changes are considered substantive.

  2. Documentation.

  3. Benchmark code. A code change may have differing impacts: is it performance neutral? Does it affect the Experimental Record? Or is the change solely to the reporting tools that consume such records, to create human-readable output? Based on these questions, code updates may be further divided as:

    1. Code changes that the subcommittee determines to be performance neutral, without dissent.
    2. Code changes that are not agreed to be performance neutral.
    3. Code changes that affect only reporting.

Policies for updates of each of these are addressed in the next sections.

2.8.3.2 Updates to run and reporting rules

Clarifications to run rules require a formal subcommittee vote for approval, but not an OSSC vote. The OSSC must be notified at least 2 business days prior to posting the change on SPEC's web site. The OSSC Chair may direct that a change that was considered by the subcommittee to be merely a clarification shall nevertheless be treated as substantitive.

Substantive changes to run rules require both a formal subcommittee vote for approval and a vote by the OSSC to approve the change.

2.8.3.3 Updates to documentation

Subcommittees are free to adopt processes as they see fit for updating documentation. For example, a subcommittee may analyze its support calls and, in cooperation with SPEC's Editorial staff, update the User Guide as posted at www.spec.org to reduce user confusion. The subcommittee shall be notified of all such updates. Updated documents shall be clearly marked with a version and/or date string, so that licensees can distinguish them.

2.8.3.4 Code changes

  1. Benchmark code changes (performance neutral): Changes that the subcommittee agrees, without dissent, to be performance neutral, require a formal subcommittee vote for approval prior to release, but not an OSSC vote. The OSSC must be notified at least 2 business days prior to posting on SPEC's web site. The OSSC Chair may direct that a change considered by the subcommittee to be performance neutral shall nevertheless be treated as non-neutral.

  2. Benchmark code changes (non-performance neutral): If any member of a subcommittee votes that a change is not performance neutral, then the change shall not be released unless the OSSC approves the change.

  3. Updates for reporting: In cooperation with SPEC's Editorial staff, subcommittees may adapt reporting as they see fit. For example, it may be decided that graphs should be generated in a format that is easier to read, and all posted results may be updated at www.spec.org. The subcommittee shall be notified of all such updates.

2.9 Benchmark Retirement

From time to time SPEC may retire benchmarks. Benchmark retirement needs the approval of the OSSC. The following provides an outline for a standard process. Adjustments may be made to the process with the agreement of the OSSC, with the exception that the minimum time noted below will usually not be reduced, because such reduction may disrupt planning cycles for licensees.

Appendix A. [deleted] Suite Development Overview

The material in this section was identified as obsolete. A portion was deleted; the remainder was updated, and moved to the new section 2.7.7

In order to avoid breaking external references to material later in the document, this placeholder is left behind.

Appendix B. Guidelines for Handling SPEC Information

B.1.0 Introduction

SPEC is an industry consortium whose membership includes many competing for-profit corporations. As described in the Bylaws, membership in SPEC is open to entities who are affected by SPEC's activities, and participation in SPEC activities is generally open to all members. However, in order to protect the interests of SPEC and its members, and to encourage free and open participation and discussion to further SPEC's objectives, public release of SPEC internal information may be prohibited without explicit approval from the SPEC organization concerned. Violators of this policy will be subject to the full force of the SPEC penalties process.

B.2.0 Anti-Trust Compliance

Since all SPEC meetings are to be conducted in compliance with all applicable laws, including anti-trust laws, the following policies shall be followed in the course of SPEC meetings, telephone meetings, and electronic exchanges: Any discussions that relate to the validity or cost of patent use should be avoided. Any discussions or any material relating to an ongoing litigation shall be avoided, except for the Board in executive session with advice of counsel to discuss matters affecting SPEC. Any discussions of pricing or issues that would violate US antitrust laws shall be avoided.

B.3.0 How Information is Classified and Unclassified

SPEC officers and the SPEC Administrator may explicitly designate certain information classified and mark it appropriately. Any member that wishes to be sure that its information is treated as SPEC Confidential should clearly label each page (electronic or hard copy) as such. It is the member's responsibility to assure that such information is appropriately labeled. Information not explicitly marked but which meets the descriptions in this policy must be taken as though it were so marked.

Classified information may become public through the authorized public release of the information by SPEC or by a member. If classified information is disclosed without authorization (leaked), that does not automatically remove its classification and relieve participants of the obligation to protect that information from further disclosure. However, if an unauthorized disclosure is widespread, then a SPEC officer may redesignate that information as public, stating the reason for so doing.

B.4.0 Classifications

SPEC identifies two levels of internal information which must be protected by SPEC members and associates, employees, and contractors, in accordance with these policies. "SPEC Confidential" is the ordinary classification of information which should not be disclosed outside SPEC. "SPEC Confidential Need-to-Know" refers to the most sensitive information described below.

B.4.1 SPEC Confidential Need-to-Know

This is information that must be restricted to only those individuals who must have it in order to carry out their SPEC responsibilities. In meetings, this information is discussed only when the board adjourns to executive session. Only the author of such a document, or the SPEC President, can change the classification or the approved reader list. Examples of Confidential Need-to-Know information includes:

B.4.1.1 Document Handling Guidelines

Documents of this sensitivity class will always be clearly labeled "SPEC Confidential Need-to-Know" on each page. The documents may be numbered for control purposes.

SPEC Private documents must have a cover page which serves to:

  1. hide the first page of the document,
  2. document the approved reader list, and
  3. document the date the information is no longer SPEC Confidential, if any,

Such documents should always be carefully transported. Documents should be kept inside an envelope, clearly labeled with the approved reader's name, when not in active use. Only persons on the approved reader list are privy to the document contents.

Email should be used sparingly for communicating such information. Messages should be sent directly to the intended recipient(s) rather than to any mail alias. Messages should be retained no longer than absolutely necessary, and should be kept separate from other messages. Encryption should be employed where possible.

B.4.2 SPEC Confidential

The following information is "SPEC Confidential":

Note that non-members of SPEC should not be on mailing aliases. Please note that SPEC offers various levels of membership to fill different needs for information access.

B.5.0 Who May Have Access to SPEC Confidential Information

Responsibility for protecting SPEC Confidential information rests with the member companies, which are expected to ensure that all employees and contractors who have access to SPEC Confidential information comply with these information handling requirements. Each company designates a primary representative, who may access SPEC Confidential information solely for the purpose of carrying out the member's SPEC activities. The primary representative may designate additional employees and contractors at the member company who have access, solely for the same purpose. A member's SPEC representatives should use whatever internal controls are appropriate in their company to ensure that the member complies with SPEC's policies on confidentiality. In addition it is expected that the primary representative will:

B.6.0 Meetings

Information about the activities and developments of SPEC are intended to be readily available to all parties with legitimate interests in the performance of advanced computer systems. The Board is responsible to carry out this policy.

The annual meeting of the SPEC membership must be open to any interested party with a legitimate interest in the work of the Corporation, including the press.

Other meetings of each SPEC group and subcommittee must be open to representatives of every member of that group (OSG, HPG, and GWPG). SPEC members who are not members of a particular group may nevertheless generally attend other groups' meetings, to encourage wider participation and exchange of ideas, unless the group has specifically decided to restrict attendance for some or all meetings. Such decisions are subject to review by the Board.

B.6.1 Non-member participation

The Chair of any SPEC meeting may invite limited participation by non-members who have a legitimate interest in SPEC's work, or who may possess information that SPEC would like to learn from. Such external participants must agree to SPEC's Guest Participation Statement. Exceptions to this requirement may be granted only by the Board.

B.6.2 Press Participation

Press representatives who are not SPEC members are generally excluded from SPEC meetings other than the annual meeting and SPEC's Public Workshops.

For other meetings, a meeting Chair may open the meeting to a representative of the press (who is not a member of SPEC) only if the press representative agrees to the confidentiality requirements in SPEC's Guest Participation Statement. Exceptions to the requirement for the participation statement may be granted only by the Board.

On the rare occasions that the press is in attendance, the Chair should announce press attendance prior to the meeting. The meeting Chair should request that the Press member use the attendance opportunity as "background" information in technical areas of performance benchmarking. SPEC is primarily an engineering organization which depends on frank and open exchange and debate of views. Views expressed in SPEC meetings are not expected necessarily to reflect official positions of SPEC member companies and organizations. SPEC asks that the press show restraint in any coverage based on SPEC meetings, so as not to jeopardize our open exchange of views.

B.6.3 Audio/Visual Recording

SPEC committees may apply restrictions on the use of audio recording, video recording, or photography equipment where they may impede free discussion, or where they are disruptive. Such restrictions should be clearly identified, in advance, to attendees.

B.7.0 Results Review

The following guidelines apply to results under review.

(a) Confidentiality during review: Benchmark results submitted to SPEC for review and publication on the SPEC web site (and not otherwise published) are considered SPEC Confidential during the review process. This includes the benchmark result, the configuration, availability dates, and tuning notes.

(b) Published results not confidential: Information which the test sponsor makes public is not considered confidential. For example, a company may issue a press release or briefing listing performance of new systems and, around the same time, submit those benchmark results for SPEC publication. In this case details included in the full SPEC disclosure under review, which were not included in the company's public announcement, are not considered SPEC Confidential.

(c) Surpassing results + Use of new optimizations: Trying to surpass a result currently under review is considered common practice and is allowed. If a new optimization method is learned from a result under review, results that use that method may not be submitted to SPEC, nor independently published, until at least one day after the normal review and publication period. This period is typically 2 weeks plus 1 day: see section 2.3 regarding timeliness of disposition and timeliness of publication. If a result is submitted that violates this guideline, the subcommittee may delay the offending result by one review cycle.

Examples. In all three of the examples that follow, a subcommittee has review cycles that begin on June 1st, 15th, and 29th. Subcommittee results are published by close of business on the 2nd, 16th, and 30th. Company A submits a result for the review cycle that starts on June 1, and Company A believes that it has used new methods to obtain its result.

Example 1. Company B learns about the methods from Company A's submission. Company B may submit a result to SPEC that uses the new methods no earlier than June 17, for the review cycle that begins on June 29th.

Example 2: Assume the same facts as Example 1, except that publication of Company A's result is delayed until August, due to pending actions or due to a hold at request of the sponsor. Company B remains free to submit results using the new methods on June 17.

Example 3: Company B submits a result to SPEC for review on June 15th that uses the same methods as Company A. On questioning, Company B engineers deny learning the methods from Company A, and describe some other way in which they learned the method. After discussion, the subcommittee decides whether to allow the submission.

(d) Confidentiality after review: Results submitted to SPEC are public information once they are published at SPEC's web site. Results submitted to SPEC (and not disclosed to the public) remain SPEC Confidential if they fail to pass SPEC technical review, if they are withdrawn, or if they are held pending at the submitter's request.

Example (Part 1). Company A runs SPECjvm2008 on the Model 42 and the tools report "1278.64 SPECjvm2008 Base ops/m". This measurement is not discussed in public, but it is submitted to SPEC. It does not pass technical review because a violation of rule 2.4.1 is noted: the full disclosure shows that it uses property file setting P, which is clearly described in the JVM Configuration Guide as unsupported.

Example (Part 2). A few weeks later, a local marketing group for Company A, apparently not having heard about the facts of Part 1, publishes the Model 42 "1278.64 SPECjvm2008 Base ops/m" result on the local web pages for Company A.

B.7.1 New Benchmark Announcements

More restrictive confidentiality rules may be applied by the SPEC groups on the occasion of results review for SPEC's announcement of a new benchmark suite, in addition to all of the preceding rules.

B.8.0 SPEC Press Relations

Press releases from SPEC must be approved by the SPEC group concerned, and by the Board.

SPEC members may occasionally be interviewed by the press, e.g. on the subject of newly released benchmarks. Normally SPEC's PR consultant will participate in such interviews in order to see that SPEC is treated fairly, and to assure all SPEC members that information is being represented in an unbiased manner. The Board Press Committee may on a case by case basis, waive this requirement, based on such factors as: degree of controversy of the subject, potential for appearance of conflict of interest by the spokesman, legal advice, advice from SPEC's PR consultant, recommendation of steering committee and technical committees and their chairs, experience of spokesman, experiences with the reporter, etc.

Appendix C. Guidelines for General Availability

One of the principal tenets of SPEC benchmarks is that the implementation of the System under Test (SUT) must be generally available, documented and supported. The run and reporting rules for each benchmark includes this statement. The available, documented, and supported trio is frequently referred to with the single term: general availability.

The purpose of including general availability requirements is to ensure that the systems and their hardware and software components actually represent real products that solve real business and computational problems. It is also intended to ensure that the benchmarked system does not contain so-called benchmark specials, which improve benchmark performance but fail one or more of the tests of general availability. These tests include:

  1. Has the system/component shipped or has the vendor committed to shipping the finished product within a time specified by the specific benchmark's run rules, usually 3 to 6 months from the date the result was made public?

    A finished product will have the vendor's endorsement that the product is suitable for production use in an environment comparable to the one represented by the benchmark. Typically this version will be designated as the final released version and not designated as a prototype or a preliminary release (i.e. alpha, beta, field test, etc.).

    Shipping the product refers to the vendor's standard process for getting said finished product into the hands of the end-user base. This can include putting it in a box and physically transporting it to end-user sites or simply advising the customer base that the product can be obtained on demand, for example by having an announced URL where the product can be downloaded or purchase information can be found.

    The commitment to ship a product within a specific time frame provided in a results disclosure must be taken as a public statement of intent. Any potential end-user should be able to confirm the vendor's intent to ship the finished product within the stated time frame and that information should not be considered secret once the result has been publically disclosed.

    Shipping a product generally requires that one or more instances of the final product be in end-user hands or enroute to end-users with delivery imminent and based on shipping method used.

  2. Is the product documented such that there are written descriptions on the product, how to install it, how to configure it and how and when to use it?

    The product documentation should be consistent with the deployment of the product in the benchmark such that there are no undocumented features employed solely for the benchmark. Availability of documentation should be concurrent with product availability.

  3. Is the level of support reasonable for the type of product or component, and consistent with its importance within the system?

    Levels of support can clearly have a wide range depending on the type of product or product component. SPEC understands that the determination of whether a level of support is "reasonable" may require a discussion and a ruling by the relevant subcommittee or by the steering committee. In such discussions, SPEC shall consider:

    1. Are the provider(s) of support clearly identified?
    2. Have clear procedures and expectations been established regarding support?
    3. Are support requests answered?
    4. If problems of a critical nature arise (i.e. those that affect the central purpose of the component, with no convenient workaround), are there mechanisms to ensure timely problem resolution?

    If the answers to the above questions are "yes", this is taken as evidence that the product or component is supported. Notes: (1) Individual benchmark run rules may impose additional support requirements. (2) Sometimes, a component might be considered to have achieved a level of maturity such that support requirements are minimal or non-existent. If so, one or more of the above tests may be waived.

    On the other hand, the requirement for paid support may exclude products for which standard benchmark results might be of interest, some of which are adequately supported by unpaid alternative mechanisms, such as support forums, bug mailing lists, wikis, and IRC channels. Such alternatives are herein designated as "community support".

    For community supported products or components, SPEC shall consider:

    1. Is there an active support mechanism? (For example, is there traffic over the last 90 days? Postings from the benchmarkers or their representatives do not count in determining whether the mechanism is active.)
    2. Is the user interface usable, so that new reports can be filed in a straightforward manner?
    3. Is it possible to observe that support requests are being addressed?
    4. Can users benefit from previous support requests, for example via a search function, or via an evolving, useful knowledge base?
    5. If problems of a critical nature arise (i.e. those that affect the central purpose of the component, with no convenient workaround), are there community members who maintain the application?

    If the answers to the above questions are "yes", that is taken as evidence that an active support exists mechanism exists, even in the absence of a paid support contract.

  4. Does the product meet the vendor's own definition of general availability, in addition to all the other tests specified by SPEC?

    Many vendors have their own internal processes for product development that set specific requirements for products to be considered generally available. If so, those requirements must be met for the product to be considered generally available by SPEC, in addition to the tests listed here. If the vendor has no such definition, then all other tests specified by SPEC must still be met.

    A vendor is not required to discuss or disclose its internal processes to SPEC. However the SPEC representative is responsible to identify the internal definition most closely matching SPEC's intent of general availability, and applying that definition to any statements to SPEC regarding general availability. Note that the vendor may use different terms, e.g. full customer availability, open shipping status, phase 4 exit, etc.

  5. Does the vendor represent the product to customers as being generally available?

    This is not a clear-cut test, as products may have several different distribution channels, and a product may not become generally available in all channels simultaneously. However, it is an indication that a product may not be generally available if the vendor specifically says in some venue that it is not. E.g.,

    • Web page saying product is not available.
    • Web page listing other configurations of the product as available, which fails to list the published configuration. (E.g., web page offers 1.5 GHz and 2.0 GHz models for sale but doesn't mention the published 2.5 GHz model.)
    • Sales person will not take orders for the product.
    • Statement by official company spokesman saying product is not available.

    It is an indication that a product may be generally available if the vendor specifically says so in some venue. E.g.,

    • Web page saying product is available.
    • Advertisement for the product
    • Sales person or reseller, possibly in a different locale, will take orders for the product.
    • Statement by official company spokesman saying product is available.

    In case of conflicting information, SPEC will weigh the preponderance of the evidence with the standard that it show the product to be generally available in some channel.

The rules for each benchmark may have slightly different requirements related to general availability and may specify other types of benchmark specials to avoid. The overall goal of ensuring fair benchmarking for the vendors submitting results and the end-users reviewing those results remains the same.

Note: The term "benchmark special" refers to any optimization which only improves benchmark performance.

[moved] Appendix D. Template SPECbenchmarkYYYY Run and Reporting Rules

SPEC maintains a template for development of run rules. The version formerly kept in this policy document was identified as obsolete, and has been moved to the SPEC members' private server.

Previous Revision History

Version V1.00.0 Approved by OSSC on June 10, 1999
Version V2.00.0 Approved by OSSC on July 20, 1999
Version V2.01.0 Approved by OSSC on July 20, 2000
Version V2.02.0 Approved by OSSC on July 25, 2001
Version V2.03.0 Approved by OSSC on March 21, 2002
Version V2.04.0 directed by BOD on January 25, 2006
Version V2.04.1 Approved by OSSC on January 11, 2007
Version V2.05 Incorporate editorial corrections approved by OSSC on 9 May 2007
Version V2.06 Add a procedure for subcommittee elections; allow subcommittees to adjust participation requirements, per OSSC 20 Sep 2007.
Version V2.07 Incorporate editorial corrections approved by OSSC on 13 Dec 2007
Version V2.08.1 6 Feb 2008
Version V2.09
25 June 2008

  • In section 2.1 about the OSSC:

    • Add a few bold words at the beginning of paragraphs, to make it easier to find topics.
    • Address, in a manner consistent with the Bylaws, affiliation of OSSC members.
    • Disallow use of proxy voting.
    • Define quorum.
  • Add section 2.3.5 on Historical Systems
  • Add section 2.3.6 on procedures for correcting published results
  • In the section on General Availability, remove the implication that the SUT must be supported by a single source. Identify criteria to judge whether "community support" counts as "supported" for the purposes of SPEC's general availability rule.
Version V2.10
3 Feb 2009
  • Clarify that associates are not eligible to run for OSSC in table 1.6.4
  • Expand section 2.3.1 on results review. Some of the updates and clarifications here include:
    • Although SPEC reviews results, they remain the responsibility of the tester.
    • A peer review process is used, with a notion of "Primary Reviewers".
    • A test sponsor may request that results be held.
    • Testers can expect certain levels of timeliness for publication.
  • Set procedures for Required Disclosures, in a newly added section 2.3.7.
  • Add new section 2.7 on benchmark development processes, addressing (among other things):
  • When New Benchmarks are released, emphasize that the OSSC will consider comments from the general membership, but remove the now-vestigial notion of a general member vote on benchmark release.
  • Set requirements for approval of New Versions of Benchmarks. Define Performance Neutrality.
  • Classify minor updates based on what they affect, and set approval processes by type, in sections 2.8.3.2 through 2.8.3.4.
  • Set procedures for notification of benchmark retirement
Version V2.11
8 Jul 2009
  • Expand language required for historical submissions, in section 2.3.5.
  • In B.1.0, reference the Bylaws regarding membership.
  • Remove an obsolete appendix.
Version V2.12
30 Jul 2009
Version V2.13
6 Jul 2010
  • Update B.5.0: Who may access confidential info, to provide specific language about confidentiality procedures.
  • Update B.7.0: Results Review, to clarify several cases regarding confidentiality and to add examples.
Version V2.14
7 Nov 2011
Version V2.15
21 Feb 2012
Version V2.16
30 Apr 2012
  • Expand discussion of checklists, then reference it from several other places.
  • Add section on responsibilities of subcommittee Initial Chair and require that the voting members and the voting method be picked by end of second meeting.
  • Explain how subcommittees are dissolved.
Version V2.20
29 Oct 2018
  • Clarified 2.1.c, and in section 2.1.d, add the obligation to attend all OSSC meetings, effective as of 29 Jan, 2019.

Copyright © 2000-2018 Standard Performance Evaluation Corporation