|
Benchmarks
Tools
Order Benchmarks
SPEC
Mirror Sites
Resources
|
The SPEC 30th anniversary
Better benchmarks since 1988
Like a lot of institutions, the Standard Performance Evaluation Corporation (SPEC) sprang out of frustration, according to co-founder John Mashey.
"All the performance folks and systems architects were just so sick of being hassled with dumb benchmarks."
Thirty years later, SPEC has grown from a handful of members and a single benchmark to 125 members in 22 countries, presiding over nearly two dozen active benchmarks and tools for measuring performance and energy efficiency, conducting research into a wide range of performance issues, and sponsoring international conferences and symposiums.
Here's a timeline encapsulating the history of the world's leading computer benchmarking organization.
1988
SPEC incorporates on November 14.
1989
SPECmark (subsequently known as SPEC CPU89) introduced with a VAX 11/780 as a reference machine and 10 computation-intensive benchmark programs (average size of 2.5 billion dynamic instructions per program).
1992
SPEC92 (subsequently known as SPEC CPU92) debuts with 20 benchmark programs and up to 10.2 billion dynamic instruction count.
1993
SPECsfs 93, a one-test NFS benchmark suite derived from the Nhfsstone workload, is introduced.
1994
The SPEC High Performance Group (SPEC/HPG) is formed.
The Graphics Performance Characterization (GPC) group releases the SPECviewperf benchmark for graphics performance.
1995
SPEC CPU95 goes live, with a SPARCstation 10 model 40 as its reference machine and 18 benchmark programs with up to 520.4 billion dynamic instruction count.
1996
SPEC/HPG releases SPEChpc96, a benchmark suite that measures the performance of HPC systems running industrial-style applications.
The GPC Group (now SPEC/GWPG) joins SPEC and releases SPECviewperf 5.0.
SPEC announces SPECweb96, the first standardized benchmark for measuring the performance of web servers.
1997
SPECsfs 97 released with major upgrades, including support for both NFS protocol version 3 and version 2.
The SPEC Application Performance Characterization (SPECapc) project group is formed under SPEC/GWPG.
1998
SPEC releases SPECjvm98, the first industry-standard benchmark that measures performance for Java virtual machine (JVM) client platforms.
SPECapc releases Solidworks 98 and Quake II application benchmarks.
SPECviewperf 6 is released with a new viewset and support for multiple primitive types and textures.
1999
SPECweb99 is released with a new framework and workload that reflect industry changes.
SPECapc releases new application benchmarks for Pro/ENGINEER and SolidWorks CAD/CAM software.
On December 30, SPEC rings in the new millennium with SPEC CPU2000, comprising 26 benchmarks and dynamic instruction counts that reach 546.7 billion.
2000
SPEC/HPG announces SPEComp2001, a new benchmark development effort based on OMP applications.
SPEC releases its first server-side Java benchmark, SPEC JBB2000.
SPECapc releases a new benchmark for Unigraphics V15.
2001
SPEC releases SPECmail 2001, a benchmark for evaluating the performance of mail servers.
SPEC OMP2001, a benchmark suite based on OpenMP applications for evaluating the performance of SMP systems, goes live.
SPECapc releases new benchmarks for systems running 3ds Max, Solid Edge and Unigraphics software.
2002
SPEC releases SPECjAppServer2001, a client/server benchmark that measures the performance of Java Enterprise Application Servers.
SPEC/HPG releases SPEC OMPL2001, a new benchmark suite that measures the performance of computing systems running large scalable applications.
New SPECviewperf 7.0 enables general state changes to be made during frames, bringing the benchmark much closer to real-world behavior.
The SPEChpc 2002 benchmark, derived from real HPC applications and practices, is released.
SPECapc updates its benchmarks for new versions of Solidworks, 3ds Max and Pro/E applications.
2003
SPECapc updates its benchmarks for new versions of Solid Edge, Solidworks and Maya applications.
SPEC/HPG releases V3.0 of the OMP benchmark.
2004
SPEC releases SPECjAppServer2004, a multi-tier benchmark for measuring the performance of Java 2 Enterprise Edition (J2EE) application servers.
SPECviewperf 8 released with four new viewsets and application tracing that gets closer to application behavior in the real world.
SPECapc updates its benchmarks for new versions of Solid Edge, Solidworks, Maya and 3ds Max applications.
2005
SPEC releases its JBB2005 benchmark for evaluating server-side Java performance.
SPECweb 2005 released with three real-world workloads and dynamic content in PHP and JSP.
SPECapc updates its benchmarks for new versions of Maya and 3ds Max applications.
2006
SPEC CPU2006 is released with 29 benchmarks exercising new areas of functionality for CPUs, memory systems, and compilers.
The SPECpower committee is formed to develop standard methods and metrics for comparing energy use of server-class computers.
New SPECviewperf 9 features two new viewsets and major code changes. A Linux/Unix version is also released.
SPECapc updates its benchmarks for new versions of 3ds Max, NX and Pro/E applications.
2007
SPECpower_ssj2008, the first industry-standard benchmark that measures power consumption in relation to performance for server-class computers, goes live.
SPEC forms a new subcommittee to develop standard methods of comparing performance for servers using the Session Initiation Protocol (SIP).
SPECviewperf 10 adds performance measurement for full-scene anti-aliasing and multithreading.
SPEC/HPG releases MPI2007 to measure the performance of parallel computing systems and clusters running Message-Passing Interface (MPI) applications.
SPECjms2007, a new benchmark designed to measure performance of enterprise messaging servers based on JMS (Java Message Service), makes its debut.
SPECapc updates its benchmarks for 3ds Max, NX, Solid Edge and Solidworks.
2008
SPECsfs2008, designed to evaluate the speed and request-handling capabilities of file servers utilizing the NFSv3 and CIFS protocols, is released.
SPEC releases SPECjvm2008, a benchmark suite for measuring the performance of a Java Runtime Environment (JRE).
SPECmail2008, designed to measure the performance of enterprise mail servers compliant with SMTP and IMAP4 protocols, makes its debut.
SPECviewperf 10 for Linux/Unix is released.
2009
The SPECpower committee embarks on a project to develop a Server Efficiency Rating Tool (SERT) for use by the EPA.
SPECmail2009, a new version of the enterprise mail server benchmark, goes live.
SPECweb2009 is released with new power/performance measurement.
SPECjEnterprise2010 is released to measure performance for Java Enterprise Edition 5 or later application servers, databases and infrastructure.
SPECapc releases new benchmark for Lightwave 3D and updates its Maya application benchmark.
2010
SPECviewperf 11 released with a new GUI and fully updated viewsets based on traces from professional applications.
SPECvirt_sc2010, SPEC's first benchmark for virtualized server consolidation, goes live.
SPEC/HPG releases MPI2007 v2.0, which adds the MPIL2007 suite to allow testing up to 2048 ranks.
SPEC and ACM launch the first International Conference on Performance Engineering (ICPE).
2011
SPEC's Research Group (SPEC RG) is formed to develop standard scenarios and workloads for emerging technologies and applications.
SPEC releases SPECsip_Infrastructure2011, designed to evaluate a system's ability to act as a SIP server.
SPECapc releases updated benchmark for the 3ds Max application.
SPEC RG establishes the annual Kaivalya Dixit Distinguished Dissertation Award.
2012
SPEC forms a new Cloud subcommittee.
SPEC releases SPEC OMP 2012, an updated benchmark for measuring performance and energy efficiency using applications based on the OpenMP 3.1 standard.
SPEC/GWPG forms a workstation performance characterization (SPECwpc) group.
The first two SPEC Kaivalya Dixit Distinguished Dissertation Awards are presented.
SPEC RG establishes the Security Benchmarking and Cloud working groups.
SPECapc releases updated benchmarks for Creo, NX and Maya applications.
2013
The Server Efficiency Rating Tool (SERT), comprising synthetic worklets to test power consumption data at different load levels, makes its debut.
SERT testing becomes mandatory part of the U.S. Environmental Protection Agency's (EPA) ENERGY STAR for Servers specification.
SPECjbb2013, developed from the ground up to measure performance based on the latest Java application features, is released.
SPECvirt_sc2013, the latest virtualized server benchmark, goes live.
SPECwpc releases the first benchmark to measure all key aspects of workstation performance based on diverse professional applications.
SPECviewperf 12 released with extensible architecture and new viewsets representing energy and medical applications.
SPECapc updates benchmarks for NX and Solidworks applications.
2014
SPEC ACCEL V1.0, a benchmark suite that measures performance with computationally intensive parallel applications, goes live.
SPEC releases the Chauffeur WDK (Worklet Development Kit) Tool, designed to simplify the development of workloads for measuring both performance and energy efficiency.
SPEC SFS2014, SPEC's updated benchmark suite to measure performance of storage configurations, is released.
Fifth annual ICPE conference held in Dublin.
SPECapc updates its benchmark for the 3ds Max application.
SPEC RG forms working groups for Big Data and DevOps Performance.
2015
SPECwpc 2.0 is released with improved storage workloads and better scalability measurement for multi-core systems.
SPECjbb2015, an enhanced Java server benchmark that now supports virtualized hosts, is released.
SPECapc updates benchmarks for Creo and Solidworks.
2016
SPEC holds benchmarking symposiums and technical meetings in Beijing, China.
SPEC Cloud_IaaS 2016, a new benchmark that measures the performance of infrastructure-as-a-service (IaaS) cloud implementations, is released.
SPECapc updates its benchmark for the NX application.
2017
SPEC releases SPEC CPU2017, an all-new version of its flagship performance evaluation suite, with 43 individual benchmarks organized into four categories.
SERT 2.0, which adds a single-value metric and numerous other improvements, is released.
SPEC establishes the Power working group.
Version 2.0 of the Chauffeur Workload Development Kit (WDK) is released.
SPECapc updates its benchmark for the Maya application.
2018
SPEC releases SPECjEnterprise2018 Web Profile, a new benchmark measuring performance of Java EE 7 Web Profile application servers.
SPEC offers HPG benchmarks (ACCEL, MPI2007, OMP2012) free of charge to qualified non-profit organizations worldwide.
SPECviewperf 13 is released with new and updated viewsets, redesigned GUI, and improved scoring and reporting methods.
SPECworkstation 3 (formerly SPECwpc) is released with new and updated workloads throughout the benchmark suite.
SPEC Cloud IaaS 2018 is released with new workload versions and a wide range of methodology updates.
EPA's ENERGY STAR Version 3.0 for Computer Servers defines active state efficiency thresholds based on SERT.
SPECapc updates its benchmark for the Maya application.
SPEC RG creates the Quality of Experience working group.
|
