OSG Benchmarks
Order Benchmarks
SPEC
Mirror Sites
Resources
|
Open Systems Group
The Open Systems Group is the oldest group within SPEC. The OSG is
the current embodiment of the original founders of SPEC. The OSG is
governed by the policies and principles described in the SPEC
Open Systems Group Policies and Procedures Document. The Open Systems
Steering Committee (OSSC) provides oversight and management for a number
of technical committees that investigate and develop the benchmarks,
metrics, run and reporting rules, and so on.
OSG focus is on component- and systems-level benchmarks for desktop
systems, workstations and servers running open operating system environments.
This is the group responsible for the processor metrics SPECint®2017
and SPECfp®2017 (and their predecessors in 2006, 2000, 1995, 1992 and the
original SPECmarks from 1989).
In addition to the SPEC CPU® 2017 benchmark suite, the
group has also developed Java benchmarks, a SIP benchmark, web server
benchmarks, mail server benchmark, file server benchmarks, a power
and performance benchmark, and the virtualization benchmarks.
More in-depth information may be found in the benchmark specific links below.
Joining SPEC/OSG
We welcome your interest in joining the Open Systems Group. For more
detailed information on benefits, membership fees, and the application
process, please go to our membership page.
The Current Benchmarks and Tools
- SPEC Cloud® IaaS 2018 benchmark suite
- SPEC's infrastructure-as-a-service (IaaS) performance benchmark for public and private cloud platforms.
- SPEC CPU® 2017 benchmark suite
- The current release of SPEC's popular processor performance tests;
the successor to the SPEC CPU® 2006 benchmark.
- SPECjbb® 2015 benchmark suite
- Server-side Java benchmark, developed from the ground
up to measure performance based on the latest Java application features.
- SPECjEnterprise® 2018 Web Profile benchmark suite
- The SPECjEnterprise 2018 Web Profile benchmark measures full-system performance for Java
Enterprise Edition (Java EE) Web Profile Version 7 or later application
servers, databases and supporting infrastructure.
- SPECjEnterprise® 2010 benchmark suite
- The full system benchmark which allows performance measurement
and characterization of Java EE 5.0 servers and supporting infrastructure
such as JVM, Database, CPU, disk, and servers.
- SPECjvm® 2008 benchmark suite
- The current Java virtual machine (JVM) benchmark, with multithreaded
workloads that represent a broad range of application areas.
- SPECpower_ssj® 2008 benchmark suite
- The SPECpower_ssj 2008 benchmark is the first industry-standard SPEC benchmark
that evaluates the power and performance characteristics of volume
server class computers. The first subset of server workloads addresses
the performance of server-side Java; additional workloads are planned.
- SPECstorage® Solution 2020 benchmark suite
- A storage benchmark suite measuring file server throughput and response time,
providing a standardized method for comparing performance across different vendor platforms.
- SPECvirt® Datacenter 2021 benchmark suite
- The next generation of virtualization benchmarking for measuring performance of a scaled-out datacenter.
It is a multi-host benchmark using simulated and real-life workloads to measure the overall efficiency
of virtualization solutions and their management environments.
- SPEC VIRT_SC® 2013 benchmark suite
- The benchmark addresses performance evaluation of
datacenter servers used in virtualized server consolidation, measuring
end-to-end performance of all system components including the hardware,
virtualization platform, and the virtualized guest operating system
and application software. In addition to major workload upgrades,
the SPECvirt web server workload has been modified to require SSL
(HTTPS) between the client system and the web server.
- Chauffeur® Worklet Development Kit (WDK) tool
- The Chauffeur WDK 2.0 tool simplifies workload development for researchers analyzing
computer performance and power consumption. Targeting both industry and academia,
the WDK is based on the SERT 2.0 suite infrastructure, providing
run-time & platform configuration support to simplify worklet development.
This major release includes significant enhancements to the hardware detection,
customization options of generating HTML reports, and developer documentation.
It has now reduced memory requirements for the Director when signing results
files and reduced the size of the result output for large systems or clusters.
Version 2.0 added worklet-specific normalization of results and an updated
list of supported operating systems including Ubuntu (14.04 LTS and 16.04 LTS)
as well as current versions of Windows Server, RHEL, SLES, AIX, and Solaris. The
WDK also includes the latest PTDaemon integration for power analyzers and temperature
sensors, along with data collection, validation and reporting.
- SPEC PTDaemon® tool
- The SPEC PTDaemon tool is used to control power analyzers in
benchmarks which contain a power measurement component.
- Results Search
- A search-engine configured across all the available OSG benchmark
results.
- Purchase a Benchmark
- Order SPEC/OSG products. The SPECjvm 2008 benchmark is also available as a free download.
- Submitting Results for
Publication
- Information on how benchmark licensees can submit their results
for publication on the SPEC web site.
Future Benchmarks
These are the projects that SPEC has currently under active development for future release.
- SPEC ML
- The OSG ML Committee was formed in 2021 to develop practical methodologies
to benchmark machine learning (ML) performance in the context of real-world platforms and
environments. The Committee also works with other SPEC committees to update their benchmarks for
ML environments. The OSG ML Committee's first benchmark, SPEC ML, will measure end-to-end performance
of a system under test (SUT) handling ML training and inference tasks.
Retired Benchmarks
- SPEC Cloud® IaaS 2016 benchmark suite
- SPEC's original benchmark addressing performance of
infrastructure-as-a-service (IaaS) cloud platforms.
- SPEC CPU® 89 benchmark suite
- Predecessor to the CPU 92 suites, now long obsolete. No material
available here at this time...
- SPEC CPU® 92 benchmark suite
- The now obsolete processor performance benchmarks.
- SPEC CPU® 95
- The popular processor performance tests. Successor to SPEC CPU 92 and
replaced by the SPEC CPU 2000 benchmarks.
- SPEC CPU® 2000 benchmark suite
- The previous release of SPEC's popular processor performance tests;
replaced by the SPEC CPU 2006 benchmarks.
- SPECjAppServer® 2001 benchmark suite
SPECjAppServer® 2002 benchmark suite
SPECjAppServer® 2004 benchmark suite
- Based on ECperf (developed under the Java Community Process),
a set of industry-standardized benchmarks designed to measure the
performance of J2EE 1.2 and 1.3 application servers.
- SPECjbb® 2000 benchmark suite
- A Java Business Benchmark - SPEC's first server-side Java
benchmark emulating middle tier business logic.
- SPECjbb® 2005 benchmark suite
- The successor to SPEC's original server-side Java benchmark emulating
middle tier business logic, JBB2000.
- SPECjbb® 2013 benchmark suite
- Server-side Java benchmark, developed from the ground
up to measure performance based on the latest Java application features.
- SPEC JMS® 2007 benchmark suite
- SPEC's benchmark for enterprise message-oriented middleware servers
based on JMS (Java Message Service).
- SPECjvm® 98 benchmark suite
- Benchmark suite for comparing Java virtual machine (JVM) client
platforms.
- SPECmail® 2001 benchmark suite
- SPEC's first industry-standardized mail server benchmark designed
to measure a system's ability to act as a mail server processing
email requests, based on the Internet standard protocols SMTP and
POP3.
- SPECmail® 2008 benchmark suite
- SPEC's industry standard benchmark designed to measure the performance
of corporate e-mail servers processing requests based on the Internet
standard protocols SMTP and IMAP4.
- SPECmail® 2009 benchmark suite
- SPEC's industry standard benchmark designed to measure the performance
of corporate e-mail servers processing requests based on the Internet
standard protocols SMTP and IMAP4, with support for SSL v3.0 and
TLS 1.0.
- SPEC SDM™ benchmark suite
- Our multi-user, UN*X commands level performance suite (sdet and
kenbus1).
- SPEC SFS® 93 benchmark suite
- "System: File Server", aka LADDIS, test of NFS performance.
Replaced by the SPEC SFS® 97 V2.0 benchmark.
- SPEC SFS 97® (2.0) benchmark suite
- "System: File Server", a test of NFS server performance.
Replaced by SPEC SFS® 97_R1 V3.0.
- SPEC SFS® 97_R1 (3.0) benchmark suite
- "System: File Server", a test of NFS server performance.
Replaced by SPEC SFS® 2008.
- SPEC SFS® 2008 benchmark suite
- "Solution File Server" benchmark, a test of server performance
utilizing NFS version 3 and CIFS. Follow-on to the SPEC SFS 97_R1 benchmark.
- SPEC SFS® 2014 benchmark suite
- SPEC's "Solution File Server" benchmark, a test of file server performance.
- SPEC SIP_Infrastructure® 2011 benchmark suite
- A standardized performance test designed to evaluate a system's
ability to act as a SIP server supporting a particular SIP application.
The application modeled is a VoIP deployment for an enterprise,
telco, or service provider, where the SIP server performs proxying
and registration.
- SPEC SMT™ 97 benchmark suite
- System-MultiTasking. The next generation for the SPEC SDM benchmarks. A flexible workload harness with
a workload based upon public domain system utilities (e.g. gcc,
groff, gzip, etc.) performing basic code development activities.
Should provide a good measure of a system's ability to manage
large numbers of active users.
- SPEC VIRT_SC® 2010 benchmark suite
- SPEC's benchmark addressing performance evaluation of datacenter
servers used in virtualized server consolidation, measuring end-to-end
performance of all system components including the hardware, virtualization
platform, and the virtualized guest operating system and application
software.
- SPECweb® 96 benchmark suite
- A standardized performance test for web servers. Succeeded by the SPECweb® 99 benchmark.
- SPECweb® 99 benchmark suite
- A standardized performance test for web servers. Succeeded by the SPECweb® 2005 benchmark suite.
- SPECweb® 99_SSL benchmark suite
- A standardized performance test for web servers utilizing SSL.
Succeeded by the SPECweb 2005 benchmark suite.
- SPECweb® 2005 benchmark suite
- A standardized performance test for WWW servers, successor of
SPECweb99 and SPECweb99_SSL, the benchmark consists of different
workloads (both SSL and non-SSL), such as banking and e-commerce,
and writes dynamic content in scripting languages to more closely
model real-world deployments. The web server also communicates with
a lightweight backend to simulate an application/database server.
- SPECweb® 2009 benchmark suite
- A standardized performance test for web servers, the benchmark
consists of different workloads (both SSL and non-SSL), such as
banking and e-commerce. The web server also communicates with a
lightweight backend to simulate an application/database server.
New in this benchmark is the addition of a power workload with a
measurement methodology taken from the SPECpower group.
Questions
General questions can be answered by SPEC staff reached at info@spec.org.
Technical questions should be directed to support@spec.org.
|