SPECworkstation 4.0 Benchmark Measures Latest Workstation Hardware, Adds AI/ML Workloads

By Chandra Sakthivel, SPECwpc Chair

I’m extremely excited that SPEC has announced the release of the SPECworkstation 4.0 benchmark, a major update to SPEC’s comprehensive tool designed to measure all key aspects of workstation performance. This significant upgrade from version 3.1 incorporates cutting-edge features to keep pace with the latest workstation hardware and the evolving demands of professional applications, including the increasing reliance on AI and machine learning (ML).

Workstations are crucial for a range of industries, including architecture, engineering & construction (AEC), energy, life sciences, product design, media & entertainment, scientific research, and software development. The new SPECworkstation 4.0 benchmark provides a robust, real-world measure of CPU, graphics, accelerator, and disk performance, ensuring that professionals have the data they need to make informed decisions about their hardware investments. Developed by a consortium of semiconductor vendors and system manufacturers working as members of the non-profit SPEC organization, the benchmark is an unbiased, real-world, application-driven tool for measuring performance across a wide range of workstation-based workloads.

Let’s look in detail at what the new SPECworkstation 4.0 benchmark offers, the importance of its support for AI/ML, and the benefits it brings to the professional computing community.

What’s new in the SPECworkstation 4.0 benchmark

The SPECworkstation 4.0 benchmark caters to the diverse needs of engineers, scientists, and developers who rely on workstation hardware for daily tasks. The benchmark includes real-world applications like Blender, Handbrake, LLVM and more, providing a comprehensive performance measure across various scenarios, and spanning seven different industry verticals, each focusing on specific use cases and subsystems critical to workstation users.

Compared to the SPECworkstation 3.1 benchmark, the SPECworkstation 4.0 benchmark provides a more accurate, detailed, and relevant measure of the performance of today’s workstations, especially when it comes to AI/ML. Key new features of the benchmark include:

• Support for AI/ML – The benchmark includes a new category of tests focusing on AI and ML workloads, including data science tasks and ONNX runtime-based inference tests, reflecting the growing importance of AI/ML in workstation environments.
• New workloads
  • Autodesk Inventor: Measures key performance metrics in Inventor, a popular software package in the architecture, engineering, and construction (AEC) segment.
  • LLVM-Clang: Measure code compilation performance using the LLVM compiler and toolchain.
  • Data Science: Represents a data scientist's workflow, including data scrubbing, ETL (extract, transform, load), and classical machine learning operations using tools such as Numpy, Pandas, Scikit-learn, and XGBoost.
  • Hidden Line Removal: Measures time taken to remove occluded edges from wireframe models.
  • MFEM: Uses finite element methods to perform dynamic adaptive mesh refinement (AMR).
  • ONNX Inference: Benchmarks AI/ML inference latency and throughput using the ONNX runtime, a popular framework for evaluating ML models.
  • Updated CPU Workloads: Updates for popular workloads like Blender, Handbrake, NAMD, and Octave improve relevance, accuracy, and compatibility for performance measurements.
  • Updated Graphics Workloads: Tests adapted from the SPECviewperf2020 v3.1 benchmark measure the performance of professional graphics cards using application-based traces.
• New Accelerator subsystem – This release introduces the Accelerator subsystem, recognizing the rapid innovation in the computing space. With accelerators playing a critical role in speeding up tasks such as AI/ML processing, video transcoding, and other compute-intensive operations, measuring and understanding their performance in the workstation environment is essential.
• Enhanced user interfaces – A completely redesigned UI simplifies the benchmarking process, making it more user-friendly, and a new command-line interface (CLI) enables easier automation.

All-in on AI/ML and Data Analytics

The release of the SPECworkstation 4.0 benchmark holds significant value for the user community, particularly those wanting to take advantage of rapidly evolving AI technologies. As enterprises increasingly focus on building infrastructure to support AI-powered use cases, having a reliable benchmark is more critical than ever as more data scientists and software developers rely on workstations for data science, data analytics, and ML.

Until now, there hasn’t been a comprehensive benchmark that accurately measures a workstation's performance across the entire data science pipeline – from data acquisition and cleaning to ETL processes and algorithm development. However, data scientists often use workstations to experiment with different algorithms and solutions before scaling to server-based model training. The workstation's performance at a system level is crucial, especially when dealing with terabytes of data.

This new benchmark addresses these challenges by providing workloads designed to measure performance across the data science pipeline. It provides the first set of performance metrics specifically designed to measure data science and ONNX-based inference workloads on workstations and serves as a useful tool for estimating the performance of workstation hardware in real-world AI applications.

ONNX Runtime (ONNX RT) is a high-performance inference engine that enables the deployment of ML models in various environments, including cloud, edge, and mobile devices. Developed by Microsoft, it supports models trained in multiple frameworks like PyTorch and TensorFlow, allowing seamless execution across different hardware platforms. ONNX RT is optimized for speed and efficiency and supports a wide variety of platforms, making it an attractive choice for running AI models in production and at scale.

The SPECworkstation 4.0 benchmark evaluates inference latency and throughput performance across different quantization approaches and batch sizes, including FP32, FP16, and INT8. This reflects the industry's ongoing evolution towards more efficient AI models. By offering a comprehensive set of performance metrics, this benchmark helps scientists and developers understand the capabilities of their workstation hardware and enables them to make informed decisions to optimize their systems for better performance in real-world AI/ML applications. The benchmark also covers classical machine learning workloads by measuring performance with widely used libraries like XGBoost and Scikit-learn. This further ensures the benchmark is relevant for a wide range of AI/ML development tasks, from data pre-processing to algorithm design.

Available for Immediate Download

Downloading and installing the SPECworkstation 4.0 benchmark is simple and straightforward, and you can begin running tests within a few minutes. You can also check out the benchmark’s published results page as they become available to compare your performance with that of other systems, or to research a system that will best balance your performance and budget requirements.

The SPECworkstation 4.0 benchmark is available from SPEC under a two-tiered pricing structure: free for the user community and $5,000 for sellers of computer-related products and services. SPEC/GWPG members receive benchmark licenses as a membership benefit.

If you're a SPECworkstation user, I highly recommend that you download this latest release.

[Back to SPEC blog]