The State of AI/ML Benchmarking: Addressing Gaps and Driving Progress

By Arthur Kang, Machine Learning Committee Chair

In the rapidly evolving field of artificial intelligence (AI) and machine learning (ML), enhancing the accuracy and computational efficiency of large models remains a critical challenge. As of 2024, there are over 100 commercial large language models worldwide, with OpenAI's ChatGPT boasting more than 500 million users. The swift development of these models is driving significant advances in AI capabilities.

As large models grow in complexity, the demand for computational power and the associated energy consumption—leading to substantial carbon emissions—has become a pressing issue. Improving model accuracy while optimizing computational efficiency is crucial to managing the high demands and costs associated with training and inference. Currently, however, there is a lack of standardized benchmarks for evaluating the computational efficiency of large models, and this lack makes it challenging to quantify performance across different models and ensure effective improvements in model capabilities, as well as overall cost reduction.

SPECworkstation 4.0 Benchmark Measures Latest Workstation Hardware, Adds AI/ML Workloads

By Chandra Sakthivel, SPECwpc Chair

I’m extremely excited that SPEC has announced the release of the SPECworkstation 4.0 benchmark, a major update to SPEC’s comprehensive tool designed to measure all key aspects of workstation performance. This significant upgrade from version 3.1 incorporates cutting-edge features to keep pace with the latest workstation hardware and the evolving demands of professional applications, including the increasing reliance on AI and machine learning (ML).

The Latest News from the SPEC Embedded Group

By Joseph Yiu, Embedded Group Chair

I’m pleased to share with you the first edition of the SPEC Embedded Group Newsletter, which will give us a regular opportunity to update the community on everything happening with SPEC Embedded Group (SPEC EG).

SPEC Embedded Group was created in October 2023 when EEMBC (Embedded Microprocessor Benchmark Consortium) was merged into SPEC. We are already moving forward in several directions, including working to update current benchmarks and starting to plan new ones

The SPECvirt Datacenter 2021 Benchmark - An Answer to the Datacenter Architect’s Dilemma

By Travis Hindley and Lisa Roderick, SPEC Virtualization Committee

An IT architect faces a variety of crucial choices in creating a robust and efficient virtualized data center.

Wrapping Up ICPE 2024: A Milestone Event in Performance Engineering

By Cristina L. Abad, William Knottenbelt and Weiyi Shang

We are delighted to share insights and highlights from the recently concluded 15th annual ACM/SPEC International Conference on Performance Engineering (ICPE) 2024, held May 7-11th in the historic and vibrant city of London. ICPE has been the flagship conference in the area of performance engineering over the years and has a tradition of bringing together both academic scholars and industrial practitioners in discussing the recent advances in the area.

This year’s event, located at Imperial College London in South Kensington, hosted more than 130 attendees who participated in a rich program spanning keynote speeches, research and industrial presentations, and collaborative workshops. The conference facilitated discussions and networking among attendees from people across the globe, including academia, industry, and research institutions.

The SPECworkstation 3.1 Benchmark — A User's Story

By Chandra Sakthivel, SPECwpc Chair

Nikki is a procurement manager at a large digital media and marketing company that offers vertical solutions for a range of industries, including media and entertainment, life sciences and ecommerce. She was tasked with purchasing over 300 new workstations to satisfy the needs of power users across the company – while staying within a finite budget.

The need for new workstations arose as employees in software development, graphics modeling and entertainment production increasingly complained that the performance of their existing systems was significantly impacting their productivity, leading to slower release cycles, missed opportunities and frustrated co-workers and customers. The existing workstations typically had older-generation CPUs, entry-level GPUs, 8GB of memory, and 512GB of disk space with DDR3 memory, so Nikki spent several days online reviewing the latest offerings from various vendors and reading reviews on several technology media sites to try to determine hardware configurations that would optimally balance performance and cost.

The SPEChpc 2021 Benchmark Suites — A User's Story

By Dr. Junjie Li, Research Associate, Texas Advanced Computing Center (TACC), and Nick Hagerty, HPC Engineer, Oak Ridge National Lab (ORNL)

The SPEChpc 2021 benchmark suites, a standard High-Performance Computing (HPC) test tool, proves valuable in various situations. In this blog, Nick Hagerty, HPC engineer from the Oak Ridge National Laboratory (ORNL) and Dr. Junjie Li, Research Associate from the Texas Advanced Computing Center (TACC), two of the largest HPC centers in the U.S., share their experiences with the SPEChpc 2021 Benchmark Suites in their research and daily data center operations.

Updated PTDaemon Interface Meets Today's Industry Needs

By Klaus-Dieter Lange, SPECpower Committee Chair

I'm pleased to announce a recent update to the PTDaemon interface. SPEC benchmarks play an increasingly important role in energy efficiency and sustainability initiatives because they enable computer vendors, enterprises, and government entities to accurately measure energy consumption by computing systems and data centers. The PTDaemon interface makes it easier for SPEC benchmarks to measure power and temperature by providing a common TCP/IP-based interface that can be integrated into different benchmark harnesses. This allows power measurement devices to be automatically incorporated into performance evaluation software and eliminates the need for users to understand and adapt to different power analyzer interface protocols and behaviors.

The SPECapc Benchmark — A User's Story

By Jessica Heerboth, SPECapc Committee Chair

Ashley, a college sophomore majoring in graphics and design, specializes in 3D modeling. Her final project at the end of her freshman year was to create and animate a 3D game character that exhibited some lifelike characteristics when running, jumping and throwing.

Her modeling tool of choice was Autodesk Maya 2024, the latest version of the popular 3D software that both professionals and enthusiasts use to create realistic characters and blockbuster-worthy effects. However, Ashley's animation process required her to frequently make small tweaks to a frame and then re-render the animation to see how that small tweak looked before making the next tweak. Unfortunately, the laptop computer she'd used since high school wasn't equipped with powerful enough hardware to support fast rendering of the animation, forcing her to sit idly for 10 or 15 minutes while each re-rendering finished – amounting to several hours of wasted time each day.