| $Id: changes.html 6738 2023-01-12 23:11:44Z JohnHenning $ | Latest: www.spec.org/cpu2017/Docs/ |
|---|
This document describes changes in SPEC CPU®2017, a product of the SPEC® non-profit corporation
(about SPEC).
This document is intended as a readable summary of the key changes in each release.
The Revisions document has
a more complete, terse list of changes.
All users
are strongly encouraged to update to the latest version,
to improve compatibility, reliability, accuracy, and clarity (as described below).
Updates are
easy to do.
PTD (the Power/Temperature Daemon) v1.10.0
PTD (the Power/Temperature Daemon)
Intel Compiler version checking
Updates to Run and Reporting Rules
500.perlbench_r / 600.perlbench_s
Updates to Run and Reporting Rules
500.perlbench_r / 600.perlbench_s
Native toolset available: SPEC CPU tools (such as runcpu, specinvoke, and specperl) are now available for RISC-V 64-bit Linux systems. The initial toolset included with v1.1.9 was built and tested on Ubuntu 22.04.1. Other versions of Linux may also work, but have not been tested. If you run into difficulties with other Linux versions, please check the list of Frequently Asked Questions and the list of Known Problems before contacting Technical Support.
Portability work remains: Please note that the SPEC CPU 2017 suite contains 43 component benchmarks. It is possible that RISC-V systems may require portability compiler options and/or portability source code changes for some of these benchmarks. Therefore, it may not be possible to do a reportable run on RISC-V until after portability problems are resolved, either by selection and approval of PORTABILITY options or by development and approval of source code patches. The procedure to suggest a source code patch is defined in the Technical Support document.
The sysinfo utility, which automatically prints information about the system under test, has been extended to report additional information, including CPU microcode version, process limits, and selected information from systemd.
PTD (the Power/Temperature Daemon) is updated to Version 1.10.0 and supports additional devices.
Starting in v1.1.8, --output_format=check is automatically included when selecting reportable runs.
Usage note: Although it is allowed to run a SPEC CPU test and fill in the system details afterwards, you may find it useful to create the description before the test, because you can use it as a checklist of whether you have configured the system in the desired fashion. It was arguably an error that reportable did not formerly include output_format=check, since the definition of reportable has always been that it should enforce the run rules whenever practical. A valid system description is required by the rules.
The presubmission tools, which were first included in 1.1.5, have been updated. As noted below, please see the README for the tools.
There are many updates, including:
localreview.pl to summarize problems and make it easy to navigate to them.tar file.sw_os, sw_state, hw_model, and hw_cpu_name are now checked.localreview.pl (Use --help to see them.)dmidecode is not availableThe sysinfo utility now
checks whether dmidecode is available in several directories, starting with whatever is on your PATH, and then
looking in /usr/local/sbin, /usr/sbin/, and /sbin/. Note that if you
install dmidecode from nongnu.org, by default it
will be installed in /usr/local/sbin.
The sysinfo utility now checks whether your version of lscpu includes the useful
--cache option.
When SPEC CPU 2017 v1.1.7 is installed on a macOS system, SPEC CPU tools will be selected that match the chip - either x86_64 or arm64 ("M1"). The config directory now includes an example config file for Apple Clang on macOS.
The portability flag SPEC_GCC_VARIADIC_FUNCTIONS_MISMATCH_WORKAROUND is available for use on macOS, where it is neeeded to address an ABI violation.
The presubmission tools, which were first included in 1.1.5, have been updated. As noted below, please see the README for the tools.
Update PTD (the Power/Temperature Daemon) to v1.9.2, which supports more devices.
The field sw_compiler has certain syntax requirements for how compiler versions are specified. Due to a change in how Intel Compiler versions are released, this field has been updated. If you are using a newer Intel compiler, you will need this update in order to avoid warnings when result files are generated.
The sysinfo utility has several fixes and enhancements. The changes include:
In addition, sysinfo output is now captured in the primary log file, which is useful if a run fails without producing any other reports.
All users of CPU 2017 are encouraged to update their copy of SPEC CPU 2017 to obtain these sysinfo changes, which may improve both accuracy and clarity of reports.
GCC version 10 requires workarounds for several benchmarks. These workarounds do not qualify as PORTABILITY (runrules.html#portability) and therefore must be applied consistently in base (runrules.html#BaseFlags). The example GCC config files have been updated to demonstrate how to apply the workarounds while complying with the rules.
Benchmarks with updated GCC notes include:
| SPECrate | SPECspeed |
|---|---|
All who use GCC with SPEC CPU 2017 are encouraged to update their copy of SPEC CPU to obtain the new example config files, which are more compatible with contemporary GCC.
Rule 2.2.4 regarding size changes was clarified, adding references to related rules.
Removed op/time_loop from the (non-timed) test workload, which was discovered to be non-portable when a compiler user reported that undefined behavior on a conversion caused the test to always fail, independent of optimization level. The test was for a case that does not arise in the (timed) reference workload.
Added portability option -DSPEC_PAREST_STD_FLUSH_WORKAROUND which works around an issue in module message_log.cc, which takes the address of std::flush and uses it in a comparison. If a compiler does not generate consistent results for the address, then the comparison fails and 510.parest_r does not generate any output.
It is also conceivable that a C++2003 compiler might forbid taking the address of std::flush, because it was not explicitly deemed acceptable to do so prior to C++20.
In either of those cases, this flag may be used. It substitutes an alternate flush function in the MessageLog namespace thereby allowing the comparison to succeed.
Results that are published at SPEC are reviewed by a series of perl scripts prior to publication. A
subset of those tools is included with SPEC CPU 2017 v1.1.5 in directory
$SPEC/bin/presubmit (Unix) or
%SPEC%\bin\presubmit (Windows).
If you use the presubmit tools, your review of results may be quicker or smoother. Please see the README.txt file in that directory.
Power was previously introduced as an experimental feature in CPU 2017 v1.0.
As of v1.1, the feature is now fully supported.
Power measurement is optional. You can include it, or not, as you may prefer.
If you wish to measure power, you will need:
In your config file, you specify the network location for the controller system; set the expected voltage and current ranges; and describe your measurement setup for readers of your results.
For more information, see the config file documentation on Power Measurement.
Provide full support for power measurements and energy metrics. See:
Removed signal handling, on the grounds that it is system-specific (non-portable) behavior, it makes debugging difficult, and the change is performance neutral (because no signals are invoked during ordinary execution). The change does not affect ordinary reportable runs.
One of the components of the (non-timed) test workload could fail if more than 1000 copies of the test workload have previously been run. Fixed tmpdir handling in lib/File/Spec/Unix.pm to avoid the problem. The change does not affect the timed (reference) workload.
Removed signal handling, on the grounds that it is system-specific (non-portable) behavior, it makes debugging difficult, and the change is performance neutral (because no signals are invoked during ordinary execution). The change does not affect ordinary reportable runs.
A syntax error comparing a pointer to an integer was corrected. The change is performance neutral: it affects one comparison which is done 16 times during a run; and empirical testing did not show any differences. (Median within less than a tenth of a percent.)
--- ...old.../510.parest_r/src/source/base/parameter_handler.cc
+++ ...new.../510.parest_r/src/source/base/parameter_handler.cc
@@ -749,7 +749,7 @@
char *endptr;
double d = std::strtod (s.c_str(), &endptr);
// assert there was no error
- AssertThrow ((s.c_str()!='\0') || (*endptr == '\0'),
+ AssertThrow ((s.c_str()[0]!='\0') || (*endptr == '\0'),
ExcConversionError(s));
Removed signal handling, on the grounds that it is system-specific (non-portable) behavior, it makes debugging difficult, and the change is performance neutral (because no signals are invoked during ordinary execution). The change does not affect ordinary reportable runs.
In file src/read_grib.c, the file time.h is included from whatever the usual location is, instead of having a hard-coded (and possibly incorrect) path. The change was made after a user reported that the hard-coded path was not present on his macOS 10.14 system. If for some unlikely reason it were actually necessary to use the hard-coded path, a portability flag could be used to request the old behavior. As with any other portability flag, such usage would need to be justified.
--- ...old...521.wrf_r/src/read_grib.c +++ ...new...521.wrf_r/src/read_grib.c #include <math.h> #include <limits.h> -#if defined(MACOS) || defined(SPEC_MACOSX) +#if (defined(MACOS) || defined(SPEC_MACOSX)) && defined(SPEC_TIME_H_ABSOLUTE_PATH) #include "/usr/include/time.h" #else #include <time.h> #endif
Removed signal handling, on the grounds that it is system-specific (non-portable) behavior, it makes debugging difficult, and the change is performance neutral (because no signals are invoked during ordinary execution). The change does not affect ordinary reportable runs.
Removed signal handling, on the grounds that it is system-specific (non-portable) behavior, it makes debugging difficult, and the change is performance neutral (because no signals are invoked during ordinary execution). The change does not affect ordinary reportable runs.
(This fix first shipped with update v1.0.5.) Fix an incompatibility with GLIBC 2.26+ found in module threads.c. The problem is that struct timespec does not get defined if you use gcc -std=c99 or gcc -std=c11. The fix increases the _POSIX_C_SOURCE to the needed level for this module.
--- ...old.../526.blender_r/src/blender/source/blender/blenlib/intern/threads.c +++ ...new.../526.blender_r/src/blender/source/blender/blenlib/intern/threads.c @@ -29,6 +29,12 @@ * \ingroup bli */ +#if defined(SPEC) && (_POSIX_C_SOURCE - 0) < 200112L + /* Needed for struct_timespec */ +# undef _POSIX_C_SOURCE +# define _POSIX_C_SOURCE 200112L +#endif + #include <stdlib.h>
Thank you to Steve Ellcey for reporting the issue.
As a consequence of the above change, an Apple-specific include located a few lines further down in threads.c ran into unknown types within /usr/include/sys/sysctl.h. The solution that was picked (and which first ships with SPEC CPU 2017 v1.1) causes macOS to follow the same path as other Unix systems. This change is expected to be performance neutral, since the threading code is not used in 526.blender_r; and empirical testing showed differences that are smaller than run-to-run variation.
/* for checking system threads - BLI_system_thread_count */ #ifdef WIN32 # include <windows.h> # include <sys/timeb.h> -#elif defined(__APPLE__) +#elif defined(__APPLE__) && !defined(SPEC) # include <sys/types.h> # include <sys/sysctl.h> #else # include <unistd.h> # include <sys/time.h> #endif
A potential data race condition was reported between multiple threads writing and reading an implicitly SHARED variable in a parallel region. To avoid the problem, line 109 of src/uv3s_update.F90 was changed to mark the variable tmp as private:
!$omp parallel do private (i, j, k, tmp)
The source code in question is executed during the reference (timed) benchmark run. Although SPEC tries to avoid changes to timed code after a benchmark has been released, in this case an exception was made because:
Removed signal handling, on the grounds that it is system-specific (non-portable) behavior, it makes debugging difficult, and the change is performance neutral (because no signals are invoked during ordinary execution). The change does not affect ordinary reportable runs.
Changes to SPEC CPU®2017 Copyright © 2019-2022 Standard Performance Evaluation Corporation (SPEC®)