Last updated: $Date: 2014-04-11 20:14:53 -0400 (Fri, 11 Apr 2014) $ by $Author: JohnHenning $
(To check for possible updates to this document, please see http://www.spec.org/cpu2006/Docs/ )
Contents
Editing result description fields
Fields allowed only in rawfiles
How do you generate a SPECrate result from a SPECspeed result? (or vice-versa)
Note: links to SPEC CPU2006 documents on this web page assume that you are reading the page from a directory that also contains the other SPEC CPU2006 documents. If by some chance you are reading this web page from a location where the links do not work, try accessing the referenced documents at one of the following locations:
This document describes various useful utilities that ship with SPEC CPU2006. It is assumed that you have already read runspec.html and config.html.
Most of the examples in this document have been written in Unix syntax. That means you'll need to use your imagination just a bit as you read this document:
| If you see: | Just pretend you saw: |
| cat | type |
|---|---|
| cp | copy |
| diff | fc |
| grep | findstr |
| ls | dir |
| / | \ |
| $SPEC | %spec% |
"configpp" is a short way to say:
$ runspec --action configpp
That switch is mentioned in runspec.html but you'll find much more about the config file preprocessor in config.html. The configuration file is specified using the --config switch, or -c for short, at the command line. For example, the following two command are equivalent:
$ runspec --action configpp -c try_utilities.cfg $ configpp -c try_utilities.cfg
There are a couple of points worth mentioning here:
The configpp utility is a good way to check your configuration file for fatal errors, as the config file is completely parsed before it's output.
(History: The configpp utility was added in SPEC CPU2006 V1.0.)
The "convert_to_development" utility removes various safety checks that ensure that benchmark code, data, and tools haven't been modified. This utility is useful if you are pursuing experimental changes that you would like to try out while still continuing to use the SPEC tool harness. For example, you might want to add your own performance instrumentation code, add some debugging code, or try out a modified workload.
To use it, type:
convert_to_development <--- Unix specperl bin\convert_to_development <--- Windows
A development tree cannnot be used for reportable runs.
Benchmark binaries built in one type of tree (reportable or development) cannot be used in the other type unless you set check_md5=0. Note that even in a development tree, the default for check_md5 remains 1.
Warning 1: If you use this utility, you will permanently render the current SPEC directory tree incapable of doing reportable runs. If you are doing both development work and reportable runs, install the suite twice in two different directory trees (for example, on different disks).
Warning 2: Because benchmark binaries built in a reportable tree cannot be used in a development tree (unless you set check_md5=0), care should be taken to avoid accidental deletion and rebuilding of benchmark binaries. Consider the following scenario, where the user only wanted to try changing input files from the workload of 403.gcc:
$ runspec --config myconfig --action build 403.gcc
$ convert_to_development
Modify the workload for 403.gcc to try out some idea,
changing files in 403.gcc/data/ref/input
$ runspec --config myconfig --action validate 403.gcc
...
Building 403.gcc base
Unless myconfig.cfg contains check_md5=0, the effect of the above command will be to immediately rebuild 403.gcc, which probably isn't your intent if you are only studying the effects of workload modifications for 403.gcc.
(History: The convert_to_development utility was added in SPEC CPU2006 V1.1.)
SPEC distributes approved source changes in packages referred to as src.alts, which are applied using the config file src.alt option. In config.html, you will also find an example showing what happens when one is applied.
The dumpsrcalt tool allows you to find out what's in a src.alt without necessarily having to apply it.
dumpsrcalt takes a packaged src.alt, an installed src.alt, or just the src.alt description file (srcalt.pm) and outputs the source changes as a unified diff file, which shows "-" for lines that are removed and "+" for lines that are added.
A src.alt package may also contain whole new files. In fact, all will contain at least one, as the presence of a "README" file is enforced by makesrcalt. The new files will also be printed, if they are available.
dumpsrcalt's command-line options control which parts of the src.alt are output. By default all information known about a src.alt is printed. This includes a summary with the name of the src.alt, the benchmark to which it applies, the versions of CPU2006 with which it may be used, and a count of new and changed files.
Here are the contents of the src.alt created in the example for makesrcalt:
$ dumpsrcalt 403.gcc.no_line.cpu2006.v1.2.tar.xz
This is src.alt "no_line" for 403.gcc.
It is usable with version 1.2 of the benchmark.
It contains 1 new file and changes to 1 existing file.
------------------------------------------------------------------------------
Index: README
===================================================================
--- original/README 1970-01-01 01:00:00.000000000 +0100
+++ no_line/README 2011-07-20 16:40:47.000000000 +0100
@@ -0,0 +1,1 @@
+Remove some of those pesky #line directives.
Index: c-parse.c
===================================================================
--- original/c-parse.c 1970-01-01 01:00:00.000000000 +0100
+++ no_line/c-parse.c 2011-07-20 16:40:47.000000000 +0100
@@ -71,8 +71,6 @@
#define CLASS 321
#define ALIAS 322
-/* #line 34 "c-parse.y" */
-
#include "config.h"
#include "system.h"
#include "tree.h"
@@ -102,10 +100,8 @@
/* Rename the "yyparse" function so that we can override it elsewhere. */
#define yyparse yyparse_1
-/* #line 67 "c-parse.y" */
typedef union {long itype; tree ttype; enum tree_code code;
const char *filename; int lineno; } YYSTYPE;
-/* #line 200 "c-parse.y" */
/* Number of statements (loosely speaking) and compound statements
seen so far. */
$
Notice above that dumpsrcalt was applied to the packaged (.xz) src.alt. The dumpsrcalt utility can also be applied to an installed src.alt, producing the same output:
$ . ./shrc
$ specxz -dc 403.gcc.no_line.cpu2006.v1.2.tar.xz | spectar -xvf -
README.403.gcc.src.alt.no_line.txt
benchspec/CPU2006/403.gcc/src/src.alt/no_line/srcalt.pm
benchspec/CPU2006/403.gcc/src/src.alt/no_line/README
$ dumpsrcalt 403.gcc no_line
This is src.alt "no_line" for 403.gcc.
It is usable with version 1.2 of the benchmark.
It contains 1 new file and changes to 1 existing file.
------------------------------------------------------------------------------
Index: README
===================================================================
--- original/README 1970-01-01 01:00:00.000000000 +0100
+++ no_line/README 2011-07-20 16:44:27.000000000 +0100
@@ -0,0 +1,1 @@
+Remove some of those pesky #line directives.
Index: c-parse.c
===================================================================
--- original/c-parse.c 1970-01-01 01:00:00.000000000 +0100
+++ no_line/c-parse.c 2011-07-20 16:44:27.000000000 +0100
@@ -71,8 +71,6 @@
#define CLASS 321
#define ALIAS 322
-/* #line 34 "c-parse.y" */
-
#include "config.h"
#include "system.h"
#include "tree.h"
@@ -102,10 +100,8 @@
/* Rename the "yyparse" function so that we can override it elsewhere. */
#define yyparse yyparse_1
-/* #line 67 "c-parse.y" */
typedef union {long itype; tree ttype; enum tree_code code;
const char *filename; int lineno; } YYSTYPE;
-/* #line 200 "c-parse.y" */
/* Number of statements (loosely speaking) and compound statements
seen so far. */
$
Finally, you can apply dumpsrcalt just to srcalt.pm. In that case, the README file is not displayed, since srcalt.pm contains only differences:
$ dumpsrcalt benchspec/CPU2006/403.gcc/src/src.alt/no_line/srcalt.pm
This is src.alt "no_line" for 403.gcc.
It is usable with version 1.2 of the benchmark.
It contains 1 new file and changes to 1 existing file.
------------------------------------------------------------------------------
New file "README" isn't present, and can't be dumped.
Index: c-parse.c
===================================================================
--- original/c-parse.c 1970-01-01 01:00:00.000000000 +0100
+++ no_line/c-parse.c 2011-07-20 16:46:43.000000000 +0100
@@ -71,8 +71,6 @@
#define CLASS 321
#define ALIAS 322
-/* #line 34 "c-parse.y" */
-
#include "config.h"
#include "system.h"
#include "tree.h"
@@ -102,10 +100,8 @@
/* Rename the "yyparse" function so that we can override it elsewhere. */
#define yyparse yyparse_1
-/* #line 67 "c-parse.y" */
typedef union {long itype; tree ttype; enum tree_code code;
const char *filename; int lineno; } YYSTYPE;
-/* #line 200 "c-parse.y" */
/* Number of statements (loosely speaking) and compound statements
seen so far. */
$
Notice that the output from dumpsrcalt is suitable for use as input to many patch (1) programs, available on Unix systems and in some add-on packages for Windows (e.g. Cygwin). If you try to apply the output of dumpsrcalt yourself, take care to notice how paths are handled by your patch program, especially if the src.alt is for a benchmark that has a complicated source tree. Many patch programs will pay attention to the filename but not the path listed on the "Index:" lines, so it may be best to apply them using the "-p1" option, to cause patch to ignore "Index:" lines and use the full paths of the files as listed in the diff.
dumpsrcalt's command-line options are intended to be self-explanatory:
$ dumpsrcalt --help
Usage: bin/dumpsrcalt [options] <benchmark> <srcaltname>
bin/dumpsrcalt [options] <path to srcalt tarball>
bin/dumpsrcalt [options] <path to srcalt.pm>
Options:
--skip-changed -- Do not output diffs of files that have changed
--skip-new -- Do not output diffs for new files
--summarize -- Do not output any diffs at all
--no-summary -- Do not output the summary for the src.alt
<benchmark> must be the full nnn.name form
<srcaltname> must match the name of the subdirectory in the benchmark's
src.alt directory
(History: The dumpsrcalt utility was added in SPEC CPU2006 V1.1.)
If you have a rawfile, you can extract the config file that was used with it. To do so, use the extract_config utility.
For example, starting with a freshly-installed SPEC CPU2006 directory tree:
% runspec --config=Example-simple --size=test --iterations=1 \ --tuning=base --output_format=text,raw bzip2 > /dev/null % cd result % ls CINT2006.001.test.rsf CINT2006.001.test.txt CPU2006.001.log lock.CPU2006
Note that we have just the single result. Let's make a temporary directory and copy the rawfile there:
% mkdir tmp % cd tmp % cp ../CINT2006.001.test.rsf . % ls CINT2006.001.test.rsf
(On Windows, that ls command would be spelt dir. You'll find some more Windows notes in the introduction.)
Now, attempt to extract the config file:
% extract_config --stdout CINT2006.001.test.rsf > tmp.cfg
Does the extracted file match the original?
% diff -u ../../config/try_utilities.cfg tmp.cfg
--- ../../config/try_utilities.cfg 2006-05-22 14:41:42.000000000 -0700
+++ tmp.cfg 2006-05-22 14:43:16.000000000 -0700
@@ -1,3 +1,6 @@
+# Invocation command line:
+# /Volumes/storage/WorkRelated/SPEC/CPU2006/Kits/cpu2006-kit91/bin/runspec
--config=try_utilities --size=test --iterations=1 --tuning=base
--output_format=asc,raw bzip2
+############################################################################
# This is a sample config file. It was tested with:
#
# Compiler name/version: Apple GCC v4.0.1 build 5245
@@ -128,15 +131,8 @@
sw_state = Default
sw_other = None
-__MD5__
-401.bzip2=base=try_utilities=default:
-# Last updated Mon May 22 14:41:42 2006
-optmd5=0666e60587f33e2aa13eb6ba0917aa60
-baggage=
-compile_options=\
-@eNqNj00LgkAQhu/7K4a9b5duoUKuJpa6C+qlS9QmYqAjaYf+faMF2cfBYQ8Dsy/P80oV6xWUxoCg\
-h4BtX2HTLRCEl2pfHqTOaU08380DoBFqCaI+35tjXRnRoGgrCpbwNVaHt6spHCZXIKXNicDHXblb\
-pTObf+A4IwkibaJ1kNLtF82ZGsI6C+Nw79OX/xqcRWGyexaa4Wrh6VKYvnMm5m+rwTfyXu4z8WNE\
-5UPBSbsHPPNh9Q==
-exemd5=d828496b92983e96408f680b55393225
%
Yes. The only differences are:
The config file can also be extracted directly to a file:
% extract_config --nostdout CINT2006.001.test.rsf Wrote "CINT2006.001.test.cfg"
or
% extract_config CINT2006.001.test.rsf Wrote "CINT2006.001.test.cfg"
Lastly, you can choose the filename if you're piping files in via stdin:
% extract_config --basename=SampleFilename < CINT2006.001.test.rsf Wrote "SampleFilename.cfg"
Instead of saying diff just above, you would say fc (and, as already mentioned, dir is the Windows spelling for what Unix calls ls.)
If you have a rawfile, you can extract the XML flags description file that the result uses. To do so, use extract_flags. Usage is similar to extract_raw, but unlike extract_raw, the extract_flags utility might not produce a flags description file identical to the original flags file. This is because EOLs (end-of-line) characters are normalized to Unix-style LF (line feed) only.
Note: If you have used a flags description file to format your result, the entire flags file is stored in the raw file (not just the part needed for result).
For example, continuing the example from extract_raw, you can extract the flags file using the following command:
$ extract_flags --stdout new.rsf > extracted.flags.xml $ diff ../myflags.xml ./extracted.flags.xml $
We see no difference between the extracted flags description file and the original. Notice that the above command asked for the flags to be written to standard out; without that switch, they would have been written to whatever filename was suggested.
If you'd like for the filename of the extracted flags file to match the rawfile, use the --nofilename option. This will cause extract_flags to ignore the flag file's "suggested filename":
% ls *rsf
cpu2006-20060901-00074.rsf cpu2006-20060904-00080.rsf
cpu2006-20060901-00075.rsf cpu2006-20060904-00081.rsf
cpu2006-20060903-00076.rsf cpu2006-20060904-00082.rsf
cpu2006-20060903-00077.rsf cpu2006-20060904-00083.rsf
cpu2006-20060904-00078.rsf cpu2006-20060904-00084.rsf
cpu2006-20060904-00079.rsf cpu2006-20060904-00085.rsf
% extract_flags --nofilename cpu2006-20060904-00085.rsf
Reading "cpu2006-20060904-00085.rsf"
Wrote "cpu2006-20060904-00085.xml"
%
(History: as of SPEC CPU2006 V1.1) You can format a result with multiple flags files. If you used multiple flags files as input to runspec then you will get multiple flags files when running extract_flags.
# extract_flags CINT2006.001.ref.rsf Reading "CINT2006.001.ref.rsf" Wrote "platform-setup.xml" Wrote "pathscale-flags.xml" # extract_flags --nofilename CINT2006.001.ref.rsf Reading "CINT2006.001.ref.rsf" Wrote "CINT2006.001.ref.xml" Wrote "CINT2006.001.ref-1.xml"
In the examples above, result 001 used two flags files. One has information about the platform (which might contain items such as BIOS settings), and the other describes the compiler.
Note: If a raw file has not been formatted with a flags description file, extract_flags does nothing.
(History: the extract_flags utility is new with SPEC CPU2006 V1.0. Several of its switches are new with SPEC CPU2006 V1.1.)
If you have a PDF, HTML, or PostScript output file, you can extract the rawfile that was used to generate it. To do so, use extract_raw. For example:
$ cp -p CINT2006.267.ref.rsf new $ ls -g new -rw-rw-r-- 1 ptg 186202 Mar 13 22:57 new $ rawformat -o html,raw --flagsurl ./myflags.xml new 2>&1 > /dev/null $ ls -g new.html new.rsf -rw-rw-r-- 1 ptg 109390 Mar 20 12:48 new.html -rw-rw-r-- 1 ptg 194193 Mar 20 12:48 new.rsf $ mkdir AppleFrancisco $ cd AppleFrancisco/ $ cp ../new.html . $ ls -g -rw-rw-r-- 1 ptg 109390 Mar 20 12:51 new.html $ extract_raw new.html new.html: Wrote "new.rsf" $ ls -g -rw-rw-r-- 1 ptg 109390 Mar 20 12:51 new.html -rw-rw-r-- 1 ptg 194193 Mar 20 12:51 new.rsf $ diff ../new.rsf new.rsf $
In the above example, we begin by copying the original rawfile (always recommended anytime you use commands that create new rawfiles). The rawformat command takes new as input and produces as output both new.html and new.rsf. We copy only new.html to a newly created directory, where we demonstrate that extract_raw can create a copy of new.rsf that is identical to the copy in the parent directory.
Note that the file you extract from can be PDF, PostScript, or HTML, just as on Unix systems, with one important exception: if you are using a browser that attempts to add value to the HTML when you save it, that browser may break the embedded rawfile. For example, with Internet Explorer Version 5.00.3103.1000, if you are viewing a SPEC CPU2006 html file and you select File -> Save As, the default for "Save as type" is shown as
"Web Page, complete (*.htm,*.html)"
Do not accept that default if you want to be able to retrieve the rawfile later on. If you take the default, your saved file will be incomplete. To save the web page INCLUDING the embedded rawfile, you must change "Save as type" to
"Web Page, HTML only (*.htm,*.html)"
(You can also invoke this utility with the spelling: 'flags_dump')
If you have an XML flag description file, such as one extracted using extract_flags, or one downloaded from SPEC, flag_dump will create a more human-readable HTML version from it. The flag_dump utility will also handle more than one XML flag description at a time.
For example:
$ cd /tmp $ mkdir myflags $ cd myflags/ $ cp $SPEC/Docs/flags/*xml . $ ls example-medium-cfg-flags.xml flags-advanced.xml flags-simple.xml $ cp flags-advanced.xml flags-for-willi.xml $ flag_dump flags-advanced.xml flags-for-willi.xml flags-simple.xml Processing flags-advanced.xml...wrote flags-advanced.html Processing flags-for-willi.xml...wrote flags-for-willi.html Processing flags-simple.xml...wrote flags-simple.html $
If you are writing a flags file, you should validate that the dump is valid HTML. If it isn't, you probably made a mistake in the HTML text that you have included inside your XML flags file. A handy HTML validator can be found at http://validator.w3.org/.
Note that flag_dump utility assumes that you are formatting flags files written by users. These files are not allowed to contain mandatory flags. If you attempt to use the flag_dump utility to format any of the suite-provided flag files such as flags_mandatory.xml or the benchmark flag files that reside in $SPEC/benchspec/CPU2006/*/Spec/ directory, the utility will exit with an error message such as the one below.
% cd $SPEC/benchspec
% flag_dump flags_mandatory.xml
Processing flags_mandatory.xml...
ERROR: The flag file at
flags_mandatory.xml
contains one or more flags in the class "mandatory", which is not
allowed.
%
(History: The flag_dump utility was added in SPEC CPU2006 V1.0.)
When you source the shrc, as described in runspec.html, the 'go' shell alias is created. go is a handy utility to navigate the directory structure of a SPEC source tree. Here are the various things it understands and the places they will take you:
| If you say | You'll end up in |
| go
go top | $SPEC |
|---|---|
| go bin | $SPEC/bin (where the tools live) |
| go config | $SPEC/config |
| go result
go results | $SPEC/result |
| go doc
go docs | $SPEC/Docs |
| go cpu | $SPEC/benchspec/CPU2006 |
| go benchmark | The top level directory for the named benchmark, if it can be found. (Examples follow) |
| go 401 | $SPEC/benchspec/CPU2006/401.bzip2 |
| go perlb | $SPEC/benchspec/CPU2006/400.perlbench Notice that abbreviated names are accepted. But if the abbreviation is ambiguous, the command will pick the first match, so pay attention to the reply: $ go g /spec/cpu2006/kit91/benchspec/CPU2006/403.gcc $ go go /spec/cpu2006/kit91/benchspec/CPU2006/445.gobmk $ |
| go nada | no change: there's no benchmark named nada* |
| go benchmark subdir | The named subdirectory of the directory tree for the referenced benchmark, for example "go perlb run". You can also name more than one level of subdirectory: "go 462 data train". (History: The go alias existed in SPEC CPU2000, but for SPEC CPU2006 the order is new when naming subdirectories, as is the ability to name more than one level of subdirectory.) |
| go subdir | If you are already in a benchmark subtree, then you can move to a different branch of the same benchmark, by naming the branch and not mentioning a benchmark. For example: $ pwd /rahul/benchspec/CPU2006/401.bzip2/run/build_base_a3.0000 $ go exe /rahul/benchspec/CPU2006/401.bzip2/exe $ |
go also understands -h and --help, and will output a short usage message in response to either of those arguments.
Note: If you leave the shell where you sourced shrc, the alias is no longer available! So if you want to use this alias, do not start a sub-shell.
Note: 'go' does not know about your settings for output_root or expid, and will not be able to take you to your run, exe, or result directories if you use those features. See ogo for something that can help.
(History: since the release of SPEC CPU2006 V1.0,) If you use a csh-like shell, please note that after you say source cshrc, there is a go alias that is set up for you, which implements the same features as described above.
Sorry, the go feature does not exist for Windows systems.
As you are preparing sets of results, you might find it handy to use scripts.misc/index. This script is NOT supported - please do not submit bug reports for it. But you're welcome to try it out, like so:
$ ls fprate.rsf intrate.rsf $ rawformat -o html *rsf > /dev/null $ ls fprate.flags.html fprate.rsf intrate.html fprate.gif intrate.flags.html intrate.rsf fprate.html intrate.gif $ $SPEC/bin/scripts.misc/index $ ls fprate.flags.html fprate.rsf intrate.gif fprate.gif index.html intrate.html fprate.html intrate.flags.html intrate.rsf $
In the above example, we have a couple rawfiles in a directory, generate html reports for them, and then add an index.
SPEC distributes approved source changes in packages referred to as src.alts, which are applied using the config file src.alt option. (The file config.html also has an example showing what happens when one is applied.)
A src.alt is a specialized difference file, and can't be made by hand. When you need to package some source changes, makesrcalt is the tool you need.
If you are a developer and you believe that changes are needed to the SPEC source code, it will probably be convenient to work in a sandbox, temporarily abandoning the tools during your development phase. (Or, you can use convert_to_development to make the whole installation into one big sandbox.) Once you have a pretty good idea of the changes that you want:
Create a subdirectory under the benchmark's src/ directory named "src.alt/".
Decide on a name for your src.alt. The src.alt name may contain only letters, digits, and underscores. For convenience, it should be short. Create a subdirectory under src/src.alt/ using the name you have chosen.
In that subdirectory, deposit any modified or new source files for the benchmark.
You must also put a file named "README" in that subdirectory. Its contents are up to you; to see examples of the kind of information that is usually placed in README files, see www.spec.org/cpu2006/src.alt. In your draft, kindly do NOT claim that it is approved; the release manager will do that at the last minute.
Once the directory structure is set up, creation of the src.alt package is simple: just type makesrcalt followed by the name of the benchmark and the src.alt name.
Contact SPEC to get the src.alt approved.
Example: We've made source changes to 403.gcc to remove preprocessor #line directives from the c-parse.c source file. This src.alt will be called 'no_line':
$ makesrcalt 403.gcc no_line Making 'no_line' src.alt for 403.gcc This will be usable with version 1.2 of the suite. Making differences and computing checksums... Writing control file... Gathering files... Wrote /spec/cpu2006/403.gcc.no_line.cpu2006.v1.2.tar.xz Finished making 'no_line' src.alt for 403.gcc. Elapsed time: 0 seconds $
If you use --help, you'll get an up-to-date list of options. Usually, you won't need any of these:
$ makesrcalt --help
Usage: bin/makesrcalt [options] <benchmark> <srcaltname>
Options:
--usewith <min>:<max> -- Set the versions of the suite that will accept
this src.alt (defaults to 1.1)
--notarball -- Skip packaging the src.alt as a tarball (created
by default)
--context <lines> -- Set lines of context for the diff. This defaults
to 3 and should not be changed.
--help -- Print this message
<benchmark> must be the full nnn.name form
<srcaltname> must match the name of the subdirectory in the benchmark's
src.alt directory
$
(History: the makesrcalt utility was improved in, and was documented for the first time with, SPEC CPU2006 V1.1.)
When you source shrc, as described in runspec.html, the 'ogo' shell alias is created. Its function is similar to the go alias with one important difference: if you use the output_root setting in your config file (also discussed in runspec.html) you can use ogo as a shorthand for moving around both the original installed benchmark tree and your output tree.
To do so, simply set the GO environment variable to the same value as your output_root. Then, 'ogo' will use the contents of that variable as the top level of the benchmark tree when looking for run directories and results.
For example, you could say something like this:
$ go config /spec/config $ grep output_root cds.cfg output_root = /cds_stuff $ GO=/cds_stuff $ ogo run perl Using value in GO for output_root: /cds_stuff /cds_stuff/benchspec/CPU2006/400.perlbench/run $
Note the helpful message that tells you which directory you have landed in. (If you define the environment variable OGO_NO_WARN the message will be shorter - the first line "Using value in GO..." will not be output.)
In this example, the benchmarks are installed in "/spec", and output_root is set to "/cds_stuff" in the config file. Any runs using that config file will write their results, executables, and run directories under "/cds_stuff". Continuing with other examples:
| If you say | You'll end up in |
| ogo ogo top |
/cds_stuff Your own private output tree |
|---|---|
| ogo result | /cds_stuff/result Your own private result directory |
| ogo 400 run | /cds_stuff/benchspec/CPU2006/400.perlbench/run Your own private run directories |
| ogo bzip exe | /cds_stuff/benchspec/CPU2006/401.bzip2/exe Your own private executables |
| ogo doc | /spec/Docs There's only one Docs directory, on the original installed $SPEC tree |
| ogo config | /spec/config The config directory is always on the original installed tree; set its protections as described in runspec.html |
| ogo 400 src | /spec/benchspec/CPU2006/400.perlbench/src Run directories are under output_root, but the benchmark src directory is still in the original installed $SPEC tree. |
ogo also understands -h and --help, and will output a short usage message in response to either of those arguments.
(History: since SPEC CPU2006 V1.1,) Saying ogo top or just ogo takes you to $GO. It used to take you to $SPEC.
Note: It's not necessary to export the "GO" variable. Declaring this variable at the command-prompt is sufficient.
$ GO=/tmp/cds
$ echo $GO
/tmp/cds
(History: The ogo alias was added in SPEC CPU2006 V1.0.)
ogo has also been implemented for cshrc. There is one restriction: to get the full feature set above, you must set OGO to the actual directory, not to a symbolic link.
Sorry, the ogo feature does not exist for Windows systems.
The port_progress utility analyzes log files and summarizes errors. It was primarily used during the benchmark porting phase for SPEC CPU2006, but can still come in handy, for example when trying the benchmarks with a new compiler. To use it:
cd $SPEC/result (or on Windows, cd %SPEC%\result) port_progress logfile...
Optionally, you can limit output by saying:
-c config only report logs that match the named config file
In the example below:
$ port_progress -c dec2 *.log
SPEC CPU2006 mydir
Tester: john
dec22a
Base Peak
Te Tr Ref Te Tr Ref
400.perlbench S S S S S S
401.bzip2 S S S S S S
403.gcc S S S S S S
410.bwaves S b? b? b? b? b?
416.gamess
429.mcf S S S S S S
433.milc
434.zeusmp
435.gromacs
436.cactusADM
437.leslie3d
444.namd
445.gobmk S S S S S S
447.dealII
450.soplex
453.povray
454.calculix
456.hmmer S S S S S S
458.sjeng S S S S S S
459.GemsFDTD
462.libquantum S S S S S S
464.h264ref S S r? S S S
465.tonto
471.omnetpp S S S S S S
473.astar S S S S S S
481.wrf
482.sphinx3
483.xalancbmk S S S S S S
998.specrand
999.specrand S S S S S S
464.h264ref run failures in logs:
079 --define OUTDISK=/export/bmk/john/out/ -c dec22a --reportable -r int
New with CPU2006 V1.2, you can remove reporting on base runs with --nobase or remove peak with --nopeak. This option may be convenient if, at a particular moment, you are only debugging one type of output.
Some benchmark executables are invoked multiple times. If you'd like to know how much time was spent on each invocation, it may not be easy to tell. You could examine "speccmds.out", but that file is not easy to interpret:
$ runspec --config=try_utilities --size=test --iterations=1 \
--tuning=base --output_format=ps 400
$ go work 400
$ cd run_base_test_try_utilities.0000/
$
$ cat speccmds.out | fold -s
running commands in speccmds.cmd 1 times
runs started at 1148417981, 220153000, Tue May 23 13:59:41 2006
run 1 started at 1148417981, 222495000, Tue May 23 13:59:41 2006
child started: 0, 1148417981, 222535000, pid=2396,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
attrs.pl'
child finished: 0, 1148417981, 339666000, sec=0, nsec=117131000, pid=2396, rc=0
child started: 0, 1148417981, 339754000, pid=2397,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
gv.pl'
child finished: 0, 1148417981, 384163000, sec=0, nsec=44409000, pid=2397, rc=0
child started: 0, 1148417981, 384240000, pid=2398,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
makerand.pl'
child finished: 0, 1148417981, 710682000, sec=0, nsec=326442000, pid=2398, rc=0
child started: 0, 1148417981, 710786000, pid=2399,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
pack.pl'
child finished: 0, 1148417981, 973602000, sec=0, nsec=262816000, pid=2399, rc=0
child started: 0, 1148417981, 973684000, pid=2400,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
redef.pl'
child finished: 0, 1148417981, 997347000, sec=0, nsec=23663000, pid=2400, rc=0
child started: 0, 1148417981, 997429000, pid=2401,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
ref.pl'
child finished: 0, 1148417982, 24001000, sec=0, nsec=26572000, pid=2401, rc=0
child started: 0, 1148417982, 24081000, pid=2402,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
regmesg.pl'
child finished: 0, 1148417982, 55465000, sec=0, nsec=31384000, pid=2402, rc=0
child started: 0, 1148417982, 55621000, pid=2403,
'../run_base_test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib
test.pl'
child finished: 0, 1148417988, 281976000, sec=6, nsec=226355000, pid=2403, rc=0
run 1 finished at: 1148417988, 282030000, Tue May 23 13:59:48 2006
run 1 elapsed time: 7, 59535000, 7.059535000
runs finished at 1148417988, 282155000, Tue May 23 13:59:48 2006
runs elapsed time: 7, 62002000, 7.062002000
$
Fortunately, SPEC has provided a tool to interpret the above for you:
$ printkids.pl
Seconds Command
0.12 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib attr
0.04 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib gv.p
0.33 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib make
0.26 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib pack
0.02 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib rede
0.03 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib ref.
0.03 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib regm
6.23 test_try_utilities.0000/perlbench_base.try_utilities -I. -I./lib test
=========
7.06 Total by adding children
7.06 Total according to speccmds.out
$
As a convenience for Windows users, you can use printpath.pl to print your path in an easier to read form. You can send the output to a file, which can then be easily edited.
In the example below, printpath.pl makes it easy to notice that cygwin (a Windows/Unix compatibility product) is in the path. SPEC recommends that your %PATH% should not include Windows/Unix compatibility products, because these have sometimes contributed to hard-to-diagnose errors.
C:\cpu2006>echo %PATH%
C:\cpu2006\bin\windows;C:\cpu2006\bin;C:\Program Files\Microsoft Visual Studio 9
.0\Common7\IDE;C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN;C:\Program Fi
les\Microsoft Visual Studio 9.0\Common7\Tools;C:\WINDOWS\Microsoft.NET\Framework
\v3.5;C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;C:\Program Files\Microsoft V
isual Studio 9.0\VC\VCPackages;C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin
;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;c:\cygwin\bin;C:\WINDOW
S\system32\WindowsPowerShell\v1.0
C:\cpu2006>specperl bin\printpath.pl
PATH=^
C:\cpu2006\bin\windows;^
C:\cpu2006\bin;^
C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE;^
C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN;^
C:\Program Files\Microsoft Visual Studio 9.0\Common7\Tools;^
C:\WINDOWS\Microsoft.NET\Framework\v3.5;^
C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;^
C:\Program Files\Microsoft Visual Studio 9.0\VC\VCPackages;^
C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin;^
C:\WINDOWS\system32;^
C:\WINDOWS;^
C:\WINDOWS\System32\Wbem;^
c:\cygwin\bin;^ <-- not recommended
C:\WINDOWS\system32\WindowsPowerShell\v1.0
C:\cpu2006>specperl bin\printpath.pl > my-easy-to-edit-pathset.bat
C:\cpu2006>notepad my-easy-to-edit-pathset.bat
C:\cpu2006>my-easy-to-edit-pathset.bat
In the example above, to remove cygwin, the user just uses notepad to remove the second last line of the file my-easy-to-edit-pathset.bat
"rawformat" is the result formatter. It can be invoked directly, by typing rawformat, or equivalently by typing runspec --rawformat. These two commands achieve the same effect because, saying runspec --rawformat just causes runspec to exit, invoking rawformat in its stead, and passing the formatter whatever other options you mentioned on the command line.
The rawformat utility is also automatically invoked by runspec at the end of a run, to generate the reports from your run.
For convenience, most rawformat options are documented in runspec.html, which see for basic information on using this command. You can also say rawformat --help to get a list of switches that are used by the formatter.
Note that when runnng rawformat, you will always get format "Submission Check", which encourages consistent formatting for various result fields when preparing final (submittable) reports. In addition, you will get the formats that you mention on the command line, or, if none are mentioned there, then you will get the defaults documented under output_format. (History: the automatic addition of subcheck to the list of outputs was added in SPEC CPU2006 V1.1)
A few additional points are worth mentioning, in the following sections.
A result rawfile may be edited to update reader fields, such as the system description. Rawfile syntax is similar to the syntax in config files, with a few exceptions:
These points are described in this section.
WARNING: Before you do anything, make a backup copy of the rawfile first!
Example: What do you do when Marketing decides to change the name from "SuperHero 4" to "SuperHero IV" just before you publish the results, or just before you're ready to submit them to SPEC? In your config file, you had
hw_model = SuperHero 4
which has caused the rawfile to contain the line
spec.cpu2006.hw_model: SuperHero 4
and you've just been told that the name is wrong. You don't have time to re-run your test. It's too late to change the config file.
SPEC does allow you to edit the rawfile to make corrections. Just make sure that you only edit the fields located before the "fence":
# =============== do not edit below this point ===================
WARNING: Make a backup copy of the rawfile first!
For example:
$ cd $SPEC/result
$ cp CINT2006.001.ref.rsf new.rsf
edit new.rsf to make your changes
$ diff CINT2006.001.ref.rsf new.rsf
9c9
< spec.cpu2006.hw_model: SuperHero 4
---
> spec.cpu2006.hw_model: SuperHero IV
$
$ rawformat -o ps new.rsf
runspec v4256 - Copyright 1999-2006 Standard Performance Evaluation Corporation
Using 'macosx-ppc' tools
Reading MANIFEST... 2565 files
Loading runspec modules...............
Locating benchmarks...found 31 benchmarks in 2 benchsets.
Locating output formats: ASCII, config, CSV, flags, html, mail, PDF,
PostScript, raw, Screen, Submission Check
Formatting new.rsf
Parsing flags for 400.perlbench base: done
Parsing flags for 401.bzip2 base: done
Parsing flags for 403.gcc base: done
Parsing flags for 429.mcf base: done
Parsing flags for 445.gobmk base: done
Parsing flags for 456.hmmer base: done
Parsing flags for 458.sjeng base: done
Parsing flags for 462.libquantum base: done
Parsing flags for 464.h264ref base: done
Parsing flags for 471.omnetpp base: done
Parsing flags for 473.astar base: done
Parsing flags for 483.xalancbmk base: done
Doing flag reduction: done
format: PostScript -> new.ps
There is no log file for this run.
runspec finished at Tue May 23 15:09:25 2006; 16 total seconds elapsed
$
Notice when editing fields that have more than one line that you need to construct the numbers with correct syntax. The syntax in the rawfile is a bit more picky than in config files (compare the following vs. the config.html section on "Field scoping and continuation". You cannot say something like this:
| WRONG: |
spec.cpu2006.sw_compiler: Intel C Compiler 9.1 for Windows XP
spec.cpu2006.sw_compiler1: Visual Studio 2005 (for libraries) |
because the tools are not prepared at this point for a mixture of numbered and un-numbered fields. If you need to have continued fields, say instead:
| RIGHT: |
spec.cpu2006.sw_compiler0: Intel C Compiler 9.1 for Windows XP
spec.cpu2006.sw_compiler1: Visual Studio 2005 (for libraries) |
Two testbed description fields are allowed in raw files that are not allowed in config files:
| sw_parallel_defeat | This field can be used to specify a reason why, despite the settings of other fields, parallelism was defeated on a particular SUT. For example, you could set: spec.cpu2006.sw_parallel_defeat: Tested with OMP_NUM_THREADS=1 This field cannot be set in config files, but can be set in raw files. If set in a config file, a warning will be issued and the setting will be ignored. The reason that you specify will be displayed in the notes section of reports. See also the discussion Parllel Reporting in config.html. (History: sw_parallel_defeat was added in SPEC CPU2006 V1.1) |
| test_date | When the tests were run. (History: As of SPEC CPU2006 V1.1,) Tthis field is populated automatically based on the clock in the system under test. Setting this in the config file will generate a warning and the setting will be ignored. If your system clock is incorrect, then the value may be edited in the raw file. It's better to avoid the necessity to edit, by setting your system clock properly. |
To generate a 1-copy SPECrate result from a SPECspeed run, copy the original rawfile to another location, and use rawformat to both generate the new rawfile and whatever other reports you want. For example:
$ ls
CINT2006.383.ref.rsf CINT2006.383.ref.txt
$ grep SPECint *txt | grep base
SPECint(R)_base2006 10.1
$ cp CINT2006.383.ref.rsf convertme
$ rawformat --output_format rsf,txt --rate convertme
rawformat v5613 - Copyright 1999-2007 Standard Performance Evaluation Corporation
Using 'solaris-sparc' tools
Reading MANIFEST... 2357 files
Loading rawformat modules............
Locating benchmarks...found 31 benchmarks in 3 benchsets.
Locating output formats: ASCII, config, CSV, flags, HTML, mail, PDF, PostScript, raw, Screen, Submission Check
Formatting convertme
Parsing flags for 400.perlbench base: done
Parsing flags for 400.perlbench peak: done
Parsing flags for 401.bzip2 base: done
Parsing flags for 401.bzip2 peak: done
Parsing flags for 403.gcc base: done
Parsing flags for 403.gcc peak: done
Parsing flags for 429.mcf base: done
Parsing flags for 429.mcf peak: done
Parsing flags for 445.gobmk base: done
Parsing flags for 445.gobmk peak: done
Parsing flags for 456.hmmer base: done
Parsing flags for 456.hmmer peak: done
Parsing flags for 458.sjeng base: done
Parsing flags for 458.sjeng peak: done
Parsing flags for 462.libquantum base: done
Parsing flags for 462.libquantum peak: done
Parsing flags for 464.h264ref base: done
Parsing flags for 464.h264ref peak: done
Parsing flags for 471.omnetpp base: done
Parsing flags for 471.omnetpp peak: done
Parsing flags for 473.astar base: done
Parsing flags for 473.astar peak: done
Parsing flags for 483.xalancbmk base: done
Parsing flags for 483.xalancbmk peak: done
Doing flag reduction: done
format: raw -> convertme.rsf
format: ASCII -> convertme.txt
format: Submission Check -> PASSED syntax check
Note: although this result has PASSED subcheck
you should be aware that:
- SPEC may have updated requirements in the version of the
checker at the SPEC submission website.
- Much (though not all) of SPEC's version is downloadable,
and can be synchronized with your version; see:
http://www.spec.org/auto/cpu2006/Docs/runspec.html#newflags
rawformat finished at Thu Jan 10 05:40:45 2008; 62 total seconds elapsed
$ grep SPECint *txt | grep base
CINT2006.383.txt: SPECint(R)_base2006 10.1
convertme.txt: SPECint(R)_rate_base2006 10.1
$
As you can see, the new file convertme.txt has the SPECrate merics.
We can also continue the above example and convert back again:
$ cp convertme.rsf backagain
$ rawformat --output_format rsf,txt --speed backagain
rawformat v5613 - Copyright 1999-2007 Standard Performance Evaluation Corporation
Using 'solaris-sparc' tools
Reading MANIFEST... 2357 files
Loading rawformat modules............
Locating benchmarks...found 31 benchmarks in 3 benchsets.
Locating output formats: ASCII, config, CSV, flags, HTML, mail, PDF, PostScript, raw, Screen, Submission Check
Formatting backagain
Parsing flags for 400.perlbench base: done
Parsing flags for 400.perlbench peak: done
Parsing flags for 401.bzip2 base: done
Parsing flags for 401.bzip2 peak: done
Parsing flags for 403.gcc base: done
Parsing flags for 403.gcc peak: done
Parsing flags for 429.mcf base: done
Parsing flags for 429.mcf peak: done
Parsing flags for 445.gobmk base: done
Parsing flags for 445.gobmk peak: done
Parsing flags for 456.hmmer base: done
Parsing flags for 456.hmmer peak: done
Parsing flags for 458.sjeng base: done
Parsing flags for 458.sjeng peak: done
Parsing flags for 462.libquantum base: done
Parsing flags for 462.libquantum peak: done
Parsing flags for 464.h264ref base: done
Parsing flags for 464.h264ref peak: done
Parsing flags for 471.omnetpp base: done
Parsing flags for 471.omnetpp peak: done
Parsing flags for 473.astar base: done
Parsing flags for 473.astar peak: done
Parsing flags for 483.xalancbmk base: done
Parsing flags for 483.xalancbmk peak: done
Doing flag reduction: done
format: raw -> backagain.rsf
format: ASCII -> backagain.txt
format: Submission Check -> PASSED syntax check
Note: although this result has PASSED subcheck
you should be aware that:
- SPEC may have updated requirements in the version of the
checker at the SPEC submission website.
- Much (though not all) of SPEC's version is downloadable,
and can be synchronized with your version; see:
http://www.spec.org/auto/cpu2006/Docs/runspec.html#newflags
rawformat finished at Thu Jan 10 05:42:32 2008; 64 total seconds elapsed
$ grep SPECint *txt | grep base
CINT2006.383.ref.txt: SPECint(R)_base2006 10.1
backagain.txt: SPECint(R)_base2006 10.1
convertme.txt: SPECint(R)_rate_base2006 10.1
$ diff CINT2006.383.txt backagain.txt
185c185
< Report generated on Wed Jan 9 16:07:02 2008 by CPU2006 ASCII formatter v5613.
---
> Report generated on Thu Jan 10 05:42:32 2008 by CPU2006 ASCII formatter v5613.
$
Notice above that the round trip is complete: the original report, from run 383, matches backagain.txt in all but the time that the formatter was run.
You cannot format a multi-copy SPECrate run as a SPECspeed run; only a single-copy run.
Also notice above that output format "Submission Check" (aka subcheck) was automatically selected. This is because rawformat is often used to check raw file edits for results that are to be submitted to SPEC for publication.
You may find that --output_format will only accept one argument at a time. If so, first create the rawfile with --output_format raw, then use the new rawfile to create the other reports.
If you rearrange mount points or rename your SPEC root directory, you will probably find confusing problems like the one in the example that follows. We used to have a working SPEC tree under /cpu2006/cpu2006-kit91, but we're going to change that:
$ cd /cpu2006 $ mv cpu2006-kit91 jamesbond $ cd jamesbond $ . ./shrc $ runspec -h -bash: /cpu2006/jamesbond/bin/runspec: /cpu2006/cpu2006-kit91/bin/specperl: bad interpreter: No such file or directory
The problem is that the first line of runspec points to a path that no longer exists:
$ head -1 bin/runspec #!/cpu2006/cpu2006-kit91/bin/specperl $
Fortunately, SPEC has provided a tool to relocate the tools in the new tree:
$ cd /cpu2006/jamesbond $ . ./shrc $ relocate Top of SPEC benchmark tree is '/cpu2006/jamesbond' $ runspec -h | head -3 runspec v4256 - Copyright 1999-2006 Standard Performance Evaluation Corporation Using 'macosx-ppc' tools Reading MANIFEST... 2565 files
Limitations: There are some limitations to this tool. Most notably, it uses perl, so if the tree has been re-arranged in such a way that perl has an identity crisis and can't find itself, you'll be in trouble. Make sure that you have sourced shrc, as in the next example; if it still fails, you may need to reinstall the CPU2006 kit.
$ cd /cpu2006
$ mv jamesbond dr_evil
$ cd dr_evil
$ bin/relocate
Your $SPEC variable is not set correctly!
Top of SPEC benchmark tree is '/cpu2006/dr_evil'
Can't locate strict.pm in @INC (@INC contains: /cpu2006/jamesbond/bin
/cpu2006/jamesbond/bin/lib .) at bin/relocate line 3.
BEGIN failed--compilation aborted at bin/relocate line 3.
$ . ./shrc
$ bin/relocate
Top of SPEC benchmark tree is '/cpu2006/dr_evil'
$ runspec -h | head -2
runspec v4256 - Copyright 1999-2006 Standard Performance Evaluation Corporation
Using 'macosx-ppc' tools
Reading MANIFEST... 2565 files
$
Important: delete the old run and build directories. When setting up run directories, the tools read benchspec/CPU2006/nnn.benchmark/run/list to see what directories already exist. (A similar method is used for build directories.) If the list contains references to directories that are not underneath the current top, the intended behavior is that such directories should cause a warning, and should not be used. For example:
Notice: Unusable path detected in run directory list file.
/brian/liberator/cpu2006/benchspec/CPU2006/998.specrand/run/list
references one or more paths which will be ignored, because they are
not subdirectories of this run directory. This condition may be a
result of having moved your SPEC benchmark tree. If that's what happened,
and if you don't need the old run directories, you can just remove
them, along with the list file. (Usually it's safe to delete old
run directories, as they are automatically re-created when needed.)
Unfortunately, it is sometimes possible to confuse the tools when the list contains directories that are not underneath $SPEC. Although the above message is printed, the tools can actually end up using a mixture of old and new directory paths, and nothing good can come of such mixtures. To avoid the confusion, you should delete the old run directories right after the relocate. Old build directories should be deleted (perhaps after backing them up if you wish to save them for reference). For example:
mkdir newdir mv $SPEC/* newdir cd newdir bin/relocate cd benchspec/CPU2006 spectar cf - */build | specxz > ~/mybuilds.backup.tar.xz rm -Rf */run */build
In order to decide whether your system got the correct answer for a benchmark, runspec writes a file called compare.cmd in the run directory, and then calls specinvoke, which calls specdiff. (The specinvoke utility is described below.)
Suppose that your results are flagged as incorrect, but you'd like to have more detail. You could ask for more than 10 lines of detail by changing the config file's default for difflines from 10 to something else, and re-running the experiment. But, you might save a lot of time and avoid the re-run of the experiment, by calling specdiff directly, with a different number of lines for its switch -l.
For example, a file chock-full of wrong answers for 444.namd has been created by hand. First, let's see how specdiff would normally be run (line wraps added for readability):
$ go run namd $ cd run_base_test_try_utilities.0000/ $ specinvoke -c 1 -f compare.cmd -n # Use another -n on the command line to see chdir commands # Starting run for copy #0 specperl /cpu2006/dr_evil/bin/specdiff -m -l 10 --abstol 1e-05 \ /cpu2006/dr_evil/benchspec/CPU2006/444.namd/data/test/output/namd.out namd.out > namd.out.cmp
The above says that normally we test for correct answers by comparing the file $SPEC/benchspec/CPU2006/444.namd/data/test/output/namd.out versus the file namd.out in the current directory. Normally, 10 lines of detail are printed. Let's change that to 15 lines, and compare versus our intentionally wrong file instead:
$ specdiff -m -l 15 --abstol 1e-05 \
/cpu2006/dr_evil/benchspec/CPU2006/444.namd/data/test/output/namd.out namd_wrong.out
0003: -355359.705797 0.000000 24229.348329
-355359.705797 0.000000 23229.338329
^
0004: 229430.745102 5781.466932 -2454.694382
229330.735102 5781.366932 -2353.693382
^
0005: 5781.466932 226184.623759 -7867.535225
5781.366932 226183.623759 -7867.535225
^
0006: -2454.694382 -7867.535225 252281.168869
-2353.693382 -7867.535225 252281.168869
^
0010: 0.000000 -0.000000 0.000000 24.034240
-0.000000 -0.000000 0.000000 23.033230
^
0012: 0.000000 -0.000000 0.000000 24.034240
-0.000000 -0.000000 0.000000 23.033230
^
0016: -355359.705797 1175488.426964 24229.348329
-355359.705797 1175388.326963 23229.338329
^
0017: 229430.745102 5781.466932 -2454.694382
229330.735102 5781.366932 -2353.693382
^
0018: 5781.466932 226184.623759 -7867.535225
5781.366932 226183.623759 -7867.535225
^
0019: -2454.694382 -7867.535225 252281.168869
-2353.693382 -7867.535225 252281.168869
^
0020: 5180.324284 277.843622 -1135.837232
5180.323283 277.833622 -1135.837232
^
0021: 277.843622 2960.457709 -681.816292
277.833622 2960.357709 -681.816292
^
0022: -1135.837232 -681.816292 4515.105462
-1135.837232 -681.816292 3515.105362
^
0023: 0.000000 -0.000000 0.000000 24.034240
-0.000000 -0.000000 0.000000 23.033230
^
0024: -0.000000 -0.000000 0.000000 0.654402
0.000000 -0.000000 0.000000 0.653302
^
0025: 0.000000 -0.000000 0.000000 24.079992
-0.000000 -0.000000 0.000000 23.079992
^
specdiff run completed
$
Here are the switches supported by specdiff:
$ specdiff -h
Usage: /spec/cpu2006/bin/specdiff <options> file1 [file2]
-l,--lines # of lines of differences to print (-1 for all)
-q,--quiet don't print lines just set return code
-a,--abstol absolute tolerance (for floating point compares)
-r,--reltol relative tolerance (for floating point compares)
-o,--obiwan allow off-by-one errors
-O,--OBIWAN *don't* allow off-by-one errors
-s,--skiptol # of differences to ignore
--skipabstol # of abstol differences to ignore
--skipreltol # of reltol differences to ignore
--skipobiwan # of off-by-one differences to ignore
-t,--calctol calculate required tolerances
-H,--histogram show a histogram of differences
-f,--floatcompare Do comparison for floating-point values even if no
tolerances are specified. (Normally this is
automatic.)
-m,--mis write file2.mis with miscompares
-c,--cw collapse whitespace (probably doesn't do what you
think it does)
-C,--CW *don't* collapse whitespace
-b,--binary Do byte-wise comparison for binary outputs
-i,--ignorecase Ignore case for string comparisons
-d,--datadir Compare against file(s) in data directory
--os Set the operating system type (this is unnecessary)
-v,--verbose Set the level of noisiness for the output
-h,--help Print this message
$
Changed (fixed) in V1.2: The specdiff program, as of CPU2006 V1.2, removes previous outputs prior to running. For CPU2006 V1.0 and V1.1, you needed to remove previous outputs by hand before your run in order to avoid confusion from output files (e.g. .mis) that were hanging around from earlier runs.
In order to run a benchmark, runspec writes a series of commands in the run directory, in the file speccmds.cmd. These commands are then interpreted by specinvoke.
When something goes wrong, it is often useful to try the commands by hand. Ask specinvoke what it did, by using the "-n" switch:
$ go namd run
/cpu2006/dr_evil/benchspec/CPU2006/444.namd/run
$ cd run_base_test_try_utilities.0000
$ ls
compare.cmd namd.input namd_base.try_utilities
compare.err namd.out.cmp speccmds.cmd
compare.out namd.out.mis speccmds.err
compare.stdout namd.out.orig speccmds.out
namd.err namd.stdout speccmds.stdout
$ specinvoke -n
# Use another -n on the command line to see chdir commands
# Starting run for copy #0
../run_base_test_try_utilities.0000/namd_base.try_utilities \
--input namd.input --iterations 1 \
--output namd.out > namd.stdout 2>> namd.err
$
In the above example, we see how namd is invoked (line wraps added for readability). If you wanted to try out your own variations on the above command by hand, you could do so.
Here are the switches supported by specinvoke:
$ specinvoke -h
Usage: specinvoke [options] [instructionfile]
-i <n> iterations
-c <n> concurrent processes (overrides -C in command file)
Do not attempt to use this flag for rate runs.
-E return non-zero exit code if child does
-s shell shell to invoke
-f file instruction file
-o file output file
-e file error file
-d dir change to dir first
-n[n] print a 'dry_run' of commands
-A return 0 error code
-S msecs sleep between spawning copies (in milliseconds)
-r don't do I/O redirection ($command already has it)
-N open null device when no input file is specified
-Z use zero-length file when no input file is specified
-C (default) close stdin when no input file is specified
-D <n> Turn on debugging at level <n>
-q Stop issuing commands as soon as one fails (default)
-k Continue issuing commands even if one fails
-h this message
The -q and -k switches above are new with CPU2006 1.2. They may be useful if specinvoke is being used separately as part of a benchmark development or debug effort (see runspec-avoidance.html.)
New with CPU2006 V1.2, the output of specinvoke -nn is expanded. It now includes information about environment variables that were set by the tools prior to the invocation, which may come in very handy if you want to run a benchmark outside the SPEC harness, for example, during a debugging or tuning effort.
Consider the following example, where OMP_NUM_THREADS is set prior to doing a (fake) invocation of runspec:
$ cd $SPEC/config $ export OMP_NUM_THREADS=1024 $ cp Example-linux64-amd64-gcc43+.cfg jeff.cfg $ runspec --config=jeff --fake --loose --size=test libquantum > /dev/null $ $ go libquantum run /home/jeff/cpu2006/benchspec/CPU2006/462.libquantum/run $ cd run_base_test_gcc43-64bit.0000/ $ grep OMP speccmds.cmd -E OMP_NUM_THREADS 1024 $ $ specinvoke -nn | grep OMP OMP_NUM_THREADS=1024 export OMP_NUM_THREADS PATH REPLYTO SHELL SPEC SPECDB_PWD SPECPERLLIB SSH_ASKPASS $
In the example above, an environment variable is set to request 1024 threads, and the setting is recorded in speccmds.cmd. When specinvoke -nn is used, the setting is printed, along with an export command that would put the setting into the environment (along with various other variables that the tools noticed in the original environment).
It is important to understand that specinvoke does not actually set any environment variables for you. Consider this simplified test case:
$ cat tmp.cmd -E MY_IMPORTANT_ENV_SETTING enabled echo "important setting: '$MY_IMPORTANT_ENV_SETTING'" $ specinvoke tmp.cmd important setting: '' $
In the example above, the echo command prints nothing for the environment variable, despite the presence of -E MY_IMPORTANT_ENV_SETTING enabled in the command file. So, one might ask, what good is the feature? Well, it's very useful, if you just redirect the output of specinvoke -nn to a file, as is commonly done:
$ specinvoke -nn tmp.cmd > doit.sh $ sh doit.sh important setting: 'enabled' $
To build a benchmark, runspec uses specmake, which is simply GNU make under a unique name to avoid possible conflicts with other versions of make on your system.
If a benchmark build fails, one of the most massively useful things you can do is to ask specmake what it did, by using the "-n" switch, which means "dry run".
For example, let's create a config file that is unlikely to work:
$ go config $ cat > tmp.cfg ext = silly FC = f66 $
and try it out:
$ runspec --config=tmp --tuning=base --action=build leslie3d
runspec v4256 - Copyright 1999-2006 Standard Performance Evaluation Corporation
Using 'macosx-ppc' tools
Reading MANIFEST... 2565 files
Loading runspec modules...............
Locating benchmarks...found 31 benchmarks in 13 benchsets.
Locating output formats: ASCII, config, CSV, flags, html, mail, PDF,
PostScript, raw, Screen, Submission Check
Reading config file '/cpu2006/dr_evil/config/tmp.cfg'
Benchmarks selected: 437.leslie3d
Compiling Binaries
Building 437.leslie3d ref base silly default: (build_base_silly.0000)
Error with make 'specmake build': check file
'/cpu2006/dr_evil/benchspec/CPU2006/437.leslie3d/run/build_base_silly.0000/make.err'
Error with make!
*** Error building 437.leslie3d
If you wish to ignore this error, please use '-I' or ignore errors.
The log for this run is in /cpu2006/dr_evil/result/CPU2006.009.log
runspec finished at Thu May 25 10:01:54 2006; 8 total seconds elapsed
$
Sure enough, it failed. Let's find the appropriate run directory where our silly build was attempted:
$ go leslie3d run /cpu2006/dr_evil/benchspec/CPU2006/437.leslie3d/run $ ls build_base_silly.0000 list $ cd build_base_silly.0000
And ask specmake just exactly what it generated:
$ specmake -n f66 -c -o tml.o tml.f f66 tml.o -o leslie3d
OK, let's enter the command by hand:
$ f66 -c -o tml.o tml.f -bash: f66: command not found $
A great convenience of the CPU2006 suite is the fact that you can now proceed to play with all the files in this run directory, to prototype different changes and try out ideas, without fear that you will break anything. As long as you confine your edits to (in this example) $SPEC/benchspec/CPU2006/437.leslie3d/run/build_base_silly.0000, you are in a private sandbox and will not harm the original sources.
Here are the switches supported by specmake:
$ specmake -h
Usage: specmake [options] [target] ...
Options:
-b, -m Ignored for compatibility.
-B, --always-make Unconditionally make all targets.
-C DIRECTORY, --directory=DIRECTORY
Change to DIRECTORY before doing anything.
-d Print lots of debugging information.
--debug[=FLAGS] Print various types of debugging information.
-e, --environment-overrides
Environment variables override makefiles.
-f FILE, --file=FILE, --makefile=FILE
Read FILE as a makefile.
-h, --help Print this message and exit.
-i, --ignore-errors Ignore errors from commands.
-I DIRECTORY, --include-dir=DIRECTORY
Search DIRECTORY for included makefiles.
-j [N], --jobs[=N] Allow N jobs at once; infinite jobs with no arg.
-k, --keep-going Keep going when some targets can't be made.
-l [N], --load-average[=N], --max-load[=N]
Don't start multiple jobs unless load is below N.
-n, --just-print, --dry-run, --recon
Don't actually run any commands; just print them.
-o FILE, --old-file=FILE, --assume-old=FILE
Consider FILE to be very old and don't remake it.
-p, --print-data-base Print make's internal database.
-q, --question Run no commands; exit status says if up to date.
-r, --no-builtin-rules Disable the built-in implicit rules.
-R, --no-builtin-variables Disable the built-in variable settings.
-s, --silent, --quiet Don't echo commands.
-S, --no-keep-going, --stop
Turns off -k.
-t, --touch Touch targets instead of remaking them.
-v, --version Print the version number of make and exit.
-w, --print-directory Print the current directory.
--no-print-directory Turn off -w, even if it was turned on implicitly.
-W FILE, --what-if=FILE, --new-file=FILE, --assume-new=FILE
Consider FILE to be infinitely new.
--warn-undefined-variables Warn when an undefined variable is referenced.
This program built for powerpc-apple-darwin6.8
Report bugs to <bug-make@gnu.org>
To check XML flags files, runspec uses specrxp, which is simply RXP under a unique name to ensure no possible conflicts with other versions of rxp on your system. RXP is a validating XML parser. If your flags file passes RXP's tests, that doesn't mean that the flags file makes sense, but it's a start.
You don't have to know anything abour specrxp, as it is called automatically. But if you want to, you can call it directly, for example:
specrxp -V -s myflags.xml
The -V switch requests validation, and -s requests that all output be suppressed except error messages.
A manpage for rxp is available.
Once you have your flags file passing specrxp validation, you should do a flag_dump, and validate that the dump is valid HTML.
If specrxp seems to hang:
If specrxp seems to hang when you run it, it may because this line in your flags file requires internet access, but your system is behind a firewall:
SYSTEM "http://www.spec.org/dtd/cpuflags2.dtd"
To resolve the problem, you can use one of these three solutions:
Copy the flags file to a system that does have access.
Define proxy environment variable(s) to allow internet access for programs on your system. For example, depending on your operating system and your shell, you might be able to enter a command such as:
export http_proxy=http://www-proxy.us.mycompany.com:80/
Or *temporarily* change the line in the flags file to point to a local copy, like so:
SYSTEM "file:///Arques/Site/Lab/carol/cpu2006/Docs/flags/cpuflags2.dtd" <-- temporary only!
Warning: Be sure to change it back to the correct location prior to attempting to submit a result to SPEC, because SPEC probably does not have access to /Arques/Site/Lab/, and the submission handler will probably bounce your submission.
If you get unexpected errors from specrxp:
If you get unexpected messages such as:
Warning: Start tag for undeclared element
check that you have correctly set the dtd version.
New: as of SPEC CPU2006 V1.2, please note that the dtd version is now cpuflags2.dtd, not cpuflags1.dtd.
(History: The specrxp utility was added in SPEC CPU2006 V1.1)
The utility spectar is simply GNU tar, built under a known set of circumstances at the same time as other tools in the SPEC toolset, and renamed to avoid conflicts with any other versions of the program on your path. GNU tar, unlike some other archiving programs, correctly handles very long path names.
The utility specxz is XZ Utils, built under a known set of circumstances at the same time as other tools in the SPEC toolset, and renamed to avoid conflicts with any other versions of the program on your path.
Commonly-used switches work analogously to those in other compression utilities, such as -d to decompress. For details, say specxz --help
Inclusion of specxz is new with CPU2006 V1.2. Switching compression programs has reduced the size of SPEC distribution media. If you have written scripts that rely on specbzip2, it will be necessary to either retain a copy of specbzip2 from your V1.1 kit, or to change your scripts to specxz. If you make the change, you will also probably notice that specxz produces smaller files.
Warning: this idiom, commonly used on Unix systems, has been observed to fail on some tested Windows systems:
specxz -dc mumble.tar.xz | spectar -xf - <-- may fail on Windows
If you see failures, break it into two commands.
specxz -d mumble.tar.xz <-- an alternative
spectar -xf mumble.tar
If you have a rawfile, you can get information about the binary toolset and the versions of all of the system-independent tools files. To do so, use the toolsver utility.
For example, starting with a freshly-installed SPEC CPU2006 directory tree:
% runspec --config=try_utilities --size=test --iterations=1 \ --tuning=base --output_format=ps > /dev/null % cd $SPEC/result % ls CINT2006.001.test.ps CINT2006.001.test.rsf CPU2006.001.log lock.CPU2006
Note that we have just the single result file. Now, attempt to extract the version information:
% toolsver CINT2006.001.test.rsf
CINT2006.001.test.rsf:
Binary toolset used was 'macosx-ppc'
Versions of Perl-based tools:
4256 runspec
4163 CPUFlagsParser.pm
4160 PSPDF.pm
4253 asc.pl
4256 benchmark.pm
4223 benchset.pm
4238 cfgfile.pl
3626 compare.pl
4243 config.pl
4247 csv.pl
4238 flags.pl
4238 flagutils.pl
3626 format.pm
4238 listfile.pm
4166 locate.pl
4160 log.pl
4214 mail.pl
3626 monitor.pl
4160 os.pl
4252 parse.pl
4160 pdf.pl
4253 ps.pl
4184 raw.pl
3626 screen.pl
4212 subcheck.pl
4251 util.pl
4196 vars.pl
(History: The toolsver utility was added in SPEC CPU2006 V1.0.)
Copyright 1999-2011 Standard Performance Evaluation Corporation
All Rights Reserved