specwebclient.jar - a load generator which runs on each client system
specweb.jar - a test manager which runs on the prime client
reporter.jar - a report generator
configuration files for the test and for each workload (banking, ecommerce, support, and power)
Redistributed Open Source Code, which consists of:
specweb_src.jar - Java source code
jfreechart-1.0.12.tar.gz - JFreeChart class file
jcommon-1.0.15.tar.gz - JCommon class file
Smarty 2.6.22
FastCGI
apache-tomcat-6.0.18
Wafgen - a file set generator which is run on the Web server to create the static files referenced by the workload.
BeSim - a Backend Simulator which runs on an independent Web server system connected to both the prime client and the system under test (SUT). BeSim includes C code which can be compiled for FastCGI, ISAPI, or NSAPI.
Power and Temperature Daemon (PTD) version 1.4.0 – software from SPECpower that allows a controller (usually the prime client) to interface with power analyzers and temperature sensors.
Dynamic scripts to be run on the Web server that implement the dynamic pages for each workload
PHP scripts and Smarty (http://www.smarty.net/) templates
JSP scripts and Java class files
ASP.NET aspx files and C# classes
Assorted "readme" and "install" documentation files to help explain how do setup, configure, and run specific components. Please review these files; most have been included as appendixes to the document for your convenience.

1.1.2 Additional Software Requirements:

Web server Software (such as IIS, Apache, Zeus, Sun One, Rock Web Server) that can support PHP, ASP.NET, or JSP scripts.
J2SE 1.6 or newer JVM (for running the test harness)
C compiler (for compiling BeSim)
PHP 5.0 or newer from http://www.php.net (if running the supplied PHP dynamic scripts)
Application Server such as Tomcat 6.0.18 (if running the supplied JSP dynamic scripts)
If a more recent PTD version is available than the one included in the kit (version 1.4.0), the user may choose to upgrade it, but is not required to do so.

1.2 Hardware Requirements

SUT: a system configured as a Web server, including storage
Client(s): one or more client systems to act as load drivers for SPECweb2009
Prime Client: test controller that must be able to connect to all the clients, the BeSim and the SUT. The prime client can also function as a load generating driver if desired.
BeSim system: a client class system configured as a Web server with BeSim installed as a FastCGI, NSAPI, or ISAPI
Network: clients must be able to connect to the SUT and the SUT must be able to connect to the BeSim system.
Controller(s): one or more systems to run the PTD. The systems can be any of the clients or other systems, but must be accessible by the prime client.
Power Analyzers accepted by SPECpower
Temperature Sensors accepted by SPECpower
Note: For debugging purposes, you could run everything on 1 or 2 systems but these instructions assume you have at least 3.

2 Setting Up The Test Environment

2.1 Running the SPECweb2009 installer

Place a copy of the provided setup.jar or setup.exe on all systems that are part of your test setup. You will need to have a JVM installed; a JDK or JRE version 1.6 or later is recommended.

To Run the Linux installer, invoke the Java installer as follows:

To run via X/GUI: java -jar setup.jar
To run via console: java -jar setup.jar –i console

To run the Windows installer:

Double-click setup.jar within Windows Explorer

If the installation of the SPECweb2009 components fails with the following message:

Errors occurred during the installation.

- JVM not found

Please set the JAVA_HOME environment variable to the root path of your JVM engine.

Linux: export JAVA_HOME=/usr/path_to_java

Windows: set JAVA_HOME=c:\path_to_java

2.2 Web Server Setup

General Directions:

Set localization on the server system to US.
Install and configure your favorite Web server with SSL and PHP, ASP.NET, or JSP support configured in. If using JSP, a separate Java application server may be needed. Check your Web server software documentation for assistance.
Run the SPECweb2009 installer (see above) and set your current working directory to the wafgen directory.
# pwd /web2009-install/wafgen

Depending on the type of scripts you want to use choose one of the following options:

If you plan to use the JSP dynamic scripts provided:

Check to see that you have either a Web server that supports JSP scripts or a Java application server (i.e. Tomcat) configured.
Follow the directions provided in the SPECweb2009 Scripts/JSP/README.FIRST and the INSTALL file from each .war file (jar xvf < wkld>.war) to install the JSP scripts in their proper locations.

If you plan to use the PHP dynamic scripts provided:

Check to see that you have a version of PHP (such as PHP 5.0 from http://www.php.net) and that your Web server is configured to support PHP dynamic scripts.
Copy the contents of the SPECweb2009/Scripts/PHP/* directory to your document root (i.e. cp -ar /web2009-install/Scripts/PHP/* /var/www/html/)
Copy the Smarty-2.6.22 directory from the folder with "redistribution" folder into a folder under document root. Verify the "bank", "ecommerce", "support", and "Smarty-2.6.22" subdirectories are now under the document root.
Ensure the web server has read and write access to these directories and subdirectories (i.e. chmod -R a+rw bank/ ecommerce/ support/ Smarty-2.6.22/ from the document root). An alternative is to use chownto assign ownership to the Web server user/group.
The stock PHP distribution ships with two ini files, php.ini-dist and php.ini-recommended. While php.ini-recommended is optimized for performance and security, for initial troubleshooting, php.ini-dist may be a better choice. This will configure PHP to display errors in the browser. If PHP comes with your OS distribution, you may only have one php.ini, in which case you should consider changing "display_errors" and "display_startup_errors" to "On". You may also want to set " error_reporting = E_ALL", which will display all errors and warnings.

If you plan to use the ASP.NET dynamic scripts provided

Check to see that you have IIS 7.0 or above installed.
Either use the IIS manager to install each of the workloads or follow the directions provided in the SPECweb2009/Scripts/ASPX/README.FIRST file to install the ASP.NET scripts in their proper locations.
Review the Wafgen README which contains the instructions for tailoring the resource files (.rc) to create the workload specific file sets for the Web server.
Edit the .rc files for the workload(s) that you wish build the file sets for. Ensure that at the very least you set SIMULTANEOUS_SESSIONS and DOCROOT to the correct values.
Run wafgen according to the directions provided.
Check that the files generated are correct and under your Web server's document root.

2.3 Besim Setup

Configure the BeSim system with Web server software installed on port 81 or another port (avoid port 80 to avoid confusion with the primary Web server).
Run the SPECweb2009 installer, do a "BeSim" or "Full" installation, and change your current working directory to the <specweb_install_directory>/Besim directory.
Review the BeSim Make_Readme
NOTE: For Windows/IIS, a precompiled ISAPI is provided (BeSim.dll); copy this to a directory with ISAPI execute permissions, and skip to step 6.
You may choose from FastCGI* for Zeus or Apache or ISAPI for Zeus or IIS or NSAPI for SUN One and edit the appropriate
MakeIncl.<type>.<OS> file to reflect your compile and linker options and select one or more Defines. Then you can build the BeSim code for your selected API. (If using FastCGI , please copy the FastCGI executable from the Redistributed Open Source code directory into the BeSim directory.)
Install this code to your BeSim Web server.
Start your BeSim Web server.
Use the test_besim_bank, test_besim_ecom, and test_besim_support scripts (included in the "besim" subdirectory of the installation) to test whether you're getting valid BeSim responses. Both Perl scripts and Linux bash scripts are provided. The Perl scripts require a Perl interpreter as well as the following Perl modules: URI, HTML-Tagset, HTML-Parser, and libwww-perl. The bash scripts require cURL to be in the path.

Invoke these scripts with the URL to your compiled BeSim API.

perltest_besim_bank.pl http://192.2.1.132:81/fcgi-bin/besim_fcgi.fcgi
bashtest_besim_bank.sh http://bsim614:81/fcgi-bin/besim_fcgi.fcgi

*Note: For more FastCGI instructions (particularly for the Apache HTTP Server), see Appendix F.

If you notice that one BeSim system has a bottleneck (CPU, disk, network, etc.), there is an option to use multiple physical BeSim systems. Follow the same steps above to install BeSim on another system, and specify multiple BESIM_SERVERs (delimited by spaces) in Test.config. See the example Test.config files for more information.

2.4 Client Setup

Install a 1.6 JVM (or later) on all client systems.
Run the SPECweb2009 installer per Section 2.1 on each client and set your current directory to the SPECweb2009 directory created by the installation.
Select one of the client systems to serve as the "Prime Client" (can also be a load generating client).
On the Prime Client, decide on an appropriate set of configuration files for your particular system. For a Red Hat Linux server running Apache and PHP, you should use the *.Unix-PHP.config files. Copy or rename these files to SPECweb_Banking.config, SPECweb_Ecommerce.config, SPECweb_Support.config, SPECweb_Power.config and Test.config, i.e.

# cp SPECweb_Banking.Unix-PHP.config SPECweb_Banking.config
# cp SPECweb_Ecommerce.Unix-PHP.config SPECweb_Ecommerce.config
# cp SPECweb_Support.Unix-PHP.config SPECweb_Support.config
# cp SPECweb_Power.Unix-PHP.config SPECweb_Power.config
# cp Test.Unix-PHP.config Test.config

For JSP, follow the same steps but use *Unix- JSP.config files instead . Similarly, for ASP.NET, follow the same steps using * ASPX.config files instead .

Review the SPECweb_<workload>.<OS/Web server>-<Script Engine>.config, Test.<OS/Web server>-<Script Engine>.config, and Testbed.config to understand the parameters in these files, with attention to those in the beginning (configurable) section of the workload and Test configuration files which may need to be updated to match your test bed.
Edit configuration files to match your Web server, BeSim configuration, Client, and network configuration. Change only those parameters listed above the section marked "# Fixed workload properties" or in Test.config above the line marked " # SPECweb2009 fixed test properties":

Test.config- general properties that apply to all workloads, such as web server name/port, BeSim name/port/API name, run times, name of client(s), etc.
SPECweb_Banking.config- parameters specific to Banking workload.
SPECweb_Ecommerce.config- parameters specific to Ecommerce workload.
SPECweb_Support.config- parameters specific to Support workload.
SPECweb_Power.config -parameters specific to Power workload.
Testbed.config- contains your test bed description, tuning disclosures, and notes.

Some tips:

Test.config: If using the PHP scripts, set the SMARTY*_DIR parameters to the absolute paths you copied the scripts to in Section 2.2.
Test.config: Change the BESIM_SERVER, BESIM_PORT, and BESIM_INIT_SCRIPT to match the setup you configured in Section 2.3.
Test.config: If you encounter errors on the prime client, try temporarily setting DEBUG_LEVEL to a high value (10 should be sufficient). This will print out more diagnostic messages that should assist in troubleshooting your issue.
SPECweb_Banking.config, SPECweb_Ecommerce.config, SPECweb_Power.config: If you have issues running any of these three workloads, try temporarily setting USE_SSL to 0. This will result in a non-compliant run, but would rule out SSL as a potential cause.
SPECweb_<workload>.config: *_DIR must be changed to match the paths that you specified in your Wafgen .rc files.

2.5 Network Setup

Configure all the test bed systems so that they can communicate with one another; make sure to update your /etc/hosts file (or equivalent). The Prime Client must be able to communicate with each Client, Besim and each Controller in the test environment. The Web server and the BeSim system must be able to communicate with each other.

2.6 Measurement Device (Power and Temperature) Setup

Set up the analyzer/analyzers according to the instructions in the Power and Temperature Hardware Setup Guide. Note that these measurement devices need to be set up for both the SUT System as well as the Storage as per the requirements outlined in the SPECweb2009 Run and Reporting Rules.

2.7 Controller Setup

PTDaemon version 1.4.0 is distributed with the kit. The latest version of PTDaemon can be downloaded from the Accepted Measurement Devices page.
PTDaemon should install into <PTD_Home> when installing from the kit. If installing a newer version, copy PTDaemon into <PTD_Home>.
Install an analyzer or a temperature sensor to the controller's interface (such as USB, Serial port etc).
Start the PTDaemon using the PTDaemon script. Use ptd –h to get a brief help. An example of the command is

<PTD_home>/ptd –p 1920 -l mylogfile 8 /dev/term/a

In the example above, the power daemon is communicating with the Yokogawa WT210, a type 8 analyzer, via /dev/term/a interface and communicates with the primary client via port 1920. All messages of the power analyzer are logged in mylogfile (logging power analyzer messages is not a requirement, so the –l option may be omitted).

For further details on hardware installation, please see the Power and Temperature Hardware Setup Guide.

2.8 Power Analyzer Setup

Please refer to Section 2 of the SPECweb2009 Run and Reporting Rules for the requirement of power measurement and consult the manufacturer's documentation for instructions that are specific to the exact analyzer used. Once you have correctly associated the device number with the corresponding integer values that apply to those devices, follow the steps listed below:

Connect the communication interface of the power analyzer to the appropriate port of the controller server with a manufacturer-approved cable.
Open runpower.bat for Windows or runpower.sh for Unix with a text editor, and locate the set DEVICE= for the runpower script respectively. Enter the correct device number for the power analyzer being used.

set DEVICE=8 (for Windows & Unix)

Now enter the DEVICE_PORT that the PTD uses to communicate with Controller system.

set DEVICE_PORT=COM1 (for Windows)

set DEVICE_PORT=/dev/tty2 (for Unix)

To verify what port the power meter is using (for Windows) you may left click my computer then click ‘Manage’. Click ‘Device Manager’ and then click ‘Ports’ this should give you a list of all the ports attached to your computer at that time. Verify the port the power analyzer is connected to.

Set power analyzer communication settings as required by the Power and Temperature Hardware Setup Guide.
Connect the SUT's AC power cord to the power receptacle provided by the power analyzer.

2.9 Temperature Sensor Setup

Please refer to Section 2 of the SPECweb2009 Run and Reporting Rules for the requirement of temperature measurement and consult the manufacturer's documentation for instructions that are specific to the exact sensor used. Then follow the steps listed below:

Place the temperature sensor in accordance with the rules outlined in this section.
Connect the temperature sensor's communication interface to the appropriate port on the controller server with a manufacturer-approved cable.
Open runtemp.bat for Windows or runtemp.sh for Unix with a text editor, and locate the set DEVICE= for the runtemp script respectively. Enter the correct device number for the temperature sensor being used.

set DEVICE=1001 (for Windows & Unix)

Also locate the set DEVICE_PORT= for the runtemp script respectively. Enter the correct port number used to connect the temperature sensor.

set DEVICE_PORT=USB (for Windows)

set DEVICE_PORT=/dev/tty1 (for Unix)

To verify what port the temperature sensor is using (for Windows) you may left-click my computer, click Manage. Next click Device Manager and then click Ports this should give you a list of all the ports attached to your computer at that time. Find the port that the temperature sensor is connected to.

3 Running SPECweb2009

3.1 Test.config Parameters

Once the Web server, clients, controllers, and BeSim system have been configured and networked, the next step is to try running a test strobe for one of the workloads.

1. To start with a short test, edit the Test.config and the workload specific config files.

In the Test.config, verify that the following parameters are appropriately defined:

TEST_TYPE=SPECweb_<test type> -
It is necessary to set TEST_TYPE to the desired workload (SPECweb_Banking, SPECweb_Ecommerce, SPECweb_Support, and SPECweb_Power).
WEB_SERVER = <SUT Name or IP>
BESIM_SERVER = <BeSim Name or IP>
BESIM_PORT = 81 [or the non-80 value set up on the BeSim Web server]
PTD_IP_NAME[i] =<IP_ADDRESS> of a controller with an index of i, starting at 0.
PTD_IP_PORT[i]=<PORT_NUM> - the same port number given in the PTD command, with an index of i, starting at 0.
PTD_TYPE[i] = "SUT" either "SUT" or "DISK" with an index of i, starting at 0.
PTD_SAMPLE_RATE[i] = 1000 – sampling rate in milliseconds of power or temperature daemon, with an index of i, starting at 0.
PTD_TIMEOUT[i] = 30 – Timeout in seconds of power or temperature daemon, with an index of i, starting at 0.
PTD_VOLT_RANGE[i]=120 – Voltage range of power analyzer with an index of i, starting at 0.
KILL_CLIENT = 1 – Kills client at the end of a test. Valid 0, 1. 0 keeps the client running after the test is complete.
DEBUG_LEVEL = <i> - Debug level output. Valid 1-10.
If in total N power analyzers and M temperature sensors are used, PTD_IP_NAME[N+M+1]=X
"EndOfPtdDefinition" must be set in Test.config file.
SIMULTANEOUS_SESSIONS = <sus> the number of user sessions to be simulated by the clients (e.g. SIMULTANEOUS_SESSIONS=100)
For running the Power workload, put the same lines at the end of the file except that SIMULTANEOUS_SESSIONS = <sus>% .
For a valid submission, <sus> for SPECweb_Power must be the same as the number used for SPECweb_Ecommerce.
When using SIMULTANEOUS_SESSIONS =100% for power metric run, the harness will first warm up the workload to 100, then generate 100,80,60,40,20, idle load levels for step runs.

There are another three parameters that must be set. Please refer to SPECweb2009 run rule document for detailed description. The following table summarizes the minimum values required for a valid run.

Name	Value (regular)	Value (power)	Comment
WARMUP_SECONDS	300	300	minimum value
THREAD_RAMPUP_SECONDS	180	180	minimum value
THREAD_RAMPDOWN_SECONDS	180	180	minimum value

3.2 Workload Specific Config Parameters

Verify the workload specific config file (SPECweb_Banking.config or SPECweb_Ecommerce.config or SPECweb_Support.config or SPECweb_Power.config) to make sure that the following parameters are defined appropriately. The list below shows the required values for a conforming run of the benchmark. In order to get a conforming run, parameters that are in the "Do not modify" section of the config file should not be modified. However, the user may want to use shorter intervals in order to test the working of the harness and get some initial test runs done before embarking on a mission to get a conforming run.

For each power analyzer device i, the parameter PTD_AMP_RANGE[i] specifies the ampere range. Appropriate ranges ensure that SPECweb2009 requirements concerning power measurement uncertainty are met. Ranges can be specified with numerical ampere values as in the following examples. These values represent upper bounds of the ranges. Alternatively, the string auto can be specified, which sets the device to automatic range detection mode. This mode is discouraged, however, as the automatic adjustments may occur unsynchronized with the measurement intervals. As shown in the following examples, workloads Banking, Ecommerce and Support require a single range value, while Power requires a concatenation of six values. If the parameter is not specified, the benchmark will run without changing the range status of the device.

PTD_AMP_RANGE[i]=2 - Amp range setting of power analyzer.
PTD_AMP_RANGE[i]=6^5^4^3^2^1 (where each number separated by a "^" represents the range to use for the load level. Since the Power workload uses 6 different levels, a compliant run will use six different values separated by "^". For all the other workloads PTD_AMP_RANGE only requires a single number as shown.

A few additional parameters have to be set as well. The following table summarizes how to set their values for a valid run. Changing the values of parameters with fixed values will not only invalidate your run, but may also cause errors during the run.

Name	Value (regular)	Value (power)	Comment
ITERATIONS	3	1	fixed value
NUMBER_OF_DISCRETE_WORKLOAD_POINTS	NA	6	fixed value (not defined for reg. runs.)
RAMPUP_SECONDS	300	-	fixed value (not used for power)
RAMPDOWN_SECONDS	300	-	fixed value (not used for power)
RUN_SECONDS	1800	600	fixed value
INTRA_INTERVAL_RAMP_SEC	-	120	min. value (not used for regular runs)
INTRA_INTERVAL_WARMUP_SEC	NA	60	fixed value (not defined for reg. runs.)

3.3 Starting the benchmark

Next, test the setup using the commands below.

You may want to use 2 windows or redirect the output of the two Java commands to files or create a script to run the harness and to save the output:

Start the specwebclient process on each client machine. Execute the following command in the directory where the client was installed, or run the start_client script provided in the client directory for more details.
· java-Xms512m -Xmx512m -jar specwebclient.jar
Start the Power and temperature daemons. Please refer to the runpower and runtemp script files in the PTDaemon directory for more details.
To start the test harness manager process on the prime client begin the test, execute the following command in the prime client directory, or modify the provided start_prime_client scripts.

Windows:
· java-Xms512m -Xmx512m -cp .;lib\jcommon-1.0.15.jar;lib\jfreechart-1.0.12.jar;bin\specweb.jar specweb

Linux:
· java-Xms512m -Xmx512m -cp .:lib/jcommon-1.0.15.jar:lib/jfreechart-1.0.12.jar:bin/specweb.jar specweb

After test has completed check the redirected output and the contents of the results. Please check whether there are power measurements in the output of the primary client as well.

Note: you may wish to set other Java parameters besides the heap parameters to optimize the specwebclient processes as you increase the test load. Check the documentation for your JVM.

Did that test work?

Yes? Great, so did these instructions "work" or were there missing details or do you have additional questions that you are now writing up to send to mailto:web2009support@spec.org?

No? Well, there should be enough information in the debug output to start debugging. Please send any corrections or requests for clarification to these instructions to the email address above. Continue reading and reviewing the Web server, BeSim, and Client Checks below, along with the related README files for the various components in the attached appendixes, particularly Appendix D for common errors and Appendix E for the Support FAQ.

Now, that the basic settings are functional, you can try tests that use more Sessions and adding more clients.

3.4 Trouble Shooting your first run

3.4.1 Web Server Checks

Have you configured your Web server to support the HTTP or HTTPS based on the workload you are running? Banking is all HTTPS (SSL), Ecommerce requires both HTTP and HTTPS, and Support uses only HTTP. Note: that you can temporarily override the use of HTTPS in Banking and Ecommerce by setting USE_SSL = 0 in the .config file. Results will be noncompliant but this may be useful when debugging.
Have you configured you Web server to support PHP, JSP or ASP.NET according to the directions provided with your Web server? If using JSP and a separate Java application server such as Tomcat also review its documentation and configure appropriately. Try running the PHP, JSP or ASP.NET "Hello World" equivalent to make sure your Web server's script engine is working.
Have you configured the wafgen.rc files for the workload and run Wafgen to create the fixed and scaling file sets? Did you set the number of SIMULTANEOUS_SESSIONS to a value equal or greater than the value you plan to use in your testing?
Can the Web server system ping the BeSim system? Connect to the BeSim Web server (telnet [BESIM_IP] [BESIM_port])?

3.4.2 BeSim Checks

Have you installed and configured a Web server on you BeSim system to run on port 81 (or other non-standard port number)?
Have you built and installed the BeSim application (FastCGI, NSAPI, ISAPI) in the appropriate directory under the Web server's document root?
Have you run the test_besim_* scripts provided with BeSim to check that the BeSim setup is working? If not, see Section 2.3.

3.4.3 Prime Client Checks

Have you checked the SPECweb_<test type>.config?
Have you checked Test.config? Verify the following parameters in section 3.1 match your test environment.

TEST_TYPE=SPECweb_<test type>
WEB_SERVER=<SUT>
BESIM_SERVER=<BeSimsystem>
BESIM_PORT=81 [or the non-80 value set up on the BeSim web server]
Verify that BESIM_INIT_SCRIPT matches the relative path of the BeSim API installed on the BeSim Web server.
Verify that the Prime Client can "ping" the other clients, controllers, the SUT and BeSim system.
Made a quick check that the prime and other clients can successfully do a "telnet [WEB_SERVER_IP] [80 and/or 443]" and that you can successfully perform a "telnet [BESIM_IP] [BESIM_port]" from the prime?
Checked that the prime (and other clients) have set per process limits for threads (i.e. max_thread_proc) and file descriptors (i.e. maxfiles) adequately for the number of simultaneous sessions you plan to test? For example, if you are running a test for 1500 simultaneous sessions and using 2 clients, each client is responsible for 750 sessions. Each session uses ~3 connections to the server, so each client process (i.e. java -jar specwebclient) will need just over 750 * 3 threads or 2100+ threads and 750 * 3 file descriptors (socket = FD) or 2100 file descriptors. If your clients' settings aren't adequate, then your test will fail and you'll get a Java error telling you it couldn't start a thread or establish a connection.

3.5 Collecting Statistics (optional)

The prime client has the ability to poll clients for data at specified intervals during a run and display it as CSV or in a GUI. Data is collected and displayed only during the warm-up and benchmark run phases. The following configuration parameters in Test.config control the collection and display of this information:

POLL_CLIENTS: Set this parameter to 1 to poll clients for data (default = 0). The below parameters have no effect if this is not set to 1.
BEAT_INTERVAL: The frequency that client data is returned to the prime client (default = 10). If USE_GUI=1 (below), the minimum value is 2. (Reminder: for SPEC compliant benchmark runs, BEAT_INTERVAL must be set to 10.)
INTERVAL_POLL_VALUES: Set to 0 to return cumulative values; set to 1 to return values between measurement intervals.
USE_GUI: Set to 1 to display a visual graph of the results. If set to 0, the data will display in comma-separated values (CSV) format on the prime client console, which could be imported into a spreadsheet.

3.6 Optimizing Performance

Once the SPECweb2009 benchmark is running the workload(s) without errors, you will want to optimize your testbed for optimal performance. Here are some tips:

Test.config: If you increased the value of DEBUG_LEVEL during troubleshooting, be sure to reduce it back to 1 or 0 to avoid the clients printing too much information during benchmark runs.
Test.config: For SPECweb_Banking, SPECweb_Support and SPECweb_Ecommerce, to reduce the traffic between the prime client and clients, turn off statistics collection by setting POLL_CLIENTS back to 0 if you set it to 1.
Test.config (PHP-specific): Try setting BESIM_PERSISTENT to 1 and see if it works with your web server configuration. This will reduce the need for the server to close and open multiple HTTP sockets to BeSim.
Test.config (PHP-specific): Try setting SEND_CONTENT_LENGTH to 0 and see if it works with your particular web server configuration. This will eliminate the need for the PHP scripts to calculate the size of the response body for the "Content-Length" response header.
php.ini (PHP-specific): If you turned "display_errors" and "display_startup_errors" to "On" per Section 2.2, turn these back off once everything is working correctly. You may also want to set "error_reporting = 0", which will turn off all error reporting.
bank.conf, ecom.conf, support.conf (JSP-specific): Each of these files has a BESIM_POOL_SIZE, which specify the maximum number of connections made to BeSim. Setting this value too low can hinder performance, but setting this value higher than the request arrival rate can cause exceptions to occur. Each setup is different, so tune this value to determine the optimal value for your particular configuration.
General system-level performance tuning: Consult your OS and Web server documentation to determine tunes to increase performance for your SUT, BeSim, and clients. Gather statistics from the benchmark, OS, and web server to identify potential bottlenecks.

4 Appendices

4.1 Appendix A - README file from SPECweb2009 client kit

Here are the basics for setting up SPECweb2009 Clients:

·   First, you should at minimum have version 1.6.0 of the java jdk and/or jre installed on the client machines running this code to take 
advantage of garbage collection optimizations in the more recent jvms.

·   Next, there are six different .config files used for SPECweb2009. There is a SPECweb_<workload>.config for each of the four 
workloads (Banking, Ecommerce, Power and Support) that contain both configurable and fixed parameters specific to the workload. 
Additionally, Test.config contains parameters that are common to all workloads and includes configurable and fixed parameters.

·   Lastly, Testbed.config contains the testbed hardware/software configuration properties.

Note that Test.config and the workload-specific configuration files mentioned above do not exist in a new installation. Instead, 
example workload and Test configuration files are provided, and you will need to copy the sample file most suited to your environment. 

Example:

·                             cp Test.Unix-PHP.config Test.config

·                             cp SPECweb_Banking.Unix-PHP.config SPECweb_Banking.config

·                             cp SPECweb_Ecommerce.Unix-PHP.config SPECweb_Ecommerce.config

·                             cp SPECweb_Support.Unix-PHP.config SPECweb_Support.config

·                             cp SPECweb_Power.Unix-PHP.config SPECweb_Power.config

"specwebclient" is the SPECweb2009 client load generator. On each client machine, after editing the .config files 
to match your test environment, you need to start this process on all client machines. From the directory where 
specwebclient.jar is installed, and at a command prompt type:

·                         java -jar specwebclient.jar

Starting "specwebclient" with tuned heap sizes and garbage collection parameters is also recommended. Specifically, for v1.6 
of the jdk on a multi-processor client, using parallel and CMS garbage collection is strongly recommended. The flag for 
ForceTimeHighResolution is highly recommended, particularly while running the client on multi-core platforms.

Example:

java -cp c:/program files/specweb2009 -Xms500m -Xmx500m -XX:+ForceTimeHighResolution -XX:+UseParNewGC -XX:+UseConcMarkSweepGC –jar c:/program files/specweb2009/specwebclient.jar

     The usage for specwebclient is:

java -jar specwebclient.jar -h

Usage: java -jar specwebclient.jar [-v] [-p port] [-n hostname] [-s servername] [-le errorlogfilename] [-lo stdoutlogfilename]

Command line flags::

-v Print version string and exit

-p <port> Override default port number 
       (allows multiple instances on same machine)

-n <hostname> Override hostname (use Alias)

-s <servername> Overrides WEB_SERVER

-le <errorlogfilename> redirects error messages

-lo <stdoutlogfilename> redirects other logging messages

"specweb" is the SPECweb2009 test manager and is run on the Prime Client. It is started after all the "specwebclient" processes have been started. 
Starting specweb should begin the test. The "specweb" process exits at the end of the test, and the "specwebclient" process can be configured to 
terminate as well (see Test.config). To start the "specweb" process, from the directory where specweb.jar is installed, at a command prompt type:

                    Windows:

java -Xms512m -Xmx512m -cp .;lib\jcommon-1.0.15.jar;lib\jfreechart-1.0.12.jar;bin\specweb.jar specweb

      Linux:

java -Xms512m -Xmx512m -cp .:lib/jcommon-1.0.15.jar:lib/jfreechart-1.0.12.jar:bin/specweb.jar specweb

      The usage for specweb is:

       Windows:

java -Xms512m -Xmx512m -cp .;lib\jcommon-1.0.15.jar;lib\jfreechart-1.0.12.jar;bin\specweb.jar specweb -h

      Linux:

java -Xms512m -Xmx512m -cp .:lib/jcommon-1.0.15.jar:lib/jfreechart-1.0.12.jar:bin/specweb.jar specweb –h

Usage:

 java -jar specweb.jar [-C] [-v] [-w workloadname] [-le errorlogfilename][-lo stdoutlogfilename]

Command line flags:

 -h Print this message and exit

 -v Print version string and exit

      -C Overrides any non-compliant settings in the .config files with the default compliant values
        
      -w Overides the TEST_TYPE in the config file (SPECweb_Banking, SPECweb_Ecommerce, SPECweb_Support, SPECweb_Power)

 -le <errorlogfilename> redirects error messages

 -lo <stdoutlogfilename> redirects other logging messages

"reporter" is the SPECweb2009 report generator and is invoked by "specweb" at the end of a test to produce .TXT and .HTML reports. It can also be invoked 
to create a merged .raw file from results from running the 4 workloads on a given testbed for a submission to SPEC; as well as generating combined .TXT and .HTML 
report files. 

Here is the usage for the reporter:

      java -jar bin/reporter.jar -h

SPECweb2009 Reporter 1.20

Copyright (c) 2009 Standard Performance Evaluation Corporation

Command line flags:

 -h Print this message and exit

 -v Print version string and exit

 -c Combine 4 separate workload files into one submittable .raw file

 -R Regenerate HTML and ASCII reports from combined rawfile

Example Reporter Usage:

Regenerate reports from a single raw file:

java -jar reporter.jar results\SPECweb_Support.20090323-102204.raw

Combine four compliant workload raw files into one submittable raw file:

java -Xms512m -Xmx512m -cp .;lib/jcommon-1.0.15.jar;lib/jfreechart-1.0.12.jar;bin/reporter.jar reporter -c bank.raw ecom.raw support.raw power.raw submit.raw

Note that raw file of SPECweb_Power run (like the power.raw of the example) must be the last raw input file when combining the 4 workload raw files.

Regenerate reports from a previously combined raw file:

java -jar bin/reporter.jar -R submit.raw

4.2 Appendix B - Wafgen README file

4.2.1 Invoking Wafgen:

  Unix/Linux:

·                             Wafgen.sh <workload_specific>.rc

  Windows:

·                           Wafgen.bat <workload_specific>.rc

 or alternately:

·                             java -Xms384m -Xmx384m -jar Wafgen.jar <workload>.rc

or:

·                             java -Xms384m -Xmx384m -classpath Wafgen.jar org.spec.wafgen.Wafgen <workload>.rc

4.2.2 Wafgen Overview

Wafgen is used to create the file sets used by SPECweb2009. Each workload has a "fixed" and a "scaling" file set. 
The fixed file set includes a set of static elements that are commonly referenced by the pages that make up a given workload. 
The scaling file set includes a set of directories each containing additional static elements for the workload. 
The number of directories created(or referenced by a test) is calculated by:

(SIMULTANEOUS_SESSIONS * DIRSCALING) = Total #directories

The total amount of storage required for a benchmark is also be dependent on
the total number of directories in the scaling file set and the size of the 
contents of each directory (see Storage Requirement below). Since the total
number of directories needed could exceed system limits for the number of files per directory (usually ~32,000), 
an additional level of subdirectories can be added using the SUBDIR* parameters. 
By default, banking and ecommerce use SUBDIR's. The scaling directories are round-robined across the specified number of subdirectories.

This kit contains the resource files (.rc) for each workload that are used as input to Wafgen. 
There is a set tailored for both Windows and Unix/Linux in the windows and unix subdirectories. 
To use these files copy the appropriate set into the upper directory and edit them as needed to match your test environment. 
For conforming results - ONLY parameters listed under Section A of the .rc file may be modified. 
Double check all section A parameters, especially:

·                             DOCROOT (if not set will default to current directory)

·                             BASEDIRNAME (see examples in <workload_specific>.rc files)

·                             BASESUBDIRNAME (if applicable)

·                             SIMULTANEOUS_SESSIONS (may be larger than the corresponding value in

·                             value in SPECweb_<workload>.config)

Also verify that the SPECweb_<workload>.config file parameters in the test harness on the Prime Client are in agreement.

The .rc files for fixed file sets include:

·                             bank_image_props.rc

·                             ecommerce_image_props.rc

·                             support_image_props.rc

The .rc files for scaling file sets include:

·                             bank_usercheck_props.rc

·                             ecommerce_productline_props.rc

·                             support_downloads_props.rc

The .rc files are annotated to provide additional guidance.
Reminder: Altering .rc file parameters listed in section B will result
in non-conforming tests, so only modify parameters in section A.

4.2.3 Storage Requirements:

Since the benchmark requires that the System Under Test (SUT) has the same set of hardware components 
that could contain the max file set among all four workloads, it is important to understand the storage requirement 
for each workload. Below are the data and formulas to help estimate storage requirements for SPECweb2009.

4.2.3.1 Banking:

The fixed file set created by bank_image_props.rc includes:

·                   bank/images - 44 files; Average size: 3570 bytes; Total size: 157096 bytes

·                   bank/dynamic_padding - 15 files; Average size: 21800 bytes; Total size: 327000 bytes

The scaling file set created by bank_usercheck_props.rc creates a number
of user0000nnnnnn directories based on the SIMULTANEOUS_SESSIONS value.

Each bank/images/{subdir00mm}/user0000nnnnnn contains - 20 files; Average size: 10431 bytes; Total size: 208625 bytes

To estimate the storage requirement for the user check directories use
the following formula:

·                    (SIMULTANEOUS_SESSIONS * CLIENT_SESSION_USER_IDS) * 208625

For Conforming results, CLIENT_SESSION_USER_IDS (alias for DIRSCALING) must be 50.

Example:

((1000 SIMULTANEOUS_SESSIONS * 50 CLIENT_SESSION_USER_IDS) * 208625) = 10,431,250,000 bytes

4.2.3.2 Ecommerce and Power:

The fixed file set created by ecommerce_image_props.rc includes:

·                    ecommerce/images - 172 files; Average size: 2032 bytes; Total size: 349557 bytes

·                    ecommerce/dynamic_padding - 11 files; Average size: 73000 bytes; Total size: 803000 bytes

The scaling file set created by ecommerce_productline_props.rc creates a number of 
productline0000nnnnnn directories based on the SIMULTANEOUS_SESSIONS value. 
Each ecommerce/images/{subdir00mm}/productline0000nnnnnn contains - 30 files; Average size: 18816 bytes; Total size: 564495 bytes

To estimate the storage requirement for the product line directories use the following formula:

(SIMULTANEOUS_SESSIONS * DIRSCALING) * 564495

For Conforming results, DIRSCALING must be 5.

Example:

((1000 SIMULTANEOUS_SESSIONS * 5 DIRSCALING) * 564495) = 2,822,475,000 bytes

4.2.3.3 Support:

The fixed file set created by support_image_props.rc includes:

·                           support/images - 31 files; Average size: 728 bytes; Total size: 22580 bytes

o   Warning: this file set contains several very small files (30 bytes) so the
path name must not be longer than 14 characters. 
So, "support/images" is OK but "supporting/image" is NOT.

·                             support/dynamic_padding - 6 files; Average size: 59616 bytes; Total size: 357700 bytes

The scaling file set created by support_downloads_props.rc creates a number of 
user0000nnnnnn directories based on the SIMULTANEOUS_SESSIONS value.
Each support/downloads/{subdir00mm}/dir0000nnnnnn contains - 16 files; Average size: 4279501 bytes; Total size: 68472021 bytes
To estimate the storage requirement for the download directories use the following formula:

(SIMULTANEOUS_SESSIONS * DIRSCALING) * 68472021

For Conforming results, DIRSCALING must be 0.25.

Example:

((1000 SIMULTANEOUS_SESSIONS * .25 DIRSCALING) * 68472021) = 17,118,005,250 bytes

Note: The heap parameters used for Wafgen are required to support the large files in the scaling file set for Support workload.

4.2.4 Tips for Building Large Filesets:

To increase the size of an existing file set to support a larger number of Simultaneous Sessions than used initially, the steps are:

·        Edit the <workload>.rc file for the scaling fileset create previously 
and change these parameters:

o   SIMULTANEOUS_SESSIONS=<new higher value>

o   FIRSTDIRECTORY=<1+highest directory number previously create>

Example:

Originally a support_downloads_props.rc file used:

SIMULTANEOUS_SESSIONS=1000

FIRSTDIRECTORY=1

When a file set to support 2000 Simultaneous Sessions is needed, change:

SIMULTANEOUS_SESSIONS=2000

FIRSTDIRECTORY=251

Then run:

·                             Wafgen <updated_workload>.rc

To run several Wafgen's in parallel to create a large file_set, copies of I the <workload>.rc file can be for each parallel 
wafgen to be run and make the changes to the SIMULTANEOUS_SESSIONS and FIRSTDIRECTORY directories. 
To calculate the correct values for FIRSTDIRECTORY in each copy use the formula:

·         Number of Directories = SIMULTANEOUS_SESSIONS * DIRSCALING 
(or for banking the DIRSCALING alias CLIENT_SESSION_USER_IDS is used).

Use the directory scaling formula for the workload (shown above) to calculate the correct values for FIRSTDIRECTORY in each copy.

·             For example, to create a support downloads file set for a total of 
   4000 Simultaneous Sessions:

 support_downloads_props.rc_1:

 	SIMULTANEOUS_SESSIONS=1000

 	FIRSTDIRECTORY=1

 support_downloads_props.rc_2:

 	SIMULTANEOUS_SESSIONS=2000

 	FIRSTDIRECTORY=251

 support_downloads_props.rc_3:

 	SIMULTANEOUS_SESSIONS=3000

 	FIRSTDIRECTORY=501

 support_downloads_props.rc_4:

 	SIMULTANEOUS_SESSIONS=4000

 	FIRSTDIRECTORY=751

 Then run 4 parallel wafgens:

 	Wafgen support_downloads_props.rc_1 &

 	Wafgen support_downloads_props.rc_2 &

 	Wafgen support_downloads_props.rc_3 &

 	Wafgen support_downloads_props.rc_4 &

(Note: depending on the number of CPUs and the I/O subsystem being used 
this sequence may not take significantly less time than just running a 
single Wafgen for the whole file set.)

To convert from a file set that was recreated without using *SUBDIR* parameters or to add additional subdirectories, the original 
file set will need to be re-created from scratch. The <workload>.rc file will need to be edited to configure these parameters. 
The parameters are preceded by the comment string "#subdirs#" if use of SUBDIRS is not the default (i.e Support).

Multiple disks or mount points may be used to hold subsets of a workload's file sets, for example: /<docroot>/bank may 
have the images directory as as separate mount point or /<docroot>/bank/images may have separate mount points for 
each of 10 subdirectories (subdir0001 thru subdir0010).

Warning:

The path names have been standardized in the wafgen <workload>.rc files and the SPECweb_<workload>.config 
files in the harness, so that in the DOCROOT there would be these directories:

·                            bank/images

·                            bank/dynamic_padding

·                            ecommerce/images

·                            ecommerce/dynamic_padding

·                            support/images

·                            support/dynamic_padding

·                            support/downloads

While it is possible to change these names in the .rc and .config files with due care (and making sure not to exceed 
path length limits for some very small files); it is recommended that users retain this naming to avoid errors due to 
typos and to simplify debugging.

Note: Using parallel wafgens to create the datasets may impact the performance.

4.3 Appendix C - BeSim Make_Readme file

How to Build BESIM:

UNIX/LINUX:

o   Type: uname -s to get name of the operating system (i.e. HP-UX)

o   In the Makefiles directory use one of the MakeIncl.<type>.<OS> files as 
a template for your MakeIncl.<type>.<your_OS> and update to match you environment.

o   To Build Fast-cgi version of BESIM: make fcgi

o   To Build NSAPI version of BESIM:  make nsapi

o   To Build Zeus ISAPI version of BESIM: make zisapi

o   To Build ISAPI version of BESIM: make isapi

Optionally you can specify other make targets by adding TARGET='list of targets'

For example:

·     make zisapi TARGET='clean'

·     make nsapi TARGET='clean all install'

The install target will use the path specified by DEST to install the besim executable or library, so it should be set to the appropriate place on your 
Besim webserver directory tree (i.e. /www/fast-cgi). If you plan to use Fast-cgi you must build and install the Fast-CGI development kit
which provides the fast-cgi library and include files before build BESIM. 
If using Apache for the BESIM webserver, that will need to include mod_fastcgi.

See: http://www.fastcgi.com/

Note: Please send any new or updated MakeIncl.* files you want included in a 
future release to web2009support(at)spec.org.

Windows:

Project files are included as an example: BeSim.def, BeSim.dsp, BeSim.dsw

4.4 Appendix D - Common Errors and Debugging Tips

· By default the DEBUG parameter is set to 1 in Test.config, if a validation error occurs this value is sufficient to print out the
ERROR message related to the invalid response. The invalid response is also printed out and that is followed by a printout
of the Request that produced the invalid response. Reading the returned HTML response and the associated HTTP Request
is key to debugging problems.

· Status 404 (file not found) errors usually mean that there is a path problem either in the .config files on on the associated
Web server. In some cases it may indicate that not a large enough file set was created by Wafgen.

· Errors Returned by BeSim are found in the body of the returned HTML

4.5 Appendix E - Support FAQ

NOTE: For the latest version of this document, please visit http://www.spec.org/web2009/docs/supportfaq.html.

4.6 Appendix F - Installing BeSim as a FastCGI for the Apache HTTP Server

There are two pieces necessary to get BeSim for FastCGI working properly under Apache:

1. Compiling the FastCGI source code and BeSim

2. Compiling/configuring mod_fastcgi, an Apache module to invoke FastCGI for specific directories

4.6.1 FastCGI source compilation

1. Compile the FastCGI source code. The source tree is uded on a BeSim (or full) installation of SPECweb2009. Here are steps to compile it:

cd <path_to_SPECweb2009>/besim/fcgi-2.4.0/

./configure --libdir=/lib

make

make install

NOTE: --libdir=/libwas added above due to the default FastCGI Makefile installing libraries to /usr/local/lib, which is not a default library path on Linux and could cause this error upon execution of the FastCGI:

besim_fcgi.fcgi: error while loading shared libraries: libfcgi.so.0: cannot open shared object file: No such file or directory

NOTE #2: For x86_64 versions of Fedora Core 3 and 4, it was observed that --libdir=/lib64 should be specified instead of /lib.

2. Create a directory for the BeSim FastCGI within your web server directory hierarchy (but preferably outside the document root). Here are some recommended locations for different setups:

SuSE: /srv/www/fcgi-bin
Red Hat: /var/www/fcgi-bin
Apache 2.x compiled from source: /usr/local/apache2/fcgi-bin

3. Compile BeSim as a FastCGI (replace /var/www/fcgi-bin/ with the directory created in the previous step):

make fcgi TARGET='clean all install' DEST=/var/www/fcgi-bin/

4. This should create besim_fcgi.fcgi and install it into DEST. If not, check the make output for errors.

4.6.2 Apache FastCGI Web server module: mod_fastcgi

These instructions show how to compile the FastCGI web server module for Apache (mod_fastcgi). For other web servers, please refer to their documentation to see whether FastCGI support is available.

NOTE for SuSE: SUSE Linux (8.1 and above) ships with a precompiled mod_fastcgi for Apache 2.x. To check if it is installed, do a`rpm -q apache2-mod_fastcgi`.If it's not, install this package from the CDs (see your OS documentation for more instructions). If this package is installed, you can skip the rest of this section.

NOTE for Red Hat: If using the httpd package that ships with the distribution, the httpd-devel RPM must also be installed, as it contains the necessary utilities and headers to compile Apache modules. To check, do a rpm -q httpd-devel.

1. Download the latest mod_fastcgi tarball from http://www.fastcgi.com/dist/ and follow the directions in the INSTALL document (INSTALL.AP2 for Apache 2.x).

NOTE: As INSTALL.AP2 states, unless you are compiling from source, you will need to modify top_dir in your make command. For example, Red Hat users should use the following 'make install' command:

make top_dir=/usr/lib/httpd install

You should now have a mod_fastcgi.so in your Apache modules directory:

      Red Hat:

-rwxr-xr-x 1 root root 161408 Sep 16 13:47 /usr/lib/httpd/modules/mod_fastcgi.so

Compiled from source:

# ls -l /usr/local/apache2/modules/mod_fastcgi.so

-rwxr-xr-x 1 root root 156434 Sep 21 10:03 /usr/local/apache2/modules/mod_fastcgi.so

Create a directory for FastCGI to store Unix socket files, and change the permissions so the Web server can have full access to it:

Red Hat:

# mkdir -p /etc/httpd/fastcgi
# chmod 777 /etc/httpd/fastcgi

Apache 2.x from source:

# mkdir -p /usr/local/apache2/fastcgi
# chmod 777 /usr/local/apache2/fastcgi

Add support for handling FastCGIs by making the following changes to httpd.conf (for Red Hat, this is in /etc/httpd/conf/):

1. Add this line in the "Dynamic Shared Object (DSO) Support" section:

· LoadModule fastcgi_module modules/mod_fastcgi.so

2. Add the following lines just after theScriptAlias /cgi-bin/ line:

ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/" #<fcgi-bin directory from Section 1, Step 2>
FastCgiIpcDir /etc/httpd/fastcgi # <FastCGI directory from previous step>

NOTE: Replace /var/www and /etc/httpd with /usr/local/apache2 if compiling from source.

3. Add the following section, preferably after the<Directory "<CGIDIR>/cgi-bin"> section:

<Directory "<CGIDIR>/fcgi-bin"> # <fcgi-bin directory from Section 1, Step 2>
    AllowOverride None
    Options +ExecCGI -Includes
    SetHandler fastcgi-script
    Order allow,deny
    Allow from all
</Directory>
AddHandler fastcgi-script fcgi

4.6.3 Testing the BeSim FastCGI

1. Start (or restart) your web server.

2. At the console, monitor your Apache error log to view any errors or warnings during FCGI execution, i.e.

# tail -f <APACHE_ROOT>/logs/error_log

3. You should initially see some FastCGI initialization messages such as:

[Wed Sep 21 10:36:39 2009] [notice] FastCGI: process manager initialized (pid 25700)

4. Test the BeSim FastCGI, using the scripts in the subdirectory of the SPECweb2009 installation. See BeSim Setup for examples.

See BeSimsections above for additional information.

The SPECweb2009 benchmark utilizes the JFreeChart library, which is licensed under the Lesser General Public License (LGPL). Source code and a copy of the license are included with the benchmark kit.

Java® is a registered trademark of Sun Microsystems.