Version 1.05
Last modified: Tuesday, October 5, 2004
Section 2 - Installing SPECjAppServer2001
Section 3 - Building and Deploying SPECjAppServer2001
Section 4 - Running SPECjAppServer2001
SPECjAppServer2001 is a J2EE™ benchmark to measure the scalability and performance of J2EE servers and containers. SPECjAppServer2001 strives to stress the middle-tier rather than the client tier or the database server tier.
This document is a practical guide for setting up and running a SPECjAppServer2001 benchmark. This user guide discusses some, but not all, of the rules and restrictions pertaining to SPECjAppServer2001. Before continuing with the benchmark, we strongly recommend you read the complete SPECjAppServer2001 Run and Reporting Rules contained in the kit. For an overview of the benchmark architecture, see the SPECjAppServer2001 Design Document also contained in the kit.
BOPS is the SPECjAppServer2001 performance metric and denotes the average number of successful Business Operations Per Second completed during the Measurement Interval. BOPS is composed of the total number of business transactions completed in the Customer Domain, added to the total number of workorders completed in the Manufacturing Domain, normalized per second.
price/BOPS is the SPECjAppServer2001 price/performance metric and is the total cost of the SUT divided by the number of BOPS.
The Domain is a logical entity that describes a distinct business sphere of operations. The four SPECjAppServer2001 domains are: Customer, Manufacturing, Supplier and Corporate.
The Driver refers to the client code that drives the benchmark, monitors requirements, and reports results.
SPECjAppServer2001 Kit refers to the complete kit provided for SPECjAppServer2001. This includes the SPECjAppServer2001 Reference Beans, the Driver, load programs, and documentation.
SPECjAppServer2001 Reference Beans refers to the implementation of the Enterprise Beans provided for the SPECjAppServer2001 workload.
The Supplier Emulator is a Java Servlet that can run inside any Java enabled web server, and emulates the process of sending and receiving orders to/from suppliers.
The SUT is the System Under Test and is comprised of all components that are being tested. This includes Application Servers/Containers, Database Servers, network connections, etc. The Driver and Supplier Emulator are not part of the SUT.
The SPECjAppServer2001 workload emulates a heavyweight manufacturing, supply chain management, and order/inventory system representative of one in use at a Fortune 500 company. This workload stresses the ability of EJB containers to handle the complexities of memory management, connection pooling, passivation/ activation, caching, etc. The SPECjAppServer2001 Design Document includes a complete description of the workload and the application environment in which it is run. This section of the user's guide describes software and hardware environment required to run the workload.
Although SPECjAppserver2001 can be run on a single machine for testing purposes, compliance with the SPECjAppServer2001 Run and Reporting Rules requires that the driver and supplier emulator be run on a machine outside the SUT. Therefore, a compliant hardware configuration must include a network and a minimum of two systems – one or more systems to run the components within the SUT and at least one system to run the driver and supplier emulator outside the SUT. A typical configuration is illustrated below.
SPECjAppServer2001 is a J2EE application that requires a J2EE 1.2 compatible application server and database server to run as part of the SUT. Outside the SUT, a Java Runtime Environment (JRE) version 1.3 or later is required for the driver and supplier emulator. A web server supporting HTTP 1.1 is also required for the supplier emulator
The SPECjAppServer2001 Kit is supplied as a tar file that should be extracted into a directory. We will call this top level directory $JAS_HOME.
SPECjAppServer2001 can be run on any Java platform. Most of the code is implemented in Java, however there are a few scripts used to start certain programs. These scripts are provided in .sh format for Unix platforms and in .bat format for Windows.
ant : The ant installation directory.
bin : Driver and database load scripts. All additional scripts except specific database related scripts will be in this directory.
classes : This is a directory for compiling Java classes and packaging them. It comes with all the necessary class files necessary to run SPECjAppServer2001.
classes/META-INF : Temporary place for holding deployment descriptors just before packaging into JAR format files.
classes/WEB-INF : Temporary place for holding web deployment descriptors just before packaging the WAR files.
config : Environment configuration and properties files used by the Driver, make and load programs. Once SPECjAppServer2001 has been successfully deployed, this is the only place where the configuration should be modified.
config/security : Security policy files used for the benchmark programs. These should not be modified.
docs : The documentation for all SPECjAppServer2001 beans generated by running 'make doc'.
jars : The JAR files created by the make files. The files distributed in this directory are driver.jar, launcher.jar, charts.jar , xerces.jar. The application server specific jars for deployment need to be generated by the user.
schema : Schema information and static data to be loaded into the database.
schema/sql : Generic SQL scripts for creating the tables, etc. in the database. These should be copied and modified as necessary for a particular database product.
src : Root directory of the source files. This contains generic and application server specific make files.
src/com : Entry into the com.sun.jini.debug and com.sun.jini.thread package source.
src/deploy : The deployment descriptors.
src/docroot : HTML and JSP pages used for the web interface.
src/org : Entry into the org.spec.jappserver package source.
src/output : Default directory in which output of runs go.
Note: For some of the popular DBMS products, DBMS specific files are in schema/<DBMS vendor> and DBMS specific SQL scripts are in schema/<DBMS vendor>/sql.
All SPECjAppServer2001 Java classes are located in the org.spec.jappserver package. The following lists the sub-packages of org.spec.jappserver:
The beans adhere to the EJB 1.1 specification. There are currently two versions for some of the beans - CMP and BMP. The CMP version is to be used with Container Managed Persistence and the BMP version must be used with Bean Managed Persistence. The BMP beans are sub classed from CMP.
The source files for the CMP and BMP are respectively named <bean>CmpEJB.java and <bean>BmpEJB.java. The kit includes pre-compiled class files for all the beans (both CMP and BMP versions) and you can package them into appropriate ejb-jars for your environment.
Of the packages listed above, corp, orders, mfg, and supplier contain the beans used for the four SPECjAppServer2001 domains respectively. Each domain package contains a distinct sub-packages for each EJB and its helper classes. The EJB package contains a sub-package named "ejb" which holds the bean implementation including the home and remote interfaces. For instance, the following shows the fully qualified class names for the OrderLineEnt entity bean:
org.spec.jappserver.orders.orderlineent.ejb.OrderLineEntHome
org.spec.jappserver.orders.orderlineent.ejb.OrderLineEnt
org.spec.jappserver.orders.orderlineent.ejb.OrderLineEntPK
org.spec.jappserver.orders.orderlineent.ejb.OrderLineCmpEJB
org.spec.jappserver.orders.orderlineent.ejb.OrderLineBmpEJB
The helper classes for the orders domain are in the helper sub-package under orders as follows:
org.spec.jappserver.orders.helper.CustomerInfo
org.spec.jappserver.orders.helper.CustomerStatus
org.spec.jappserver.orders.helper.InsufficientCreditException
org.spec.jappserver.orders.helper.ItemQuantity
org.spec.jappserver.orders.helper.OrderStatus
Besides the four domains, several other packages are included:
The common package contains the source code for helper classes used by all the beans (i.e. it is common across different domains as well as beans)
The ruleengine package contains the rule parser used by the corp domain to evaluate order discounts
The util package contains the beans used to generate primary keys
The webbeans directory contains Java Beans and support classes used by the web interface
The supplier.emulator package contains the Supplier Emulator Servlet.
The charts package contains a stand-alone Java application that can be used to chart transaction throughput during a SPECjAppServer2001 benchmark run (source not included). Note: To run the charts application you need JClass Chart from Sitraka Software
The driver package contains the SPECjAppServer2001 drivers.
The load package contains the database data generation and load programs required to load the SPECjAppServer2001 database(s) according to the benchmark scaling rules.
xerces.jar is used by Delivery Servlet in Supplier domain and the Supplier Emulator Servlet. Add this jar file in the Application server classpath if XML DOM Parser packages are not part of your application server. In the case of J2EE RI server this jar should be added in the J2EE_CLASSPATH by editing J2EE_HOME/bin/userconfig.sh. To include this jar in the Tomcat server, copy xerces.jar file to $TOMCAT_HOME/lib directory.
There are several components required to build, deploy, test and run SPECjAppServer2001. These are :
Building and deploying the benchmark requires ant, which is available from http://jakarta.apache.org/ant.
Several steps must be accomplished prior to running the SPECjAppServer2001 benchmark:
For the Standard Workload, you can create a single database that houses all the 4 Domains. For the Distributed Workload, you must create 4 separate databases, one for each domain.
Standard SQL scripts for creating the database schema are provided in schema/sql. These are intended to give a starting point for creating schemas for other database products.
For convenience database creation scripts are provided for Oracle 8 & 9i, DB2 and Sybase in schema/<DBMS>. These are example scripts and can be modified. However, see section 1.4 of the SPECjAppserver2001 Run and Reporting Rules for modification requirements.
This section describes the steps required to build and deploy SPECjAppServer2001 on the J2EE Reference Implementation (RI). These directions assume that the J2EE RI version 1.3.1 is used and that it is installed in the directory $J2EE_HOME.
This example also uses an Oracle database assumed to be running on host 'orahost' with instance name 'specdb' and accessed through the Oracle Thin driver.
Set the J2EE_CLASSPATH variable in the $J2EE_HOME/bin/userconfig.sh file to include your JDBC Driver class libraries as well as the XML DOM Parser library - $JAS_HOME/jars/xerces.jar.
Example for Oracle (assuming that Oracle is installed in $ORACLE_HOME on the same machine):
J2EE_CLASSPATH=$ORACLE_HOME/jdbc/lib/classes12.zip:$JAS_HOME/jars/xerces.jar
Use the RI's j2eeadmin tool to add a new JDBC driver, URL and the Data Source jdbc/SPECDB.
Example for Oracle:
$ cd $J2EE_HOME/bin
$ j2eeadmin -addJdbcDriver oracle.jdbc.driver.OracleDriver
$ j2eeadmin -addJdbcDatasource jdbc/SPECDB jdbc:oracle:thin:@orahost:1521:specdb
Note: By default SPECjAppServer2001 assumes that the account used to access the database has username 'spec' and password 'spec'. If you are using different username and/or password you should include these in the JDBC URL above as follows:
$ j2eeadmin -addJdbcDatasource jdbc/SPECDB \ jdbc:oracle:thin:username/password@orahost:1521:specdb
Make sure that the file $JAS_HOME/config/appserver contains "ri.1.3.1" (as by default).
Edit the $JAS_HOME/config/ri.1.3.1.env properties file.
You need to set the following properties:
Ensure you can get to the ant binaries by including ant in your PATH.
Unix: $ export PATH=$JAS_HOME/ant/bin:$PATHWindows: C:\>set PATH=%JAS_HOME%\ant\bin;%PATH%
$ cd $JAS_HOME
$ ant clean // makes sure the workspace is clean
// before starting to build
$ ant // build the SPECjAppServer2001 application
components using the default build.xml
Edit the $JAS_HOME/config/*db.properties files to establish the correct JDBC connection properties for your database.
Example for Oracle:
dbURL = jdbc:oracle:thin:@orahost:1521:specdb
dbDriver = oracle.jdbc.driver.OracleDriver
jdbcVersion = 2
Note: In case you are using different database username/password than the default spec/spec you will also have to edit the dbUser and dbPassword properties in the above files.
Now load the database:
$ cd $JAS_HOME
$ ant loaddb
Note: To load the database for higher Injection Rate runs use the bin/loaddb.sh script.
$ cd $J2EE_HOME/bin
$ j2ee
To deploy the SPECjAppServer EJBs to the running J2EE RI server on $JAS_HOST do the following:
$ cd $JAS_HOME
$ ant -f ri.xml
To deploy the Emulator to the running J2EE RI server on $EMULATOR_HOST do the following (for simplicity we assume that the Emulator is deployed in the same J2EE RI Server as the EJBs):
$ cd $JAS_HOME
$ ant -f ri.xml emulator-deploy
or
To build and deploy emulator on Tomcat server, edit the $JAS_HOME/config/tomcat.env properties file.
$ cd $JAS_HOME
$ ant -f tomcat.xml
$ cp jars/Emulator.war $TOMCAT_HOME/webapps
$ rm -rf $TOMCAT_HOME/webapps/Emulator // if it is a redeployment
This section describes the general steps necessary to build and deploy SPECjAppServer2001 on an application server from scratch. Additional information on configuring SPECjAppServer2001 for use with specific application servers is available from:
http://www.spec.org/osg/jAppServer2001/vendor_setup/index.html
When these are working, you should package up the mango.xml, config/mango.env, and src/deploy/mango/* files, as well as any of the confg/*db.env files that may have native JDBC drivers referenced in them. For convenience, include instructions on how to deploy SPECjAppServer2001 for your AppServer.
Using the CMP version of SequenceEnt requires that UtilDataSource defined in util.xml have its isolation level set to SERIALIZABLE. The CMP version will only work if your Container/DBMS support the Serializable isolation level correctly. Consequently, we suggest that you use the BMP version of SequenceEnt.
The web client and emulator are deployed as .war files. If your AppServer does not support these formats, then you will have to make appropriate changes to <appserver>.xml to compile and deploy them.
Now that the database and tables are created, and the load programs are compiled, follow these steps to load the database:
Edit the $JAS_HOME/config/appsserver file and replace the current name with the name of the application server being used.
If the file $JAS_HOME/config/<appsserver>.env does not exist, copy it from $JAS_HOME/config/ri.env
Edit the $JAS_HOME/config/<appsserver>.env file and ensure the JDBC_CLASSPATH variable points to a valid JDBC driver and $JAS_HOME points to root of SPECjAppServer2001 distribution. Note that the other variables are not used during the load.
Uncomment the JDBC URL and Driver class in the $JAS_HOME/config/*db.properties files depending on what database and driver you are using to access it. In the centralized configuration, the contents of all files are the same.
Run the bin/loaddb.sh script from the $JAS_HOME directory to load all the databases. Read the script to understand the order in which loads must be done. The LoadCorp program writes out data to temporary files in /tmp. These files are then read by the other load programs to populate their respective domains. If any of the load programs fail, follow these steps:
- If LoadCorp failed, all of the domains will have to be re-loaded. Re-run bin/loaddb.sh.
- If one of the other domain loads fail, you can selectively re-load only this domain as long as the /tmp/*pipe files created by LoadCorp have not been deleted.
Note: If you reload the database you need to restart the AppServer as the SequenceSes bean caches primary key values to insert new rows in the database.
Assuming the application server is configured properly and the beans have been successfully deployed, you should be able to test the functionality of the SPECjAppServer2001 application using the web interface. If the deployment descriptor puts the web pages under the SPECjAppServer2001 context as defined by the deployment descriptors supplied for the J2EE RI, you should be able to access the web page by pointing to the URL
http://<JAS_HOST>:<JAS_PORT>/SPECjAppServer2001/
Test that all the functionality is in place by trying the following transactions through the web interface:
In Orders Domain:
In Manufacturing Domain:
To make sure Delivery and Emulator servlets are functioning you should be able to connect to the following URLs.
http://<JAS_HOST>:<JAS_PORT><JAS_PREFIX>Supplier/DeliveryServlet
Note: In most cases the JAS and EMULATOR PREFIX will be '/'
If all of the above run successfully, you are ready to run the benchmark using the Driver.
Running SPECjAppServer2001 requires that the driver be configured for your environment and particular application server. Configuring the driver properly requires understanding how the driver works, which is described below.
The SPECjAppServer2001 Driver consists of several Java programs and is designed to run on multiple client systems, using an arbitrary number of JVMs to ensure that the Driver has no inherent scalability limitations. Note that none of the client systems are part of the SUT.
The Master machine: This is the machine from which you drive the benchmark run. It is typically one of the client systems.
The Agent machines: These are the client systems on which the various workloads run. The Master machine can also be an Agent machine.
The Driver consists of the following components:
The actual applications that implement the workload are defined in the specifications. These are OrderEntry, PlannedLine and LargeOrderLine.
There is one Agent type for each of the applications. The Agents are OrdersAgent, MfgAgent and LargeOLAgent. You can configure as many OrdersAgent and MfgAgent agents as you wish on any number of client systems. There must be at least one of each Agent type on the Master machine if OrdersAgent and/or MfgAgent are to be run.
There is one Controller which runs on the Master machine and with which all the Agents register.
There is one Driver which runs on the Master machine and which is responsible for doing a benchmark run.
The Driver communicates with all the Agents using RMI. The Driver reads the run properties and configures the Agents appropriately. Each Agent will then run as many threads of their respective workload. For example, the OrdersAgent will run OrderEntry. The number of threads is determined by the scaling rules of the specification and are equally distributed amongst all Agents. Each thread runs independently, executing the workload according to the rules defined in the spec. When the run completes, the Driver co-ordinates with the Agents to retrieve all the stats and prints out the reports.
In the past, some people have had problems with the way the Driver gets its InitialContext and lookups. This section explains exactly how this is done.
The Driver obtains the JNDI InititalContext by executing the following code in each of the client threads in Agent processes :
try {
context = new InitialContext();
} catch (NamingException ne) {
throw new RemoteException("InitialContext failed. " + ne);
Each of the bean home interfaces that the applications require is looked up using the absolute home name of the bean. An example is shown below :
OrderSesHome orderSesHome =
(OrderSesHome) PortableRemoteObject.narrow
(context.lookup("OrderSesHome"),
OrderSesHome.class);
If your app server needs extra properties to be set for the InitialContext, this can be done by setting them in jndi.properties in the CLASSPATH or changing the variable JAVA in config/<appserver>.env. The example below shows a generic implementation. Specific values for several application servers are provided in the subsequent sections.
JAVA="$JAVA_HOME/bin/java
-Djava.naming.factory.initial=vendor naming factory goes here
-Djava.naming.provider.url=vendor url goes here
iPlanet
-Djava.naming.factory.initial=com.sun.jndi.cosnaming.CNCtxFactory
-Djava.naming.provider.url=iiop://${JAS_HOST}:${IIOP_PORT}
WebLogic
-Djava.naming.factory.initial=weblogic.jndi.WLInitialContextFactory
-Djava.naming.provider.url=t3://${JAS_HOST}:${JAS_PORT}
WebSphere
-Djava.naming.factory.initial=com.ibm.websphere.naming.WsnInitialContextFactory
-Djava.naming.provider.url=iiop://${JAS_HOST}:${IIOP_PORT}"
Follow these steps to configure the Driver for you environment and particular application server:
Edit $JAS_HOME/config/appsserver to contain the name of the application server you want to run.
Edit $JAS_HOME/config/<appsserver>.env to contain the correct values for the following variables :
- JAVA_HOME - Path name of your JDK installation
- <APPSERVER>_HOME - Path to your app server installation
- JAS_HOME - Path to the root directory of SPECjAppServer2001 distribution
- CLASSPATH - Path to all the library and client jar files needed to run an EJB client for your particular application server
- BINDWAIT - Time to wait for the RMI server to bind when starting the Driver; defaults to 5 seconds
- JAVA - The Java executable to run. Some application servers require the client to call a wrapper instead of the java command. If this is not set, it defaults to $JAVA_HOME/bin/java
Edit $JAS_HOME/config/run.properties and set the SPECjAppServer2001 run parameters appropriately. The parameters are described in comments in this file. Ensure that dumpStats is 0 if you are not running the charting application, as otherwise the Driver will hang waiting for the reader on the pipe.
Edit $JAS_HOME/config/agent.properties. There are currently no values you can change in this file.
Check that the method used by the Driver to get its InitialContext and do lookups will work for your application server.
Now, run the driver by running the script bin/driver.sh [<driver host>] from the $JAS_HOME directory.
You can abort a run at any time by pressing Ctrl-C. This will cause all the processes to exit gracefully.
If there were errors during pervious run, M_largeorder table may have entries which will affect the BOPS of the run. So make sure before each run the M_largeorder table is empty by reloading the database or by deleting entries from M_largeorder table using a SQL command.
If no <driver host> is specified the default local host is used.
We have sometimes experienced binding exceptions at the start of the first time run. For now, just cancel the script (by pressing Ctrl-C) and rerun it again. This problem usually does not persist to the second run.
The driver.sh script starts all the Agents on the Master machine. If you want to start the Agents on different client systems, the script needs to be edited appropriately. "bin/driver.sh <driver host>" must be started on the other client systems after starting the script on the Master machine. <driver host> is the name of the Master machine. The properties file must be configured correctly on the client systems, with the appropriate configurations in the $JAS_HOME/config/run.properties for the OrderAgent and the MfgAgent to be run on the client systems.
A driver.bat script is provided for running SPECjAppServer2001 on MS Windows systems.
Section 2.2 of the SPECjAppServer Run and Reporting Rules specifies the requirements for atomicity of the transactions. The code to run the various atomicity tests is implemented via the Debug class in the OrderEnt and LargeOrderEnt beans. Debug level 4 is used for the atomicity tests. To run Atomicity Tests 1 and 2, change the environment variable debuglevel of OrderEnt to 4 in ejb-jar.xml in the META-INF directory of the Orders EJB JAR orders.jar. Similarly, to run Atomicity Test 3, change debuglevel of LargeOrderEnt to 4 in ejb-jar.xml in the META-INF directory of the Manufacturing EJB JAR mfg.jar.
See the following URL for the latest tips and troubleshooting information on SPECjAppServer2001:
After completing a run, the SPECjAppServer2001 driver will generate a result.props file in the output directory containing the raw results from the benchmark. The contents of result.props are used in the Submission File from which the SPECjAppServer2001 report is generated.
The Submission File contains a detailed description of the SUT in text format that is used by the SPECjAppServer2001 reporter to produce a report suitable for publication.
To create a SPECjAppServer2001 Submission File:
SPECjAppServer2001 includes a utility for generating a results page called the reporter. The report contains a summary of the key results and a complete system description in a format designed to be consistent with other SPEC reporting pages.
To run the reporter, use the following command:
# java -classpath reporter.jar reporter [-a] <submission file>.txt
Where <submission file>.txt is the Submission File created in the previous step.
An HTML file named <filename>.report.html is created by default.
The "-a" option will create a text report page named <filename>.report.txt.
Once you have a successful run, you can submit the results to the SPEC OSG Java subcommittee for review. To submit a result to SPEC:
Every submission goes through a minimum two-week review process, starting on a scheduled SPEC OSG Java sub-committee conference call. During the review, members of the committee may ask for additional information or clarification of the submission. Once the result has been reviewed and approved by the committee, it is displayed on the SPEC web site at http://www.spec.org/.
Copyright (c) 2002 Standard Performance Evaluation Corporation