Performance

Configuration

Common Workload Notes

SUT Notes

• 1 SAS 10K 300GB disk for OS, 3 SAS 10K 300GB for SVM metadb
• 1 SAS 10K 300GB disk for benchmark kit and web server
• The 8-port (2x4) 6GBs SAS PCIe Host Adapters installed in slots 2, 3, 6, and 7 on the server.
• The Sun Storage F5100 Flash Array units each have 4 domains (80 FMOD in each unit).
• The 8 domains in the two units is attached to port 0 and 1 of the four 8-port (2x4) 6GBs SAS PCIe Host Adapters.
• Each domain has 20 FMODs.
• Sun Volume Manager used to create 5 (3 FMOD) volumes in four domains and 4 (3 FMOD) volumes in the other four domains. Total 36 volumes
• 16 FMODs (2 per domain) are striped together for the logs volume.
• Support, Banking, and Ecommerce, has SVM interlace of 4MB, logs volume has interlace of 1MB.
• Support, 36 x 67GB filesystems newfs with -C 128 and mounted with 'nologging,noatime,ro'.
• Banking, and Ecommerce 36 x 67GB filesystems mounted with 'nologging,noatime,ro'.
• Access log filesystem mounted with 'nologging,noatime'.
• The 10 GbE Dual Port fiber PCIe cards installed in slots 0, 1, 8, and 9.
• Total of 11 networks, three onboard 1 Gigabit ports and eight 10 GbE over 4 dual port cards.
• 1x Arista DCS-7124s 24-10GbE port switch for 6 server 10 GbE networks and 3 clients per network.
• 1x Foundry Networks EdgeIron 8x10G, 8-10GbE port switch for 2 server 10GbE networks and 3 clients per network.
• 1 x Cisco 2970 series (WS-C2970G-24TS-E) switch for the
• networks 1, 2 and 3 (onboard 1Gbit NIC each), BeSIM traffic
• network 4 through 11 (10Gbit each), 3 clients per network (Prime client also load generating)

Operating System Notes

• set rlim_fd_max=2000000
• set rlim_fd_cur=2000000
• set sq_max_size=0
• set maxphys=1048576
• set ufs:smallfile=240000000
• set ip:ip_squeue_fanout=1
• set ip:ip_soft_rings_cnt=4
• set ip:ip_soft_rings_10gig_cnt=16
• ndd -set /dev/tcp tcp_smallest_anon_port 2048
• ndd -set /dev/tcp tcp_conn_req_max_q 81920
• ndd -set /dev/tcp tcp_conn_req_max_q0 81920
• ndd -set /dev/tcp tcp_max_buf 4194304
• ndd -set /dev/tcp tcp_xmit_hiwat 800000
• Patch 146019-02

Web Server Software Notes

• Oracle iPlanet Web Server 7.0.9
• MPSSHEAP=256M; export MPSSHEAP
• Web server process started in FX class using priocntl

HTTP Script Notes

• SPEC-provided JSP scripts used without modification

Client Notes

• Twenty four load generating clients. Each client driving 1x 10GbE network.
• One acting as prime client.
• System Tuning:
• set sq_max_size=0
• set rlim_fd_max=32768
• set rlim_fd_cur=32768
• set ip:ip_soft_rings_10gig_cnt=10
• ndd -set /dev/tcp /dev/tcp tcp_max_buf 8388608
• ndd -set /dev/tcp /dev/tcp tcp_cwnd_max 4194304
• ndd -set /dev/tcp /dev/tcp tcp_recv_hiwat 800000
• ndd -set /dev/tcp /dev/tcp tcp_xmit_hiwat 800000
• /usr/java/jre/lib/security/java.security:
• Disable SunPKCS11 security provider for Java
• Reorder remaining security providers by decrementing provider number by 1
• Communication between Prime client and BeSIM occurs over the 3 ports on the Quad Gigabit Ethernet Express module.

BESIM Notes

• set rlim_fd_max=500000
• set rlim_fd_cur=500000
• set sq_max_size=0
• set ip:ip_soft_rings_cnt=8
• set ip_squeue_soft_ring=1
• set pcplusmp:apic_intr_policy=1
• ndd -set /dev/tcp tcp_smallest_anon_port 2048
• ndd -set /dev/tcp tcp_conn_req_max_q 81920
• ndd -set /dev/tcp tcp_conn_req_max_q0 8192
• ndd -set /dev/tcp tcp_xmit_hiwat 800000
• HTTP Software Notes:
• Web server process started in FX class using priocntl -c FX -e
• One web server instance listening on port 81
• magnus.conf:
• disable loading libj2eeplugin.so
• Init fn='init_besim' LateInit='no'
• MaxProcs 4
• server.xml:
• platform: 32bit web server
• Error Log: log-level failure
• file-cache: disabled
• java: disabled
• ssl-session-cache: disabled
• Performance Keep-alive: threads 4
• Performance Keep-alive: max-connections 8192
• Performance thread-pool: queue-size 8192
• Performance thread-pool: min-threads 520
• Performance thread-pool: max-threads 520
• port 81 listener parameters:
• ip *
• acceptor-threads 4

Other Notes

• Tuning descriptions see Tuning Disclosure: Oracle_Tuning_20101129.txt

Notes for Banking Workload

Operating System Notes

• set segmap_percent=36
• Bank: /usr/sbin/ksslcfg create -f pem -i /etc/kssl/keypair.pem -p /etc/kssl/passphrase -x 8080 -z 500000 443
• Bank: /usr/sbin/ksslcfg create -f pem -i /etc/kssl/keypair.pem -p /etc/kssl/passphrase -x 8081 -z 20000 445
• Bank: /kernel/drv/ixgbe.conf:
• Bank: tx_queue_number=3; set the number of the transmit queues, default 8
• Bank: rx_queue_number=4; set the number of the receive queues, default 8
• Bank: intr_throttling=2000; set interrupt throttling for 10GbE NICs, default 200
• Bank: Two 88 CPU processors sets created, one using CPUs 0-87 and the other using 128-215.
• Bank: Interrupts disabled within the processor sets.
• Bank: Used pcitool to reassign interrupts for PCI devices on /pci@400 to CPUs 88-127 and interrupts for PCI devices on /pci@500 to CPUs 216-255.
• Bank: One device interrupt per CPU, except for devices not used during testing, these devices were divided between two CPUs.
• Bank: Interrupts from the same device were grouped to interrupt CPUs in the same core were possible.
• Bank: "cores count up from CPU 0 to 7, 8 to 15, etc.

Web Server Software Notes

• Bank: Two web server instances with same tuning parameters except for listening IP address and ports.
• Bank: Web server instance 1 bound to CPUs 0-87 has 4 listeners, listening on the IP addresses of NICs in slots 0 and 8 on port 8080
• Bank: Web server instance 2 bound to CPUs 128 - 215 has 4 listeners, listening on the IP addresses of NICs in slots 1 and 9 on port 8080.
• Bank: Web server instance 2 has fifth listener.
• Bank: Instance 2's fifth listener, listens on the IP address of the first network of NIC in slot 0 on port 8081 for benchmark web server initialization
• Bank: magnus.conf: Init fn=pool-init block-size=2097152
• Bank: server.xml:
• Bank: General settings: platform 64bit web server
• Bank: General settings: stats disabled
• Bank: Error Log: log-level info
• Bank: file-cache: max-age 3600
• Bank: file-cache: max-entries 2000000
• Bank: file-cache: max-open-files 2000000
• Bank: file-cache: sendfile true
• Bank: file-cache: replacement false
• Bank: file-cache: max-heap-file-size 65536
• Bank: file-cache: max-heap-space 30000000000
• Bank: file-cache: max-mmap-file-size 1000001
• Bank: file-cache: max-mmap-space 1
• Bank: thread-pool: queue-size 270000
• Bank: thread-pool: min-threads 224
• Bank: thread-pool: max-threads 224
• Bank: keep-alive: timeout 120
• Bank: keep-alive: max-connections 270000
• Bank: keep-alive: threads 224
• Bank: keep-alive: poll-interval 0.05
• Bank: listener parameters
• Bank: port
• Bank: ip
• Bank: acceptor-threads 3
• Bank: acceptor-threads 2, Instance 2 only, fifth listener.
• Bank: listen-queue-size 81920
• Bank: blocking-io false
• Bank: servlet-container: log-level failure
• Bank: servlet-container: encode-cookies false
• Bank: http: etag false
• Bank: JVM OPTIONS: -server -Xrs -Xmn7500m -Xms15g -Xmx15g -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:ParallelGCThreads=10 -XX:+DisableExplicitGC -XX:+AggressiveOpts
• Bank: obj.conf: Service fn=send-file
• Bank: default-web.xml:
• Bank: JSP page compiler and execution servlet parameters
• Bank: param-name: reload-interval param-value: -1
• Bank: param-name: development param-value: false
• Bank: param-name: trimSpaces: true
• Bank: set session-timeout to 3

Client Notes

• ndd -set /dev/tcp tcp_smallest_anon_port 1200
• Bank: java -d64 -server -XX:+DisableExplicitGC -Xm2048m -Xms2048m specwbclient
• Bank: 9 client processes started per client system
• Bank: java -jar -d64 -server -XX:+DisableExplicitGC -Xmx2048m -Xms2048m /export/home/web2005-1.20/Harness/specweb.jar

Other Notes

• Bank: CHECK_IMAGE_SUBDIRS=2016, default 10
• Bank: BESIM_POOL_SIZE = 1024, default 128

Errors for Banking Workload

Quality of Service Errors

•   No QOS Errors Found

Validation Errors

•   No Validation Errors Found

Notes for Ecommerce Workload

Operating System Notes

• set segmap_percent=80
• Ecom: /usr/sbin/ksslcfg create -f pem -i /etc/kssl/keypair.pem -p /etc/kssl/passphrase -x 8080 -z 500000 443
• Ecom: /usr/sbin/ksslcfg create -f pem -i /etc/kssl/keypair.pem -p /etc/kssl/passphrase -x 8081 -z 20000 445
• Ecom: /kernel/drv/ixgbe.conf:
• Ecom: tx_queue_number=3; set the number of the transmit queues, default 8
• Ecom: rx_queue_number=4; set the number of the receive queues, default 8
• Ecom: intr_throttling=2000; set interrupt throttling for 10GbE NICs, default 200
• Ecom: Two 88 CPU processors sets created, one using CPUs 0-87 and the other using 128-215.
• Ecom: Interrupts disabled within the processor sets.
• Ecom: Used pcitool to reassign interrupts for PCI devices on /pci@400 to CPUs 88-127 and interrupts for PCI devices on /pci@500 to CPUs 216-255.
• Ecom: One device interrupt per CPU, except for devices not used during testing, these devices were divided between two CPUs.
• Ecom: Interrupts from the same device were grouped to interrupt CPUs in the same core were possible.
• Ecom: "cores count up from CPU 0 to to 7, 8 to 15, etc.

Web Server Software Notes

• Ecom: Two web server instances with same tuning parameters except for listening IP address and ports.
• Ecom: Web server instance 1 bound to CPUs 0-87 has 8 listeners, listening on the IP addresses of NICs in slots 0 and 8 on ports 80 and 8080
• Ecom: Web server instance 2 bound to CPUs 128 - 215 has 8 listeners, listening on the IP addresses of NICs in slots 1 and 9 on port 80 and 8080.
• Ecom: Web server instance 2 has ninth listener.
• Ecom: Instance 2's ninth listener, listens on the IP address of the first network of NIC in slot 0 on port 8081 for benchmark web server initialization
• Ecom: magnus.conf: Init fn=pool-init block-size=2097152
• Ecom: server.xml:
• Ecom: General settings: platform 64 bit web server
• Ecom: General settings: stats disabled
• Ecom: Error Log: log-level info
• Ecom: file-cache: max-age 3600
• Ecom: file-cache: max-entries 2000000
• Ecom: file-cache: max-open-files 2000000
• Ecom: file-cache: sendfile true
• Ecom: file-cache: replacement false
• Ecom: file-cache: max-heap-file-size 65536
• Ecom: file-cache: max-heap-space 3400000000
• Ecom: file-cache: max-mmap-file-size 1000001
• Ecom: file-cache: max-mmap-space 1
• Ecom: thread-pool: queue-size 350000
• Ecom: thread-pool: min-threads 512
• Ecom: thread-pool: max-threads 512
• Ecom: keep-alive: timeout 300
• Ecom: keep-alive: max-connections 350000
• Ecom: keep-alive: threads 128
• Ecom: keep-alive: poll-interval 0.05
• Ecom: listener parameters
• Ecom: port
• Ecom: ip
• Ecom: acceptor-threads 2
• Ecom: acceptor-threads 2, Instance 2 only, ninth listener.
• Ecom: listen-queue-size 81920
• Ecom: blocking-io false
• Ecom: servlet-container: log-level failure
• Ecom: servlet-container: encode-cookies false
• Ecom: JVM OPTIONS: -server -Xmn9000m -Xms18g -Xmx18g -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:ParallelGCThreads=20 -XX:+DisableExplicitGC -XX:+AggressiveOpts
• Ecom: obj.conf: Service fn=send-file
• Ecom: default-web.xml:
• Ecom: JSP page compiler and execution servlet parameters
• Ecom: param-name: reload-interval param-value: -1
• Ecom: param-name: development param-value: false
• Ecom: param-name: trimSpaces: true
• Ecom: set session-timeout to 2

Client Notes

• ndd -set /dev/tcp tcp_smallest_anon_port 1200
• Ecom: java -d64 -server -XX:+DisableExplicitGC -Xm2048m -Xms2048m specwbclient
• Ecom: 9 client processes started per client system
• Ecom: java -jar -d64 -server -XX:+DisableExplicitGC -Xmx2048m -Xms2048m /export/home/web2005-1.20/Harness/specweb.jar

Other Notes

• Ecom: PRODUCT_IMAGE_SUBDIRS = 2016, default 10
• Ecom: BESIM_POOL_SIZE = 1024, default 128
• Ecom: LG_BUF_SIZE = 102400, default 22528

Errors for Ecommerce Workload

Quality of Service Errors

•   No QOS Errors Found

Validation Errors

• 1 Validation Errors for Iteration 1  :   b9-21:1500 - customize3   

• 1 Validation Errors for Iteration 1  :   b9-23:1200 - customize3   

• 1 Validation Errors for Iteration 1  :   b9-24:1500 - customize3   

• 1 Validation Errors for Iteration 1  :   b8-66:1300 - customize3   

• 1 Validation Errors for Iteration 1  :   b9-27:1600 - customize3   

• 1 Validation Errors for Iteration 1  :   b9-28:1200 - customize3   

• 1 Validation Errors for Iteration 1  :   b8-70:1400 - customize3   

• 2 Validation Errors for Iteration 1  :   b8-66:1300 - cart   

• 1 Validation Errors for Iteration 1  :   b9-28:1700 - cart   

• 1 Validation Errors for Iteration 1  :   b9-21:1200 - login   

• 1 Validation Errors for Iteration 1  :   b9-25 - login   

• 1 Validation Errors for Iteration 1  :   b9-30:1300 - login   

• 1 Validation Errors for Iteration 1  :   b8-78:1400 - login   

• 1 Validation Errors for Iteration 1  :   b9-23:1100 - login   

• 1 Validation Errors for Iteration 1  :   b8-66:1300 - login   

• 1 Validation Errors for Iteration 1  :   b8-77 - login   

• 1 Validation Errors for Iteration 1  :   b9-28:1100 - login   

• 1 Validation Errors for Iteration 1  :   b9-28:1700 - login   

• 1 Validation Errors for Iteration 1  :   b8-71:1800 - shipping   

• 1 Validation Errors for Iteration 1  :   b9-30 - billing   

• 1 Validation Errors for Iteration 1  :   b8-71:1800 - billing   

• 2 Validation Errors for Iteration 1  :   b9-30 - confirm   

• 1 Validation Errors for Iteration 1  :   b9-30:1200 - confirm   

• 2 Validation Errors for Iteration 1  :   b8-71:1800 - confirm   

• 1 Validation Errors for Iteration 2  :   b9-22:1700 - customize3   

• 1 Validation Errors for Iteration 2  :   b8-78 - customize3   

• 1 Validation Errors for Iteration 2  :   b8-70:1100 - customize3   

• 1 Validation Errors for Iteration 2  :   b8-73:1300 - customize3   

• 1 Validation Errors for Iteration 2  :   b8-73:1400 - customize3   

• 3 Validation Errors for Iteration 2  :   b8-74:1100 - customize3   

• 1 Validation Errors for Iteration 2  :   b9-22:1700 - cart   

• 2 Validation Errors for Iteration 2  :   b8-78 - cart   

• 1 Validation Errors for Iteration 2  :   b9-28:1100 - cart   

• 1 Validation Errors for Iteration 2  :   b8-69:1300 - cart   

• 1 Validation Errors for Iteration 2  :   b9-22:1700 - login   

• 1 Validation Errors for Iteration 2  :   b8-78 - login   

• 1 Validation Errors for Iteration 2  :   b9-23:1200 - login   

• 1 Validation Errors for Iteration 2  :   b8-66 - login   

• 1 Validation Errors for Iteration 2  :   b9-28:1100 - login   

• 1 Validation Errors for Iteration 2  :   b8-69:1300 - login   

• 1 Validation Errors for Iteration 2  :   b8-78:1400 - billing   

• 1 Validation Errors for Iteration 2  :   b9-29:1100 - confirm   

• 2 Validation Errors for Iteration 2  :   b8-78:1400 - confirm   

• 1 Validation Errors for Iteration 2  :   b8-74 - confirm   

• 1 Validation Errors for Iteration 3  :   b9-25:1700 - customize3   

• 1 Validation Errors for Iteration 3  :   b8-71:1800 - customize3   

• 1 Validation Errors for Iteration 3  :   b8-75:1500 - customize3   

• 1 Validation Errors for Iteration 3  :   b9-27:1100 - customize3   

• 1 Validation Errors for Iteration 3  :   b9-22:1100 - cart   

• 1 Validation Errors for Iteration 3  :   b9-26:1300 - cart   

• 1 Validation Errors for Iteration 3  :   b8-68:1600 - cart   

• 1 Validation Errors for Iteration 3  :   b9-27:1100 - cart   

• 1 Validation Errors for Iteration 3  :   b8-73:1100 - cart   

• 2 Validation Errors for Iteration 3  :   b8-73:1300 - cart   

• 1 Validation Errors for Iteration 3  :   b9-22:1100 - login   

• 1 Validation Errors for Iteration 3  :   b9-25:1700 - login   

• 1 Validation Errors for Iteration 3  :   b9-26:1300 - login   

• 1 Validation Errors for Iteration 3  :   b8-68:1600 - login   

• 1 Validation Errors for Iteration 3  :   b8-71:1100 - login   

• 1 Validation Errors for Iteration 3  :   b9-27:1100 - login   

• 1 Validation Errors for Iteration 3  :   b8-73:1100 - login   

• 1 Validation Errors for Iteration 3  :   b8-73:1300 - login   

• 1 Validation Errors for Iteration 3  :   b9-22:1200 - shipping   

• 1 Validation Errors for Iteration 3  :   b8-66:1600 - shipping   

• 1 Validation Errors for Iteration 3  :   b9-22:1200 - billing   

• 1 Validation Errors for Iteration 3  :   b8-78:1300 - billing   

• 1 Validation Errors for Iteration 3  :   b8-66:1600 - billing   

• 1 Validation Errors for Iteration 3  :   b8-76:1500 - billing   

• 2 Validation Errors for Iteration 3  :   b9-22:1200 - confirm   

• 2 Validation Errors for Iteration 3  :   b8-78:1300 - confirm   

• 1 Validation Errors for Iteration 3  :   b8-66:1600 - confirm   

• 2 Validation Errors for Iteration 3  :   b8-76:1500 - confirm   

• 1 Validation Errors for Iteration 3  :   b8-70 - confirm   

Notes for Support Workload

Operating System Notes

• set segmap_percent=36
• Supp: /kernel/drv/ixgbe.conf:
• Supp: tx_queue_number=4; set the number of the transmit queues, default 8
• Supp: rx_queue_number=6; set the number of the receive queues, default 8
• Supp: intr_throttling=2000; set interrupt throttling for 10GbE NICs, default 200
• Supp: Two 48 CPU processors sets created, one using CPUs 0-47 and the other using 128-175.
• Supp: Interrupts disabled within the processor sets.
• Supp: Used pcitool to reassign interrupts for PCI devices on /pci@400 to CPUs 48-127 and interrupts for PCI devices on /pci@500 to CPUs 176-255.
• Supp: One device interrupt per CPU, except for devices not used during testing, these devices were divided between two CPUs.
• Supp: Interrupts from the same device were grouped to interrupt CPUs in the same core were possible.
• Supp: "cores count up from CPU 0 to 7, 8 to 15, etc.

Web Server Software Notes

• Supp: Two web server instances with same tuning parameters except for listening IP address and ports.
• Supp: Web server instance 1 bound to CPUs 0-47 has 4 listeners, listening on the IP addresses of NICs in slots 0 and 8 on port 80.
• Supp: Web server instance 2 bound to CPUs 128 - 175 has 4 listeners, listening on the IP addresses of NICs in slots 1 and 9 on port 80.
• Supp: Web server instance 2 has fifth listener.
• Supp: Instance 2's fifth listener, listens on the IP address of the first network of NIC in slot 0 on port 445 for benchmark web server initialization
• Supp: magnus.conf: Init fn=pool-init block-size=1048576
• Supp: server.xml:
• Supp: General settings: platform 32bit web server
• Supp: General settings: stats disabled
• Supp: Error Log: log-level info
• Supp: file-cache: max-age 3600
• Supp: file-cache: max-entries 1000000
• Supp: file-cache: max-open-files 1000000
• Supp: file-cache: sendfile true
• Supp: file-cache: sendfile-size 131070
• Supp: file-cache: replacement false
• Supp: file-cache: max-heap-file-size 32768
• Supp: file-cache: max-heap-space 2000000000
• Supp: file-cache: max-mmap-file-size 16000001
• Supp: file-cache: max-mmap-space 1
• Supp: thread-pool: queue-size 140000
• Supp: thread-pool: min-threads 620
• Supp: thread-pool: max-threads 620
• Supp: keep-alive: timeout 120
• Supp: keep-alive: max-connections 140000
• Supp: keep-alive: threads 64
• Supp: keep-alive: poll-interval 0.001
• Supp: listener parameters
• Supp: port
• Supp: ip
• Supp: acceptor-threads 4
• Supp: acceptor-threads 1, Instance 2 only, fifth listener.
• Supp: listen-queue-size 81920
• Supp: blocking-io false
• Supp: servlet-container: log-level failure
• Supp: servlet-container: encode-cookies false
• Supp: http: etag false
• Supp: JVM OPTIONS: -server -Xrs -Xmn750m -Xms1500m -Xmx1500m -XX:+AggressiveHeap -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:ParallelGCThreads=10 -XX:+DisableExplicitGC -XX:+AggressiveOpts
• Supp: obj.conf: Service fn=send-file
• Supp: default-web.xml:
• Supp: JSP page compiler and execution servlet parameters
• Supp: param-name: reload-interval param-value: -1
• Supp: param-name: development param-value: false
• Supp: param-name: trimSpaces: true
• Supp: set session-timeout to 2

Client Notes

• Supp: java -server -XX:+DisableExplicitGC -Xm800m -Xms800m specwbclient
• Supp: 10 client processes started per client system
• Supp: java -server -XX:+DisableExplicitGC -Xmx256m -Xms256m specweb.jar

Other Notes

• Supp: DOWNLOAD_SUBDIRS= 36, default 10
• Supp: BESIM_POOL_SIZE= 1024, default 128

Errors for Support Workload

Quality of Service Errors

•   No QOS Errors Found

Validation Errors

• 1 Validation Errors for Iteration 1  :   35-135:1100 - download