Performance Summary:

Performance Details:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• Firmware settings used:
• TurboModeEnable=Enable
• TurboBoost=Traditional
• HardwarePrefetcher=Disable

Storage Notes

• All storage accessed with 2 x 4 Gb Fibre Channel links. Each of the 8 DS3400 units houses 12 73-GB SAS drives and connects (daisy chained) to 3 EXP3000 SAS enclosures. Each EXP3000 enclosure houses 12 73-GB SAS drives. Total SAS drive count is 384. There are 48 SAS RAID5 arrays, each consisting of 8 drives. There are 12 x EXP810 disk enclosures attached to the 1 x DS4800 via 4 x 4 Gb drive ports. Two sets of 6 x EXP810 enclosures are daisy-chained together to form two loops with the DS4800 unit. Each EXP810 holds 16 x 73 GB FC drives. Total FC drive count is 192. There are 18 FC RAID5 arrays, each consisting of 8 drives (48 FC drives are unused). For all arrays (FC and SAS), each one has one LUN defined for one Tile. On the first array, there is a second LUN defined for hypervisor boot disk. Each LUN defined for a Tile is divided into 12 logical volumes (2 per VM) using LVM. Of the 66 total LUNs defined, 10 are unused.

Virtualization Software Notes

• The following boot options were used:
•   elevator=deadline intel_iommu=on iommu=pt clocksource=tsc idle=halt
• The following services are disabled:
•   atd, autofs, avahi-daemon, cgconfig, cron, iptables, ip6tables, iscsi, iscsid, ksm, ksmtuned, postfix, rhnsd, rpcbind, sysstat
• Network configuration is as follows:
•   One software bridge is created (br0) and the second Broadcom Ethernet port is assigned
•   to the bridge (eth1 to br0). First Broadcom port (eth0) is for host access and the third
•   and fourth Broadcom ports (eth2 and eth3) are unused. Every VM's virtIO network device is
•   assigned to br0. Kernel module for Intel x520 10 Gigabit Server adapter is loaded with
•   these options to create 360 virtual functions: max_vfs=60,60,60,60,60,60.
•   224 of these virtual functions are assigned (one per VM) to the Web, App
•   DB, and Infra VMs. When a virtual function is present on a VM, a script
•   is executed to shut down the first virtio_net device and migrate the IP
•   settings to the virtual function device. This is done to facilitate
•   easy IP/dns management when using a mix of VMs with and without
•   virtual functions. The script, convert_to_vf.sh is available in the
•   disclosure archive. On the host, Ethernet devices eth4 to eth7 are the four physical
•   functions for the active Intel 10Gbps Ethernet ports (eth8 and eth9 not active). These
•   are only used by the host to provide physical transport for the virtual functions.
• Web and App VMs are launched with nice utility (-n -2) to adjust their scheduling priority
• All VMs of tiles 1,9,17,25,33,41,49 are bound to NUMA node 1 and have a preferred memory allocation node of 1
• All VMs of tiles 2,10,18,26,34,42,50 are bound to NUMA node 2 and have a preferred memory allocation node of 2
• All VMs of tiles 3,11,19,27,35,43,51 are bound to NUMA node 3 and have a preferred memory allocation node of 3
• All VMs of tiles 4,12,20,28,36,44,52 are bound to NUMA node 4 and have a preferred memory allocation node of 4
• All VMs of tiles 5,13,21,29,37,45,53 are bound to NUMA node 5 and have a preferred memory allocation node of 5
• All VMs of tiles 6,14,22,30,38,46,54 are bound to NUMA node 6 and have a preferred memory allocation node of 6
• All VMs of tiles 7,15,23,31,39,47,55 are bound to NUMA node 7 and have a preferred memory allocation node of 7
• All VMs of tiles 8,16,24,32,40,48,56 are bound to NUMA node 0 and have a preferred memory allocation node of 0
• All VMs are created with the following command line format (options in [] are for VMs which have PCI device assignment):
•   /usr/libexec/qemu-kvm -name $name -M rhel6.0.0 -enable-kvm -drive file=$dev1,if=virtio,boot=on,cache=none
•   -drive file=$dev2,if=virtio,boot=off,cache=none -netdev tap,id=$name,ifname=$name,script=/etc/kvm/qemu-ifup-br0,vhost=off
•   -device virtio-net-pci,netdev=$name,mac=$macaddr [-pcidevice host=$pcidev] -m $mem -vnc $vnc_port -cpu qemu64,+x2apic
•   -no-kvm-pit-reinjection -smp $cpus,sockets=$cpus,cores=1,threads=1 -monitor telnet::$monitor_port,server,nowait
•   -serial telnet::$serial_port,server,nowait -daemonize
• After boot, all block I/O devices changed to use deadline I/O scheduler via sysfs interface

Web Server VM Notes

• The following boot options were used:
•   selinux=0 elevator=deadline
• The following services are disabled:
•   atd, avahi-daemon, cgconfig, cron, iptables, ip6tables, iscsi, iscsid, postfix, rhnsd, sysstat
• ulimits for nofile and noproc set to 102400
• httpd.conf changes:
•   KeepAlive on
•   MaxKeepAliveRequest 0
•   ServerLimit 10240
•   MaxClients 10240
•   MaxRequestsPerChild 1000
•   CustomLog logs/access_log common
• Web VM used NFS to access files from infra VM
• No options were specified in the NFS mount
• root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback
• ixgbevf module option used: InterruptThrottleRate=2000

Application Server VM Notes

• The following boot options were used:
•   selinux=0 elevator=deadline
• The following services are disabled:
•   atd, avahi-daemon, cgconfig, cron, iptables, ip6tables, iscsi, iscsid, postfix, rhnsd, sysstat
• ulimits for nofile and noproc set to 102400
• There are two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator
• Heap size for AppSrv01 has been changed to 1280 MB
• root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback
• ixgbevf module option used: InterruptThrottleRate=2000

Mail Server VM Notes

• The following boot options were used:
•   selinux=0 elevator=deadline
• The following services are disabled:
•   atd, avahi-daemon, cgconfig, cron, iptables, ip6tables, iscsi, iscsid, postfix, rhnsd, sysstat
• ulimits for nofile and noproc set to 102400
• Dovecot was compiled with default options and the following dovecot.conf was used:
•  base_dir = /var/run/dovecot/
•  ssl = no
•  disable_plaintext_auth = no
•  listen = *
•  log_path = /var/log/dovecot
•  mail_debug = yes
•  mail_location = dbox:/mailstore/dovecot/maildir/%u
•  mdbox_rotate_size = 2048
•  passdb {
•   args = /etc/passwd.dovecot
•   driver = passwd-file
•  }
•  protocols = imap
•  service auth {
•   user = root
•  }
•  service imap-login {
•   inet_listener imap {
•    port = 143
•   }
•   process_min_avail = 10
•  }
•  userdb {
•   driver = passwd
•   driver = passwd
•  }
• root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback

Database Server VM Notes

• The following boot options were used:
•   hugepages=416 selinux=0 elevator=deadline
• The following services are disabled:
•   atd, avahi-daemon, cgconfig, cron, iptables, ip6tables, iscsi, iscsid, postfix, rhnsd, sysstat
• ulimits for nofile and noproc set to 102400
• Database tuning options:
•   db2 update db cfg for specdb using DATABASE_MEMORY 200000
•   db2set DB2_LARGE_PAGE_MEM=DB; db2set DB2_PINNED_BP=ON
•   alter bufferpool IBMDEFAULTBP size 160000
•   alter tablespace roll_space NO FILE SYSTEM CACHING
•   alter tablespace C_cust_space NO FILE SYSTEM CACHING
•   alter tablespace C_supp_space NO FILE SYSTEM CACHING
•   alter tablespace C_site_space NO FILE SYSTEM CACHING
•   alter tablespace C_parts_space NO FILE SYSTEM CACHING
•   alter tablespace C_custinv_space NO FILE SYSTEM CACHING
•   alter tablespace M_parts_space NO FILE SYSTEM CACHING
•   alter tablespace M_bom_space NO FILE SYSTEM CACHING
•   alter tablespace M_wo_space NO FILE SYSTEM CACHING
•   alter tablespace M_lo_space NO FILE SYSTEM CACHING
•   alter tablespace M_inv_space NO FILE SYSTEM CACHING
•   alter tablespace O_cust_space NO FILE SYSTEM CACHING
•   alter tablespace O_ords_space NO FILE SYSTEM CACHING
•   alter tablespace O_ordl_space NO FILE SYSTEM CACHING
•   alter tablespace O_item_space NO FILE SYSTEM CACHING
•   alter tablespace S_comp_space NO FILE SYSTEM CACHING
•   alter tablespace S_sc_space NO FILE SYSTEM CACHING
•   alter tablespace S_po_space NO FILE SYSTEM CACHING
•   alter tablespace S_site_space NO FILE SYSTEM CACHING
•   alter tablespace S_supp_space NO FILE SYSTEM CACHING
•   alter tablespace U_space NO FILE SYSTEM CACHING
•   update db cfg for specdb using logprimary 64
• root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback
• ixgbevf module option used: InterruptThrottleRate=2000

Infraserver VM Notes

• The following boot options were used:
•   selinux=0 elevator=deadline
• The following services are disabled:
•   atd, avahi-daemon, cgconfig, cron, iptables, ip6tables, iscsi, iscsid, postfix, rhnsd, sysstat
• ulimits for nofile and noproc set to 102400
• httpd.conf changes:
•   Listen port to 81
•   ServerLimit 10240
•   MaxClients 10240
•   CustomLog logs/access_log common
• /etc/httpd/conf.d/fcgi.conf added:
•   LoadModule fastcgi_module modules/mod_fastcgi.so
•   ScriptAlias /fcgi-bin/ /var/www/fcgi-bin/
•   FastCgiIpcDir /etc/httpd/fastcgi
•   <Directory /var/www/fcgi-bin>
•     AllowOverride None
•     Options +ExecCGI -Includes
•     SetHandler fastcgi-script
•     Order allow,deny
•     Allow from all
•   </Directory>
•   AddHandler fastcgi-script fcgi
•   FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• Standard Linux NFS server is enabled to serve files to Web VM
• /etc/exports file contains:
•  /support-download *(rw,no_subtree_check,crossmnt,no_root_squash)
• root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback
• ixgbevf module option used: InterruptThrottleRate=2000
• Fcgi package built from Fedora 14 source RPM, which includes patch to compile with GCC 4.4

Idle Server VM Notes

• The following boot options were used:
•   selinux=0 elevator=deadline
• The following services are disabled:
•   atd, avahi-daemon, cgconfig, cron, iptables, ip6tables, iscsi, iscsid, postfix, rhnsd, sysstat
• ulimits for nofile and noproc set to 102400
• root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback

Client Driver Notes

• All prime clients and prime controller are VMs hosted with Red Hat Enterprise Linux 6.0. The first seven client systems (6 x x3650M2, 1 x x3250M3) each host 8 prime client VMs. The eighth client system hosts the prime controller VM and any unused prime client VMs (for tiles 57 and greater). Each VM has 2 vCPUs, and prime client VMs' vCPUs are bound to sibling threads in a processor core. Paravirtual clock is used for all client VMs. 1 virtio_net device, and 1 virtio_blk device are used in all client VMs. For x3650M2 hosts, there are 4 1Gbps Ethernet ports (4 active) on each physical client. For x3250M3 hosts, there are 6 x 1Gbps Ethernet ports (4 active). On all clients, first Ethernet port is used for host access and NFS. The remaining 3 active ports are used for software bridges, up to 3 VMs per bridge. Client VM images are provided from an NFS server.

Other Notes

• A single Blade Network Technologies G8000 switch is equipped with 2 x G8000 XGE SFP+ modules and 2 x SFP+ 10GBase-SR transceivers and is used for all physical network connections. Two QLogic 5600 FC switches are used for all fibre channel access.