JVM Options:
-server -Xms18g -Xmx18g -Xmn16g -Xss256k -XX:MetaspaceSize=200M -XX:+AggressiveOpts
-XX:+UseParallelOldGC -XX:ParallelGCThreads=9 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-XX:InitialCodeCacheSize=200m -XX:ReservedCodeCacheSize=200m -XX:+UseCompressedOops -XX:-UseBiasedLocking
-XX:+AlwaysPreTouch -XX:InitialSurvivorRatio=24 -XX:SurvivorRatio=24 -XX:TargetSurvivorRatio=90
-XX:-UseAdaptiveSizePolicy -XX:+UseLargePages -Djava.net.preferIPv4Stack=true
-Djaxws.transport.streaming=true -XX:-TieredCompilation -Dweblogic.ProductionModeEnabled=true
-Dweblogic.management.discover=false -Dweblogic.diagnostics.debug.DebugLogger.DISABLED=true
-Doracle.jdbc.defaultRowPrefetch=200
-Djavax.xml.parsers.DocumentBuilderFactory=weblogic.xml.jaxp.RegistryDocumentBuilderFactory
-Dweblogic.threadpool.MinPoolSize=30 -Dsun.net.inetaddr.ttl=0 -Dnetworkaddress.cache.ttl=0
-Dcom.sun.xml.ws.spi.db.BindingContextFactory=com.sun.xml.ws.db.glassfish.JAXBRIContextFactory
-Djavax.xml.bind.JAXBContext=com.sun.xml.bind.v2.ContextFactory
-Dweblogic.MuxerClass=weblogic.socket.NIOSocketMuxer -Dweblogic.SocketReaders=2

Server instances were started using numactl binding 1 instance per 9 cores.

Oracle patch for WebLogic Server Standard Edition 12.1.3 20720853.

-showversion -server -Xms10g -Xmx10g -Xmn5g -Xss272k
-XX:MetaspaceSize=200M -XX:+UseLargePages
-XX:+AggressiveOpts -XX:+DisableExplicitGC -verbosegc -Xloggc:emugc.log
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintCommandLineFlags
-XX:-TraceClassUnloading -Djava.net.preferIPv4Stack=true
-Dweblogic.MuxerClass=weblogic.socket.NIOSocketMuxer -Dweblogic.SocketReaders=1
-Dhttp.keepAlive=true -Dhttp.keepAliveCache.lifeTime=6000000 -Dweblogic.management.discover=false
-Dweblogic.diagnostics.debug.DebugLogger.DISABLED=true
-Djavax.xml.parsers.DocumentBuilderFactory=weblogic.xml.jaxp.RegistryDocumentBuilderFactory
-Dsun.net.inetaddr.ttl=0 -Dnetworkaddress.cache.ttl=0 -XX:-TieredCompilation

Server instances were started using numactl binding 2 instances per chip.

db_name                       = specdb
control_files                 = +DATA/cntrlspecdb
aq_tm_processes               = 0
compatible                    = 11.2.0
db_block_size                 = 2048
db_block_checking             = FALSE
db_block_checksum             = FALSE
db_cache_size                 = 60G
db_cache_advice               = OFF
db_4k_cache_size              = 10G
db_8k_cache_size              = 4G
db_keep_cache_size            = 6G
db_file_multiblock_read_count = 128
db_files                      = 256
ddl_lock_timeout              = 30
deferred_segment_creation     = false
dml_locks                     = 1000
filesystemio_options          = DirectIO
log_buffer                    = 536870912
log_checkpoint_interval       = 0
log_checkpoints_to_alert      = TRUE
open_cursors                  = 2400
parallel_max_servers          = 100
pga_aggregate_target          = 0
processes                     = 4000
query_rewrite_enabled         = FALSE
replication_dependency_tracking = FALSE
session_cached_cursors        = 8000
sessions                      = 6000
shared_pool_size              = 16G
statistics_level              = BASIC
timed_statistics              = FALSE
trace_enabled                 = FALSE
transactions                  = 6000
transactions_per_rollback_segment = 1
undo_management               = AUTO
undo_retention                = 30
undo_tablespace               = undo_ts

Grid Infrastructure was installed to use Oracle ASM for DATA and LOG disk groups

-server -Xms3200M -Xmx3200M -Dsun.net.inetaddr.ttl=0 -Dnetworkaddress.cache.ttl=0
-Dhttp.keepAliveCache.lifeTime=600000 -Djava.net.preferIPv4Stack=true -verbosegc
-Dcom.sun.xml.ws.spi.db.BindingContextFactory=com.sun.xml.ws.db.glassfish.JAXBRIContextFactory
-Djavax.xml.bind.JAXBContext=com.sun.xml.bind.v2.ContextFactory

BIOS Setting:
COD (Cluster on Die) is enabled in BIOS SW 1.2, with availability date of June 2015.

Added to /etc/sysctl.conf:
kernel.shmall = 10737418240
net.core.netdev_max_backlog = 600000
net.core.optmem_max = 40000000
net.core.rmem_default = 40000000
net.core.rmem_max = 40000000
net.core.somaxconn = 60000
net.core.wmem_default = 40000000
net.core.wmem_max = 40000000
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_max_syn_backlog = 60000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_mem = 40000000 40000000 40000000
net.ipv4.tcp_rmem = 40000000 40000000 40000000
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_wmem = 40000000 40000000 40000000
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
vm.zone_reclaim_mode=1
vm.nr_hugepages = 60000
vm.hugetlb_shm_group = 502

echo NO_GENTLE_FAIR_SLEEPERS > /sys/kernel/debug/sched_features

For each of the NICs:
set arp_announce = 2
set arp_ignore = 1
set txqueuelen 60000
enable tso and gso
On NICs used for driver and emulator communication:
set rx-usecs 300
On NICs used for database communication:
set rx-usecs 64
Bind interrupt to a core corresponding to the server instance

Server instances were started using numactl binding 2 instance per chip.
Stop iptables.
Stop irqbalance.

Mirrored SSDs were used for JMS and server log files.

Added to /etc/sysctl.conf:
fs.file-max = 6815744
fs.aio-max-nr = 1048576
kernel.shmall = 10737418240
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.core.netdev_max_backlog = 400000
net.core.optmem_max = 30000000
net.core.rmem_default = 30000000
net.core.rmem_max = 30000000
net.core.somaxconn = 40000
net.core.wmem_default = 30000000
net.core.wmem_max = 30000000
net.ipv4.ip_local_port_range = 9000 65535
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_mem = 30000000 30000000 30000000
net.ipv4.tcp_rmem = 30000000 30000000 30000000
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_wmem = 30000000 30000000 30000000
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
vm.zone_reclaim_mode=1
vm.nr_hugepages = 90000
vm.hugetlb_shm_group = 502

For each of the NICs:
set arp_announce = 2
set arp_ignore = 1
set txqueuelen 60000
enable tso and gso

4 Oracle listener processes were started, 2 processes bound per processor.
Bind each listener's network interrupts to its local processor.
Stop iptables.
Stop irqbalance.

External storage configuration:
1x Oracle Server X5-2L configured with 4x 1.6 TB NVMe PCIe SSDs,
2x 600GB Disks, and 1x Sun 16Gb FC Dual Port HBA for the database files.
1x Oracle Server X5-2L configured with 14x 600 GB disks, 1x Sun 16Gb FC
Dual Port HBA, and onboard PCI HW Raid controller for redo logs.

Added to /etc/sysctl.conf:
fs.file-max = 524288
kernel.shmall = 10737418240
kernel.shmmax = 68719476736
kernel.sem = 250 32000 100 128
net.core.netdev_max_backlog = 400000
net.core.optmem_max = 30000000
net.core.rmem_default = 30000000
net.core.rmem_max = 30000000
net.core.somaxconn = 40000
net.core.wmem_default = 30000000
net.core.wmem_max = 30000000
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_mem = 30000000 30000000 30000000
net.ipv4.tcp_rmem = 30000000 30000000 30000000
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_wmem = 30000000 30000000 30000000
vm.zone_reclaim_mode=1

For each of the NICs:
set arp_announce = 2
set arp_ignore = 1
set txqueuelen 240000
enable tso and gso

8 Systems were used as drivers, each with 1x 10Gbit NIC.
1 System was used as Emulator, with 1x 10Gbit NIC and configured with 22500 hugepages.