Defect identified in SPECjbb®2013
affects comparability of results

December 9, 2014 - The Standard Performance Evaluation Corporation (SPEC) has identified a defect in the SPECjbb®2013 benchmark suite. SPEC has suspended sales of the SPECjbb®2013 benchmark and is no longer accepting new submissions of SPECjbb®2013 results for publication on SPEC's website (www.spec.org).

SPEC is advising SPECjbb®2013 licensees and users of the SPECjbb®2013 metrics that a defect recently uncovered impacts the comparability of results. This flaw can significantly reduce the amount of work done during the measurement period, resulting in an inflated SPECjbb®2013 metric. SPEC recommends that users not utilize SPECjbb®2013 results for system comparisons without a full understanding of the impact of this defect on each benchmark result.

SPECjbb®2013 results published on SPEC's website have been marked Code Defect (CD) to alert readers to this issue. The SPEC OSG Java subcommittee is working to revise the benchmark to correct this defect, add additional validation safeguards, and release a new version as soon as possible. Current licensees will receive a free copy of the new version when it becomes available.

Below is a summary of the SPECjbb®2013 defect and its secondary effects. Please be sure to review this before utilizing any SPECjbb®2013 data or running the SPECjbb®2013 benchmark for system comparisons.

Primary defect: Partial Purchase Requests result in a workload that can vary significantly

Purchase transactions are the main transactions executed in SPECjbb®2013. The Transaction Injector (TxI) will initialize a purchase request to one of the available Supermarkets. Each purchase request will try to purchase an average of 60 items before checking out at the register. As load increases on large systems and latencies get longer, the store may have run out of inventory and the purchase request will not be able to purchase all intended items. This partial transaction occurs because the replenish request triggered when the inventory for a given item goes below 10% is not completed in time and the Supermarkets inventory reaches zero for that item. The threshold for a successful purchase was set too low (~1% of original target) and, in some circumstances, purchases with as few as a single item are declared successful. Since this flaw can affect results to varying degrees, the metrics obtained from these results are not comparable.

Secondary effects of above defect

1. Lightweight HQ Datamining Requests:
   As part of a Purchase Request, a receipt is sent to the HQ for the Supermarket. Due to the effect described above with partial transaction receipts having fewer than 60 items purchased, the HQ datamining requests iterating through the partially fulfilled requests are lighter weight than expected, resulting in further inflated SPECjbb®2013 metrics.
2. SPECjbb®2013 v1.0 compared to v1.01:
   Due to a bug fix in SPECjbb®2013, the above defect resulted in further performance inflation. Due to this, SPECjbb®2013 v1.0 and v1.01 results cannot be compared.

For more information, contact SPEC.