SPECmail2001 Frequently Asked Questions

Version 1.00
Last modified: January 16, 2001

Q1: What is SPECmail2001?

A1: SPECmail2001 is an industry standard benchmark designed to measure a system’s ability to act as a mail server compliant with the Internet standards Simple Mail Transfer Protocol (SMTP) and Post Office Protocol –Version 3 (POP3). The benchmark models consumer users of an Internet Service Provider (ISP) by simulating a real world workload. The goal of SPECmail2001 is to enable objective comparisons of mail server products.

Q2: What is the user profile for SPECmail2001?

A2: A consortium of experts within SPEC has defined the following user profile:

A user sends 2 messages per day to 2 recipients.
A user checks mail 4 times a day. After the first mail check, there are 3 successive checks performed by the client mail program.
The average message size is 25 kilobytes, with a distribution ranging from 1 kilobyte to 2 megabytes.
90% of users would be connected via 56 kilobit connections, the remaining 10% at high speeds.
SPECmail2001 simulates the peak hour of the day which is 15% of the daily activity.
Users send 90% of their messages to remote addresses, 10% to local addresses.
90% of received mail originates from remote users, 10% from local users.
1% of user sessions disconnect mid-stream during mailbox check to simulate inadvertent modem disconnects.

Q3: What is the performance metric for SPECmail2001?

A3: SPECmail2001 expresses performance in terms of SPECmail2001 messages per minute (MPM). For example:

Messages per minute = ((Messages sent per day * Number of Users) * Percentage Peak Hour)/60 minutes
MPM can be translated into a user count as follows: one MPM equals 200 SPECmail users. For example: 1,000 users = 5 MPM, 10,000 users = 50 MPM, 100,000 users = 500 MPM, 1,000,000 users = 5000 MPM, etc.
SPECmail2001 requires the reported throughput number (MPM) to meet the following Quality of Service (QoS) criteria:

for each mail operation, 95% of all response times recorded must be under 5 seconds
95% of all messages transferred to/from the mail server must transfer at a minimum rate of half the modem speed plus 5 seconds
95% of all messages sent to remote users must be received by the remote server during the measurement period
95% of all messages to local users must be delivered in 60 seconds
the mail server can produce no more than 1% of errors during the measurement period

Q4: Where can I find published results for SPECmail2001?

A4: SPECmail2001 results are available via SPEC’s Web site: http://www.spec.org/.

Q5: Who developed SPECmail2001?

A5: SPECmail2001 was developed by the Standard Performance Evaluation Corporation (SPEC), a consortium of leading hardware and software vendors and university representatives. The Mail Server subcommittee within SPEC included participants from major mail server vendors.

Q6: What are the limitations of SPECmail2001?

A6: The first release of SPECmail2001 does not support Internet Message Access Protocol (IMAP) or Webmail (email accessible via a browser). The plan is to include IMAP in a future SPEC mail server benchmark. Currently, there are no plans to address Webmail.

Q7: Why are message attachments not measured in SPECmail2001?

A7: While ISP POP consumers do receive message attachments (i.e. MIME, Multipurpose Internet Mail Extensions), these attachments are a client-side issue under the POP3 and SMTP protocols. Since the email client is responsible for parsing MIME attachments within a message, there is no impact on the mail server except for message size. SPECmail2001's message size distribution includes a consideration for attachments.

Q8: Can I use SPECmail2001 to determine the size of the mail server I need?

A8: SPECmail2001 cannot be used to size a mail server configuration, because it is based on a specific workload. There are numerous assumptions made about the workload, which may or may not apply to other user models. SPECmail2001 is a tool that provides a level playing field for comparing mail server products that are targeted for an ISP POP consumer environment. Expert users of the tool can use the benchmark for internal stress testing, with the understanding that the test results are for internal use only.

Q9: How do I obtain the benchmark?

A9: It can be purchased on CD-ROM. A discount price is available for universities and other non-profit organizations. To place an order, please contact SPEC at: http://www.spec.org/spec/contact.html

Q10: What is contained on the SPECmail2001 CD?

A10: The SPECmail2001 CD contains: Benchmark software including Java source code, documentation, and configuration files.

Q11: What additional software is required to run SPECmail2001?

A11: SPECmail2001 requires a mail server that supports POP3 and SMTP. A Java Runtime Environment (JRE) version 1.1.8 or later must be installed on the client machines.

Q12: What hardware is required to run the benchmark?

A12: In addition to the hardware for the system under test, one or more client machines are required. The number and size of client machines required by the benchmark will depend on the size of the workload.

Q13: What if I have questions about running the SPECmail2001 benchmark?

A13: Refer to the Trouble-Shooting section of SPECmail2001 User Guide or send questions to support@spec.org.

Q14: How can I submit SPECmail2001 results?

A14: SPEC members can submit for free and non-members pay a fee. For more information about submitting results, please contact SPEC.

Q15: Where can I go for more information?

A15: SPECmail2001 documentation consists of four documents: User Guide, Run Rules, Architecture White Paper and this FAQ. The documents can be found on SPEC’s Web site: http://www.spec.org/.