SPECmail2008 is an industry standard benchmark designed to measure a system's ability to act as a mail server compliant with the Internet standards Simple Mail Transfer Protocol (SMTP) and Internet Mail Application Protocol, Version 4 (IMAP4). The benchmark models business user behavior by simulating a real world workload experienced by enterprised based email services. The goal of SPECmail2008 is to enable objective comparisons of mail server products.
Only two are used. The benchmark assumes a single corporate domain holds all local e-mail accounts - e.g. user134@yourdomain.co.us. It uses a second domain to generate external e-mail traffic loads - e.g. someuser55@branch1.yourdomain.co.us. Note: The benchmark considers the subdomain (i>branch1.yourdomain.co.us a separate e-mail domain.
The benchmark must have at least 200 users to fulfill all of the message size and folder heirarchy distribution requirements.
The benchmark has been tested up through 4000 users. That is not a hardcoded limit. The main constraint has been disk storage.
More users in a single domain should work. However, the workload should not be extrapolated into the 10's of thousands or higher levels. That is usually the realm of outsourced e-mail services. The interaction level between any two users is usually determined by corporate bounds. This is reflected in the size of mail distribution lists and recipient counts. A 5000 user corporation has different workload characteristics than 100 corporations of 50 users.
The benchmark works with only 1 user. We suggest using 2, to make the SMTP and IMAP work correctly - one to send and one to receive.
The benchmark uses the IMAP4 command set defined by RFC 2060, only. To maximize interoperability chances, the benchmark does not use any IMAPv4 extended commands or parameters.
The SPECmail2008 benchmark uses different threads to emulate behaviors for particular mail clients: multi-session, single-session; both long and short duration mailbox activities; and short duration probes.
A command seuqence is a series of IMAP commands geared toward a specific purpose. There are 5 command sequences: 2 primary and 3 secondary. Primary command sequences are persistent thru the life of the user while secondary command sequences are transient, and will start/end as needed during the run.
A Client Type is a unique combination of IMAP Command Sequences that approximate observed behavior in real world e-mail IMAP4 clients.
Client types model unique behaviors found in collected IMAP sessions during the SPECmail2008 benchmark development. These sequences are attributable to unique e-mail clients. Each client type uses at least one (1) of the primary command sequences. The ratio of client types is configurable within the benchmark using the config key CLIENT_TYPE_DISTRIBUTION.
Each load generator creates a user thread assigned to a client type. Each user thread generates one or more Command Sequence session into the SUT, according to the rules of that Client Type. Some Command Seqences perform as many operations as possible, throttled only by the response time of the SUT. Other IMAP command sequences follow predefined timings derived from real world data.
We suggest a multi-CPU, not just multi-core hosts with 1 GB RAM for each CPU (not core). Keep in mind that each load generator emulates many users - 1 to hundreds. Each of these user threads must remember the status of each user's IMAP folders. Each user thread group must also have one persistant connection, and at least one other IMAP connection. All of these users must share the same host but perform the complex work of what is normally done by a dedicated computer.
The initial reaction is to run a single load generator on such
multi-CPU/core host. However, JVM limitations restrict how well the load
generator utilizes the physical box. We suggest running multiple load
generators instead. Start each with the "-p
Confirm that the benchmark manager and load generators can talk directly to the SMTP and IMAP servers on the SUT. The SMTP server will also need the ability to initiate SMTP connections to the designated SMTP sink host (usually the manager). This may require some DNS settings.
Yes. Due to a special "feature" of Java RMI, the configuration settins should use simple hosts names, without domain references. These host names should be the primary names defined in either the hosts table or in the DNS entries. Hostname aliases causes the benchmark components to not rendevous at the correct points, at various benchmark run phases.
The SPECmail2008 benchmark sequence goes through three distinct phases: initialization, benchmark loadtest runs, and generating SPECmail2008 reports from the collected data. A compliant run consists of the verification and load test phases. The resulting output.raw file is used to create the official Disclosure.
The mail store must be initialized before a compliant run that will be published. This very long process depends on the number of users.
One possible short cut is to initialize the mail store once and then backup this data set. Future tests can use this same backup image (restored of course) for the same number of test users. This means the benchmark really needs 900MB disk space per user.
A compliant run consists of: verify, ramp-up, steady-state (100%), data collection
The workload is derived from the total number of users (USER_END minus USER_START), the PEAK_PCT_USERS, and the CLIENT_TYPE_DISTRIBUTION.
The SUT may or may not be at 100% utilizaiton.
Sun Java from java.sun.com, 1.5 or later is recommended
Default installed Javas are not recommended
The benchmark dynamically generates multi-part MIME encoded messages based on the defined message MIME parts and defined message size distributions.
Yes. The Client Type distribution in the SPECmail_fixed.rc file defines the numbers of each type of command sequence generated. Changing this distribution changes the workload characteristics.
No these are embedded in the source
The workload can be changed in the following ways:
This config key is used only for the non loadtest benchmark phases - initialization, cleaning, verification. This value determines the number of concurrent threads the benchmark uses during these maintenance tasks.
The number of simultaneous sessions active during the course of the test period is a function of the Traffic Pattern distribution and the number of active users (USER_MAX minus USER_MIN times the PERCENT_ACTIVE).