Purpose

To test and benchmark several common SMTP servers for Unix. This is inspired by the work done by Matthias Andree at his Postfix vs. qmail - Performance page. His information, while likely still highly relevant is based heavily around file system I/O and not overall performance of the MTA. Additionally, more MTAs and OSs will eventually be tested here.

First Test Machine

• OS: Solaris 9 for x86 (SunOS sun.jrh.org 5.9 Generic_117172-07 i86pc i386 i86pc)
• CPU: Pentium 4 1.8ghz
• RAM: 768MB PC1600
• Drives: 2 80GB Samsung IDE drives with 2MB buffer
• Partitioning: The drives were laid out using Solaris 9's default UFS values, /var does have logging enabled. /opt, where the binaries are, is on a striped RAID1 using Disksuite, but no data is written to that partition.
• Software: Postfix 2.1.5, Exim 4.42, Sendmail 8.13.1, Netqmail 1.05

Second Test Machine

• OS: FreeBSD 6.0-STABLE
• CPU: AMD XP2200+
• RAM: 1024MB PC2100
• Drives: 1 Western Digital 120GB
• Partitioning:

Third Test Machine

• OS: Solaris 10 for x64

DNS was not run locally and the smtp connections were all made from a machine on the same LAN, less than 1ms away. The DNS server was also less than 1ms away. Each machine has a 100mbit connection to the local network and no traffic from this test ever went out on to the Internet.

First Postal Test Setup

The first test uses Postal a free "SMTP and POP benchmark program." For the test, the most basic usage was used, in the form:

postal -s 0 sun.jrh.org users -

The users file was quite small, only 4 users to test to, but in this case, keeping it simple seemed best. The test was allowed to run for 5 minutes, removing the first run from the results.

First Postal Test Results

Postal outputs a fairly basic format:

time,messages,data(K),errors,connections,SSL connections

For this test we always hope to see the errors and SSL connections at 0 and we did. As no errors occured, they are not noted in the results. Also, the time is a one minute counter, not how long the operation took, so that is also not included.

First SMTP-Source Test Setup

The second part of this test was to use the smtp-source program, which is part of the Postfix testsuite. For this, a more simple test was used, since smtp-source doesn't have an easy way of doing more than one To: at a time. The command generated 5000 messages, 5k in size each:

time smtp-source -c -l 5000 -t testuser@sun.jrh.org -s 5 -m 5000 sun.jrh.org

This test was basically just to see how long it took to deliver that many messages, using 5 concurrent senders. In this case, also measured was the average CPU time on the receiving host, as there wasn't much else to measure.

First SMTP-Source Test Results

smtp-source itself doesn't output any information per se, so the only results that can be measured need to use another program. In this case, just the Unix time(1) command was used, but disk access and other things could certainly be measured.

Future

In the future, this test will be run on a wider variety of operating systems to benchmark how much of a role filesystems, network stacks and other various facets of the OS impact mail delivery.

Josh Howard Last modified: Sun Sep 19 22:33:41 PDT 2004