A Brief History

Around 1999 or so, the ucolick.org mail server was re-built by De Clarke and Will Deich around Solaris 7 and sendmail. Previously, it was running on SunOS, and a custom-compiled sendmail.

That machine, an Ultra 10, died an ignoble death in January 2002. It was replaced by another Ultra 10 running essentially the same software, except built around Solaris 8. Eventually, MailScanner (to allow the use of Sophos) and SpamAssassin were installed to perform virus and rudimentary spam filtering. Two things became clear over time:

With those principles in mind, a gradual migration to postfix, courier, and dspam began early in 2005 with the transition to only IMAP and POP for mail delivery, and was successfully completed in May 2005. In September 2006, this system was augmented with an additional two servers that each provide SMTP, IMAP, and POP services, with coarse load balancing via round-robin DNS. The remaining third machine was then dedicated completely to dspam.

The Implementation

Various tasks are split between different machines in our mail delivery scheme. There are currently three machines involved in the mail delivery path; the end-user services (smtp, imap, and pop are split between the hosts hunan and zilan, and delivery (popoca) for spam scanning. We are also using Maildir for mail storage, which (among other benefits) allows us to leverage the /home/public NFS server for user mail rather than store mail locally on the mail server. The accepted addresses to use for the various services are smtp.ucolick.org, imap.ucolick.org, and pop.ucolick.org. There are multiple ethernet interfaces on the mail hosts, and multiple hosts providing these services via round-robin DNS, and only the interface corresponding to the appropriate name is granted appropriate access via the firewall.

hunan and zilan handle:

popoca handles:

Other machines handle:

What path does your average inbound message take to get from some external sender to the user's inbox?

  1. Inbound connection hits one of the smtp hosts on port 25, and is handled by the postfix-in instance of postfix on that box. The message is checked against the list of valid recipients, and is either rejected or accepted for delivery.

  2. The accepted message is handed off to clamsmtpd via the localhost interface on port 10025. clamsmtpd queries against the running clamav daemon, which receives hourly updates to its virus definitions. The message is filtered into /var/clamsmtp if it is unacceptable, otherwise it is delivered to a postfix-out instance of postfix via the localhost interface on port 10026.

  3. The postfix-out instance does one of two things at this point: deliver the message to delivery if it is for one of the domains or hostnames we accept messages for, otherwise, it delivers it to the proper MX for the message recipients. The delivery to delivery subverts the normal evaluation of MX records.

  4. When a message is accepted on delivery, it first goes through alias translation and .forward processing. If the translation means the message needs to go off to some other mail server, it will do so at this point. If it needs to go to some other local user, it is correctly aimed at that username at this point.

  5. The postfix instance on delivery will now attempt to deliver the message to the local user. It is configured to use dspam as its local delivery agent; dspam will first take the message, perform its analysis, and then decide whether the message should be delivered to a user's spam quarantine, or to their normal inbox. If it is delivered to the spam quarantine, it is handed off to procmail, using a truncated configuration file that delivers all messages to $HOME/Maildir/.spam.$year.

  6. Messages destined for normal delivery to the inbox are also handed off to procmail, except that this delivery path will consult the user's .procmailrc file, and follow any instructions and additional filtering specified therein. In the average case, this means the message will now be delivered to the user's inbox.

The Installation

There are hooks all over the place for this software installation. Complete instructions for how to set up each software components are listed in the CONFIGURE file in that software package's main directory; for example, the configuration directives for postfix on delivery are listed in /local/postfix./CONFIGURE.

Run-time configuration directives are typically within the subdirectory for each individual program, with the exception of postfix; the postfix configuration directives are in /etc/postfix, or on the smtp hosts, /etc/postfix-in and /etc/postfix-out. Startup scripts, as one might expect, are all in /etc/init.d.

The Results

On an average day, the load on delivery is somewhere around 0.1, though it may get as high as 0.5 if the machine is in the middle of an intense wave of e-mail. The load on the smtp and imap servers tends to be in a similar regime, with ~2GB of system memory in active use.

dspam requires a lot of hardware for it to perform acceptably. Typically, the mysql backend will occupy something like 4GB of system memory, as each user profile in dspam is some 300-500k lines in the dspam tables. Each incoming message is tokenized and compared to that database, and each relevant token has a score that is incremented for each message matching that particular token. For typical messages, the dspam system can process 4-7 messages per second. Atypical messages, such as messages with 2MB inline bodies, can take up to 15 minutes to process if that user's profile is not in the active mysql cache. Note that does not include attachments-- if a message has attachments, they are ignored by dspam, and do not affect the message processing time.

Even with all the additional overhead, average mail delivery time is quite a bit shorter than it was with the sendmail-based system. This is primarily due to postfix's asynchronous processing, as opposed to sendmail's queue-based processing. Rather than wait for the appointed time to roll around to deliver messages, postfix will deal with any queued messages if it has free threads to do the dirty work. In practice, this means that backlogs are cleared in minutes, where with the old MailScanner system it could take half an hour for even the most basic backlog to be processed.

Postfix writes very verbose log info out to syslog; where sendmail would output maybe two lines per message delivery, postfix outputs something more akin to eight. Combine that verbosity with the substantial number of additional hops in the delivery chain, and you've got a whole lot more log info to slog through when slogging is required.

Common Tasks

Systems/Mail (last edited 2008-07-17 20:33:15 by YudhvirSidhu)