Sendmail, Procmail and SpamAssassin email filtering How-To

Why do this:
Because when I first started trying to use SpamAssassin, I did the usual and it didn't work. I Googled, scoured mailing lists, read the FAQ's and docs in /usr/share/doc and didn't get what I needed. The SpamAssassin website is rather difficult to navigate- it's poorly laid out, and the search function is basically useless. Great product, crappy site, 'nuf said. After I started wanting to do some more advanced tricks with SpamAssassin, I joined the SA-Talk mailing list- a very very helpful (mostly) and agreeable bunch. However, I noticed a LOT of folks who would pop in and ask the same sort of questions. Often getting flamed for asking "that one again" because it had been recently been covered (for the 40th time). The "old timers" on the list get tired of answering the same questions over and over again. And the newbies keep coming.
Plus, I like to put up docs; for one thing, I often refer back to them myself (cause who likes to solve the same problem more than once?) when I've forgotten how I made something work 2 years after the fact.

First things first: A lot of people get all cornfused when trying to get SA running because they don't really understand how their email system is working in the first place... It's like wanting to add a turbocharger to a car, without having the foggiest idea of what makes an internal combustion engine run. You're in for a frustrating experience until you have a solid understanding of what happens to the little piece of text that we call an email on it's journey across the 'net. So here's a (painfully brief) explanation of the various pieces and parts that make an email system:

Spamassassin: TAGS Spam, but IT DOES NOT DO ANYTHING ELSE TO IT!!! It Can't delete your spam, it can't reply to it. It can't forward it. It looks at it, and either marks it as spam or not. That's it. Period.

Procmail: Procmail can (and will, if you ask it to) filter, scan, delete and otherwise adulterate your email. Procmail is also how your email will get sent to SpamAssassin (more on that later). Procmail is a Mail Delivery Agent, or MDA.

Sendmail: Sendmail is the program that receives the email coming into your server from another server or mailer. Sendmail can be configured to use SpamAssassin directly via a Milter, but that's another subject. Sendmail is a Mail Transport Agent, or MTA.

NOTE: I use Sendmail on Mandrake. Mandrake installs Postfix by default. Postfix sucks (IMO). It's a resource hog, and it's slow. The first thing I do when I install a new Mandrake system is install sendmail (urpmi sendmail sendmail-cf), fix the stoopid typo in /etc/mail/sendmail.mc (the path to m4.cf is wrong) and rpm -e postfix. Postfix can use SpamAssassin (via procmail or Maildrop) but there's loads of documentation out there on that setup.

The way things work on a "standard" Sendmail-Procmail system:
Sendmail receives the message, determines to whom it is supposed to be delivered to, and hands it off to Procmail.

Procmail then takes the message and delivers it to the users' mailbox (wherever that may be, usually /var/spool/mail/username).

User logs in to check their mail (via their Mail User Agent, or MUA such as Eudora or (bleah) Outlook) and receives their spam payload (oh, and if they're lucky, a couple of legit messages too).

The way things WILL work with SpamAssassin installed:
Sendmail will hand the message off to Procmail.

Procmail will look at it's config and realize that it's supposed to run it through SpamAssassin (either via spamassassin, or spamc if you're using the daemonized version).

SpamAssassin/ spamc will then RETURN THE MESSAGE TO PROCMAIL, after determining whether the mail is spam or not.

Procmail will look for the appropriate headers (X-SPAM-STATUS=yes) and act accordingly. It can simply send it on and let the end users' email client filter on those headers, or it can take a variety of actions, from dumping the message altogether to putting it in a different mailbox to, well, you get the picture.


Now, How-to do all this?
I like to build my own RPMs rather than install from CPAN due to issues with CPAN stomping all over my installed RPMs. An alternate to this (but you'll royally screw up your RPM database) is to run: [root]# perl -e shell -MCPAN
cpan> install Mail::SpamAssassin
You'll likely need a whole slew of other perl packages, like Net::DNS, HTML::Parser etc. as well... all can be installed from within CPAN. Good luck...

However, as I said, I strongly prefer the method below...
First, get the current stable tarball of SpamAssassin from the SA website.
Run the following as root (note that the wget command is split into 2 lines- you'll need it to be all one line for it to work.): [root]# wget http://www.mirror.ac.uk/sites/spamassassin.taint.org/\
spamassassin.org/released/Mail-SpamAssassin-x.x.tar.gz (get the most recent version)
[root]# rpm -ta ./Mail-SpamAssassin-{version}.gz
That will create 3 RPMS for you to install: perl-Mail-SpamAssassin, spamassassin, and spamassassin-tools. I use Mandriva Linux 2007, so your paths may vary- in particular, RedHat puts the RPMs in /usr/src/RPM/RPMS/i386)
Next: [root]# rpm -ivh {/path/to/your/RPMS/*}
If you run into dependancies, dig around on RPMfind.net or (and this is why I use Mandrake and have no intention of changing) simply urpmq {some substring of the package name}, and when you have a name, urpmi it into your system. If you don't have the contribs for your version of Mandrake configured as a urpmi source, may I suggest you go do that right now? Editorial If you haven't aquainted yourself with urpmi, do so- it's a completely fantastic tool. It solves all the frustrations that I used to have on my (numerous) RedHat systems of updates/ bug or security fixes etc. Urpmi just works, and works well. IMHO, it's reason enough to switch from RedHat to Mandrake all by itself.

Now that SpamAssassin is installed on your system, you need to tell Procmail to use it. One of the most common issues people have is getting confused when they don't have a particular file on their system: /etc/procmailrc is typical of xNix systems; if you're just calling procmail with it's default parameters, you simply don't need a config file. So you often won't find one. If, however, you do need to change the parameters, simply add the file. Procmamil will find it, read it, and the changes will take effect immediately. Nice, huh?
A word of caution regarding your procmailrc file (or any config file on a xNix system for that matter...): Don't edit this file on a Windoze system! DOS (Read: Windoze) and xNix use a different character for marking end-of-line / newline. As a result, one of the most common breakdowns with this is people using notepad (why? Beats the crap outta me...) to create the file and FTPing it over to their xNix machine (FTP as root? Zoicks...). Do yourself a favor and gain a working knowledge of VI. Create the file in place on the system that will use it, and all will be happy.

My /etc/procmailrc is below:

:0fw
* < 256000
| spamc # Change to spamassassin if you aren't running spamd...

# Work around procmail bug: any output on stderr will cause the "F" in "From"
# to be dropped. This will re-add it.
:0 H
* ! ^From[ ]
* ^rom[ ]
{
LOG="*** Dropped F off From_ header! Fixing up. "

:0 fhw
| sed -e 's/^rom /From /'
}

You can use procmail to do fun things like summarily drop executable attachments, search for strings in the messages etc.. Read # man procmailrc. Now, test it all out... find a piece of spam (there's a sample in /usr/share/doc/spamassassin-{VERSION}) and do the following:

[root]# cat /usr/share/doc/spamassassin-{VERSION}/sample-spam.txt | spamassassin

You should see your message return to stdout (your terminal screen) with a bunch of extra messages added by spamassassin (example below):

#> cat /usr/share/doc/spamassassin-2.60/sample-spam.txt | spamc

Subject: Test spam mail (GTUBE)
Message-ID:
Date: Wed, 23 Jul 2003 23:30:00 +0200
From: Sender
To: Recipient
Precedence: junk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Spam-Report:
* 1000 GTUBE BODY: Generic Test for Unsolicited Bulk Email
X-Spam-Status: Yes, hits=1000.0 required=5.0 tests=GTUBE autolearn=no
version=2.60
X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on
some.server.com
X-Spam-Level: **************************************************

This is the GTUBE, the
Generic
Test for
Unsolicited
Bulk
Email

If your spam filter supports it, the GTUBE provides a test by which you
can verify that the filter is installed correctly and is detecting incoming
spam. You can send yourself a test mail containing the following string of
characters (in upper case and with no white spaces and line breaks):

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

You should send this test mail from an account outside of your network.
[root]#

Now that it's installed and working, you'll want to set up procmail to do something about those spammy messages (this is my ~/.procmailrc): # Mails with a score of 5 or above get sent to my Junk mailbox

# Note that I use IMAP... this won't work if you use POP...

:0:
* ^X-Spam-Level: \*\*\*\*\*
Junk

So, there you have it... a working installation of SpamAssassin, teamed up with Sendmail and Procmail! I hope this was helpful; if so great (send me a line on the feedback page if you feel like it) or an email to rbennett_at_DIESPAMMERS_thatitguy.com. Suggestions on improvements welcome as well!

NEW!!! A friend and I have been working on a system to get rid of the Junk folder called the SpamDigester. Basically, the Digester takes messages tagged as Junk by SpamAssassin and drops them into a holding area (actually, a MySQL database), and then sends periodic digests to the users for whom it's holding messages. Each Digest contains a synopsis of the messages for that user, and an HTML form asking what to do with each of them (Delete/ Deliver/ Deliver and Whitelist). See more here: http://spamdigester.org

New New!!! We have largely stopped development work on the SpamDigester, because we found Untangle (http://untangle.com). Untangle does pretty much everything that the SpamDigester does, but we don't have to maintain the codebase. Sweet.