An Intro to ProcMail

Recently, I've found that several AOans don't seem know much about ProcMail, and some have never even heard of it. For those of you aspiring to be (or already are) system administrators, ProcMail will become a very handy tool at one point or another. It has been called an email-filtering program for the Unix environment, but it's really much, much more than that. It has the ability to send auto-responses, sort email, and even trigger external scripts. That being said, this will only be a BRIEF tutorial on the wonders of ProcMail designed only to whet your appetite for learning more about it. This is not an all inclusive document. Also, be advised that electronic mail has some of the most complicated formatting standards of any other protocol. As such, some of your first recipes may produce some undesirable results when they encounter a uniquely formatted email (HTML, MIME-encoded, PGP-encrypted, etc). I HIGHLY encourage all of you to test your recipes before implementing them (this process is explained at the end), and to continue learning about this tool as its intricacies are vast. Here are some simple examples where ProcMail could be useful:

* Formatting mailing-list email for a webpage (like BugTraq info)
* Filtering out spam messages
* Sorting your incoming email into folders
* Creating an auto-responder to automatically report spammers to their ISPs
* Remote system administration (through the use of ProcMail kicking off scripts based on your commands)

ProcMail's first version was released on 12/7/1990. Back then, most people that didn't work with computers would say "What-mail? E-mail? What's the E stand for?" Those who did use email were usually confined to some sort of UNIX email program...like, um...mail. As e-mail grew in popularity, mail programs like mh, mutt, elm, and pine became a bit more complicated to handle larger volumes of email. ProcMail came along just in time to perfectly compliment those old text-only mail programs. Only sysadmins have pushed it to do so much more! The basic concept behind ProcMail is to do something upon the arrival of an email. The way ProcMail knows that an email has arrived is that a ProcMail process is forked by Sendmail through the use of a .forward file. Most of you are familiar with a .forward file, but for those of you who are not, it's just a file sitting in your home directory that Sendmail checks for. If one is available, Sendmail will route the email to wherever the .forward tells it. For example, here's a typical .forward file to re-route all incoming email to another address:

Code:

myjunkemailaccount@hotmail.com

Simple enough? A .forward file can send email to another email address, a group of email addresses, or to a program. .forward files do not do any kind of sorting or filtering, it just does a raw dump of all email to the new location. In the above example, everything going to the example account would actually re-direct "myjunkemailaccount@hotmail.com". To kick off a program with a forward file, you must simply open a "pipe" to the program:

Code:

"|vacation [username]"

The above example will kick off the UNIX vacation program (a simple email auto-responder) for every incoming email. To kick off ProcMail, simply open a pipe to the ProcMail binary:

Code:

|/usr/local/bin/procmail

Once ProcMail is executed, it looks in your home directory for a configuration file that contains your instructions for what to do with the new email. These instructions (called recipes) along with your environment variables are contained in a file called .procmailrc. Now we get into the meat of what ProcMail can do. Let's take a look at a simple example .procmailrc file, and then dissect it afterwards. This one demonsrates quite a few of ProcMail's talents:

Code:

# RoswellProcMailRC (tm) Patent Pending # Copyright 2000 # # Environment and Variable Assignments SHELL=/bin/sh PATH=/bin:/usr/bin:/usr/bin HOME=/home/roswell DEFAULT=$HOME/missplaced.txt LOGFILE=$HOME/.proclog # Recipes (start with a cup of butter... =) # Recipe 1. :0 H: *^TO_:.*junk@w00t.net /dev/null # Recipe 2. :0 H: *^From:.*bilbobaggins ^From:.*frodobaggins lotr # Recipe 3. :0 H *^From:.*roswell *^Subject:.*accounts { :0 c: accounts :0 ! accounting@w00t.net } # Recipe 4. :0 H : *^From:.*roswell *^Subject:.*majordomo majordomo # Recipe 5. :0 *^From:.*updates@linuxthingys.com |/home/roswell/bin/webupdate.pl # Recipe 6. :0: $DEFAULT

First, let's go over the beginning section on Environment and Varible Assignments. These are just some initial declarations. The ones listed (SHELL, PATH, HOME, DEFAULT, LOGFILE) are probably the most important. SHELL operates much like the #! of any shell script. It just tells ProcMail which shell to use as a default. PATH defines the search path for any referenced binaries in your recipes. HOME defines the path to your home directory. DEFAULT defines where to send the mail if all else fails. If you don't set this, ProcMail will assume your inbox to be the default destination. You normally don't have to mess with this one. I defined it here simply because I did not want any mail to end up in my INBOX if it didn't have a match in one of my recipes. LOGFILE is a file that ProcMail will report it's problems to. These are just a few of the variables that you can assign in ProcMail. Here are some of the others:

$#, $$, $-, $1 etc., $=, $?, $@, $_, B, COMSAT, DEFAULT, DELIVERED, DROPPRIVS, EXITCODE, H, HOST, INCLUDERC, LASTFOLDER, LINEBUF, LOCKEXT, LOCKFILE, LOCKSLEEP, LOCKTIMEOUT, LOG, LOGABSTRACT, LOGFILE, MAILDIR, MATCH, MSGPREFIX, NORESRETRY, ORGMAIL, SENDMAIL, SENDMAILFLAGS, SHELL, SHELLFLAGS, SHELLMETAS, SHIFT, SUSPEND, TIMEOUT, TRAP, UMASK, VERBOSE

The next section is for the recipes. Each recipe will have a few syntax standards that make it a recipe:

Code:

:0 * (condition) (action)

The first line must begin with a :0 so that ProcMail knows this is the beginning of a recipe. There are also several flags you may include on this line that will alter how ProcMail will process this recipe. In the above .procmailrc file, for example, you'll notice that nearly all the recipes have an H listed with them. This is to denote that when parsing the condition statement, ProcMail will only interrogate the headers of the email looking for a match. A B in the same location instruct ProcMail to interrogate the body of the email. Alternately, a lowercase h tells ProcMail to only interrogate the headers of the email when parsing the action statement, just as a lowercase b would mean to interrogate the body of the email when parsing the action statement.

The second flag that is common in the above .procmailrc file is simply a colon. The colon at the end of this line tells ProcMail to use a lockfile when processing this recipe. This is useful when the action of the recipe outputs the message to a file. Since it is possible that multiple instances of ProcMail are running at the same time, this protects your file by preventing the multiple instances of ProcMail from accessing the same file at the same time. Use this flag when outputting the message to a file or mail folder.

The last flag that I will cover here can be seen in the above .procmailrc only once in Recipe 3. The c flag is used when you wish to create a copy of the message. In the example of Recipe 3, I use the c flag to create a copy of the message to save to a local mail folder, and then I forward the original message on to an external email address. There are 11 additional flags you can use in the initial line of your recipe that can modify how ProcMail parses your recipe, but their uses are a bit more involved and out of scope for this tutorial.

The next line in a ProcMail recipe begins with an asteriks *. This denotes a condition statement. Think of each * line as an if statement. In the above .procmailrc, Recipe 4 has 2 condition statements. Both are designed to match patterns using a basic regular expression syntax. The first conditional statement in Recipe 4 is:

Code:

*^From:.*roswell

This statement is designed to search for the word "From" at the beginning of a line (^) in the headers of the incoming email message followed by a colon (:), some additional characters (.*), and then the username "roswell". The next conditional statement in Recipe 4 is:

Code:

*^Subject:.*majordomo

The second statement is designed to search for the word "Subject" at the beginning of a line (^) in the headers of the incoming email message followed by a colon (:), some additional characters (.*) and then the string "majordomo".

Each conditional statement in your recipe acts as an AND logical operator, that will only return a successful match if all the conditions are met. You can have as many conditional statements as you wish after the first line of the recipe. You can even omit the conditional statement completely as I have in Recipe 6 in the above .procmailrc file. Omitting the conditional statement, however will match on ALL email messages, so you may want to only use this method when defining a catch-all recipe. Remember to put any catch-all recipe at the very END of your .procmailrc file. ProcMail uses a first-match method of delivery meaning that the mail is delivered to the first recipe that is matched in the .procmailrc. If your catch-all recipe is first in your .procmailrc, you have just "caught-all" of your email messages, and none of your other recipes would even be interrogated.

If you wish to create a recipe that contains an OR logical operator (such as a match on
"hotmail.com" OR "yahoo.com" for an anti-spam recipe), simply put the alternate statement on a new line without the preceding asteriks, as in the following example (also in Recipe 2):

Code:

*^From:.*roswell ^Subject:.*majordomo

The final line of a ProcMail recipe is the action statement. This defines what ProcMail is to do with any messages that match the condition statements. For this line, you can have the mail sorted into mail folders, output to a text file or some other device, forwarded to another email address, you can execute a shell command, or you can have a series of actions. More than one action should be enclosed within braces. Each seperate action line may have its own beginning colon line with seperate flags if necessary. In the above .procmailrc, there is an example of each of these examples.

In Recipe 1, the action line sends all mail that is sent to "junk@w00t.net" to /dev/null,
effectively deleting it as soon as it arrives. You could substitute any valid filename for
/dev/null in this example. It is also permitted to specify directory names using the form:

Code:

/path/to/directory/.

The "/." at the end of the path to the directory is required. When sending to a file or a
directory, using a lockfile is highly recommended. If you do not specify the full path to a
specific filename, it is assumed that the filename is a folder in your default mail handler. If you are set up to use PINE, for example, the message will be appended to the PINE foldername you specify.

To send the matched incoming email off to another address, use the "!" to begin your action line:

Code:

! comments@w00t.net

An example of this can also be seen in Recipe 3 above. Additional addresses may be supplied as well. Simply list them after the first with one space in between each address. Finally, you can kick off a process upon the successful match of an incoming email. Just like you used in the .forward file, use a pipe to execute shell commands in ProcMail. From Recipe 5:

Code:

|/home/roswell/bin/webupdate.pl

The above example would feed the message as standard input to the "webupdate.pl" script. Once the process had completed (either with an error or with a success) the process exits and control reverts back to ProcMail for further processing. If additional switches are required after the initial command, simply list them after the command name as you would on the command line. Occasionally, you may want to execute more than one command. This is also possible by seperating each command with a double ampersand (&&). If you have to continue your commands on another line for readability or necessity, use the backslash (\) to designate the end of the line, but not the end of the command.

Code:

|/home/roswell/bin/webupdate.pl >> /home/roswell/www/database.txt &&\ /home/roswell/bin/rehash-web.pl

** Testing ProcMail Recipes **

Before setting any .procmailrc file into place you may want to test it with a dummy email message ahead of time. The easiest way to do this is to establish an alternate .procmailrc file called whatever you wish, then create a dummy email message in a text file (or export one of your real emails from your email program to a file). When you are ready to test your new rc file, use the following command:

Code:

procmail /path/to/proctest.rc < mail.msg

Where "/path/to/proctest.rc" is the path to and the filename for your test rc file and "mail.msg" is your dummy email message. This should operate just as if ProcMail was kicked off from the .forward file in your directory.

I hope this tutorial has given you a quick introduction to the use of ProcMail. I would like to reiterate that ProcMail has quite a few additional intricacies that I did not cover here. I invite you to learn more about ProcMail, and here are a few good resources:

ProcMail's Home Page - Get your latest version here.
The ProcMail Documentation Project - An overwhelming resource of good tips and explanations.
ProcMail Quick Reference Guide - A great primer for ProcMail syntax
Timo's ProcMail Tips and Recipes - A great source of example recipes.

PM me or post here if you have any additional questions or comments. Have fun!

--roswell1329