Well, you /could/ simply put a sniffer out there and log all packets... It sounds like that box is just doing simple sendmail relaying. If so, it should basically only have inbound connections to the SMTP port and outbound connections to other people's SMTP ports, along with high-port to port 53 DNS traffic and inbound/outbound ident/auth. I would strongly encourage you to not run POP or IMAP on the sendmail relay - particularly if you are exposing those services to the net as well (which you really shouldn't be, to tell you the truth).
In the case of a compromised box, simply trying to patch it back to normal is often a complete waste of time and is asking for trouble unless you know exactly what was changed/added/etc. (and, if you're not running Tripwire or equivalent, I very much doubt you can be so positive about it).
Also, once you have a box compromised, you should be suspicious of at least everything on the same network segment.
