Pls Help / Spammer on Web Server, Maybe My Local Computer

Hello and thanks in advance for any help you can give.....

A friend of mine has a fairly large website of information, monthly magazine, forums, etc. almost as big as AO site. He has little knowledge of server security. I just started helping him as a web developer. Other developers work with him, none of which are concerned with security at all. I am, but am just learning; by no means an expert.

I ran a trial version of eEye Retina against his site and found many suspicious ports open. The server looks like it's been hacked, but nothing appears to have been destroyed. My friend told me people have said they've received spam through his IP. I don't know if they're into anything else, but I guess his system would be a dream for a spammer to find. It's attached to a huge amount of bandwidth, he's frequently used telnet, very weak passwords, default directory structure, insecure scripts.... the list of security holes goes on and on and probably was no problem to break in to.

Anyway, the server is a Sun RaQ4 running Linux w/ Apache. I'll list the most interesting ports the hackers have opened and Retina's descrip:
143-IMAP
444-SNPP Simple Network Paging Protocol
484-Integra Software Management Environment
3000-UNKNOWN
3001-NessusD
5101-Talarian_TCP/_UDP
These are in addition to standard email protocols, none of which should be running.

If we had adequate, up-to-date backups, I'd be inclined to say reformat & reinstall. I want to do a complete site backup, but don't want to subject my local computer to intrusion and I also don't want to download his trojans and replicate the problem on a new install.

In fact, I may already be infected locally because I'm typing this message for the second time after my first browser window died suddenly and suspiciously after I had almost typed the whole message out. I'm now typing this in a text editor so I can save it as I go. My local machine is a Mac G4 and I'm behind a stock DSL switched router with a firewall. I just tried to find the router's browser-based config system, and it looks like it may have been removed. It didn't come up, anyway.

Can anyone offer ideas about the complete scope of what's going on here and what I can do about it?

I may not be any help But it sounds to me like the intruder is on it right now, there has got to be a way for you to log activity, doesn't linux have utilities for intruder detection? I was just reccomended Snort by another user but my network is al M$.

I'm pretty sure snort runs on linux also (almost positive, but too lazy to look). Anyway, if it has been hacked (sounds like a good possibility) the only way to be sure that every thing is clean is to reinstall. I would backup data files (no executables that you can't verify) and start over. If the server was setup without security in mind it will probably be very difficult to clean things up without starting over. Also, if you do try to clean it up don't trust the executables on the machine boot with a rescue disk and work from there (these are easy to find on the net if you don't have one).

If you do start clean I'd recommend using tripwire (www.tripwire.com), and setting it up before the server goes online.