my deb server got hacked, little help?

Hello all.
I got debian server which was haxed today. (Wondered why it lagged and checked auth.log and noticed that german ip had logged in as root which should be impossible because no one else has access to server than me).
It´s Linux 2.2.20-compact.
I had run apt-get update and apt-get dist-upgrade yesterday so everything should be up to date.
So I ran chkrootkit. Nothing critical came up.
I checked roots bash history, nothing.
Then i unplugged it from internet.
I´d like to know:
1. How to see how that hacker got in/got my root pass?
2.How to see what he did when he was inside?
3.What i should do before reinstalling?

Thx for all who answer.
<Lorvija>

Wow.. old kernel. I'm not surprised you got compromised.

Sounds like the attacker removed/deleted logs. Have you checked syslog, dmesg and other logs in /var/log (I'm not familar with debian so I don't know the default structure)?

Look for hidden files/directories in root and elsewhere.

Before answering question 3, the question is what do you want to do? Legal action or not? If not, plan as to how you'll prevent this. If you want to do legal action, that box will be evidence. Alteration will make it hard to use.

What services are you running ? Are you running an IDS ? Do you keep a backup of your logs ? Do you have system integrity checkers ? And I think it's time to update your kernel...Check logs for suspicious activities...run a ps -ax just to be sure no malicious services have been started by the attacker..but beware, he could have trojaned a few commands on your system.....Just a little info for the future always secure a box before putting it on the internet, well if it isn't in a dmz that is...And for dealing with the kiddies, if he is one that is, a simple chattr +a /var/log/messages will do more wonders than you think....

I checked all logs in /var/log and found nothing.

Is ther any command to search new files made since ** and search for hidden files/dirs?
I havent thinked of legal actions. I just wanna know how he did it and how to prevent it. If he did anything serious then maybe legal actions.

Im running apache(1.3), php(4.3.2), ventrilo(voice communication server www.ventrilo.com), eggdrops, psybnc, sendmail, and maybe something else I cant remember now.

I dont have any backups of logs.
Im not sure what you mean with system integrity checkers but I think that chrootkit is one.

Thx for this fast help, any other ideas?

<Lorvija>

Just noticed that same guy had been in october 1. too. Perhaps he came back to check something?

<Lorvija>

The find command can do that.

Quote:

---

options for find command

-amin n
File was last accessed n minutes ago.

-anewer file
File was last accessed more recently than file was modified. -anewer is affected by
-follow only if -follow comes before -anewer on the command line.

-atime n
File was last accessed n*24 hours ago.

-cmin n
File's status was last changed n minutes ago.

-cnewer file
File's status was last changed more recently than file was modified. -cnewer is
affected by -follow only if -follow comes before -cnewer on the command line.

-ctime n
File's status was last changed n*24 hours ago.

---

You'd probably want something like find / -ctime 1 -exec ls -a {} \;

Chrootkit is more of a trojan detector. You'd want something like tripwire to do system integrity checks.

Your apache is quite old since it's 1.3 (newer version are up to 1.3.27 and 2.x). That might be one way in. Others include the eggdrops, php (again it's an old version) and sendmail.

Is BIND running? FTP?

And if he's come back that means the compromise was done far earlier and he's checking to see if you've fixed it or if he can continue using your server for whatever it is he's using it for (DoS, DDoS, sharing files, etc.)

BIND isnt running but I have FTP runnning. Its proftpd-1.2.8
Tried tripwire but couldnt understand how it works( im too noob apparently).

So how to check how he came in?Or does that old kernel have many security flaws?
So its just because its old?

<Lorvija>

There's a variety of things that you seem to be running that are old. The reality is that every release of kernel, application, etc. a new flaw will probably be found in a short period of time. So staying viligent and up-to-date can be helpful. Kernels older than 2.4.x are vulnerable to some serious ptrace() exploits amongst other things. I suspect however that the attacker used an application to get into the server.

By any chance was this running a live website on it? And if so, was the code tested to ensure that someone couldn't used it as a method in?

It had one php site which had one login field which php script parses and i didnt test it. So that is the only possibility what the hacker could have used, but I think it wouldnt be possible to get root pass with it

He doesn't need to get root access. Just access. Once able to setup an account, then he can continue. Have you checked your /etc/passwd for any "unusual" accounts?

Well there was users called backup and operator. Im not sure if they were created by computer. But I think they werent there.

operator is a usual account (at least on the *nixes I've seen) but I haven't seen backup. That could be a new one or a default one. What is the UID/GID for it?

If they are those numbers just after the name the are:34/34

That's most likely a default account then.. You still need to look for hidden files and you should probably check the history files for other users (not just root). Might show you some info that would indicate how he got in. Remember to write everything down that you do so you aren't repeating steps and so that you have a record of what has happened.

Ok. Is there any "usual" places they put their hidden folders?

Nope. :D

It's a hide-n-seek thing. Never assume that things are put in "usual" places. When machine is compromised, anything is possible. You do need to do some work however to find out what they did (given that some of the logs are gone).

Ahh.. great :mad: One thing is for sure, I will keep my comp updated. Would it be possible to get the lost logs back? Where I could find that kind of program?

Can be hard but a forensics program like FAUST or The Coroners Toolkit (aka TCT). Might also want to do some google searches on undeleting with ext2 (there is a howto on this). Check also Securityfocus.com for more file recovery applications.

Ok, thx. TCT is now running. Readme says it takes hours to finish that prog. So I´ll come back here when I got some more new info. Thx a LOT guys for ur help so far!!!

<Lorvija>

Ignore me if I'm wrong, but MsMittens mentioned checking your passwd file for weird logins..., that is good, but I'm wondering if there are any normal logins that suddenly got passwords.

I'm a vi-junky, so I only know vipw, but on the systems I've used you get * for accounts without password hashes and really long seemingly non-sensical alpha-numeric strings for accounts with passwords. If operator and backup have *, that's cool, if they suddenly became actual logins, then that could be a problem. Also, on some systems accounts that can't login have their shells pointing to something not allowed in the shells file. (That sounds confusing to me..., sorry)

How do you login to the box? Locally via keyboard, or do you use telnet or ssh? telnet is bad, ssh not configured correctly is bad too.

Check your environment with "env" make sure that weird aliases haven't appeared that make it so you are running programs you didn't intend to.

I'm with MsMittens though, that machine is out-of-date. If you do reinstall, make sure that you know exactly what you install next-time. Some Linux installs scare the crap out of me with there desire to install as much as possible.

My knowledge resevoir has run dry, MsMittens and many other members here are much more experienced with these sorts of things, I'm just happy with my BSD box sitting quitely behind a firewall with all incoming connections denied.

Good luck,
Dhej

One thing you can do for the future is to setup syslog to save log files to another machine do you have another machine you can use for logging? this will help when keeping the log files from being tampered with or deleted entirely also you should probably setup an IDS (intrusion detection system) something like SNORT is very good this can be used to detect any attack attempts that may occur.

if you want to undelete a file from an ext2 file system.

check out this link

Linux ext2fs undeletion

If your ftp server is not set up correctly there are exploits to hack Linux through there,also hope you arnt running ssh or telnet,Um if you have a X11 server running go to your Xservers script make sure you have the command set -nolisten tcp.Check your apache script shut down anything on it you dont need,asume running Mysql? if so change the passwords, to make things easy get the latest webmin set up for ssl and local use only.
Also of course use netstat -all find out whats open then use tcp wrappers(host.allow,hosts.deny)to block ports you dont need on the net just for local.Hope that helps.As for firewalls shorewall is the best i have found to date.

Also check /etc/inetd.conf for any abnormal entry that uses /bin/sh as its program etc....possibly looking for lkm rootkits as well