I think I've been 0wn3d...

Printable View

September 17th, 2002, 04:15 AM
NetSec

I think I've been 0wn3d...

Hi all!

I'm pretty sure my linux box (RH7.1) has been compromised, and I'm writing this as I'm trying to pull down 1.1GB of data off of it before it gets smoked...

I logged on this weekend to check something or other, and noticed that someone was on the same time as me... 1st indication something was not right... Then my history was smoked, and my lastlogin info... D@mmit... Tried to run chkrootkit, and it would just hang at checking "aliens", and go no further... Nuther bad sign...

Then tonight I found a sh.zk.cgi in my cgi-bin while doing a:
find / -user root -perm -4000 -print
Don't know what the h*ll that is...
(Should I rm it, or leave it til I get my files off??? I don't want p!ss off whoever has as much control over my box as me... Doesn't that suck...)

Double d@mmit...

Okay, so now that I've told my sob story, and am preparing to have this box reloaded, what should I be looking for to find out what happened??? chkrootkit made it through all the file checks saying all the binaries were clean, and my logs appear clean (or maybe have been cleaned)... I know I'll not be able to hunt the guy who did this, but I sure would like to know where my vulnerablities are... Is there some other chrootkit type app that might run when chkrootkit wont???

I know, first time poster, tons of questions... Sorry... (But you guys know everything, right??? :) )

James...
September 17th, 2002, 04:54 AM
THEJRC

best recommendation, yank whatever internet connection you have to that box. You can always retrieve the files via lan.

also, hopefully you used some sort of file checking application (such as tripwire) but chances are your data for that isnt updated... double check every file you decide to keep, burn the box down as quick as possible, and get on with life. You may look into some IDS systems with a more proactive approach in the future. but the fact is... sh*t happens eh.....
September 17th, 2002, 05:33 AM
NetSec

Thanks for the quick reply THEJRC!!!

If I could, I would pull the plug on it, but it is a dedicated box at a "major hosting" place, and when I called them RE: this, they said they would have their SAs look at it... Then when I called them back again, they were like "Didn't we just tell you that we'd have our SAs take a look at it? They should be in in the morning..."... I'm pretty sure that if I took it down myself, they'd react by rebooting it... The night crew isn't exactly on the ball...

Now I just noticed all my /bin files are now dated today... Great... ps, netstat, everything running cracked...

I guess I'm learning the hard way what preventative steps I _should_ have taken to reduce the chances of being in this position... Now I just have 5 bajillion PHP scripts to wade through looking for alterations...

Reminds me of when I got CIH virus a few years ago... Never really had need to be concerned about viruses... Your future actions are usually based upon what's happened in the past... I sure as hell am going to do everything I can to prevent this from happening again... There probably aren't too many people interested in intrusion prevention who didn't get into it by being cracked... Am I right??? Well, maybe not...

Lot to learn, lot to learn... TripWire=Good, FTP services=BAD... Much to go...

Thanks...

James...
September 17th, 2002, 05:47 AM
droby10

pull the plug - you've found the evidence, now you need to analyze without interruption. look at time stamps for the entire system based on the ctime of the setuid cgi script. were the web logs tampered with? if not, then what is the relevance of the entries there (including timestamps for access to that cgi verses the responses from your find by date output)?

i wouldn't completely trust ps, ls, or netstat output. do a file hash comparison against a clean box with the same build and patch level (if available).

==========
edit...heh i was right on the rootkit...just a little late in the response.

also by all means do not delete the cgi.
September 17th, 2002, 05:58 AM
NetSec

What happens if I delete the CGI??? Other than they will be aware that I'm aware??? I googled the sh.zk.cgi and found nothing... Can't really tell what it is as it is a binary, and I have no desire to copy it to a clean box to see what it does... Maybe after I take it down, I'll copy it and take a look at it... Judging by the name, it provides a shell through a browser???

Thanks for your replies...

James...
September 17th, 2002, 06:56 AM
problemchild

I agree with the others. Get that machine off the net ASAP because it's a hazard to the rest of the Internet. You can make an image of the hard drive and study it later to find out what happened.

This won't help you with your current situation, but when you get your system reloaded you can make the system much more difficult to rootkit by using chattr to make your binaries like ps, netstat, ls, du, top, etc. immutable. Then you can remove the chattr command from the system by putting is on a floppy disk so an intruder can't reset the attributes.
September 17th, 2002, 07:11 AM
NetSec

Thanks for the tip on chattr... Any info to help prevent a recurrence is greatly appreciated...

This may be a stupid question, but if somebody had an exploit to gain root access to a box, could they copy chattr to the box and undo it??? Or will chattr-ing these files prevent the root access???

As you may be able to tell, my copy of "Counter Hack" is still in the mail... :) What timing, I gain an interest in security, and days later I'm pushed into the "accelerated program"...

Thanks problemchild!

P.S.: Does anyone know why not to delete the CGI???
September 17th, 2002, 07:48 AM
problemchild

Yes, a knowledgeable attacker can definitely replace the command, but the name of the game is to make it more trouble than it's worth. Most attackers are looking for easy targets and will move on if they hit something like that that's too much trouble.

But that's all for later. Right now you need to get the hard drive archived somehow for forensic analysis later. I think droby was saying not to delete the CGI because the timestamp on it will tell you roughly when your box was compromised and you can use that as a reference point for looking at timestamps on the rest of your system.

My advice: Pull the plug immediately and don't touch anything. Archive the drive exactly as it is right now so you can study it at your leisure later. Don't do anything that might alert the intruder that you're onto him, because he might decide to rm -rf / and delete any evidence on the drive. Then you can rebuild the box and get it back online quickly.
September 17th, 2002, 05:59 PM
NetSec

Still waiting for my host to pull the box to reload... I've called numerous times, and their attitude sucks... And they're only gonna charge me $175 to slap a new image on it... Grrr... I'm getting tempted to do the rm -rf / myself...

I tried to pull down the CGI, and NAV wouldn't let me, stating that it was infected with Linux.RST.A... SARC has no info on it though... Remote Shell Trojan???

So here's what I'm gonna do once I get reloaded... Please let me know if I'm missing something...

1. Disable Telnet
2. Disable FTP (will use sftp through ssh?)
3. Setup IPTABLES to block all ports but 22, 80, 443, 25, & 110 ???
4. Reinstall all IDs w/ new passwords
5. Reload all data & tables (checking PHPs for changes)
6. chattr my binaries

Should this be enough to keep kiddies out while I'm figuring out how to use Tripwire & LIDS???

Does mysql need to have a /bin/bash account?

Don't you like it when I cram 38 lame questions in 1 post?

Thanks everyone for your help...

James...
September 17th, 2002, 08:00 PM
droby10

sorry for the delay...been away.

yes, i was saying not to delete it for two reasons both relating to maintaining the state of the system. 1) the timestamp as noted and 2) the contents of the file. seeing that you've already identified the contents if you are satisfied with knowing what it is instead of how it works then document the timestamp and delete...but if you find other changes to the system and wonder what part of the attack it was (pre-root, post-root, to-get-root) you'll be minus one element of evidence (there may be several features that can be traced back to this cgi).

your measures for restoration are fine...but they do they take into account how this attack occured? have you been able to locate the problem? without trustworthy logs, this will be difficult. if you could sandbox everything, set it back up exactly the way it was...then you could find out the next go around. if it's a new exploit, then it'll become more public and frequently used...if it's an old exploit, then i would focus on patching proceedures.
September 17th, 2002, 08:24 PM
NetSec

droby10:

I'm not quite sure what the point of entry was, but I'm going to make certain that all the apps running are the latest version w/ patches, etc...

I've been looking at some of the logs and am finding some clues perhaps...

Like this:
Sep 14 15:16:51 DHP01103 kernel: mIRKfORCE-glibc uses obsolete (PF_INET,SOCK_PACKET)
Sep 14 15:16:51 DHP01103 kernel: device eth0 entered promiscuous mode
Sep 14 15:18:50 DHP01103 kernel: device eth0 left promiscuous mode
Sep 14 15:19:31 DHP01103 kernel: device eth0 entered promiscuous mode
-- That doesn't look good... Any ideas???

And this:
Sep 14 14:16:30 DHP01103 portsentry[601]: attackalert: Connect from host: 211.121.xxx.xxx/211.121.xxx.xxx to TCP port: 111
Sep 14 14:16:30 DHP01103 portsentry[601]: attackalert: Ignoring TCP response per configuration file setting.
-- That certainly doesn't look good... Dunno why the config file says to just ignore... Any ideas???

I think I had also seen some :443 connect errors, which leads me to believe that it may be the exploit w/ my 0.9.6a OpenSSL... I'll put the latest ver. on new build...

I wish I had the time to figure it all out before I have to rebuild... But unfortunately, that's not an option... I just hope I can fill enough holes to keep it clean & alive while I figure all this out...

Thanks again...

James...
September 17th, 2002, 09:02 PM
droby10

Quote:

Sep 14 15:16:51 DHP01103 kernel: mIRKfORCE-glibc uses obsolete (PF_INET,SOCK_PACKET)
Sep 14 15:16:51 DHP01103 kernel: device eth0 entered promiscuous mode
Sep 14 15:18:50 DHP01103 kernel: device eth0 left promiscuous mode
Sep 14 15:19:31 DHP01103 kernel: device eth0 entered promiscuous mode
-- That doesn't look good... Any ideas???

the guys hosting your site should have acted quicker...i'd notify them that anyone logging into anywhere in that network segment between now and the time stated above needs to change their password asap - they could have been broadcast to any number of irc boards, or just to a single irc server operated or used by the attacker.

Quote:

And this:
Sep 14 14:16:30 DHP01103 portsentry[601]: attackalert: Connect from host: 211.121.xxx.xxx/211.121.xxx.xxx to TCP port: 111
Sep 14 14:16:30 DHP01103 portsentry[601]: attackalert: Ignoring TCP response per configuration file setting.
-- That certainly doesn't look good... Dunno why the config file says to just ignore... Any ideas???

could have been an rpc exploit.

but if the timestamps for the ssl attack are between the promiscous and the above, then that is likely the problem. ssl flaws have been publicized greatly over the last few weeks - so it's a hotspot.
September 17th, 2002, 11:32 PM
detoxsmurf

There's a lot of good information provided on this topic already by forum members. One thing I didn't see suggested was a scan of localhost using nmap. If the attacker opened a backdoor on a high number port you would be able to see it. If the attacker got root its hard to recommend a simple solution as all files are suspect including the kernel.
September 18th, 2002, 06:24 AM
THEJRC

eh ftp services arent necessarily bad,

however configuring FTP in the most efficient and secure way can be a pain in the ass!! set up FTP and web services on a bastion host (unsecured host) and well... plan on it getting hacked... that way if it does get whacked... your out nothing.
September 18th, 2002, 03:50 PM
NetSec

You're not going to believe this, but my box has STILL not been taken offline... My incident has "been sent to the provisioning department" and should be addressed this morning... Two days after I asked/told them to take it off the network!!! I've called them many times, and have been hung up on a few times... Their slogan for their hosting side should be "Dude, you're getting a trojan", or something to that effect... I'm not naming names though...

droby10:
I sent them snips of my logs and snips of your comments regarding their network being at risk, and was sent a reply:

"Please be advised that if you want us to do any sort of extra security work on your system, there will be a $175/hr charge."

I haven't spoken to anyone there who would be worth $1.75 an hour, and they do not seem concerned that their network is at risk...

detoxsmurf:
An Nmap currently shows the following... I think they're gonna run out of ports soon...

1/tcp open tcpmux
11/tcp open systat
15/tcp open netstat
21/tcp open ftp
22/tcp open ssh
23/tcp open telnet
25/tcp open smtp
79/tcp open finger
80/tcp open http
81/tcp open hosts2-ns
110/tcp open pop-3
119/tcp open nntp
143/tcp open imap2
443/tcp open https
540/tcp open uucp
587/tcp open submission
635/tcp open unknown
1080/tcp open socks
1524/tcp open ingreslock
2000/tcp open callbook
3306/tcp open mysql
6006/tcp open X11:6
12345/tcp open NetBus
12346/tcp open NetBus
31337/tcp open Elite
32771/tcp open sometimes-rpc5
32772/tcp open sometimes-rpc7
32773/tcp open sometimes-rpc9
32774/tcp open sometimes-rpc11
54320/tcp open bo2k

Could there be any more reason to take it down???

THEJRC:
By disabling FTP, and only using SFTP over 22 the passwords should never be in clear text, right??? I'm not sure how to set up FTP without sending clear text passwords, but from your post it sounds like that may be difficult... Please let me know if I'm wrong... If any PWs are sent clear text, like through FTP or Telnet, they could be sniffed out from a compromised box in my network segment??? Not sure, but that's what I gathered from droby10's post...

Thanks again for all your help...

James...
September 18th, 2002, 09:04 PM
NetSec

Hey, they finally got my box reloaded... Then I Nmap'd it before I did anything else, and guess what??? I still have 2 NetBus ports and the bo2k port open...

Does this mean that they just reloaded without wiping the drive??? Or are there valid reasons for running services on 12345/6 & 54320??? I read about TrendMicro using 12345 for their virus scanner, but I'm pretty sure this box isn't running that... :)

Is it possible somebody cracked right back into the fresh box (RH7.3 w/ iptables) in a matter of a few minutes, or is it more likely they just rm -rf /'d it instead of formatting???

I've just started moving some of our accounts off to shared servers w/ another host... Didn't want to do that, but these guys are jackballs...
September 19th, 2002, 07:26 PM
detoxsmurf

NetSec I have to admit that I am concerned by your nmap scans. Did you run these scans from your webserver (behind there firewall) or from your home computer? Sysadmin 101 teaches you to disable stuff like finger. Also I would not have much faith in sysadmins that use an X server to administer your website. I'm afraid to ask what version of bind and sendmail they are running. You may want to ask them what is running on those ports. Perhaps they are running some sort of monitoring or even RAT tool that actually uses those ports.
September 20th, 2002, 04:22 AM
guero61

Not sure how late this is, but there are a few ways you can disconnect [hard] from the net without actually touching the box; in order of increasing desperation (for next time).

1. /etc/rc.d/init.d/network stop (unless hacked, should stop all network resources).
2. use ifconfig to take the ethernet (?) interface down
3. If using lkms (assumed since you're using RH), you may be able to unload the ethernet module using rmmod.

About the x-sessions to administer a system -- forgive my ignorance here, but I was under the impression that if you opened X over a properly encrypted ssh session, X was encrypted too... If not, I guess it's good I only do this over a private subnet!
September 21st, 2002, 05:11 AM
NetSec

Thanks for all your help everyone... I finally gave up on even trying with these guys, and cancelled my contract with them... Had to move a friggin ton of accounts over to shared hosting elsewhere... I don't think this will work out though as the servers I'm on are slowww dogs...

Anyone know of a good source for dedicated servers, where they have some idea about networking/security/etc??? Anyone says "D*llHost" and I'll reach through the screen and strangle you!!! :)

detoxsmurf: The scans were run from home... Still messed up... Still up... (And from what I can tell, their monitoring consists of FTP connects every 5 mins. No connect = you're down)

guero61: Thanks for the tips... I'll jot those down for next time... I'd prolly get in trouble if I connected to a box I'm no longer paying for and smoked eth0...

Thanks again...

James...
September 22nd, 2002, 10:53 PM
Ratman2

Quote:

Originally posted here by NetSec
Thanks for all your help everyone... I finally gave up on even trying with these guys, and cancelled my contract with them... Had to move a friggin ton of accounts over to shared hosting elsewhere... I don't think this will work out though as the servers I'm on are slowww dogs...

Anyone know of a good source for dedicated servers, where they have some idea about networking/security/etc??? Anyone says "D*llHost" and I'll reach through the screen and strangle you!!! :)

detoxsmurf: The scans were run from home... Still messed up... Still up... (And from what I can tell, their monitoring consists of FTP connects every 5 mins. No connect = you're down)

guero61: Thanks for the tips... I'll jot those down for next time... I'd prolly get in trouble if I connected to a box I'm no longer paying for and smoked eth0...

Thanks again...

James...

Try these guys....http://www.powweb.com/ I believe they offer dedicated hosting if you ask for it. Give them a call.