We had a report in yesterday that one of our servers had been compromised.
True to form I've got no experience of incident management (compromised machines) and my boss was out all day. So I got left with the authority but not the brains. The proceedure we have for 'incidents' are quite general and not a lot of help.
The machine that was suspected of being hacked was a citrix boxed used to access a finance application. The application tech was working on the machine as it had been getting a bit unstable on checking the logs he found attempted logins on the local administrator user and as several other usernames some of which do not exist.
The tech who reported the incident is known as a bit of a 'drama queen' who loves a good incident and would play it up as much as possible. On speaking with the head of our infrastructure team she played down the incident as much as possible pretty much regarding it as a non-event.
Some of the items produced as evidence of a compromise I was able to dismiss quite quickly but not the local admin attempts.
I asked for the box to be taken off of the network immediately and for a copy of the logs to be made and sent to me. The Applications tech is trawling the logs to try to narrow down a date/time for the occurrance of any attempts to get access and it going to pass these to out firewall chap who will then try to find corroberating evidence within the firewall logs.
He is also going to check the other citrix boxes on the network for evidence of tampering.
Infrastructure keep things quite tight round here we are behind heavy firewalls and other point of entry are tightly controlled, the applications and development teams are known to be more 'relaxed' with security.
(Just had a thought we had a pen test two weeks ago....)
Boss is back today and there is going to be an emergence meeting of the various groups.
Did I do ok?
What should I have done differently?
May 27th, 2005, 12:16 PM
Well I have no work experience in the matter, but I have studied the matter in great detail.
What I would have done is made an image of the drive on the compromised machine, since it was taken down anyway. This image could then be used to set up the compromised host in vmware, or could be mounted and inspected as if you were actually on the machine. This would not only reveal the presence and type of any rootkits, but also make possible the recovery of files which had possibly been deleted.
Also, you could make a resonse script which would automatically gather any evidence you want like network state, current connections, new user accounts, open files, etc... The Helix livecd is designed specifically for this purpose, (however their static banaries are prone to segmentation faults) although I am unsure of how useful it would be on a Citrix machine.
My general method (in a lab, anyway) is to run a response script on the host to get as much volatile information as possible, and then, where possible, take the host off the network and image all the hard drives. Since you already took the compromised host offline, imaging the hard drives is (to me, anyway) a given.
A program which monitors log files and reports anything out of the ordinary can also be run on the compromised machine, such as logwatch
So my two suggestions would be:
1) Have a response toolkit with trusted tools and scripts to automate the process of gathering information ready
2) Once the machine is offline, image the hard drives to ease future investigation.
May 27th, 2005, 12:44 PM
Drive imaging was one of the first things I thought of but I was told by powers that be that there was no easy way to do this with the facilities at hand. Again I've no experience of doing that on anything other than a single drive machine and here I wouldn't be permitted to do such a thing as I am not technical staff.
The suggestions you made all sound reasonable but again not being a tech I have to ask others to perform functions on my behalf.
I was just hoping that procedurally I'd done ok.
We had the meeting this morning and it looking increasingly like many of the attempts were accidental from other develpment areas but the applications guy is looking into it further (but at least he's calmed down a bit).
Have you got an alternative link to logwatch, I'm getting an access denied on it?
May 27th, 2005, 01:09 PM
Logwatch runs on port 81 for some assanine reason, which is probably blocked at your border. Maybe I can find something not on 81 for it.
And I have also attached a wget of the website for you if that helps.
May 27th, 2005, 01:51 PM
1) Always gather as much evidence as possible before pulling the system offline. While log files and firewall logs, etc can and are often good enough for prosecution, actually showing the live connection on your system is quite a bit better and harder to explain away. To do this quickly and effeciently, you should already have a toolkit put together. Helix is a very good choice; however, what I don't like about it is that you have to boot the system for it to load, thus losing everything in memory and potentially overwriting very incriminating evidence.
2) Absolutely avoid anything that would alter the system. No unnecessary logging in/out, running programs, changing configs, deleting/moving/renaming files, everything should remain as unchanged as possible until evidence can be gathered (another reason not to disconnect it from the network until evidence is gathered). Any evidence gathering tools that you use should absolutely be used from removeable media and any data should be written to removeable media...absolutely avoid writing anything to the hdd. You should also take the time to compute MD5 sums on the images and record those, they can be useful in proving the gathered information has not been altered.
3) Timeline. Record everything that happens, noting the time, who you talked to, what was found, etc. They are very useful in prosecuting cyber crimes and are also good for your own reference to make sure you haven't overlooked something.
4) Tools ? There are tons...the two I use the most are memdump (for gathering in memory data) and dd (there are windows and unix versions). If you don't have a removeable hdd big enough to store all the data, put netcat on your incident toolset CD and use it to transfer the data (I can elaborate if needed, but this is the way recommended in the SANS Track 8 class) over the network. Once you obtain a valid image, make a copy and play with the copy, don't touch the original. There are of course other scripts and tools that can be run to get other things (like Fport, pstools, WFT, TCT, etc) that can gather alot of information about processes, ports, logs, users, netusers, and files being accessed at the time, most of which are run from the command line and can be strung together using batch files (or shell scripts in a unix environment)...once again, write all data to an external drive...don't write to the hdd.
5) Once you get your data, leave the system up but disconnect the network cable and connect to an isolated hub/switch (in case any of the hacker's kits look for a down connection) and try to leave it alone.
You now have the system in a pristine state with very good information gathered that you can analyse and you have very minimally altered the system state (running the processes does change it some). For messing around with the disk image, I recommend Autopsy...but there are other tools like ncase that can do the same thing (commericial software) and more.
So general methodology, gather as much evidence first with the goal of altering as little as possible, then analyze the data gathered and leave the system in question as isolated as possible.
May 27th, 2005, 02:07 PM
The incident response scripts on the Helix CD can actually be run without booting off it. Of course, if the attacker has hidden stuff in the directory where you mount the CD, this will disappear. The provided script uses the binaries on the CD, however some work may be required to do so.
And I'd never thought of disconnecting the system and reconnecting it to a hub to check for activity which may be triggered by a downed connection. Good idea there Nebulus.
/me suggests moving this thread to the Forensics forum
May 27th, 2005, 02:58 PM
Looks like Tiger Shark updated his thread on this subject :
Most the information that has come back to me today is point towards a lot of the evidence being generated by internal errors. Developers pointing at the wrong machines etc.
I'm getting a crash course on the firewall on Monday so I'll have a go at the logs then.
I'll certainly be reading TS tut very carefully and I'll try to develop a less general procedure to follow.
May 28th, 2005, 10:43 PM
A possible solution to imaging in your scenario would be to use an external usb hard drive large enough to fit the drive you're imaging. The compromised system could be booted with a livecd and the drive could be imaged without ever taking it out of the machine. You then have a portable image ready to go where you need it.
This may be a bit redundant at this point, but maybe you should bring it up next time you're told that there's "no easy way" to image a drive. I assume they mean by this that removing hard drives and imaging them that way is not easy. This solution might work for you. Of course if you need to image huge drives the size of which usb drives haven't been made yet, that's a whiole other bag of beans.
Even if you don't have the technical expertise to do this, I'm sure somebody there does.
Just an idea to throw at you.
May 30th, 2005, 11:24 AM
Had a discussion with the boss late on Friday.
Unofficially the org has a policy of not aiming to prosecute intruders into the system.
The process of gathering forensics (of a quality suitable for court) is too disruptive and expensive considering the likelyhood of any prosecution is negligable.
The aim of incedent response (for us anyway) is to confirm if an intrusion has occurred, how that intrusion occurred, what the intruder did while in the system and identifying what steps are needed to restore the system to a secure working state.
A lot of the info you've given is still applicable for the process of identifying the weakness in the system but for us gathering evidence for prosecution is way down the list of priorities. I suppose by default if gathering evidence for prosecution is not top of your list then it really isn't on the list at all.