Incident Management

We had a report in yesterday that one of our servers had been compromised.

True to form I've got no experience of incident management (compromised machines) and my boss was out all day. So I got left with the authority but not the brains. The proceedure we have for 'incidents' are quite general and not a lot of help.

The machine that was suspected of being hacked was a citrix boxed used to access a finance application. The application tech was working on the machine as it had been getting a bit unstable on checking the logs he found attempted logins on the local administrator user and as several other usernames some of which do not exist.

The tech who reported the incident is known as a bit of a 'drama queen' who loves a good incident and would play it up as much as possible. On speaking with the head of our infrastructure team she played down the incident as much as possible pretty much regarding it as a non-event.

Some of the items produced as evidence of a compromise I was able to dismiss quite quickly but not the local admin attempts.

I asked for the box to be taken off of the network immediately and for a copy of the logs to be made and sent to me. The Applications tech is trawling the logs to try to narrow down a date/time for the occurrance of any attempts to get access and it going to pass these to out firewall chap who will then try to find corroberating evidence within the firewall logs.
He is also going to check the other citrix boxes on the network for evidence of tampering.

Infrastructure keep things quite tight round here we are behind heavy firewalls and other point of entry are tightly controlled, the applications and development teams are known to be more 'relaxed' with security.

(Just had a thought we had a pen test two weeks ago....)

Boss is back today and there is going to be an emergence meeting of the various groups.

Did I do ok?
What should I have done differently?

Well I have no work experience in the matter, but I have studied the matter in great detail.

What I would have done is made an image of the drive on the compromised machine, since it was taken down anyway. This image could then be used to set up the compromised host in vmware, or could be mounted and inspected as if you were actually on the machine. This would not only reveal the presence and type of any rootkits, but also make possible the recovery of files which had possibly been deleted.

Also, you could make a resonse script which would automatically gather any evidence you want like network state, current connections, new user accounts, open files, etc... The Helix livecd is designed specifically for this purpose, (however their static banaries are prone to segmentation faults) although I am unsure of how useful it would be on a Citrix machine.

My general method (in a lab, anyway) is to run a response script on the host to get as much volatile information as possible, and then, where possible, take the host off the network and image all the hard drives. Since you already took the compromised host offline, imaging the hard drives is (to me, anyway) a given.

A program which monitors log files and reports anything out of the ordinary can also be run on the compromised machine, such as logwatch

So my two suggestions would be:

1) Have a response toolkit with trusted tools and scripts to automate the process of gathering information ready
2) Once the machine is offline, image the hard drives to ease future investigation.

Drive imaging was one of the first things I thought of but I was told by powers that be that there was no easy way to do this with the facilities at hand. Again I've no experience of doing that on anything other than a single drive machine and here I wouldn't be permitted to do such a thing as I am not technical staff.

The suggestions you made all sound reasonable but again not being a tech I have to ask others to perform functions on my behalf.

I was just hoping that procedurally I'd done ok.

We had the meeting this morning and it looking increasingly like many of the attempts were accidental from other develpment areas but the applications guy is looking into it further (but at least he's calmed down a bit).

Have you got an alternative link to logwatch, I'm getting an access denied on it?

Logwatch runs on port 81 for some assanine reason, which is probably blocked at your border. Maybe I can find something not on 81 for it.

/me looks around

Yes, of course, they have a freshmeat project page at http://freshmeat.net/projects/logwatch/

And I have also attached a wget of the website for you if that helps.

1) Always gather as much evidence as possible before pulling the system offline. While log files and firewall logs, etc can and are often good enough for prosecution, actually showing the live connection on your system is quite a bit better and harder to explain away. To do this quickly and effeciently, you should already have a toolkit put together. Helix is a very good choice; however, what I don't like about it is that you have to boot the system for it to load, thus losing everything in memory and potentially overwriting very incriminating evidence.

2) Absolutely avoid anything that would alter the system. No unnecessary logging in/out, running programs, changing configs, deleting/moving/renaming files, everything should remain as unchanged as possible until evidence can be gathered (another reason not to disconnect it from the network until evidence is gathered). Any evidence gathering tools that you use should absolutely be used from removeable media and any data should be written to removeable media...absolutely avoid writing anything to the hdd. You should also take the time to compute MD5 sums on the images and record those, they can be useful in proving the gathered information has not been altered.

3) Timeline. Record everything that happens, noting the time, who you talked to, what was found, etc. They are very useful in prosecuting cyber crimes and are also good for your own reference to make sure you haven't overlooked something.

4) Tools ? There are tons...the two I use the most are memdump (for gathering in memory data) and dd (there are windows and unix versions). If you don't have a removeable hdd big enough to store all the data, put netcat on your incident toolset CD and use it to transfer the data (I can elaborate if needed, but this is the way recommended in the SANS Track 8 class) over the network. Once you obtain a valid image, make a copy and play with the copy, don't touch the original. There are of course other scripts and tools that can be run to get other things (like Fport, pstools, WFT, TCT, etc) that can gather alot of information about processes, ports, logs, users, netusers, and files being accessed at the time, most of which are run from the command line and can be strung together using batch files (or shell scripts in a unix environment)...once again, write all data to an external drive...don't write to the hdd.

5) Once you get your data, leave the system up but disconnect the network cable and connect to an isolated hub/switch (in case any of the hacker's kits look for a down connection) and try to leave it alone.

You now have the system in a pristine state with very good information gathered that you can analyse and you have very minimally altered the system state (running the processes does change it some). For messing around with the disk image, I recommend Autopsy...but there are other tools like ncase that can do the same thing (commericial software) and more.

So general methodology, gather as much evidence first with the goal of altering as little as possible, then analyze the data gathered and leave the system in question as isolated as possible.

The incident response scripts on the Helix CD can actually be run without booting off it. Of course, if the attacker has hidden stuff in the directory where you mount the CD, this will disappear. The provided script uses the binaries on the CD, however some work may be required to do so.

And I'd never thought of disconnecting the system and reconnecting it to a hub to check for activity which may be triggered by a downed connection. Good idea there Nebulus.

/me suggests moving this thread to the Forensics forum

Looks like Tiger Shark updated his thread on this subject :

http://www.antionline.com/showthread...hreadid=255811

good luck. Let us know what you find

~Halv

Most the information that has come back to me today is point towards a lot of the evidence being generated by internal errors. Developers pointing at the wrong machines etc.
I'm getting a crash course on the firewall on Monday so I'll have a go at the logs then.

I'll certainly be reading TS tut very carefully and I'll try to develop a less general procedure to follow.

Aspman,

A possible solution to imaging in your scenario would be to use an external usb hard drive large enough to fit the drive you're imaging. The compromised system could be booted with a livecd and the drive could be imaged without ever taking it out of the machine. You then have a portable image ready to go where you need it.

This may be a bit redundant at this point, but maybe you should bring it up next time you're told that there's "no easy way" to image a drive. I assume they mean by this that removing hard drives and imaging them that way is not easy. This solution might work for you. Of course if you need to image huge drives the size of which usb drives haven't been made yet, that's a whiole other bag of beans.

Even if you don't have the technical expertise to do this, I'm sure somebody there does.

Just an idea to throw at you.

Had a discussion with the boss late on Friday.
Unofficially the org has a policy of not aiming to prosecute intruders into the system.

The process of gathering forensics (of a quality suitable for court) is too disruptive and expensive considering the likelyhood of any prosecution is negligable.
The aim of incedent response (for us anyway) is to confirm if an intrusion has occurred, how that intrusion occurred, what the intruder did while in the system and identifying what steps are needed to restore the system to a secure working state.


A lot of the info you've given is still applicable for the process of identifying the weakness in the system but for us gathering evidence for prosecution is way down the list of priorities. I suppose by default if gathering evidence for prosecution is not top of your list then it really isn't on the list at all.

Quote:

---

I suppose by default if gathering evidence for prosecution is not top of your list then it really isn't on the list at all.

---

Absolutely......

Quote:

---

Originally posted here by Aspman
Had a discussion with the boss late on Friday.
Unofficially the org has a policy of not aiming to prosecute intruders into the system.

The process of gathering forensics (of a quality suitable for court) is too disruptive and expensive considering the likelyhood of any prosecution is negligable.
The aim of incedent response (for us anyway) is to confirm if an intrusion has occurred, how that intrusion occurred, what the intruder did while in the system and identifying what steps are needed to restore the system to a secure working state.


A lot of the info you've given is still applicable for the process of identifying the weakness in the system but for us gathering evidence for prosecution is way down the list of priorities. I suppose by default if gathering evidence for prosecution is not top of your list then it really isn't on the list at all.

---

I think you guys are thinking the imaging process is actually more complicated than it really is and are thinking taking an image may not be helpful...

If you have all your stuff together and the disk isn't that big, you can image a drive in a few minutes...assuming a fast external drive (they are dirt cheap now...150 $US for a 120GB external HDD?) or a good network connection to another sytem (using netcat). Even if the disk is enormous, imaging still doesn't take all that long... If you are not shooting for a prosecution (bad idea IMHO, but to each organization its own) this really only effects what you do with the compromised system afterwards ... my best recommendation in that respect is to rebuild/restore it from backups to a date prior to the incident and then any data that was missed since, to review it carefully before adding it back (for evidence of trojaned files or altered data). Unless you rebuild it you can't be garunteed to have complete control of the system back.

But back to the original point, the image of the compromised system is INVALUABLE for evaluating what really happened...you can tell what files changed recenty, view DELETED files, see what files were altered and in what order and generally tell EXACTLY what happened to the system...without this you are merely guessing and hoping you see it all. In a UNIX environment this can be especially interesting since everything in UNIX is essentially a file...and it is quite useful in a Windows environment as well.

Can't stress it enough...taking an image does not require removing the HDD (please note you must make sure you have good binaries that haven't been altered (ie, rootkit, which is why an incident response kit is suggested) and in some cases it may be necessary to boot off of a CD to make sure the rootkit isn't interfering (which is where Helix shines) ) and it is very useful in determining exactly what happened...rootkit or no...deleted files or no.

EDIT: Forgot to add, about the only time I would consider NOT imaging the system was if it was a massive file server or SAN/RAID and then the requirements for storage of the image are alot higher...still very useful though and would image if at all possible.

Hello -

Quote:

---

Had a discussion with the boss late on Friday.
Unofficially the org has a policy of not aiming to prosecute intruders into the system.

The process of gathering forensics (of a quality suitable for court) is too disruptive and expensive considering the likelyhood of any prosecution is negligable.
The aim of incedent response (for us anyway) is to confirm if an intrusion has occurred, how that intrusion occurred, what the intruder did while in the system and identifying what steps are needed to restore the system to a secure working state.


A lot of the info you've given is still applicable for the process of identifying the weakness in the system but for us gathering evidence for prosecution is way down the list of priorities. I suppose by default if gathering evidence for prosecution is not top of your list then it really isn't on the list at all.

---

Just curious - but was the incident confirmed as being an internal or external?

Also, as far as procedures go - does your organization have a process/procedure to handle these types of incidents? Or would that fall under DRP/BCP?

Don't know yet. There should be a follow up meeting today or tomorrow when I should get some more details.

The procedures that are in place are a bit general. They are for any incident that covers the CIA rather than specifically for intrusions.

These incidents are very rare here, 1 intrusion and 1 virus outbreak in the last 3 years , that I've been told about (I've been here 6 months).

Note to self:

Check SANS!

http://www.sans.org/rr/whitepapers/incident/

Another one if any is looking:

http://www.forensicfocus.com/chain-of-evidence-model