Hackattack Response #2: STEP by STEP forensics for Linux and Windows

This has got potential to be a really good thread.

Let's imagine the following scenario.

One of your Red Hat servers has been compromised. The hacker has got root access. You want to know how did he get in.

What are the steps and methodology that you apply as soon as you are sitted in the console in order to get the big picture as fast as possible? Have you got this scripted?

EDIT: Let's assume this has to be a live analysis, we cannot afford to take it offline as it's a production server and every minute the server is down is worth several thousand dollars in cost.

My steps. Feel free to add yours.

[redhat@linux FC4]# w
[redhat@linux FC4]# who
[redhat@linux FC4]# ps -aux
[redhat@linux FC4]# lsof -i
[redhat@linux FC4]# netstat -an
[redhat@linux FC4]# dmesg
etc...

I would probably take the machine offline, pull the hdd, mount it in another machine with noexec, readonly, nosuid and nodev, then dd a backup off it and hash the backup to verify its contents later. Then I would copy the dd'd backup to another machine mount it noexec... and start your forensics with your favorite set of tools. If you had a network ids up somewhere on the same network you may be able to find out exactly how they got in. With the copy of the filesystem you should be able to find out a lot about what they did once on the machine.

Initially, you said the hacker has root; therefore, you are wasting your time typing those commands since you should operate under the assumption a rootkit has been applied/installed. You should have a CD on standby with statically linked versions of those commands, that way its a little harder for the commands to be hijaacked. Regardless, before I turned the system off, I'd use memdump to dump out everything thats running to removeable media. Browsing through memory can provide useful places for you to begin your search when you do go through the filesystem.

After you get the memdump, you can use something like helix to boot from to get your image of the hdd. After that, as h3r3tic said, mount the image (though I'd add to mount it read-only and to take a md5 hash of both the memory image and the hdd image and would also consider making two copies and only playing with the second). You can then go to town looking for the evidence (helix comes with several tools) but you can also use something like autopsy (if you don't have the funds for something like encase) and use it to browse the not only the file system but slack space and through deleted files and for looking at MAC times (modify/access/created), which can be very useful for backtracking what the hacker did while in the system.

Good tips there.

But I forgot to include an IMPORTANT detail here.

This is a production server, we are dealing with a critical server with a big database for an ecommerce application. Every minute that the server is offline costs the company several thousand dollars.

From a business perspective, we cannot simply afford to take it offline, even assuming root compromise or that a rootkit has been installed.

The first approach is to identify the incident, then contain it and then eradicate it. While always keeping it online, unless we detected that a major event has happened, such as credit numbers stolen.

So let's assume we cannot take it offline and we have to find out as fast as possible what has happened and how to contain it.

Let's see what are the steps and methodology we'd follow so that we can add detailed actions, tips, tricks and so on to the process.

Hi devantionline

Are you a former Enron employee?

Quote:

---

This is a production server, we are dealing with a critical server with a big database for an ecommerce application. Every minute that the server is offline costs the company several thousand dollars.

---

You have now transcended IT, and your decision making ambit..............time to call in the corporate lawyers and look at damage limitation exercises..............as well as your government law enforcement agencies?

You should have had a redundant back up server(S) shouldn't you?

If you are the type who believes in fighting fires on the fly, then I would FIRE you! :p

Your advice is verging on the criminally cretinous............you have obviously never had a proper job.

:D

Quote:

---

Your advice is verging on the criminally cretinous............you have obviously never had a proper job.

---

Judging by the scenario, most likely a student with an assignment due Monday....

:D

Quote:

---

Judging by the scenario, most likely a student with an assignment due Monday....

---

Could be?..............hell a "compromised " e-commerce server, and leave it online :eek:

Take it down immediately; or get sued by everyone who has their identity, credid card, bank account and other details stolen?

That is why you pay corporate taxation..................for the FEDs?

And if you p1$$ about like he is suggesting, you are compromising a CRIME SCENE!

Ack! phtt!

:D

Quote:

---

Originally posted here by nihil
Take it down immediately; or get sued by everyone who has their identity, credid card, bank account and other details stolen?

---

EXACTLY! Why in the world would you leave that timebomb online?

I think Relyt was right, it may be an assignment that is due soon. If this is a real e-commerce server, please let me know which company this is so I never use their site again! Cripes!

-Deeboe

Could someone do us all a favor and move this to tech humor.....

Obviously we have a young wo/man with a healthy imagination and the desire to be a hero....

Methinks, when s//herhe takes his high school computer science class s/he will see the error of his/her ways.... ;)

Wow, I think I've seen everything now. A newbie with a first post, posting way outta his/her depth. Getting a huffy replies from _nihil_, for crying out loud. AND, in serious danger of being eaten by Tiger Shark.

I can die in peace now.

;)

Well lets see, taking my limited security and pc forensic knowledge into account: it seems to me that the server is already pretty messed up if someone got root access. Even with my limited know-how i would have a back up server ready to go, and since i am always looking for a reason to smash things, i recently let a container of frosting go bad so i could use it as a baseball, i would most likely back up what i could/needed, take the comprimised server off-line, put the backup on-line, make a new back-up sever, then go out back and play full contact golf with the old server. problem fixed.

Quote:

---

Originally posted here by whizkid2300
With the server still on. The attacker has access to it. They control the box.

---

Instead of pulling the network connection you say turn the power off? Wow good idea, that way if they put something in init to erase any trace of them it's gone!

Quote:

---

It is better to drop a server for 3-5 hours to find out what's wrong with it, fix it and put it back up.

---

I don't think anyone is going to disagree with me that this is horrible advice at best. Root meaning they have root, back up whats needed and reformat. "Fixing" a box that could have been modified with a new backdoored ls and kernel...

Quote:

---

And w and who do the same ****.

---

Uhhhh no they don't. Who lists who's on the machine and what they are using. W lists uptime and a more brief spot on what's being used.

Gore:

Quote:

---

And if you take it down and I've patched the Kernel with a back door, no you can't fix it and pop it back up

---

why not?
I would think that anything that can be changed like that can be fixed as long as you know the premises behind the patchs logic. Granted i gues you would have to know there is a back door there in the first place, but wouldn't it still be possible, albeit harder, to fix and put back up?

It might be possible, but every Unix security book I own says if root is compirmised, you should back up what you need and format it because you can't tell what has beenpatched to let them back in with a back door. Ls for example and the Kernel are targets I've seen before. It's very risky to try and use a box after root has been taken over. You'd probably have to go over the source code of every app on the server...

So basically its possible, but so time consuming and next to impossible to assure a complete fix that you are better off just backing up starting over.

Here's a question for ya, can a kernal be replaced? I ask this question under the assumption that the kernal is like built in proc, if this assumption is wrong please correct me.

The amount of work replacing a Kernel, as i said before, you'd be better off just formatting, WAY less time, and this is a server mind you, so you'd want the quickest solution possible, to get it back up.

Well i you have the means to image the drive and work on an exact copy, Yes pulling the power cable would be a possible solution, and as you quite rightly said though, pulling a network cable, or the power one, I mean that HD is going to get turned back on eventually and if they added this to init:

rm -rf /

or

rm -rf /var

You have nothing to read in the form of logs.

I can see what you are trying to say gore but your lack of practical experience is showing a bit. You cant just format a server willy nilly because you think someone has put a backdoor on it. A host or stand alone yes but not a server.

A server by its very nature has a lot of important informtion on it as well an controlling a lot of network apps/protocols - so it should be backed up you will probably say - yes, it should and if it has, then there is no rush to get it formatted and back online as you said earlier. Simpily take it down, put the back-up online and you have all the time in the world to examine the compromised one.

Nokia:

Understood.

Whiz:

I banned you not Negative. You got your wish from a while back when you said "You might as well ban me because I'm not going to stop posting about you and what I think of you" so can you quit PM Neg? He didn't do it, I did. You attacked me for no reason and even Tiger Shark who never takes up for me has asked you to stop. Now knock it off, if you want to come back fine but that attitude towards everyone being below you has GOT to stop. If you actually read what I've said once in a while there wouldn't be a problem.

Nokia:

I haven't dealt with more than 5 servers and the books I have say you should always format. As you said, my inexperience shows.

Hmmmm,

Hey folks, this is obviously an academic exercise so please stand back a bit and look at the bigger picture.

1. It is a production server
2. It is an e-commerce deployment

This introduces a number of practical considerations, particularly business continuity and legal requirements

From a business viewpoint, you would have back up hardware, preferably at more than one location. You would be able to do a seamless shift from one to the other.

From a legal viewpoint you need to be able to preserve the potential crime scene and determine what the extent of the compromise was.

In REALITY the technical IT/Computer Forensics aspect is IRRELEVANT ...........we are talking about the very survival of your business operation.

This is not a question about forensics, this is a question of your livelihood and all of your employees. It is not about getting your server back on line, it is about covering your a$$ when people take you to court, and keeping your business running.............in a way it is a "trick question" ;)

OH!....................and I happen to like sea otters :D

Format is good sense....

Production servers do not usually go as "nicely" as you would like when you try to reconstitute from scratch.... That's "experience" with no offense meant.....

Whizz is an ass... Read that Whizz, weep if you like - I won't hold my breath.... But he is "righter" then you here Gore....

Reinstate and if he immediately comes back as a dick give me the trigger for half a second..... I'll have fun saving you the effort and won't bother you about it again.....

Hey Guys,

Might we get back to the OP's subject line?

It is an academic question about what to do if a server is compromised. Important facts are:

1. It is a production server
2. It is an e-commerce operation
3. There is a lot of money involved

I have been suggesting that the first emphasis should be on the business and legal considerations, rather than the technical side?

:)

OK Guys,

Following the receipt of PMs, I have attempted to reconstruct this thread along the lines of devantionline 's original....................it is an interesting question IMHO, so I would wish to extend the courtesy of some sensible replies and discussion :)

Thanks ;)

This is a lot more complicated than a few simple commands. We're talking about a mission-critical server in an e-commerce operation. There are some assumptions to be made.

Assume: The system is the lead system, not clustered, with a staged backup server and access to real-time replicated data. It is either not using a SAN (or is using a SAN but not for the OS), the boot drives are mirrored and not RAID5.

Possible solution: Power off (pull power cord), bring up backup server and LUN the data drives. Pull boot drive(s) from compromised server and connect to forensics system. DD or WinHex and create 2 images. Lock up one image. Set original drive(s) aside for law enforcement. Have at the image.

Problem with the assumption is that about the only realistic thing is the boot drives being mirrored (RAID0+1)--maybe.

In my world, a mission-critical server would be either clustered or in a web-farm behind load balancers. All this behind 2 firewalls (the web interface being in the DMZ and access to the data through a secure channel out of the backside of the DMZ to the data network) and other protections and with real-time log analysis running, as well as IDS and traffic shaping. Besides the SAN, there would be (given money and other resouces) a backup set of servers that could be brought up in the unlikely event that the cluster was brought down.

What I've found is that most forensics texts don't consider these kinds of setups when addressing these configurations. Most of the models apply just fine to workstations or simple server configurations. Very few apply practically to high-availability, fault-tolerant, high-throughput systems.

Although, I'm open to ideas on this.

rapier57

That is along the lines I was trying to come from, like what about real life and real World?

Otherwise, why are the factors of e-commerce and $'000's per hour mentioned?

:)

STOP....
Breathe..
Don't do anything stupid..
remain calm..
STEP AWAY FROM THE COMPUTER...
An Incident has occured --
Triage --stop the bleeding
contain the threat
notify your chain of command
investigate --see below
Look for other compromised boxes with the signature you've created from the attack.
eradicate the threat
recover --return to normal business
follow up --wtf happened mate?, what are you doing to prevent it from happening again?


Those are your first steps.
In a production environment where a system can't be taken offline? Your best bet is to pull out your incident response plan --You do have one right?
Don't have one? Write one.
In that plan under: "production systems that can't be taken offline"

You should see something to this effect:
Insert linux CD containing statically linked binaries that are useful in incidents.
Follow the order of volatility. Collect evidence from each.
- registers, cache

- routing table, arp cache, process table, kernel statistics,
memory

- temporary file systems

- disk

- remote logging and monitoring data that is relevant to the
system in question

- physical configuration, network topology

- archival media

Don't have one of those? Well if this is an e-commerce biz that stands to lose a lot of money from downtime and public embarrassment write a proposal for TechPathways Prodiscover Incident Response. (approx $8000) Use PDIR to collect the above.

God I love this job!