Production servers do not usually go as "nicely" as you would like when you try to reconstitute from scratch.... That's "experience" with no offense meant.....
Whizz is an ass... Read that Whizz, weep if you like - I won't hold my breath.... But he is "righter" then you here Gore....
Reinstate and if he immediately comes back as a dick give me the trigger for half a second..... I'll have fun saving you the effort and won't bother you about it again.....
January 24th, 2006, 01:13 AM
Might we get back to the OP's subject line?
It is an academic question about what to do if a server is compromised. Important facts are:
1. It is a production server
2. It is an e-commerce operation
3. There is a lot of money involved
I have been suggesting that the first emphasis should be on the business and legal considerations, rather than the technical side?
January 24th, 2006, 03:38 AM
Following the receipt of PMs, I have attempted to reconstruct this thread along the lines of devantionline 's original....................it is an interesting question IMHO, so I would wish to extend the courtesy of some sensible replies and discussion :)
January 24th, 2006, 04:46 AM
This is a lot more complicated than a few simple commands. We're talking about a mission-critical server in an e-commerce operation. There are some assumptions to be made.
Assume: The system is the lead system, not clustered, with a staged backup server and access to real-time replicated data. It is either not using a SAN (or is using a SAN but not for the OS), the boot drives are mirrored and not RAID5.
Possible solution: Power off (pull power cord), bring up backup server and LUN the data drives. Pull boot drive(s) from compromised server and connect to forensics system. DD or WinHex and create 2 images. Lock up one image. Set original drive(s) aside for law enforcement. Have at the image.
Problem with the assumption is that about the only realistic thing is the boot drives being mirrored (RAID0+1)--maybe.
In my world, a mission-critical server would be either clustered or in a web-farm behind load balancers. All this behind 2 firewalls (the web interface being in the DMZ and access to the data through a secure channel out of the backside of the DMZ to the data network) and other protections and with real-time log analysis running, as well as IDS and traffic shaping. Besides the SAN, there would be (given money and other resouces) a backup set of servers that could be brought up in the unlikely event that the cluster was brought down.
What I've found is that most forensics texts don't consider these kinds of setups when addressing these configurations. Most of the models apply just fine to workstations or simple server configurations. Very few apply practically to high-availability, fault-tolerant, high-throughput systems.
Although, I'm open to ideas on this.
January 24th, 2006, 05:09 AM
That is along the lines I was trying to come from, like what about real life and real World?
Otherwise, why are the factors of e-commerce and $'000's per hour mentioned?
January 27th, 2006, 03:54 PM
Don't do anything stupid..
STEP AWAY FROM THE COMPUTER...
An Incident has occured --
Triage --stop the bleeding
contain the threat
notify your chain of command
investigate --see below
Look for other compromised boxes with the signature you've created from the attack.
eradicate the threat
recover --return to normal business
follow up --wtf happened mate?, what are you doing to prevent it from happening again?
Those are your first steps.
In a production environment where a system can't be taken offline? Your best bet is to pull out your incident response plan --You do have one right?
Don't have one? Write one.
In that plan under: "production systems that can't be taken offline"
You should see something to this effect:
Insert linux CD containing statically linked binaries that are useful in incidents.
Follow the order of volatility. Collect evidence from each.
- registers, cache
- routing table, arp cache, process table, kernel statistics,
- temporary file systems
- remote logging and monitoring data that is relevant to the
system in question
- physical configuration, network topology
- archival media
Don't have one of those? Well if this is an e-commerce biz that stands to lose a lot of money from downtime and public embarrassment write a proposal for TechPathways Prodiscover Incident Response. (approx $8000) Use PDIR to collect the above.