Live System Forensics

Quote:

---

Having the system baselined beforehand completely removes the need for live forensics and should be used in all production environments.

---

I suspect that is what I would (in its physical manifestation) call a "reference machine". It reflects the standard desktop and is what you would use to test upgrades, new applications etc. before moving them into production.

Quote:

---

live forensics is by definition on a live system or "a system running in its normal configuration" which is still flawed as it is possible for the security policy (or elements of the user interface for that matter) to interfere with such forensics

---

When you are analysing the effects of malware, that is something you should be interested in, as you want to know how the malware interacts with your live system?

My thoughts are based on general concepts of the software development cycle. Unit Test - System Test - User Acceptance Test. Part of the back end of the system testing would include "end-to-end testing" and "compatibility testing" where you are basically verifying the integration, both internally and externally.

Now, I tend to look on malware analysis as testing an (albeit unwanted) application, just like any other. I would therefore want to know what the "beast" does in a "sterile" environment, as that would tell me its intentions. However, I would also want to know what it did on a "reference machine" (mirror of my production environment) and both pieces of information are of interest.

As for interference with the forensics, I would say that is a separate issue, as it concerns the effectiveness of the tools being used.

:)

Quote:

---

I suspect that is what I would (in its physical manifestation) call a "reference machine". It reflects the standard desktop and is what you would use to test upgrades, new applications etc. before moving them into production.

---

So you agree that more or less all organizations should have this?

Quote:

---

When you are analysing the effects of malware, that is something you should be interested in, as you want to know how the malware interacts with your live system?

---

Nope: isolate, discover how it got there, remove. No reason to understand how it works, that adds no value to the organization.

Quote:

---

As for interference with the forensics, I would say that is a separate issue, as it concerns the effectiveness of the tools being used.

---

I still hold that live forensics is silly for anyone other than the home user... and even for the home user I'm not sure how useful it is, but it makes users feel in control and allows them to be a little lazier so, why not?

cheers,

catch

Quote:

---

So you agree that more or less all organizations should have this?

---

Absolutely! it should be de riguer, the norm, policy etc.......... :D

Hey, we had a great case in point only a few days ago when Trend Micro distributed that dud update..............if you had run that on a reference machine, you would have seen that there was a problem and saved yourself a lot of grief? That is probably why Trend Micro have not been sued...........contributory negligence? (OK AV not required in a properly secured system, but please remember our previous discussion of corporate a$$ covering?)

Quote:

---

Nope: isolate, discover how it got there, remove. No reason to understand how it works, that adds no value to the organization.

---

Hey, I am sure that you stole that line from me :eek: How it works is interesting only insofar as it might indicate a flaw in your policy-process-procedure triad? I am assuming that you have been infected? and that one of the final stages in full lifecycle development is "lessons learned"? Remember, we are only interested in the ones that get through! Which should be few and far between?

I certainly use the "flush and forget" concept with private customers, as they will not stand the cost, which is what you are saying with "adds no value"

Quote:

---

I still hold that live forensics is silly for anyone other than the home user... and even for the home user I'm not sure how useful it is, but it makes users feel in control and allows them to be a little lazier so, why not?

---

A couple of exceptions would be people working for anti-malware companies and journalists. They need to respond quickly, and have a very wide audience with all sorts of security policies in place. They have to be generalistic so they are forced to concentrate on what malware tries to do. Live forensics is a pretty good way to get that information quickly?

Anyway, I like torturing nasties.............so there :p

Some of the above comments suggest to me that live forensics may not be necessary. I disagree with that idea though it is better to perform forensics on an offline system. It is easier for one.

Why would one need to perform forensics on a live system?
1. You cannot shut the system down. I have seen cases where a large (10 TB+) server was compromised and is running a business critical system and the Admins pleaded that they could not bring it down any time soon. I had to agree that we didn't want to make this company a victim twice by taking them offline while we performed forensics on this system. An additional off this topic idea is how you image a 10Tb drive. Right now you don't unless you have a portable 10tb server. You have to pick and choose what files and items you wish to "image/backup" etc. I think that is called sparse evidence files, where you document the file system and unallocated clusters and pick the files you really need to image and leave the rest. You don't need the /drivers directory if there is nothing of information or evidentiary value in there. How do you know though for sure? Really I guess it is doing the best you can at the time with what you have.

2. There maybe times when a system needs to image live, say the hard drive contains an encrypted volume that is accessible right now, but if you turn it off, you will lose the virtual drive as it is natively encrypted on the hard drive.

3. Of course, the comments on malware above are very interesting as the only way to learn how/what the malware does is live and in a controlled environment.

4. You are responsible for monitoring you company's network, and need to inspect machines across the network (the world) live in lieu of traveling to that location (cost reason).

There are many more reasons why you may need to perform live forensics.

I prefer offline "conventional" forensics as Murphy is usually bored by these routines but loves to accompany me on the new areas of forensics.

Rp

Interesting stuff RP305, I do think that live forensics is another example of a mediocore solution in an already bad situation.

1. This is bad for two reasons. One: no organization should have such a huge single point of failure. Two: administrators should have no say that violates the incident response policy.

2. I've never seen an instance requiring an encrypted drive, though even if such a situation did arise, there is no reason for the drive to be executable. In fact executables should never be encrypted for this very reason.

3. "the only way to learn how/what the malware does is live and in a controlled environment." seems to contradict to me. Wouldn't a controlled environment be the opposite of a production environment (I mean in theory they should be the same ;) )?

4. They can mail you the drive.

By my definition, a live system is a system that is operational... running and online performing its task. By running diagnostics on such a system you increase the chances of a cascading failure across the environment. The best bet is still to unplug the system... either from the network or from the wall depending on your resources and objectives and start from there.

This pull immediately approach removes a whole slew of risks, including misdiagnosis. It is too big of a risk to just rely on whatever admin's spidey senses about what might be wrong with the system to decide what to do. A moment can be the difference in a single system and an entire network.

cheers,

catch

From what I understand, forensicsis the use of science to answer questions from a legal standpoint. On this basis analisess of a live system must be flawed. How can forensic evidence be gained from a constantly changing enviroment.

What ever looking at a live system is, I would suggest it is not forensics.

I’ve been following this discussion, and I think some of you may be over simplifying when you say there is not a need/benefit for live system forensics. While I certainly would advocate live forensics in many cases (for reasons I will explain), I recognize there are other cases where you are better off to pull the plug on a system. For instance, if you have some reason to believe the attacker is someone malicious intent (i.e. a former employee, an employee with axe to grind, or you are a “big target” because people don’t like you or your company) – pull the plug. If the system is attacking other systems – pull the plug. Actually in this case, I might pull the network connection (or null route it), run live forensics, then pull the plug -- but I would definitely act to minimize damage. Obviously this can have consequences if an attacker is looking for this fact, but it is pretty rare.

Live forensics, I would contend serves several valuable needs.

1) In an incident, you are not always aware what is going on. Sure, some times it is obvious you’ve been hacked. Other times it is not as obvious. I consider live forensics the first step of incident response. It helps you identify there is indeed a problem. If you work in a big organization, most of the time the first flag that something is wrong comes from a user. You may or not know the particular system well, so a live response lets you decided if this system is worth analyzing. I personally use this to help filer out the machines really need an in-depth analysis from the ones that don’t. Many people use a tool like WFT, to help them do this pre-processing. I know several people who give a pre-built WFT CD out to some of their very junior IT people and say go run this. Then the more senior people decide based on what it collects if it is worth wasting their time doing an in-depth analysis.

2) There are also cases where the signs of malicious attack may be entirely in memory. I will use for example the code red worm (not that it is the only example). It inserted itself directly into the memory of a running IIS server without ever writing itself to disk. While it was pretty easy to tell a box had code red, what if the malware was less obvious. Dll injection is becoming more popular and is a prime candidate for this argument. The minute you reboot, you just lost any data that may have existed in memory. Sure you might get luck and find pieces of it written to disk via the swap file, but if you collected the system memory before downing the box you would have a lot more information. More and more malware is either encrypted, packed, or otherwise obfuscated, but many times the complete running memory of the process is usually unencrypted. Sure, you may be able to get some of thois post-mortem on a test box, but there is also lot of other interesting info there for the taking on a victim system before it is downed – processes, loaded dlls, open network ports, etc. Once you pull the plug these are all lost. There can be a lot of valuable information in this data. Many times it gives an analyst clues as to how the malware might function, other machines that are compromised, or even information about the attacker. A good root kit however can greatly change the value of this method, but using the 90% rule I still think it adds value.

While I know there are good arguments for both doing live forensics and not, I think it is definitely useful in many cases. I recognize this almost equates to a religious debate among many people, but I don’t see it as a simple black or white answer. I think there are definite shades of gray that we must move between as the situation changes – but that is just my 2 cents on the subject. Opinions obviously will differ.

Monty

It seems to me that there is no clear concensus on the meaning of "live forensics"

Some seem to think that it means the production system, others that it is simply a mirror of that in a laboratory environment.

Until that issue is resolved, the whole discussion is irrelevant.

I would point out that the Op has made it clear that he was referring to the laboratory rat scenario?

:)

Now that I have a few minutes..I'd like to drag this thread out of its grave...

One of the long standing debates in forensic circles is "pull the plug" or "capture data then yank it". Pulling the plug is what NIST suggests we do, however there can potentially be a gold mine of information to be collected on a live system. The answer to this is generally a judgement call. Can you capture data from a live system without modifying it, or modifying it in a proven manner such that any thing that gets modified won't decapitate your investigation? If a live forensics tool is documented to show that it only changes certain system files, then your case could perhaps be bolstered by the information collected on a live system.

Live forensics is what some people might call incident response. It's the act of responding to a live system and collecting pertinent data to determine if it's simply a security event or an incident that requires taking the box offline for further analysis. Calling live forensics a waste is foolish due to the fact that when an RCA(root compromise analysis) is neccessary(and you all do this every time you respond to a real incident don't you?), you are more than doubling your work without conducting a live investigation of the system.
In a criminal forensics incident, you'd be looking for data that helps you work the case. Examples would be traces of a shredding program running, pulling print jobs from the spooler, or grabbing an encryption key from memory.

Live forensics largely consists of gathering process lists, physical memory dumps, active network connections, pagefile contents and user lists as well as several other types of data.
The tool that Monty has created(WFT) does a wonderful job of illustrating the state of the computer, before you pull the plug.

There are a number of tools on the Market that do "live forensics" Here's a few:
WFT - Windows Forensics Toolchest
FSP - Forensic Server Project - by Harlan Carvey get it at: windows-ir.com
Encase (as was mentioned)
Pro-discover - techpathways.com
Wetstone Gargoyle - wetstone technologies - wetstonetech.com(right near mah house!)
OnlineDFS - ATC-NY http://oracorp.com/Products.html(right near mah house too!)
Helix -FRED & netcat - e-fense.com/helix

There is something to be said for an ounce of prevention...but this is forensics, not network security. Forensics is the act of responding to an incident(whether criminal or otherwise), it's not about prevention, it's about finding out how the net/sys admins ****ed up and helping to prevent it from happening again. Or it's about responding to some sick ****ing pedo or murder or any other criminal incident.

Take my current case right now where an organization may have to shell out $500,000 because of one stupid mistake. **** happens, it's my job to help show how it happened so it doesn't happen again.

As for foolmoon pimping out his product, it's a welcome addition in my books. We need more people like Monty busting their asses providing valuable resources for free. I don't make enough to purchase products like Encase so I depend on free software that some very talented people put together.