Live System Forensics

:hello:

I was just wondering what kind of tools/resources people use for live forensics? I know and use a lot of tools from sysinternals.com for malware analysis, I was wondering if anyone else knew of some powerful, reputable tools that can be used, as well as documents / resources on the process specifically.

Malware by Ed Skoudis goes over it a lot, I would like to hear about what people have to say about VMware and the forensic process. Does VMware allow you view action on the target OS, like files being opened, written to, processes, network activity, and the like? Or do you still have to bring in seperate software and install it?

Thanks in advance.

Hi,

http://digilander.libero.it/zancart

The product is called WinSonar. It monitors for unknown background processes and has a variety of tools.

Be careful with its autoprotection though................it wont even let you into task manager :cool: Not a problem because it detects internet connections and asks if you want to block unknown processes.................if you are using it as a forensics tool just say "no"

It prompts you when it detects an unknown process and lets you add it to a list of "good guys" so you can tune it rather like a firewall.

It protects itself against getting switched off. I use WinPatrol from BillP Studios, which gets round the task manager problem. WinSonar still gives a pop-up asking if you really want to close it.

I have actually used it to resolve a couple of really frustrating memory leak problems ;) not quite what it is intended for but I judge a tool by what it does, not what it says in the manual.

Cheers

PS Don't forget that SpyBot has some useful tools if you run it in advanced mode

http://www.foolmoon.net/security/wft/

Windows Forensics toolchest

http://e-fense.com/helix
I can't say enough about this yet..

Those are just a couple...

If you want a pretty comprehensive list of tools go to http://e-evidence.info . The site isn't the best looking one in the world..but who cares..it has more information than anywhere else.


As for VMware and the forensic process, I wouldn't bother. The only thing I use VMware for is hacking away at viruses and finding out their behavioral patterns. For that I use a number of freebie tools and some not so freebie tools(IDA pro). I typically get a disk containing an image of a compromised machine and I do a few things with it.
1) make 2 backup images
2) if it's been ghosted I will load it in vmware because I find it easier to identify malware from a live system than it is going through MAC times.
3) work off the backup images on my forensic station & vmware (but I typically have to crack a password or two)

To directly answer your question about what you can see...You can see everything.

Depending on what the machine was used for..I don't bother with vmware.

HTH
-hog

I wanted to post a quick note to anyone reading this post to let you know a new version of WFT was posted to my website (http://www.foolmoon.net/security/) as of last night. It has a number of new more powerful features over the previous version.

Monty McDougal (the author of WFT)

Live forensics is a bad idea. It is too low assurance to make it worth the effort. The problem is that the attcker (malware, rootkit, whatever) can use the security policy against the investigator. This means you'll need to do forensics again with the security policy disabled, so you'd might as well just start there.

cheers,

catch

Quote:

---

Live forensics is a bad idea. It is too low assurance to make it worth the effort. The problem is that the attcker (malware, rootkit, whatever) can use the security policy against the investigator. This means you'll need to do forensics again with the security policy disabled, so you'd might as well just start there.

---

I think that is an oversimplification/generalisation. It all depends on the forensic analysis you are carrying out.

If you are just investigating malware then it is highly unlikely that it will be using the security policy against the investigator. Most malware relies on an inadequate security policy in the first instance. In those circumstances live forensics have their uses. Sure you can release it on an unprotected machine to see what it does, but part of the exercise might also to be to determine how your policies/defences were circumvented.

Now, in the case of a cognoscient attack it is a different situation.

The reason I make this distinction is that the previous posts appear to be more focussed on malware/viruses rather than hacking. In other words, a relatively low technology and probably automated enemy.
:)

Monty (FoolMoon),

I was going to start a new thread pointing members of this site to your updated software as well as the other utilities you have written.. You have disapointed me by SPAMMING this forum.. How many more threads are you going to dredge up here and on other boards to promote your product? IF you software is good your subscription list alone would have popped threads on boards around and about, and it would have been far more effective for your products than this SPAMMING.
I trust the other boards treat you more kindly.

und3rtak3r

Quote:

---

It all depends on the forensic analysis you are carrying out.

---

And how do you know what you are looking for with any certainty without carrying out forensics?

That aside, if a system is acting unpredictably... it needs to be isolated and analyzed, even the discovery of simple malware may not be the full extent of the problem. Leaving this system connected is likely unacceptably dangerous.

cheers,

catch

Glenn (aka und3rtak3r),

Sorry if I have somehow offended you by posting a grand total of 5 messages about WFT to a few of the sites that I get visitors from. Please delete my mail, WFT, and your entire hard drive if you like since it is now obviously tainted with my “SPAM”. I posted to let a few people know (who might actually be interested) that I released an update to a program I have literally spent hundreds of hours developing (and then gave away for free). Obviously this effort has been wasted on you. And yes, I hope I am treated better on the other boards. It is crap like this that makes good people stop making useful tools.

Have a nice day. You have spoilt mine.

Monty

Quote:

---

And how do you know what you are looking for with any certainty without carrying out forensics?

---

Experience. The first question is always "hardware, software, malware, hacker?" A lot of malware is not very sophisticated in that it loads strange files, runs strange processes, tries to send mail, tries to connect to the internet, tries to edit the registry etc.

That sort of stuff is generally pretty susceptible to forensic tools.

Anyway you know what you are looking for..............the cause of whatever symptoms have made you look at the box in the first place.

:)

It wasn't in the OP, but I wanted tools to observe the behavior of an executable. I wasn't very clear on that, sorry.

Short of reverse engineering the file, I was looking for ways to see what parts of the OS were touched so I would know how to clean up.

In my case, I wouldn't be analyzing the entire system, only a binary provided to me. So you folks may be debating about different situations (comprimised system, single file)

Nihil, that is what is known as a process with no maturity whatsoever. :)

cheers,

catch

Soda:

Did you see this

A few thoughts.

First - WFT is a GREAT tool. There isn;'t anything else like it that allows you to collect realtime process data on a system in a forensically sound manner. Secondly, anyone scoffing at WFT OBVIOUSLY hasn't read the configuration file. If you had any idea how much time it takes to defin all of the commands in the config file, to make sure they work, and to find better (and sometimes best) tool that collects a spcific amount of information, then you would know what a valuable tool it is.

Second - anyone scoffing at live system forensics hasn't worked in a preassure cooker like a University where we get whacked boxes frequently. Live forensics allows us to make an inspection and within a half an hour make the "rebuild/remediate" decision w/ some intel. WFT has proven itself amply well over the past year here at my school. So much so that I developed a course for my sfaff on live system forensics, and now I have my job back.

To wrap =>

IF you need a tool, buy Encase and then use WFT.
If you want to know where to look w/ Encase, than run WFT first, else you will spend three days analyzing the system becuse you don't know where to look based on current data.

End of Line.

Foolmoon: 5 messages is starting to sound a whole lot like spam... if you want to do it the right way, submit your link here - that's what it's for.

Negative: I think he only posted to two threads here. He posted a total of 5 times on various boards... if I understand correclty. I only saw two posts here...

Both threads that he posted in are relevant. maybe he just didn't know that they would be bumped up to the main page. a new thread would have been appropriate... but whatever. I'm happy that he posted the update here.

Encase has a version for live forensics.

Encase enterprise edition

The general public version is the Enterprise Edition and is used to protect an EnterPrise network.

It is expensive, but works well for documenting rogue processes, rogue ports along with being able to view the true file system. It understands many common filesystems such as NTFS, EXT,HFS, UFS, BSD.

RP

Quote:

---

anyone scoffing at live system forensics hasn't worked in a preassure cooker like a University where we get whacked boxes frequently. Live forensics allows us to make an inspection and within a half an hour make the "rebuild/remediate" decision w/ some intel. WFT has proven itself amply well over the past year here at my school. So much so that I developed a course for my sfaff on live system forensics, and now I have my job back.

---

Anyone praising live system forensics is wasting their organization quite a bit of money in training, time, and just all around inefficient system configuration.

At my work, all the desktop systems are set up identically, if a system has a problem it is shut down, a spare preconfigured drive is plugged in and the system is returned to a live status. All in five minutes on the outside.

The other drive can then be analyzed outside the scope of its security policy when time permits. This approach is faster, cheaper, and higher assurance.

cheers,

catch

catch Once again I think your argument is sparked by semantic and terminological differences, at least I certainly hope so :p

No-one in their right mind is going to leave a suspect compromised machine on their network. Over here, that is what we call the "production environment". It is just the same for a hardware failure, you pull the defective kit from the production environment and replace it. That lets the user get on with what they are supposed to be doing.

As for "live forensics", that would be understood to be a system running in its normal configuration, but obviously not in the production environment. Other approaches would include examining hard drives on a forensics machine or in a lab, decompiling malware, and examining malware induced system changes.

The actual theme of this thread is not about security, it is about tools that let you run malware, and tell you what it tries to do. That is "live" forensics, as opposed to a post mortem, a biopsy rather than a necropsy :D Obviously all that takes place in a controlled laboratory environment.

Nihil, live forensics is by definition on a live system or "a system running in its normal configuration" which is still flawed as it is possible for the security policy (or elements of the user interface for that matter) to interfere with such forensics. This means that in order to ensure a clean system you must perform the forensics on the system in a dead state.

The ONLY reason you'd need to look at a live system is to figure out exactly where the problem was IF you don't have a map of what the system should look like. Having the system baselined beforehand completely removes the need for live forensics and should be used in all production environments.

cheers,

catch

Quote:

---

Having the system baselined beforehand completely removes the need for live forensics and should be used in all production environments.

---

I suspect that is what I would (in its physical manifestation) call a "reference machine". It reflects the standard desktop and is what you would use to test upgrades, new applications etc. before moving them into production.

Quote:

---

live forensics is by definition on a live system or "a system running in its normal configuration" which is still flawed as it is possible for the security policy (or elements of the user interface for that matter) to interfere with such forensics

---

When you are analysing the effects of malware, that is something you should be interested in, as you want to know how the malware interacts with your live system?

My thoughts are based on general concepts of the software development cycle. Unit Test - System Test - User Acceptance Test. Part of the back end of the system testing would include "end-to-end testing" and "compatibility testing" where you are basically verifying the integration, both internally and externally.

Now, I tend to look on malware analysis as testing an (albeit unwanted) application, just like any other. I would therefore want to know what the "beast" does in a "sterile" environment, as that would tell me its intentions. However, I would also want to know what it did on a "reference machine" (mirror of my production environment) and both pieces of information are of interest.

As for interference with the forensics, I would say that is a separate issue, as it concerns the effectiveness of the tools being used.

:)

Quote:

---

I suspect that is what I would (in its physical manifestation) call a "reference machine". It reflects the standard desktop and is what you would use to test upgrades, new applications etc. before moving them into production.

---

So you agree that more or less all organizations should have this?

Quote:

---

When you are analysing the effects of malware, that is something you should be interested in, as you want to know how the malware interacts with your live system?

---

Nope: isolate, discover how it got there, remove. No reason to understand how it works, that adds no value to the organization.

Quote:

---

As for interference with the forensics, I would say that is a separate issue, as it concerns the effectiveness of the tools being used.

---

I still hold that live forensics is silly for anyone other than the home user... and even for the home user I'm not sure how useful it is, but it makes users feel in control and allows them to be a little lazier so, why not?

cheers,

catch

Quote:

---

So you agree that more or less all organizations should have this?

---

Absolutely! it should be de riguer, the norm, policy etc.......... :D

Hey, we had a great case in point only a few days ago when Trend Micro distributed that dud update..............if you had run that on a reference machine, you would have seen that there was a problem and saved yourself a lot of grief? That is probably why Trend Micro have not been sued...........contributory negligence? (OK AV not required in a properly secured system, but please remember our previous discussion of corporate a$$ covering?)

Quote:

---

Nope: isolate, discover how it got there, remove. No reason to understand how it works, that adds no value to the organization.

---

Hey, I am sure that you stole that line from me :eek: How it works is interesting only insofar as it might indicate a flaw in your policy-process-procedure triad? I am assuming that you have been infected? and that one of the final stages in full lifecycle development is "lessons learned"? Remember, we are only interested in the ones that get through! Which should be few and far between?

I certainly use the "flush and forget" concept with private customers, as they will not stand the cost, which is what you are saying with "adds no value"

Quote:

---

I still hold that live forensics is silly for anyone other than the home user... and even for the home user I'm not sure how useful it is, but it makes users feel in control and allows them to be a little lazier so, why not?

---

A couple of exceptions would be people working for anti-malware companies and journalists. They need to respond quickly, and have a very wide audience with all sorts of security policies in place. They have to be generalistic so they are forced to concentrate on what malware tries to do. Live forensics is a pretty good way to get that information quickly?

Anyway, I like torturing nasties.............so there :p

Some of the above comments suggest to me that live forensics may not be necessary. I disagree with that idea though it is better to perform forensics on an offline system. It is easier for one.

Why would one need to perform forensics on a live system?
1. You cannot shut the system down. I have seen cases where a large (10 TB+) server was compromised and is running a business critical system and the Admins pleaded that they could not bring it down any time soon. I had to agree that we didn't want to make this company a victim twice by taking them offline while we performed forensics on this system. An additional off this topic idea is how you image a 10Tb drive. Right now you don't unless you have a portable 10tb server. You have to pick and choose what files and items you wish to "image/backup" etc. I think that is called sparse evidence files, where you document the file system and unallocated clusters and pick the files you really need to image and leave the rest. You don't need the /drivers directory if there is nothing of information or evidentiary value in there. How do you know though for sure? Really I guess it is doing the best you can at the time with what you have.

2. There maybe times when a system needs to image live, say the hard drive contains an encrypted volume that is accessible right now, but if you turn it off, you will lose the virtual drive as it is natively encrypted on the hard drive.

3. Of course, the comments on malware above are very interesting as the only way to learn how/what the malware does is live and in a controlled environment.

4. You are responsible for monitoring you company's network, and need to inspect machines across the network (the world) live in lieu of traveling to that location (cost reason).

There are many more reasons why you may need to perform live forensics.

I prefer offline "conventional" forensics as Murphy is usually bored by these routines but loves to accompany me on the new areas of forensics.

Rp

Interesting stuff RP305, I do think that live forensics is another example of a mediocore solution in an already bad situation.

1. This is bad for two reasons. One: no organization should have such a huge single point of failure. Two: administrators should have no say that violates the incident response policy.

2. I've never seen an instance requiring an encrypted drive, though even if such a situation did arise, there is no reason for the drive to be executable. In fact executables should never be encrypted for this very reason.

3. "the only way to learn how/what the malware does is live and in a controlled environment." seems to contradict to me. Wouldn't a controlled environment be the opposite of a production environment (I mean in theory they should be the same ;) )?

4. They can mail you the drive.

By my definition, a live system is a system that is operational... running and online performing its task. By running diagnostics on such a system you increase the chances of a cascading failure across the environment. The best bet is still to unplug the system... either from the network or from the wall depending on your resources and objectives and start from there.

This pull immediately approach removes a whole slew of risks, including misdiagnosis. It is too big of a risk to just rely on whatever admin's spidey senses about what might be wrong with the system to decide what to do. A moment can be the difference in a single system and an entire network.

cheers,

catch

From what I understand, forensicsis the use of science to answer questions from a legal standpoint. On this basis analisess of a live system must be flawed. How can forensic evidence be gained from a constantly changing enviroment.

What ever looking at a live system is, I would suggest it is not forensics.

I’ve been following this discussion, and I think some of you may be over simplifying when you say there is not a need/benefit for live system forensics. While I certainly would advocate live forensics in many cases (for reasons I will explain), I recognize there are other cases where you are better off to pull the plug on a system. For instance, if you have some reason to believe the attacker is someone malicious intent (i.e. a former employee, an employee with axe to grind, or you are a “big target” because people don’t like you or your company) – pull the plug. If the system is attacking other systems – pull the plug. Actually in this case, I might pull the network connection (or null route it), run live forensics, then pull the plug -- but I would definitely act to minimize damage. Obviously this can have consequences if an attacker is looking for this fact, but it is pretty rare.

Live forensics, I would contend serves several valuable needs.

1) In an incident, you are not always aware what is going on. Sure, some times it is obvious you’ve been hacked. Other times it is not as obvious. I consider live forensics the first step of incident response. It helps you identify there is indeed a problem. If you work in a big organization, most of the time the first flag that something is wrong comes from a user. You may or not know the particular system well, so a live response lets you decided if this system is worth analyzing. I personally use this to help filer out the machines really need an in-depth analysis from the ones that don’t. Many people use a tool like WFT, to help them do this pre-processing. I know several people who give a pre-built WFT CD out to some of their very junior IT people and say go run this. Then the more senior people decide based on what it collects if it is worth wasting their time doing an in-depth analysis.

2) There are also cases where the signs of malicious attack may be entirely in memory. I will use for example the code red worm (not that it is the only example). It inserted itself directly into the memory of a running IIS server without ever writing itself to disk. While it was pretty easy to tell a box had code red, what if the malware was less obvious. Dll injection is becoming more popular and is a prime candidate for this argument. The minute you reboot, you just lost any data that may have existed in memory. Sure you might get luck and find pieces of it written to disk via the swap file, but if you collected the system memory before downing the box you would have a lot more information. More and more malware is either encrypted, packed, or otherwise obfuscated, but many times the complete running memory of the process is usually unencrypted. Sure, you may be able to get some of thois post-mortem on a test box, but there is also lot of other interesting info there for the taking on a victim system before it is downed – processes, loaded dlls, open network ports, etc. Once you pull the plug these are all lost. There can be a lot of valuable information in this data. Many times it gives an analyst clues as to how the malware might function, other machines that are compromised, or even information about the attacker. A good root kit however can greatly change the value of this method, but using the 90% rule I still think it adds value.

While I know there are good arguments for both doing live forensics and not, I think it is definitely useful in many cases. I recognize this almost equates to a religious debate among many people, but I don’t see it as a simple black or white answer. I think there are definite shades of gray that we must move between as the situation changes – but that is just my 2 cents on the subject. Opinions obviously will differ.

Monty

It seems to me that there is no clear concensus on the meaning of "live forensics"

Some seem to think that it means the production system, others that it is simply a mirror of that in a laboratory environment.

Until that issue is resolved, the whole discussion is irrelevant.

I would point out that the Op has made it clear that he was referring to the laboratory rat scenario?

:)

Now that I have a few minutes..I'd like to drag this thread out of its grave...

One of the long standing debates in forensic circles is "pull the plug" or "capture data then yank it". Pulling the plug is what NIST suggests we do, however there can potentially be a gold mine of information to be collected on a live system. The answer to this is generally a judgement call. Can you capture data from a live system without modifying it, or modifying it in a proven manner such that any thing that gets modified won't decapitate your investigation? If a live forensics tool is documented to show that it only changes certain system files, then your case could perhaps be bolstered by the information collected on a live system.

Live forensics is what some people might call incident response. It's the act of responding to a live system and collecting pertinent data to determine if it's simply a security event or an incident that requires taking the box offline for further analysis. Calling live forensics a waste is foolish due to the fact that when an RCA(root compromise analysis) is neccessary(and you all do this every time you respond to a real incident don't you?), you are more than doubling your work without conducting a live investigation of the system.
In a criminal forensics incident, you'd be looking for data that helps you work the case. Examples would be traces of a shredding program running, pulling print jobs from the spooler, or grabbing an encryption key from memory.

Live forensics largely consists of gathering process lists, physical memory dumps, active network connections, pagefile contents and user lists as well as several other types of data.
The tool that Monty has created(WFT) does a wonderful job of illustrating the state of the computer, before you pull the plug.

There are a number of tools on the Market that do "live forensics" Here's a few:
WFT - Windows Forensics Toolchest
FSP - Forensic Server Project - by Harlan Carvey get it at: windows-ir.com
Encase (as was mentioned)
Pro-discover - techpathways.com
Wetstone Gargoyle - wetstone technologies - wetstonetech.com(right near mah house!)
OnlineDFS - ATC-NY http://oracorp.com/Products.html(right near mah house too!)
Helix -FRED & netcat - e-fense.com/helix

There is something to be said for an ounce of prevention...but this is forensics, not network security. Forensics is the act of responding to an incident(whether criminal or otherwise), it's not about prevention, it's about finding out how the net/sys admins ****ed up and helping to prevent it from happening again. Or it's about responding to some sick ****ing pedo or murder or any other criminal incident.

Take my current case right now where an organization may have to shell out $500,000 because of one stupid mistake. **** happens, it's my job to help show how it happened so it doesn't happen again.

As for foolmoon pimping out his product, it's a welcome addition in my books. We need more people like Monty busting their asses providing valuable resources for free. I don't make enough to purchase products like Encase so I depend on free software that some very talented people put together.