Examining a compromised server.

I'll be working on 31st it seems. I was on audit and found that while running a quick scan on a critical server there were lot of files that were being scanned (mostly with sys extension) but when I ran a search for the same files, I couldn’t find anything. I was sort of expecting that because file names are hide_evr2.sys which according to lot places is a rootkit. Now the nature of the server is "extremely critical". Seeing this server compromised to an extent that Symantec Endpoint can see the file but can't detect it (I am not sure if they have def's for it but in any case it's freaking out stuff).

I'm writing this at a point where I’m finding almost every server I examine with the same malware so I’ll come down to the question

What should be my next step in examining the server?

Please note that we don't have a "real" incident response team or plan. I know my next level of escalation and I will be doing that accordingly. My question simply is how I examine these server from here on. I will of course create a copy of the HDD and examine there so beyond that please let the suggestions flow.

I know this case is related to malware infection only but I’ve started it in Microsoft Security Discussions the reason being it’s a server running windows OS (2003) and this thread may server as a point of reference.


PS: Thanks a lot and HAPPY NEW YEAR IN ADVANCE.

did you actually discover malware or any malicious activity or are you only looking at smoke at this point? if symantec didn't tell you the server was owned, get a second opinion so you're not caught saying 'i took the server down because google said this file is a rootkit'.

other than that - examining the server might not be your next step, but beginning to build up its replacement might be a higher priority to ensure continuity.

To be frank, I don't have a single file from the one's i mentioned. But everytime I have scanned I can see them in the list of file's being scanned. I am sure that it's unusual that files that i have found in the scan but not on the system when i search for them are malicious. I mean all the file names i searched for are all part of either a rootkit or trojan.

Also all drives have autorun.inf file in the drive root file but i can't find any of them in search (using systeminternal tools / GMER rootkit detector). I can see symantec showing them up during scan but not on the machine.

My hunch is beyond google saying yes to the file. Because most of these servers were infected in past 6 months and there was never an analysis done to find the root cause. How did they get infected was never found. Also infection was not limited to 1 or 2 files. It was like one day symantec's auto protect went crazy and found 100's of files infected. When a full scan was run more files were found all infected with a single malware. This pattern is consistant for almost all servers.

I'll post more soon. Thanks anyway.

Hi there ByTe and a happy New Year!

I am afraid I don't know how Endpoint works, but I presume that it produces a summary report when it has completed its scan? If it does not include these items then I would say that what you are seeing whilst the scan is in progress is what it is scanning FOR, not the files actually being scanned.

You mention hide_evr2.sys, which is a rootkit and has quite a few variants. My problem is that is a rather old bit of malware (2006/7) and really should have been blocked, particularly as the name is a dead give-away?

I would certainly try some other products and possibly HijackThis!

It is my belief and experience that if an anti-malware product finds suspected malware it will report it. It may not be able to clean or remove it, but it would give you a hardcopy report ;) Otherwise I would expect it to throw up a warning message on the screen.

You mentioned that these servers have been compromised earlier and that no follow-up took place. This gives me another thought.........are you detecting "traces" rather than the real thing? I can't really tell you what a "trace" is, but I would think orphan Registry entries, temporary files and the like. That would explain why you cant find the executables?

The best product I know for detection and removal of those is A-Squared

http://www.emsisoft.com/en/software/free/

Unfortunately, it is only free for private use, but you might be able to get an evaluation/trial copy ;););)

Good Luck!

Happy New Year.

How is it possible that on all the server that I run a scan with SEP, I see autorun.inf file in at least one of the drives? How is it also that every time I ran GMER (www.gmer.net) anti-rootkit detector the machine crashed.

Also all server has been infected in last 6 months without any reason being recorded. It was like an administrator logged in the morning and found Symantec screaming of how it found a malware when he was away. Also the malware detected is mostly W32.Imaut.CN (http://www.symantec.com/security_res...022015-1923-99). I have submitted not less than 20 different samples of this same malware to Symantec when I have seen machine having same foldername.exe file inside almost every folder.

It's tiring right now with the kind of load I’ve been given. I am having a hard time investigating because I know there is a malware on the machine that is not getting detected. I know Symantec uses veritas technology in their corporate version of AV to find rootkit's. But here it is getting fooled. I can take a print screen of Symantec scanning all the file's I’ve mentioned and put it up if someone wants to verify my cause for this thread but beyond that I have nothing much to give. I'll be going through systeminternals utility suit now to see if there is any way I can pick up files hidden to the OS. I remember ADS was the way to go but I need to refresh myself.


Anyway thanks again for your help.

Happy New year (again).


PS: Nihil Symantec does not store names of every file it scanned in the report. It just store's any infected file it found.

ByTeWrangler ;

I am curious as to how this turns out, so just to update:

By now you have cloned the drive of a suspected server, and hopefully
have a replacement server in place while you verify that there is
indeed a compromise of the system, and are working on a cloned drive.

The problem I see is that you mention two different types of malware,
one hiding inside ADS and one which may be kernel level.

For the ADS, you have used
HijackThis as nihil suggested,
or, since you are using Sysinternals Suite , you have used the utility
Streams to discover the ADS.
What did you discover? Were any malicious?

Since there may be a kernel level rootkit, you have also booted the system
to a CD ( to use a known good kernel ) mounted the cloned drive, then looked
for the mysterious files? Did you find any?

Your hide_evr2.sys may be Symantic's
Infostealer.Snifula.B
So were there any of the suspicious files or reg entries that either of the
Symantic sites indicate?

Any indication of unusual ports open?
( I would test from both within, using something like TCPView and from outside
using something like Nmap and then maybe Wireshark ,
or whatever you have available to you. )


If something was found, did the file dates give any clue as to when or how
the box was compromised? That may be important when trying to discover
the extent of the problem and how to avoid it reoccurring.
( I know this goes without saying, but I would not try to repair a box that had
been rooted. And depending how it was done, even the replacement box could
now be compromised. )

You are much more knowledgeable then I,
but I think maybe the enormity of the task ahead has got you riled.
Just go back to basics:

1) Put on a pot of coffee
2) Order pizza
3) take it one step at a time
4) document everything you do

Good hunting.
---------------------------------------

I just find the timing of this strange:
What's up with port 12174? Possible Symantec server compromise?

Hmmmmm..................

Quote:

---

It was like one day Symantec's auto protect went crazy and found 100's of files infected. When a full scan was run more files were found all infected with a single malware. This pattern is consistent for almost all servers.

---

So, if it is supposed to "autoprotect" why didn't it?.................if it really can detect them? I am wondering if this happened right after a Symantec update?

I would certainly use another AV product or online scan to check out the Symantec findings.

When you want to check individual files then please send them to these:

http://virusscan.jotti.org/en

http://www.virustotal.com/

And get an opinion of them scanned by multiple AV products ;)

I would mention that I have rarely had satisfactory responses from AV vendors............they are way too busy trying to cover their own arses to care:mad:

My next step would be:

1. CCleaner
2. Eusing Free Registry Cleaner

Those will get rid of "rubbish" ;)

3. A-Squared
4. Malwarebytes Anti-Malware
5. Spybot S&D

They should get rid of any traces and other crap that are in there :)

Then the "rat hunt" starts:lildevil:

1. Rootkit Revealer:

http://technet.microsoft.com/en-us/s.../bb897445.aspx

2. Blacklight:

http://www.f-secure.com/en_EMEA/secu...ght/index.html

About this time I would re-run my AV and see if it still finds things?

Then HijackThis! to see what is running on the machine.

http://download.cnet.com/Trend-Micro...-10227353.html

Send the results here to begin with and look at what they say is bad or unknown:

http://www.hijackthis.de/

As I am sure that you are aware, any backups they have made will probably be quite useless as they will have backed up the problems as well :(

The only reliable route is to back-up the data (scan it afterwards for malware) then wipe the disk and reinstall.............great fun on a critical server?.................but I don't suppose you want to go there?

As for "autorun.inf" what do Symantec say about these files? If you can find them just submit them to the first two links in this post.

I have just checked this stand alone machine and I have 9 instances of that file scattered around it.

Try this: http://www.softpedia.com/progDownloa...oad-85585.html

But be careful with all this stuff as they might find false positives. That is why I linked the two free scanning sites first.

Our IT department is really fragmented here. Just to get a clone of the server my request is in consideration. I had to brief our Assistant CIO to speed things up. I haven't actually got lot of time to run any utilities till now.

However GMER crashes the server. I ran streams but didn't find the *whatever* to go through the results since i ran them somewhere before i passed out for the day.

I am going to take IKnowNot's suggestion and get myself fresh for this. I know this is going to be long. I'll be posting everything I do here so anyone else referring to this can either use the same steps or improve it.

Nihil i'm going to go step by step and compile a report on what i've found.

Here is what i'm planning to do:

1. List all autoruns - Autoruns by systeminternals
2. List all open ports - TCPView by systeminternals
3. List all shares - Sharenum by systeminternals
4. List all process and and all attached DLL's to them - different process tools by systeminternals

once i'm done with that i'm again going to go for coffe + pizza routine. Then I shall run ADS and file analysis utilities etc.. :)

Iknownot thanks for the suggestion again.. I think thats what i really needed.

We will not be cleaning the server of course and we will be going for a new installation.

Don't mind my replies. I've hardly slept. I shall put a more rational (if thats the word) reply once i get fwe things done and get a shower and new sets of cloth's..

Thanks again to everyone :)

Alright I have run for few reports. It’s through Microsoft remote desktop. On a live server (not in production environment anymore.)

I randomly stopped the scan and it showed file name "twext.exe”. Guess what I searched for the file and it wasn't there on the system. I searched using GMER and it wasn't there. I searched for the file name online and turns out to be an info stealer as per prevx (http://www.prevx.com/filenames/46985...TWEXT.EXE.html). I did find the same name .dll file.

I also found to VBS scripts. Virusremoval.vbs and newvirusremoval.vbs. As for all the files that I’ve searched for it's not on the system. If Symantec is showing them up while scanning but they are not on the system I wonder what’s going on.

I also found an IP that the scanner shows just before stopping! I am not sure if this is the host file or the DNS records. I am confirming that in sometime (need coffee).

I am really concerned. I have saved all the reports that I have mentioned in the previous post. Any member that wants to have a look at the reports can PM me. At this moment I'm not willing to put them here. I will go through them ensure they have nothing important that can (further I guess) compromise security and post it here.

My eyes are getting heavy again. I'll write something once I get some coffee again.

Oh Nihil, sorry mate.. forgot to mention this but both F-secure and Trend's rootkit detector found nothing.. :|

Hi there ByTe,

Quote:

---

If Symantec is showing them up while scanning but they are not on the system I wonder what’s going on.

---

That has got me thinking mate! :D..................... looking at it from a reciprocal viewpoint: "What can Symantec see that I cannot, using conventional tools?"

The ones that spring to mind are:

1. RAM
2. The page file

I don't know about alternate data streams and "slack space" or "cluster tips"? If it can't detect something in the latter it isn't much good, I would have thought?

I am not yet sure where I would go from here :confused:

On my desktop machines I usually set the Registry value to have Windows overwrite the page file on shutdown. That isn't realistic with a server which is on 24/7 :(

At this point I would set the page file to minimum size. Because this is used by Windows for mini-dumps, I guess malware would not be able to access it.

I would then run "Eraser" [link below] to wipe "free space" as this would get what what was in the former page file and it also wipes alternate data streams and cluster tips.

That then leaves you with the question of how this is happening? It really has to be something that they are not scanning? Possibly:

1. Portable storage media
2. Portable devices
3. Printer Servers
4. Mail Servers.............sure the e-mails are scanned, but what about the server itself?
5. "Orphan" clients (PCs in conference & training rooms, libraries, reception areas etc.) These are sometimes overlooked when nobody has direct responsibility for them?

Good Luck!

EDIT:

http://eraser.heidi.ie/

Quote:

---

Works with Windows 98, ME, NT, 2000, XP, Vista, Windows Server 2003 and Server 2008. Eraser is Free software and its source code is released under GNU General Public License.

---

EDIT 2:

Sorry ByTe, you did title your post "examining" so here you go:

http://www.jsware.net/jsware/sviewer.php5

A tool for examining ADS and deleting nasty stuff. Sorry, but it only works with NT4.0, 2000, XP and 2003 Server.

This one will work with Vista and Win 7, but you have to pay $11 for the commercial version that lets you delete stuff. I think it is called "ADS Scanner Engine".

http://www.freesoftwaretoolbox.com/repository/

Whilst you are on that site you might like to scroll down and get "Hidden File Scanner". It is the same deal as the one above...."Look for free, pay to touch" :D

Quote:

---

Hidden File Scanner also does a quick scan at start up to detect the appearance of autorun.inf files on all devices including removable medias. If such an autorun.inf file is found, a dialog box will pop up where you can either delete, unhide or inspect the content of the autorun.inf. This tool will automatically rate the autorun.inf files as normal, hidden, suspicious or dangerous file.

---

I am sorry, but I don't know of a utility to examine cluster tips. I guess I would use a Hex Editor or Disk Investigator.

http://www.theabsolute.net/sware/dskinv.html

It works with 2000 and XP but I don't know about 2003 Server.

Nihil all these files have a path! If they are in the swap file (which if I am not wrong cannot be scanned by an AV) the path would be to that file. I am sure they are not in the RAM because I know what's loaded onto the memory (It's highly unlikely that a VBS and INI files are loaded in the memory and I cannot see them). I've used systeminternals to go through the current processes and all the files attached to that process.

I am planning to have a meeting with some senior Symantec engineer who can assist us ruling out a product glitch (which would be really pathetic). Every time I open a case with Symantec I have to run the stupid load point analysis and diagnostic tool.. LONG STORY !


Anyway I've uploaded few jpeg's showing the scan window and the file names. I’ll hopefully sanitize the longs so I can upload them here. Because ADS and other logs have come back with lot of data but not a single file which looks suspicious.


Thanks again mate. I'll go through your post again tomorrow (today).. I'm sleepy, it's (originally 001114 Hrs before i started uploading the print screens) 0125 Hrs here.

I spent 10 minutes trying to get 2 print screens up. Either it's me (sleepy) or the attachment policy that needs an overhaul.

Hi there ByTe,

Please check your Private Messages.............I have sent you an e-mail address that accepts attachments up to 10 megabytes.;) We might find that useful in future?

I will raise the issue of attachment policies, as we have reverted to the vBulletin defaults since the " night of the great upgrade" :eek:

Obviously, I do not want to take discussions "offline" of the forums.....it is simply an issue of how to handle data over a long distance. Hey, ByTe, you must be about 9,000 miles away from me? :D

Now, to the case in hand:

ByTe, are you saying that Symantec is reporting malware at a particular location (path) and that when you go there, you cannot find it?

Quote:

---

I am planning to have a meeting with some senior Symantec engineer who can assist us ruling out a product glitch (which would be really pathetic)

---

Hmmmmmm......................................:halo:

OK, let's have another look?

1. Malware might attack a normal ( visible) system by three basic methods:

a) append to an existing item
b) prepend to an existing item
c) inject into an existing item

2. Your AV is finding stuff that quite sophisticated analytical tools can't?

What are the Symantec messages?

Johnno

I've presented my report. Basically (preparation, detection, containment and current analysis). I'll update soon (possibly 2 or 3 days). I met a symantec rep who wasn't too happy with the findings, nor was the central anti-virus head; knowing we had almost ALL servers down with the same thing.

:)

I am not sure if anyone has paid any attention to this but since this incident involves Windows Server 2003, I have to compile a report on the security aspect of the OS itself.

I was going through secunia to see how many vulnerabilities have been reported and so on and honestly this shocked me. There are - 14 unpatched vulnerabilities!. I was hoping Microsoft to be little more responsible towards server grade OS's but no! I just wanted this to be a part of the thread since it might help someone else prepare a report. Also my break-in (or at least the date i noticed it) was close to some popular site's getting DDOS'ed. While these servers have been infected in the past without any reason’s being found. It is still worth noting.

I wanted to get few files for analysis and I tried using Knoppix live CD but it didn't work because for some odd reason knoppix is not able to mount the drive. Server uses a SCSI interface.

I get the following error, anyone who can assist please let me know.. I will google it once i get home (leaving office right now)


(i)org.freedesktop.hal.device.volume.unknown(/i)

I was using Knoppix CD version.

Quote:

---

Originally Posted by ByTeWrangler

(i)org.freedesktop.hal.device.volume.unknown(/i)

I was using Knoppix CD version.

---

I believe there is an easy fix for this. At the boot prompt, type in "knoppix noideraid" without the quotes.

Knoppix cheatcodes

The ftp site I gave you should have the latest and great copy of that LiveCD. Just edit the ftp URL to "ftp://ibiblio.org/pub/linux/distributions/knoppix"

Quote:

---

If you need additional modules for starting controllers needed at boot
time, just copy the corresponding *.ko files from /lib/modules/* over to
/modules in the initial ramdisk (remaster needed).

---

You will need to uncompress initrd to a temp directory; make your changes; compress back into a initrd image; remaster the CD. Initrd howtos are available on the interweb.

Sad as it is. Spending new years eve and other time doing stuff that ... (anyway)


This is the reply from Symantec:

*******************************************************************
> Question/Issue:
How does Active (Quick) Scan function?
> Symptoms:
Active Scan is scanning files that cannot be found on my machine.
> Cause:
The file is not located on the machine, it is part of a script of most common file types, viruses, and file names that the active scan is searching for on your machine.
> Solution
The Active Scan, scans the system memory and all the common virus and security risk locations on the computer very quickly. The scan includes all processes that run in memory, important registry files, and files like config.sys and windows.ini. It also includes some critical operating system folders.
Memory Examples:
- The processes that are located in Task Manager.
Common infection locations and viruses would be for example: - C:\Windows\System32\dll.dll
Common registry keys Example:
- HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run
Scans the common infection locations in addition to all files or the types of files or directories that you selected
These locations are all well-known virus and security risk locations in addition to all files or the types of files or directories that you selected. This will quickly assess if your machine has one of the more common viruses located in the common locations.

******************************************************************


There is still 2 servers with confirmed infection, i'll send updates as we progress from here.
Also these files showed up while full scan, that is due the fact (feature) that all full scan's start with active scan.

Hi there ByTe,

Quote:

---

I would say that what you are seeing whilst the scan is in progress is what it is scanning FOR, not the files actually being scanned.

---

(post #4 in this thread)

Quote:

---

The file is not located on the machine, it is part of a script of most common file types, viruses, and file names that the active scan is searching for on your machine.

---

Looks like my initial suspicion was pretty much correct then?

Nice to know how it works.............I am sure to come across this, and now, thanks to your efforts, I have some sort of an answer.:)

Cheers

I honestly felt disgusted when our second meeting with Symantec took place. In the first meeting it was decided that we declare it as an incident and initiate our place (which anyway don't exist), anyway second meeting one guy walks up with a print out of how their scan technology works. This has not been documented. They created this post the case I opened.

I asked them if the first set of engineers who responded would not know this feature and their simple answer was, "maybe, not". That’s why I didn't write anything in the post here.

I spend hours trying to see if I can detect anything (which wasn't there). Sigh. The reason I kept on trying was because their first set of engineers didn't know of the feature and also asked us to submit few files they found in their load point analysis log.

Thanks to everyone who helped.

Life is at a low right now.

Hey ByTe!

Quote:

---

Life is at a low right now.

---

In the mail tomorrow will be a job offer from Semantec! :lildevil:

I know...........it doesn't help if they don't know their own products?

;)

my lawyer will eat this mutha alive.

Think of the positive side of things. you now have a complete toolkit to help you when you have a real problem thanks to all the links supplied by nihil. As to service engineers normally the first line of "engineers" are there to buy time till the big boys can have a look at the problem.
P.S Hi Nilih

Hi MURACU, and a happy New Year mi buchum,

How are things in "Gay Paree"?

Long time no hear? are you well, and all of the family?

Cheers mate! pop in more often huh?

Johnno

Alright, I need some help with getting this server booted with live CD. Knoppix DVD is not working because server is not able to boot with a DVD (its a HP DL360 G2[http://h20000.www2.hp.com/bizsupport...dTypeId=15351]) server. It uses Smart Array 5i Controller.

Does anyone have any recommendation for which Live CD to be used? I have only used knoppix which works perfect. What else should I do? My purpose is to mount the HDD's and later the file system to get 2 files to analysis. We are looking at RAID 5 SCSI interface.


Thanks in advance.

Hi there ByTe,

As far as I am aware, the Knoppix DVD just contains a whole lot of other open source/freeware, and the core functionality is still contained on a CD.

http://www.knoppix.net/

I also think that you can use a flash memory stick if your server will recognise one of those?

:)

It doens't recognise DVD drive.. Thats what the server management team said.. ~!~

The next best thing is to grab a Ext. USB CD Drive - the HP bios will support the USB Optical Drive after a reboot - I like to keep one handy for those computers that do not have a CD Drive or it is broken.

USB Boot Drives are OK too but sometimes can be fickle to get working, plus compare the time it takes to make a USB boot device to simply connecting the USB CD Drive and hitting reboot.


CTO

You could also try to generate a windows FE live Cd. It is a bit limited on the tools you can put on it and I am only just starting to experiment so not sure just how usefull it will be. Still my first attempt at creating the boot CD did work on an Apple laptop. I used a windows 7 machine as the base but you can use a vista or windows 2003 server to make the iso also. You will also need to download the windows automatic installation kit.
cheers Muracu

Happy new year to you too Nihils,

It has been a while but I'm good and still in Paris. How are things with you? Hope you've been kkeping well.
Byte sorry for the thread jacking :)