-
December 30th, 2009, 03:33 PM
#1
Examining a compromised server.
I'll be working on 31st it seems. I was on audit and found that while running a quick scan on a critical server there were lot of files that were being scanned (mostly with sys extension) but when I ran a search for the same files, I couldn’t find anything. I was sort of expecting that because file names are hide_evr2.sys which according to lot places is a rootkit. Now the nature of the server is "extremely critical". Seeing this server compromised to an extent that Symantec Endpoint can see the file but can't detect it (I am not sure if they have def's for it but in any case it's freaking out stuff).
I'm writing this at a point where I’m finding almost every server I examine with the same malware so I’ll come down to the question
What should be my next step in examining the server?
Please note that we don't have a "real" incident response team or plan. I know my next level of escalation and I will be doing that accordingly. My question simply is how I examine these server from here on. I will of course create a copy of the HDD and examine there so beyond that please let the suggestions flow.
I know this case is related to malware infection only but I’ve started it in Microsoft Security Discussions the reason being it’s a server running windows OS (2003) and this thread may server as a point of reference.
PS: Thanks a lot and HAPPY NEW YEAR IN ADVANCE.
Parth Maniar,
CISSP, CISM, CISA, SSCP
*Thank you GOD*
Greater the Difficulty, SWEETER the Victory.
Believe in yourself.
-
December 30th, 2009, 08:43 PM
#2
did you actually discover malware or any malicious activity or are you only looking at smoke at this point? if symantec didn't tell you the server was owned, get a second opinion so you're not caught saying 'i took the server down because google said this file is a rootkit'.
other than that - examining the server might not be your next step, but beginning to build up its replacement might be a higher priority to ensure continuity.
-
December 31st, 2009, 12:43 PM
#3
To be frank, I don't have a single file from the one's i mentioned. But everytime I have scanned I can see them in the list of file's being scanned. I am sure that it's unusual that files that i have found in the scan but not on the system when i search for them are malicious. I mean all the file names i searched for are all part of either a rootkit or trojan.
Also all drives have autorun.inf file in the drive root file but i can't find any of them in search (using systeminternal tools / GMER rootkit detector). I can see symantec showing them up during scan but not on the machine.
My hunch is beyond google saying yes to the file. Because most of these servers were infected in past 6 months and there was never an analysis done to find the root cause. How did they get infected was never found. Also infection was not limited to 1 or 2 files. It was like one day symantec's auto protect went crazy and found 100's of files infected. When a full scan was run more files were found all infected with a single malware. This pattern is consistant for almost all servers.
I'll post more soon. Thanks anyway.
Parth Maniar,
CISSP, CISM, CISA, SSCP
*Thank you GOD*
Greater the Difficulty, SWEETER the Victory.
Believe in yourself.
-
January 1st, 2010, 04:09 PM
#4
-
January 1st, 2010, 06:24 PM
#5
Happy New Year.
How is it possible that on all the server that I run a scan with SEP, I see autorun.inf file in at least one of the drives? How is it also that every time I ran GMER (www.gmer.net) anti-rootkit detector the machine crashed.
Also all server has been infected in last 6 months without any reason being recorded. It was like an administrator logged in the morning and found Symantec screaming of how it found a malware when he was away. Also the malware detected is mostly W32.Imaut.CN (http://www.symantec.com/security_res...022015-1923-99). I have submitted not less than 20 different samples of this same malware to Symantec when I have seen machine having same foldername.exe file inside almost every folder.
It's tiring right now with the kind of load I’ve been given. I am having a hard time investigating because I know there is a malware on the machine that is not getting detected. I know Symantec uses veritas technology in their corporate version of AV to find rootkit's. But here it is getting fooled. I can take a print screen of Symantec scanning all the file's I’ve mentioned and put it up if someone wants to verify my cause for this thread but beyond that I have nothing much to give. I'll be going through systeminternals utility suit now to see if there is any way I can pick up files hidden to the OS. I remember ADS was the way to go but I need to refresh myself.
Anyway thanks again for your help.
Happy New year (again).
PS: Nihil Symantec does not store names of every file it scanned in the report. It just store's any infected file it found.
Parth Maniar,
CISSP, CISM, CISA, SSCP
*Thank you GOD*
Greater the Difficulty, SWEETER the Victory.
Believe in yourself.
-
January 1st, 2010, 09:33 PM
#6
ByTeWrangler ;
I am curious as to how this turns out, so just to update:
By now you have cloned the drive of a suspected server, and hopefully
have a replacement server in place while you verify that there is
indeed a compromise of the system, and are working on a cloned drive.
The problem I see is that you mention two different types of malware,
one hiding inside ADS and one which may be kernel level.
For the ADS, you have used
HijackThis as nihil suggested,
or, since you are using Sysinternals Suite , you have used the utility
Streams to discover the ADS.
What did you discover? Were any malicious?
Since there may be a kernel level rootkit, you have also booted the system
to a CD ( to use a known good kernel ) mounted the cloned drive, then looked
for the mysterious files? Did you find any?
Your hide_evr2.sys may be Symantic's
Infostealer.Snifula.B
So were there any of the suspicious files or reg entries that either of the
Symantic sites indicate?
Any indication of unusual ports open?
( I would test from both within, using something like TCPView and from outside
using something like Nmap and then maybe Wireshark ,
or whatever you have available to you. )
If something was found, did the file dates give any clue as to when or how
the box was compromised? That may be important when trying to discover
the extent of the problem and how to avoid it reoccurring.
( I know this goes without saying, but I would not try to repair a box that had
been rooted. And depending how it was done, even the replacement box could
now be compromised. )
You are much more knowledgeable then I,
but I think maybe the enormity of the task ahead has got you riled.
Just go back to basics:
1) Put on a pot of coffee
2) Order pizza
3) take it one step at a time
4) document everything you do
Good hunting.
---------------------------------------
I just find the timing of this strange:
What's up with port 12174? Possible Symantec server compromise?
" And maddest of all, to see life as it is and not as it should be" --Miguel Cervantes
-
January 1st, 2010, 10:04 PM
#7
-
January 2nd, 2010, 09:23 AM
#8
Our IT department is really fragmented here. Just to get a clone of the server my request is in consideration. I had to brief our Assistant CIO to speed things up. I haven't actually got lot of time to run any utilities till now.
However GMER crashes the server. I ran streams but didn't find the *whatever* to go through the results since i ran them somewhere before i passed out for the day.
I am going to take IKnowNot's suggestion and get myself fresh for this. I know this is going to be long. I'll be posting everything I do here so anyone else referring to this can either use the same steps or improve it.
Nihil i'm going to go step by step and compile a report on what i've found.
Here is what i'm planning to do:
1. List all autoruns - Autoruns by systeminternals
2. List all open ports - TCPView by systeminternals
3. List all shares - Sharenum by systeminternals
4. List all process and and all attached DLL's to them - different process tools by systeminternals
once i'm done with that i'm again going to go for coffe + pizza routine. Then I shall run ADS and file analysis utilities etc..
Iknownot thanks for the suggestion again.. I think thats what i really needed.
We will not be cleaning the server of course and we will be going for a new installation.
Don't mind my replies. I've hardly slept. I shall put a more rational (if thats the word) reply once i get fwe things done and get a shower and new sets of cloth's..
Thanks again to everyone
Parth Maniar,
CISSP, CISM, CISA, SSCP
*Thank you GOD*
Greater the Difficulty, SWEETER the Victory.
Believe in yourself.
-
January 2nd, 2010, 01:41 PM
#9
Oh Nihil, sorry mate.. forgot to mention this but both F-secure and Trend's rootkit detector found nothing.. :|
Parth Maniar,
CISSP, CISM, CISA, SSCP
*Thank you GOD*
Greater the Difficulty, SWEETER the Victory.
Believe in yourself.
-
January 3rd, 2010, 01:01 PM
#10
Hi there ByTe,
If Symantec is showing them up while scanning but they are not on the system I wonder what’s going on.
That has got me thinking mate! ..................... looking at it from a reciprocal viewpoint: "What can Symantec see that I cannot, using conventional tools?"
The ones that spring to mind are:
1. RAM
2. The page file
I don't know about alternate data streams and "slack space" or "cluster tips"? If it can't detect something in the latter it isn't much good, I would have thought?
I am not yet sure where I would go from here
On my desktop machines I usually set the Registry value to have Windows overwrite the page file on shutdown. That isn't realistic with a server which is on 24/7
At this point I would set the page file to minimum size. Because this is used by Windows for mini-dumps, I guess malware would not be able to access it.
I would then run "Eraser" [link below] to wipe "free space" as this would get what what was in the former page file and it also wipes alternate data streams and cluster tips.
That then leaves you with the question of how this is happening? It really has to be something that they are not scanning? Possibly:
1. Portable storage media
2. Portable devices
3. Printer Servers
4. Mail Servers.............sure the e-mails are scanned, but what about the server itself?
5. "Orphan" clients (PCs in conference & training rooms, libraries, reception areas etc.) These are sometimes overlooked when nobody has direct responsibility for them?
Good Luck!
EDIT:
http://eraser.heidi.ie/
Works with Windows 98, ME, NT, 2000, XP, Vista, Windows Server 2003 and Server 2008. Eraser is Free software and its source code is released under GNU General Public License.
EDIT 2:
Sorry ByTe, you did title your post "examining" so here you go:
http://www.jsware.net/jsware/sviewer.php5
A tool for examining ADS and deleting nasty stuff. Sorry, but it only works with NT4.0, 2000, XP and 2003 Server.
This one will work with Vista and Win 7, but you have to pay $11 for the commercial version that lets you delete stuff. I think it is called "ADS Scanner Engine".
http://www.freesoftwaretoolbox.com/repository/
Whilst you are on that site you might like to scroll down and get "Hidden File Scanner". It is the same deal as the one above...."Look for free, pay to touch"
Hidden File Scanner also does a quick scan at start up to detect the appearance of autorun.inf files on all devices including removable medias. If such an autorun.inf file is found, a dialog box will pop up where you can either delete, unhide or inspect the content of the autorun.inf. This tool will automatically rate the autorun.inf files as normal, hidden, suspicious or dangerous file.
I am sorry, but I don't know of a utility to examine cluster tips. I guess I would use a Hex Editor or Disk Investigator.
http://www.theabsolute.net/sware/dskinv.html
It works with 2000 and XP but I don't know about 2003 Server.
Last edited by nihil; January 3rd, 2010 at 02:34 PM.
Similar Threads
-
By cheyenne1212 in forum Miscellaneous Security Discussions
Replies: 7
Last Post: February 1st, 2012, 02:51 PM
-
By Tiger Shark in forum The Security Tutorials Forum
Replies: 5
Last Post: March 4th, 2004, 05:00 PM
-
By NullDevice in forum The Security Tutorials Forum
Replies: 21
Last Post: December 17th, 2003, 10:03 PM
-
By -DaRK-RaiDeR- in forum Newbie Security Questions
Replies: 9
Last Post: December 14th, 2002, 08:38 PM
-
By Dome in forum Other Tutorials Forum
Replies: 11
Last Post: August 21st, 2002, 03:38 PM
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
|