Examining a compromised server.
Page 1 of 4 123 ... LastLast
Results 1 to 10 of 31

Thread: Examining a compromised server.

  1. #1
    StOrM™
    Join Date
    Aug 2004
    Posts
    1,003

    Examining a compromised server.

    I'll be working on 31st it seems. I was on audit and found that while running a quick scan on a critical server there were lot of files that were being scanned (mostly with sys extension) but when I ran a search for the same files, I couldnít find anything. I was sort of expecting that because file names are hide_evr2.sys which according to lot places is a rootkit. Now the nature of the server is "extremely critical". Seeing this server compromised to an extent that Symantec Endpoint can see the file but can't detect it (I am not sure if they have def's for it but in any case it's freaking out stuff).

    I'm writing this at a point where Iím finding almost every server I examine with the same malware so Iíll come down to the question

    What should be my next step in examining the server?

    Please note that we don't have a "real" incident response team or plan. I know my next level of escalation and I will be doing that accordingly. My question simply is how I examine these server from here on. I will of course create a copy of the HDD and examine there so beyond that please let the suggestions flow.

    I know this case is related to malware infection only but Iíve started it in Microsoft Security Discussions the reason being itís a server running windows OS (2003) and this thread may server as a point of reference.


    PS: Thanks a lot and HAPPY NEW YEAR IN ADVANCE.
    Parth Maniar,
    CISSP, CISM, CISA, SSCP

    *Thank you GOD*

    Greater the Difficulty, SWEETER the Victory.

    Believe in yourself.

  2. #2
    Member d34dl0k1's Avatar
    Join Date
    Mar 2007
    Posts
    58
    did you actually discover malware or any malicious activity or are you only looking at smoke at this point? if symantec didn't tell you the server was owned, get a second opinion so you're not caught saying 'i took the server down because google said this file is a rootkit'.

    other than that - examining the server might not be your next step, but beginning to build up its replacement might be a higher priority to ensure continuity.

  3. #3
    StOrM™
    Join Date
    Aug 2004
    Posts
    1,003
    To be frank, I don't have a single file from the one's i mentioned. But everytime I have scanned I can see them in the list of file's being scanned. I am sure that it's unusual that files that i have found in the scan but not on the system when i search for them are malicious. I mean all the file names i searched for are all part of either a rootkit or trojan.

    Also all drives have autorun.inf file in the drive root file but i can't find any of them in search (using systeminternal tools / GMER rootkit detector). I can see symantec showing them up during scan but not on the machine.

    My hunch is beyond google saying yes to the file. Because most of these servers were infected in past 6 months and there was never an analysis done to find the root cause. How did they get infected was never found. Also infection was not limited to 1 or 2 files. It was like one day symantec's auto protect went crazy and found 100's of files infected. When a full scan was run more files were found all infected with a single malware. This pattern is consistant for almost all servers.

    I'll post more soon. Thanks anyway.
    Parth Maniar,
    CISSP, CISM, CISA, SSCP

    *Thank you GOD*

    Greater the Difficulty, SWEETER the Victory.

    Believe in yourself.

  4. #4
    Super Moderator: GMT Zone nihil's Avatar
    Join Date
    Jul 2003
    Location
    United Kingdom: Bridlington
    Posts
    17,190
    Hi there ByTe and a happy New Year!

    I am afraid I don't know how Endpoint works, but I presume that it produces a summary report when it has completed its scan? If it does not include these items then I would say that what you are seeing whilst the scan is in progress is what it is scanning FOR, not the files actually being scanned.

    You mention hide_evr2.sys, which is a rootkit and has quite a few variants. My problem is that is a rather old bit of malware (2006/7) and really should have been blocked, particularly as the name is a dead give-away?

    I would certainly try some other products and possibly HijackThis!

    It is my belief and experience that if an anti-malware product finds suspected malware it will report it. It may not be able to clean or remove it, but it would give you a hardcopy report Otherwise I would expect it to throw up a warning message on the screen.

    You mentioned that these servers have been compromised earlier and that no follow-up took place. This gives me another thought.........are you detecting "traces" rather than the real thing? I can't really tell you what a "trace" is, but I would think orphan Registry entries, temporary files and the like. That would explain why you cant find the executables?

    The best product I know for detection and removal of those is A-Squared

    http://www.emsisoft.com/en/software/free/

    Unfortunately, it is only free for private use, but you might be able to get an evaluation/trial copy

    Good Luck!
    If you cannot do someone any good: don't do them any harm....
    As long as you did this to one of these, the least of my little ones............you did it unto Me.
    What profiteth a man if he gains the entire World at the expense of his immortal soul?

  5. #5
    StOrM™
    Join Date
    Aug 2004
    Posts
    1,003
    Happy New Year.

    How is it possible that on all the server that I run a scan with SEP, I see autorun.inf file in at least one of the drives? How is it also that every time I ran GMER (www.gmer.net) anti-rootkit detector the machine crashed.

    Also all server has been infected in last 6 months without any reason being recorded. It was like an administrator logged in the morning and found Symantec screaming of how it found a malware when he was away. Also the malware detected is mostly W32.Imaut.CN (http://www.symantec.com/security_res...022015-1923-99). I have submitted not less than 20 different samples of this same malware to Symantec when I have seen machine having same foldername.exe file inside almost every folder.

    It's tiring right now with the kind of load I’ve been given. I am having a hard time investigating because I know there is a malware on the machine that is not getting detected. I know Symantec uses veritas technology in their corporate version of AV to find rootkit's. But here it is getting fooled. I can take a print screen of Symantec scanning all the file's I’ve mentioned and put it up if someone wants to verify my cause for this thread but beyond that I have nothing much to give. I'll be going through systeminternals utility suit now to see if there is any way I can pick up files hidden to the OS. I remember ADS was the way to go but I need to refresh myself.


    Anyway thanks again for your help.

    Happy New year (again).


    PS: Nihil Symantec does not store names of every file it scanned in the report. It just store's any infected file it found.
    Parth Maniar,
    CISSP, CISM, CISA, SSCP

    *Thank you GOD*

    Greater the Difficulty, SWEETER the Victory.

    Believe in yourself.

  6. #6
    Senior Member IKnowNot's Avatar
    Join Date
    Jan 2003
    Posts
    792
    ByTeWrangler ;

    I am curious as to how this turns out, so just to update:

    By now you have cloned the drive of a suspected server, and hopefully
    have a replacement server in place while you verify that there is
    indeed a compromise of the system, and are working on a cloned drive.

    The problem I see is that you mention two different types of malware,
    one hiding inside ADS and one which may be kernel level.

    For the ADS, you have used
    HijackThis as nihil suggested,
    or, since you are using Sysinternals Suite , you have used the utility
    Streams to discover the ADS.
    What did you discover? Were any malicious?

    Since there may be a kernel level rootkit, you have also booted the system
    to a CD ( to use a known good kernel ) mounted the cloned drive, then looked
    for the mysterious files? Did you find any?

    Your hide_evr2.sys may be Symantic's
    Infostealer.Snifula.B
    So were there any of the suspicious files or reg entries that either of the
    Symantic sites indicate?

    Any indication of unusual ports open?
    ( I would test from both within, using something like TCPView and from outside
    using something like Nmap and then maybe Wireshark ,
    or whatever you have available to you. )


    If something was found, did the file dates give any clue as to when or how
    the box was compromised? That may be important when trying to discover
    the extent of the problem and how to avoid it reoccurring.
    ( I know this goes without saying, but I would not try to repair a box that had
    been rooted. And depending how it was done, even the replacement box could
    now be compromised. )

    You are much more knowledgeable then I,
    but I think maybe the enormity of the task ahead has got you riled.
    Just go back to basics:

    1) Put on a pot of coffee
    2) Order pizza
    3) take it one step at a time
    4) document everything you do

    Good hunting.
    ---------------------------------------

    I just find the timing of this strange:
    What's up with port 12174? Possible Symantec server compromise?
    " And maddest of all, to see life as it is and not as it should be" --Miguel Cervantes

  7. #7
    Super Moderator: GMT Zone nihil's Avatar
    Join Date
    Jul 2003
    Location
    United Kingdom: Bridlington
    Posts
    17,190
    Hmmmmm..................

    It was like one day Symantec's auto protect went crazy and found 100's of files infected. When a full scan was run more files were found all infected with a single malware. This pattern is consistent for almost all servers.
    So, if it is supposed to "autoprotect" why didn't it?.................if it really can detect them? I am wondering if this happened right after a Symantec update?

    I would certainly use another AV product or online scan to check out the Symantec findings.

    When you want to check individual files then please send them to these:

    http://virusscan.jotti.org/en

    http://www.virustotal.com/

    And get an opinion of them scanned by multiple AV products

    I would mention that I have rarely had satisfactory responses from AV vendors............they are way too busy trying to cover their own arses to care

    My next step would be:

    1. CCleaner
    2. Eusing Free Registry Cleaner

    Those will get rid of "rubbish"

    3. A-Squared
    4. Malwarebytes Anti-Malware
    5. Spybot S&D

    They should get rid of any traces and other crap that are in there

    Then the "rat hunt" starts

    1. Rootkit Revealer:

    http://technet.microsoft.com/en-us/s.../bb897445.aspx

    2. Blacklight:

    http://www.f-secure.com/en_EMEA/secu...ght/index.html

    About this time I would re-run my AV and see if it still finds things?

    Then HijackThis! to see what is running on the machine.

    http://download.cnet.com/Trend-Micro...-10227353.html

    Send the results here to begin with and look at what they say is bad or unknown:

    http://www.hijackthis.de/

    As I am sure that you are aware, any backups they have made will probably be quite useless as they will have backed up the problems as well

    The only reliable route is to back-up the data (scan it afterwards for malware) then wipe the disk and reinstall.............great fun on a critical server?.................but I don't suppose you want to go there?

    As for "autorun.inf" what do Symantec say about these files? If you can find them just submit them to the first two links in this post.

    I have just checked this stand alone machine and I have 9 instances of that file scattered around it.

    Try this: http://www.softpedia.com/progDownloa...oad-85585.html

    But be careful with all this stuff as they might find false positives. That is why I linked the two free scanning sites first.
    If you cannot do someone any good: don't do them any harm....
    As long as you did this to one of these, the least of my little ones............you did it unto Me.
    What profiteth a man if he gains the entire World at the expense of his immortal soul?

  8. #8
    StOrM™
    Join Date
    Aug 2004
    Posts
    1,003
    Our IT department is really fragmented here. Just to get a clone of the server my request is in consideration. I had to brief our Assistant CIO to speed things up. I haven't actually got lot of time to run any utilities till now.

    However GMER crashes the server. I ran streams but didn't find the *whatever* to go through the results since i ran them somewhere before i passed out for the day.

    I am going to take IKnowNot's suggestion and get myself fresh for this. I know this is going to be long. I'll be posting everything I do here so anyone else referring to this can either use the same steps or improve it.

    Nihil i'm going to go step by step and compile a report on what i've found.

    Here is what i'm planning to do:

    1. List all autoruns - Autoruns by systeminternals
    2. List all open ports - TCPView by systeminternals
    3. List all shares - Sharenum by systeminternals
    4. List all process and and all attached DLL's to them - different process tools by systeminternals

    once i'm done with that i'm again going to go for coffe + pizza routine. Then I shall run ADS and file analysis utilities etc..

    Iknownot thanks for the suggestion again.. I think thats what i really needed.

    We will not be cleaning the server of course and we will be going for a new installation.

    Don't mind my replies. I've hardly slept. I shall put a more rational (if thats the word) reply once i get fwe things done and get a shower and new sets of cloth's..

    Thanks again to everyone
    Parth Maniar,
    CISSP, CISM, CISA, SSCP

    *Thank you GOD*

    Greater the Difficulty, SWEETER the Victory.

    Believe in yourself.

  9. #9
    StOrM™
    Join Date
    Aug 2004
    Posts
    1,003
    Alright I have run for few reports. It’s through Microsoft remote desktop. On a live server (not in production environment anymore.)

    I randomly stopped the scan and it showed file name "twext.exe”. Guess what I searched for the file and it wasn't there on the system. I searched using GMER and it wasn't there. I searched for the file name online and turns out to be an info stealer as per prevx (http://www.prevx.com/filenames/46985...TWEXT.EXE.html). I did find the same name .dll file.

    I also found to VBS scripts. Virusremoval.vbs and newvirusremoval.vbs. As for all the files that I’ve searched for it's not on the system. If Symantec is showing them up while scanning but they are not on the system I wonder what’s going on.

    I also found an IP that the scanner shows just before stopping! I am not sure if this is the host file or the DNS records. I am confirming that in sometime (need coffee).

    I am really concerned. I have saved all the reports that I have mentioned in the previous post. Any member that wants to have a look at the reports can PM me. At this moment I'm not willing to put them here. I will go through them ensure they have nothing important that can (further I guess) compromise security and post it here.

    My eyes are getting heavy again. I'll write something once I get some coffee again.
    Parth Maniar,
    CISSP, CISM, CISA, SSCP

    *Thank you GOD*

    Greater the Difficulty, SWEETER the Victory.

    Believe in yourself.

  10. #10
    StOrM™
    Join Date
    Aug 2004
    Posts
    1,003
    Oh Nihil, sorry mate.. forgot to mention this but both F-secure and Trend's rootkit detector found nothing.. :|
    Parth Maniar,
    CISSP, CISM, CISA, SSCP

    *Thank you GOD*

    Greater the Difficulty, SWEETER the Victory.

    Believe in yourself.

Similar Threads

  1. Windows Error Messages
    By cheyenne1212 in forum Miscellaneous Security Discussions
    Replies: 7
    Last Post: February 1st, 2012, 02:51 PM
  2. Central Secure Logging in a Win2k Environment
    By Tiger Shark in forum The Security Tutorials Forum
    Replies: 5
    Last Post: March 4th, 2004, 05:00 PM
  3. Understanding DoS
    By NullDevice in forum The Security Tutorials Forum
    Replies: 21
    Last Post: December 17th, 2003, 10:03 PM
  4. Newbies, list of many words definitions.
    By -DaRK-RaiDeR- in forum Newbie Security Questions
    Replies: 9
    Last Post: December 14th, 2002, 08:38 PM
  5. How To Set Up An IRC Server (IRCD) Tutorial
    By Dome in forum Other Tutorials Forum
    Replies: 11
    Last Post: August 21st, 2002, 04:38 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •