Example Forensics SOP/Procedure
I just finished the initial draft of the "Incident Response" SOP that is required to comply with federal law. Since it is an exhaustive document that took a while to compile I thought I should post it here to assist others in both the process and the documentation.
If anyone can see any glaring omissions or errors I would appreciate a *heads up* before I publish this to the SOP file here at work. All the tools should be "Googleable" by just their name. If you have a problem finding them send a response and I'll point you to them.
I have attached an .rtf of the file for those of you that like "pretty".... ;)
STANDARD OPERATING PROCEDURE
NAME OF SOP: INITIAL PRODECURES IN THE EVENT OF A SUSPECTED NETWORK INTRUSION.
PURPOSE: While it is the intent of the COMPANY X’s IT Department to ensure that the potential for intrusion into the COMPANY X’s Wide Area Network, (WAN), and it’s associated networks from the outside are minimized and that the unavailability of the data of the participating Agencies be maintained from unauthorized viewing from the inside of the WAN it is recognized that there will always be the potential for compromise. This document outlines the procedures to be carried out in the event that there is suspicion and/or evidence of such activities.
It is important to understand that compromise can come from both without and within the network. It is further important to understand that the perpetrators second task, after the initial penetration, is to hide their activity by deleting logs, preventing other access to the compromised machine, installing “back doors” to give them unfettered access to it, “sniffing” the network to catch login/password data sent in clear to help them elevate their privileges within the domain, installing “kits” that allow them to carry out other actions and even to install “rootkits” at the user or even the kernel level so that while the machine appears to be cooperating with you it is very subtly hiding the presence of the perpetrator.. In short, depending upon the skill level of the attacker, everything will be done to make the investigator(s) job more difficult or even impossible. The process documented below follows the steps it does because the more sophisticated attack systems may have self protection built into them. An example would be a backdoor listening on port 1234. It will reply to a simple SYN with a SYN/ACK, (which it is supposed to), but, if the returning ACK packet does not contain certain data in the payload, (there should be no payload on a standard ACK packet), then communication will cease, or, worse yet a clean-up routine begins where the compromised machine protects itself, (drops all packets from the network and ignores console input), while it removes all evidence of itself and then reboots to a “clean” system. While I know of no working systems such as this currently “in the wild” the potential is certainly there and as the level of sophistication rises the probability of such systems is quite high.
APPLICATION: This policy applies to all IT staff of COMPANY X and it’s associated agencies.
PROCEDURES: There are occasions where a machine may give the impression that it might be compromised and there will be occasions where it is quite clear that compromise has taken place. The first act in the case of suspicion of compromise is to consult with the MIS/designated security person in the IT department to determine whether the steps this policy outlines should proceed. Prior to consultation make sure you note down exactly why your suspicions were aroused, any error messages that appeared, (the entire text), activity that was apparent etc. so that the lengthy process that follows is not gone through in vain. In short, think it through - is this really a potential compromise or is there a viable explanation for the activity, (but don’t be too quick to pass it off as a “glitch”, often the only indication of a compromise is a very subtle sign or signs). At this point great care must be taken to not alter the machine or carry out actions that might begin what are known as “kill processes” that clean the machine. Do the absolute minimum you can to confirm your suspicions. If you are at all unsure how to proceed at this point do nothing. Make the appropriate notes regarding how you were alerted, what you have done and call COMPANY X’ MIS for advice on how to proceed.
Every step taken is to be logged. The log sheet is available from here. Logs will be created for each asset and it’s components that are being investigated. Logs entries are to contain investigators initials, (legibly), date, time, action, tool, tool purpose, result, evidence location, (file name, printed document etc.). If floppy disks are used the log should indicate which floppy the evidence is located on and it’s file name. Floppy Disks are to be clearly labeled EVDISKX where X is the disk number, dated and the machine name being investigated, (eg. EVDISK1 12/21/03 RM123P). CD-ROM’s are to be similarly labeled. Make sure that there is no confusion caused by disk naming, (if there are 8 floppies and one CD-ROM, label them in the order of creation, Thus if the CD-ROM was the fourth disk created it should be labeled number 4). If components are removed from the computer such as the hard drive this action must be logged and a manifest created detailing date, time, action performed, signature. The drive is to be clearly labeled with the name of the machine it was removed from, the date and time of removal and the name of the person that removed it. When it is placed in or removed from storage, passed on to another person, or forensic tests are done on it the manifest must detail everything. In this way we create a chain of evidence that may be used if legal action is deemed necessary. All floppy disks should have their data backed up to a trusted and secure computer, (standalone), as soon as possible to prevent the possibility of data loss through bad floppies. When sufficient data is collected it should be written to CD-ROM to preserve it. Data written to CD-ROM for preservation purposes should be checked to ensure the write was good and labeled as “FINAL EVDISK 12/21/03 Rm123P). The original evidence disks are to be secured along with the original copy of the Incident Response Log. The Incident Response Logs should also be photocopied at the completion of each sheet. At the completion of the investigation and after the “of” sequence numbers have been assigned to the original logs the entire log for each asset is to be photocopied and secured in a location separate from the originals. The importance of these logs cannot be over-emphasized - log it, log it, log it………
The following are the steps to be taken on each computer or server that is considered to be compromised. This document is to be printed and followed by each member of the IT response team in the order it is written unless the team leader determines that an alternative route is justified. The printed document should be shredded upon completion of the investigation.
These procedures should be carried out prior to any investigative procedures since they make no contact with the suspect machine(s) whatsoever and rely upon totally passive methods or information that already exists.
1. The suspect machine(s) is to be left “as-is” unless, in the opinion of the MIS the damage/risk associated with such action is unacceptable. The machine therefore should remain switched on, connected to the network and no attempts are to be made to glean information from the computer locally. In short - Leave it alone until the IT Response Team leader arrives.
2. A log is to be started to document every action taken with regard to each machine and it’s components. This is especially important should the Agency consider legal action against the perpetrator(s).
3. The COMPANY X Computer Incident Response Team should be notified immediately. Team Members are detailed here.
4. The existing log files are to be secured. They are located on the workstation named XXXXXXXX in COMPANY X’s MIS’ office in the folder E:\Syslog\logs. The file is named with that days date with a .txt extension, (eg. 2003-09-10.txt). Depending upon the time of day this file can be large, (> 50Mb), so it should be copied to a remote computer. Once the copy is complete remove the network cable from the machine that the copy went to. Do not stop the logging process or disconnect the XXXXXXXX computer from the network. It needs to continue to do it’s job. XXXXXXXX’s login is XXXXXXXXX with a password XXXXXXXX where X is the key combination <ALT> and the keypad numbers XXX.
5. In the subfolder of XXXXXXXX’s E:\Syslog\logs folder called Old logs you will find copies of previous days logs. On COMPANY X’ MIS’ workstation in a folder called c:\log analysis\old you will find similar copies of these files. Compare the file sizes of each file on both machines. If they are all the same, XXXXXXX has a CD writer, cut all the files in that folder to CD, label it appropriately and secure it. If the files differ in size the compare the files of the first two copies of the logs to the final copy of the logs on COMPANYXBU in folder h:\Information system\xxxxxxx\Security Archives\firewall logs. If COMPANYXBU’s files and COMPANY X’ MIS’ files are the same size cut either to CD. If all three are different cut all three to CD for future review. Label and secure all CD’s cut.
6. At the appropriate gateway to the network Ethereal is to be started with a filter applied of “host xxx.xxx.xxx.xxx”, (without quotes), where xxx.xxx.xxx.xxx is the address of the internal machine to determine if the machine is communicating with the internet. Gateway monitors are available via computers rm258, (COMPANY X MIS’s PC), and xxxxBU, (Backup Domain Controller in xxxxxxxx). If Ethereal indicates traffic to and from this machine Ethereal is to be left running until such time as deemed fit by the IT response team leader. The data collected is to be kept as forensic evidence.
7. If traffic is detected it may be useful to place a second version of Ethereal running on the local subnet of the target machine to determine if it is communicating with other assets inside the WAN. If it is then this data is also to be kept as forensic evidence. It may not be possible to use Ethereal on the local subnet due to the use of switches. It will be the decision of the IT response team leader whether to quickly rewire the affected machine(s) through a hub so that Ethereal can be used.
8. Secure the computer’s last AIDA32 inventory file if present. This will be found on the X: drive under AidaReports. The filename will be the computer’s name with a .csv extension. This details the exact hardware and much of the computer’s state the last time an audit could successfully be carried out and may even contain the user name the perpetrator logged on as.
Non-Invasive Remote data gathering Procedures
These procedures are those that are carried out from a remote workstation on the network that use tools that are active and either request information from the suspect machine(s) or glean information from it’s responses to probes and scans, (or lack of responses). If possible run these tools from a workstation that can see the target machine and that all traffic can be seen by one of the Ethereal sniffers so that the packets themselves are logged for future reference. Note that the tests could be run from the appropriate Ethereal sniffer set up in the first phase though there is a small risk of packet loss. Better to use a laptop or other machine connected to the same hub as the Ethereal sniffer. Further information on the tools, their use and the command lines to execute can be found in the section “Remote Tools, their purposes, output and use”.
1. NMapWin: This tool is used first in the stealth mode to glean as much information as possible about the computer without making a complete connection. If the computer has been set up with a “kill process” if unauthorized attempts to connect to it take place this should not set the process off. Note: All tools after this point elevate the risk of triggering a “kill process”.
2. FScan: This will run a full connect scan to every port and grab any information it can from open ports. From here on, run the tools, log the output, check the output for items of interest and decide if this process should continue or whether the situation requires an alternative plan of action. Any alternative plan of action may only be authorized by the IT Response Team leader.
3. Cerberus Internet Scanner
Non-Invasive Local Information Gathering Procedures.
From this point onwards it is imperative to remember that you can trust nothing on the computer(s) being investigated. Your attitude must be that every existing piece of code on the target machine has been subverted. You can’t even trust the command prompt that you will use for many the following tools. There is a trusted copy of many common applications on the CD, USE THEM.
Use floppy disks to save the data to. If no, or insufficient floppies are available, and a network connection is still operable create a share on a remote computer that you have only write access to and redirect the output there. Preferably use floppies but if using recycled floppies is required, format them on a separate, trusted computer, send the output to them and immediately copy the files to another trusted, (preferably not connected to the network), computer.
Run each tool twice. The first time redirect the output to file and examine it on a different computer. The second time run it without redirection so the output comes to the screen. Run a quick comparison between the output to file and the output to screen to ensure they are consistent. If the output is consistent no further action is necessary. If the output is different then run the same tool three further times redirecting the output to file and number each file appropriately, (for example Netstatarm123-1.txt).
You may also notice that some of these tools may duplicate information that others provide. That is deliberate. Please do not skip steps simply because you think you have the information already….. You don’t.
Finally, many of these tools are not capable of changing the state of the target computer. Others most certainly are. The goal at this stage is to glean as much information as possible about the current state and configuration of the target machine. Run the tools exactly as requested unless authorized by the IT Response Team leader. If you are uncertain whether the tool will change the system or not consult with the team leader prior to executing the tool.
Start by spawning a command prompt by selecting Start-Run and manually typing in “%path%\cmd.exe” where %path% is the location of the trusted tools. Then run the following command lines where %path% is the path to the location of the trusted tools and %path2% is the path to the output resource, (floppy, write only remote share).
NOTE: A “*” indicates a tool that has the functionality to alter the configuration of the target. Make sure the command lines are correct prior to running the tool.
1. %path%\ipconfig /all > %path2%ipconfigrm123.txt
2. %path%\ipxroute.exe > %path2%ipxrouterm123.txt This will determine if IPX has been enabled.
3. (*) %path%\arp -a > %path2%arprm123.txt
4. %path%\hostname > %path2%hostnamerm123.txt
5. %path%\mem /c > %path2%memrm123.txt
6. (*) %path%\net accounts > %path2%netaccountsrm123.txt
7. (*) %path%\net localgroup > %path2%netlocalgroup.txt
8. (*) %path%\net share > %path2%netsharerm123.txt
9. %path%\net statistics server > %path2%netstatsserverrm123.txt
10. %path%\net statistics workstation > %path2%netstatsworkrm123.txt
11. (*) %path%\net time <\\rm123> > %path2%nettimerm123.txt (Substitute the target computer name for \\rm123 in the command line)
12. (*) %path%\net use > %path2%netuserm123.txt
13. (*) %path%\net user > %path2%netuserrm123.txt
14. %path%\net view > %path2%netviewrm123.txt
15. (*) %path%\route print > %path2%routeprintrm123.txt
16. %path%\fport > %path2%fportrm123.txt
17. %path%\listdlls > %path2%\listdllsrm123.txt
18. %path%\promiscdetect > %path2%\promiscrm123.txt
19. (*) %path%\psservice > %path2%\psservicerm123.txt
20. %path%\psgetsid > %path2%\psgetsidrm123.txt
21. dir /s > %path2%\dirXrm123.txt (do for each volume, X = volume letter including mapped drives)
22. (*) date > %path2%\daterm123.txt
23. (*) time > %path2%\timerm123.txt
24. vol > %path2%\volXrm123.txt (do for each volume, X = volume letter including mapped drives)
25. Tree > %path%\treeXrm123.txt (do for each volume, X = volume letter including mapped drives)
Those are the command line tools completed whose output has to be redirected to text files on trusted media. The following tools are more interactive and monitor either highly detailed activity in real time or more detailed configuration. The files can become rather large quite quickly if the activity on the machine is high so a floppy disk may be filled up quite quickly. If at all possible save these files to a remote, write only share, then cut them to CD.
1. %path%\Autoruns: This shows all the programs that are called to start during system start up. It documents the programs and the path to them.
2. %path%\filemon: This program shows file activity in real time. This file can get quite large if a lot of things are running. Run it for 5 minutes or so and save the output to the write only remote folder then cut it to CD.
3. %path%\ntpmon: This monitors the processes and what they are doing, (opening threads, closing them etc.). If nothing is really going on then this may remain pretty much empty. It’s a judgment call as to when to close it. If you are getting a lot of consistent activity from certain processes that looks “abnormal” then saving the data and closing it quite quickly may be fine and it might fit on a floppy. Remember though, it’s better to have more data then less. If you aren’t comfortable with the output or the amount, seek assistance.
4. %path%\procexp: This is a process explorer. It shows what processes are running and what sub-processes they have spawned. Clicking on the individual process or it’s sub-processes will show all the handles or dll’s the item uses. In the view menu select “Show DLL’s”. Create a subfolder on the remote, write only drive called Procexprm123 and then select each process and on the file menu select “save as”. It will put the appropriate file name there for you just make sure you redirect it to the new folder on the write only share. Yes, this could be a long process but it shows all the files being used, their versions etc. and may be helpful later on.
5. %path%\regmon: This is a registry monitor. It monitors all access to the registry by all systems. It fills up quite quickly so again this may be a judgment call as to when to save the data with the rider again that too much is better then too little.
6. %path%\tdimon: This is a network interface monitor that details TCP and UDP activity at a very low level. Like other tools above it can generate a lot of log s on a busy system. Save the data away to the write only share when you feel you have enough.
This completes the formalized data gathering process from the machine(s) itself. At this point the IT Response Ream Leader will make initial review of the data and decide if other or more data of certain types is required. When no other data is required the computer is to literally have the power cord pulled from the back of the machine at the power supply inlet. The computer is not to be shut down, logged off or any other way of closing it down. When computers are closed down cleanly they rewrite two entire hives of the registry and make numerous other “housekeeping” changes that are saved to the drive prior to actual shutdown. Additionally, shutdown routines could be in place to sanitized the machine in the event of a good but unexpected, (by the perpetrator), shutdown. Thus we “kill” the machine without giving it chance to alter information stored on the drive.
The final acts of this phase is to stop all the Ethereal sniffing, save the data and cut it to CD-ROM and saving all the data from any write only shares that were used to CD-ROM and removing, labeling, logging and securing the drive(s) those shares reside on.
Once the power has been removed the hard drive(s) are to be removed from the system, labeled to indicate what they are, the act is to be logged and all the relevant details are to be noted. Two Disk images are then to be made of each disk and labeled appropriately. The act is to be logged with the appropriate details. The original drive(s) and one copy are to be secured and when, where and who secured them is to be logged. The second image can then be placed on a machine as a slave. It must not be booted to as this will change the image itself. Should the image be changed in such a way as to compromise the investigation this fact is to be logged and a new image is to be created from the stored image, (avoid using the original disk(s) ever again - that’s why we made two images in the first place). Each time a disk is moved, handed from person to person, secured or brought out of secure storage the logs are to be updated to reflect who, when, where and why the change in situation took place.
From here on it is impossible to lay down any investigative procedures since they will rely entirely upon the evidence gathered in the preceding phases. When checking the evidence you should use the “FINAL CD’s” rather than the media the evidence was originally stored on to ensure that no changes to the evidence are inadvertently made. The original evidence media is to be logged and secured in the same fashion as the hard drive from the investigated machine.
Remote Tools, their purposes, output and use.
COMPANY X’s MIS has access to all these tools. They are either on his person, on his workstation, laptop or on a CD-ROM labeled “Forensic Toolkit” in his office. COMPANY X’s MIS is familiar with the use of these tools. If you are instructed to use a tool that you are unfamiliar with you are to ask for assistance prior to their use. Similarly, if you are tasked with reading the output and are unfamiliar with that you are seeing request assistance. Decisions are made throughout the process as to how to proceed that depend upon the information gained from the different systems used. If the output is misinterpreted or misrepresented an incorrect decision could be made that could render the investigation useless.
Remote, Non-Intrusive Tools
NOTE: Some of these tools require that WinPCap be installed. This is a windows packet capture driver that, at the time of writing is a version 3.2.1 and is available by searching Google for “WinPCap”
NOTE 2: Some of these remote detection tools may create alerts in the WAN’s Intrusion Detection Systems.
NOTE 3: Tools with a “*” after their name can successfully be run against machines where administrative rights are not available. In some cases the information will be complete, in other cases it will show only information that is available without administrative rights, in other circumstances all you may get is “Access Denied” Log it anyway, it’s important to know what can and can’t be done and may show that steps have been taken by the perpetrator to protect the computer from use by people other than himself.
1. Ethereal*: Ethereal is a packet sniffer that is used to sniff all traffic on a network segment. It only functions effectively from a hub or a switch that can be configured to have a “bridging” port so that all traffic can be seen on the local segment. The COMPANY X WAN has two machines set aside with this capability, (rm258 and FortBU), to cover the two gateways to the network. Packets can be captured from all machines of a filter can be set to only capture traffic of certain types. These filters are based on the TCPDump syntax and can be found at <http://home.insight.rr.com/procana/> in the document named Designing Capture Filters for Ethereal. There is a hard copy in the black binder labeled “Security Texts” in the MIS’ office. Under normal circumstances Ethereal captures packets in the background and simply shows a graph of the packets captured. In a forensic investigation it it useful to select the button to “Update the window in real time” to see the nature of the traffic involved as it occurs. When saving the data save it in the default TCPDump format which allows the data be be reloaded into many other analysis tools and even to rerun the entire session if necessary.
2. NMapWin*: NMapWin is the WIN32 port of the venerable NMap by Fyodor. It is the most powerful scanner/OS detection system currently available. For forensic purposes the following settings are advised. Select a SYN Stealth Scan from the Scan tab, “Don’t ping” from the discover tab, “OS Detection” and “Very Verbose” from the Options tab and select Output file and name it as a:\NMaprm123p.txt. Put the IP Address of the target in the Host line and begin the scan.
3. FScan*: FScan is a command line port scanner that has some interesting features. From it’s home directory type “fscan -?” for a list of all it’s parameters. A recommended command line would be fscan -bp 1-65535 -u 1-65535 -o a:\fscanrm123p.txt -s rm123. This would scan all TCP and UDP ports from 1 through to 65535 and grab the available banners of any open ports, show any RST’s returned from the target, (a lack of which may imply there is a “firewall” in place), and write all the results to a file called fscanrm123p.txt on the floppy disk.
4. Cerberus Internet Scanner*: This tool looks at services commonly available and extracts as much information as is available with or without administrative rights. With administrative right a lot can be determined about the general “look” of the target and some useful information comes with only guest rights. The report is written to the home folder’s “Reports” folder and is named using the IP address of the target with an HTML extension. The report file should be moved to an evidence floppy.
5. Currentstate.vbs: This is a script the MIS wrote himself to automatically extract information remotely and in a non-intrusive manner from a computer you have administrative rights over. Run it from it’s home directory using the command line “cscript currentstate.vbs” and you will be prompted for the IP address of the target machine, the full name of the output file, your full name, the administrators login name and the administrators password. This pulls a lot of relevant information regarding the current state of a remote machine right down to which processes belong to which threads and could be very useful in a forensic investigation.
6. PSInfo: PSInfo gives some additional information regarding the makeup of a machine that others above may not. Rights are required to the target machine. Recommended command line is “psinfo \\rm123p > a:\PSInform123p.txt”. Note that the results have to be piped to the output file.
7. PSList: PSlist dumps the running processes, their PID’s, kernel time, user time, idle time etc. Recommended command line is “pslist \\rm123p > a:\SPListrm123p.txt”. It requires rights to the target machine.
8. PSLoggedon*: PSloggedOn can reveal information about locally and remotely logged on users without administrative privileges on the target machine. Recommended command line is “psloggedon \\rm123p > a:\psloggedonrm123p.txt
9. PSLogList: PSlogList can retrieve the entire existing event logs of a remote computer with administrative rights, (it may also be able to with limited rights). Recommended command line is “psloglist \\rm123p > a:\psloglist.txt