Incident Scenario

Ok gang,
I've been gone for quite a while (out paying the bills) so here's a little fun for anyone that wishes to participate. This will be a multi-phase incident response and forensics scenario.

PART I




2:30 PM
you receive a frantic voice mail message while in the car from a network engineer that was monitoring a branch office network.

"It looks like there's one system acting up in your area....wait check that, a scan revealed 9 more systems with rogue FTP servers, give me a call"


From 2:30 to 3:00 you attempt to call the network engineer and the entire network engineering team (4 people in all). You are unable to reach a single one of them.


3:00 PM
you arrive back at your office. You decide to call the director of network engineering.

You: "Hey bud, I keep trying to call your engineers but can't get through to any of them, I got a call about a branch office with a lot of rogue ftp servers, know anything about it?"

Director: "Ah yeah, the entire team is in an infrastructure meeting, they won't be out until 4:00. I haven't heard much other than what you just told me. I'll let them know you called."

You: "Ok, thanks"

You hang up the phone.

You are an Incident Handler in a distributed CSIRT. There is one other handler in the immediate area that is on your team but less experienced. You have no in depth knowledge of the branch office network configuration and you've never been on site.


What do you do?

You didnt get over your head with that resume you posted on monster.com did you? lol

The actions you should take are going to depend on a policy that you've decided BEFORE the incident occurs. Which is more important to your organisation; uptime / productivity or catching the intruder. It's the policy that's going to decide, you've given WAY to little information.

Well first.. If the fact you have rogue FTP servers on your ---hang on-- you didnt mention you said a system-- is that a server or a workstation/s
I would suspect that would be enough to stuff the importance of the infrastructure meeting, and get that teams sweet little a$$'s out and start looking into the problem.. next put your ol mate not so experienced on standby..
Your ol mates are forst going to have to determine the true extent of the problem.. and pulling plugs etc as mentioned, depends on your response policy..

Gentlemen...Please stop getting hung up on MY policy. This is ficticious and not about me or MY policy, it's about you and YOURS.

Und3rtak3r: Good question. I said system for a reason....the network engineer didn't tell you and since you've never been to the branch office, you don't know yourself.

Neptune0z: Sometimes...this is all the information you get. Nothing gets served on a silver platter when responding to incidents.


According to your policy, what do you do?

hmmm ok like my first day at my current job..."There it is.. Go to it".

First step will be to determine what machines are troubled,
I would be going through my network information to learn what I can about the Branch office. A basic map would be a great start..
A sniff of the Gateway traffic, just for the record.. and clues
Also a check for unusual activity at the other branches.

My situation is I AM the branch office. and I have to report ALL pc's on the local network, domain members and not (yes we have a gaping security risk ).. The Admins in our OZ headOffice know our setup.. (But when I started they knew only a little more than I, like the IP address of our server)

Check the sniff for the FTP traffic, this should tell me what machines are affected, also perhaps endpoint(?..probably not reliable) information,

set about to block all FTP traffic at the remote site's gateway.

Isolate the affected stations,

On my situation, the Loss/leaking of data is more important than a users PC being off line for an hour or so. Now is where my mate on the spot is then to physicly check the machines..

excuse my rough approach.. Havent put too much thought in, and not read enough in this area.. yet

I'd first do a scan to see if it's valid ftp traffic versus someone putting a service on a non-standard port (such as sshd running on port 80 to bypass simple port blockers).

Then I'd scan those 9 rogue "ftp servers" and get the mac addresses on all of them and go ahead and deny those to the internet globally.

Then I'd have port 20 and 21 blocked at the closest level of these machines (let's say they're all on the same vlan or something) and then I'd hunt those rogue machines down, and club the nearest people standing by.

Going ahead with that, I'd get some network-based application recognition and really lock things down as well as putting simple acls on the routers.

But I'm not very good with network stuff, and it's about all I can think of.

Hmmm so this is it for a response? I'll post part two some time tomorrow.

Assumption 1: FTP servers are not allowed without express permission.
Assumption 2: None of the recorded incidents have a recorded permission to run an FTP server.

Step 1; Lock out machines 1-9 from all internet access.
Step 2; Retrive HEAD information on the FTP servers (TELNET or equivelant)
Step 3; Attempt to identify through the use of HEAD information, if known to be used for viral activity, quarantine machines. (Lock out from local router).

Step 4; Aquire log information on machines 1-9 to check for potential breach, if found, identify the method used and attempt to innoculate the network. (IP tables or KIX script inoculation).

Step 5; Restore machines 1-9 using image technology or equivelant.


Thats what I'd do, but it depends a little from case to case, there mgiht be certain company rules that would stand in the way of certain actions or would require you to do other things.

300+ views and only 4 responses..disappointing.

Anyways..
Here are some key steps that should be taken.

- Contact the other CSIRT member, arrange to meet them in the lobby or parking lot of the branch office.

- Contact the Branch office. Let them know you'll be coming and that there is a possible service affecting problem.

- Pack a jump bag containing your incident response toolkit and forensics tools.

- Travel to the site.

- Identify the affected systems. You can't react appropriately unless you know what you are dealing with.


PART II

4:00 rolls around and while travelling to the branch office you get a call the network engineer that called you at 2:30.

Engineer: "Hey, sorry about that we got called in to an infrastructure meeting and couldn't get out of it"

You: "yeah yeah, so what's happening at this office?"

Engineer: "Well, here's what I know. We got an alert from one of the hosts today that transferred about 6GB of data since midnight. That host usually hovers around 10MB of WAN traffic per day. That host is 102.13.84.199. I have the following IP's running rogue FTP servers on high numbered ports. 102.13.84.43, .46, .55, .58, .143, .179, .199, .210, .215. The problem is, I've got three different banners showing up and the ports differ. Some are on port 65000, others on 5883. We've blocked the .199 host at the perimeter so it can't get out but the system is still live. The others don't seem to be overly active at this moment. Want me to put in a block on them at the router?"

You:[1]


You've contacted your partner and arranged to meet in the parking lot at the branch office. You're partner contacted the network administrator at the branch office ahead of time to let him know you were coming.

[B]4:40[B]


When you arrive at his office you gather the following information.

They run a 10BaseT infrastructure at the branch office.
They have two Class C networks. 102.13.84.0/24 and 102.13.26.0/24.
They run a DHCP server that allows anyone to get an IP address, including unknown hosts.
They have two FreeBSD firewalls, one per subnet, that block the reserved port range of 0-1024, but allows traffic on every other port. Logging is NOT enabled.
This branch office deals with customer records for an e-commerce business.

You show him the list of IP addresses you received from the network engineer and this is what he tells you.

.43 is a workstation running windows xp. The main operator is an intern.
.46 is a workstation running windows xp The main operator is an intern.
.55 is a domain controller running windows 2003
.58 is a domain controller running windows 2003
.143 is a workstation running windows xp. The main operator is an HR officer.
.179 is a workstation running windows xp. The main operator is a data analyst.
.199 is a workstation running windows xp. The main operator is a data analyst.
.210 is a domain controller running windows 2003
.215 is a domain controller running windows 2003

The 2 domain controllers run snort IDSCenter and mysql. The snort sensor is placed inside the firewall. They are also connected to a MiSAN (http://www.cybernetics.com/backup_so...san/misan.html)
In addition you discover that the domain controllers are multihomed and have IP addresses on the other network
[2]

The network administrator tells you that the intrastructure is old and they've had to install ad-hoc switches to expand the network over time. A wiring upgrade is due to occur next year.
You visit the BDF which measures (12x20) and see no less than 8 switches mounted to the walls instead of the racks. You see an excess of 50' of CAT 3 cable bundled on the floor rising up to the patch panels. You see no less than 4 patch panels spread throughout the room.[3]


At this point, your partner decides to remind you of something you almost forgot about..

Partner: "You realize that with this many systems we may have to deal with the notification laws."

You:"****....that's right".
http://www.cscic.state.ny.us/security/securitybreach/
[4]

[5]


[1] Well, what do you say?
[2] How do you prioritize which system to visit? What is your process for analyzing the systems? What tools do you run? How do you run them? Do you take any systems offline? What do you do about the multihomed systems? Which logs do you collect and analyze?
[3] Due to the mess of cabling and the network administrator's slowness, it takes an estimated 5 hours to physically locate each affected system.
[4] Does this change your tactics?
[5] If this post continues to receive poor response then I will not write a Part III.

Hrmm, seems with the topology that this was just waiting to happen.

Since were dealing with customer information and most likly credit card info, ACH info, ect. (Probably a great plains type system) security is going to become a huge issue.

I would start out by tracing the boxes on the network and use a line tone kit and those cool cable tracers that can pick up the tone on the line to back track it as far as possible. Once i have located on of the boxes, most likly one of the interns, I would remove any programs that contain any business stuff and then take it for tracking. I would also check the SMS server logs to see if I could pick up any program changes on the box. Assuming you use Systems Management Server, which I do. I would then take the other boxes off-line and fire up the backup domain controller (you do have a backup dc right? And a seperate one for running auth requests?) I would also start a logd of all traffic to and from the subnets until this dliema is solved. I would also force an immediate update of the NAVCorp server and then force a doscan on the entire network.

Once the effected systems are down, the traffic to the ftp's have ceased, and the DC's are up and clean, and logs are running on the subnets, we can begin the investigation.

I would setup a snort box and logd's on a machine, isolate the selected interns computer on a seperate subnet, and let it go to work. Lets see whats really going on with it.

Based on what I find on that machine will determine the next steps.

I would not nitify the state of anything yet because we have not yet determined the true nature of why the ftps are on. Could just be a 3rd party program on the systems trying to update themselves with a shitty updater that got caught in a loop.

I guess thats it for starters.

No SMS server is implemented.
SAVCE 10.x is installed on every machine, however they are not in managed mode.