A Tale of Two Logfiles, (Part III)
Subtitle: How Proper Procedure and Comprehensive Logging make an Administrators job easier.
This is a story. It's fictional and not necessarily factually/technically correct in all cases but I am using it to demonstrate two things that are very important to an administrator, the procedure and the logs. Both go hand in hand in the event of a compromise and both must be in place prior to the event itself. The proper planning prior to the event will speed up the investigation and save time and therefore money in the "clean-up" and mitigation of the breach.
Throughout the story you will find numbers in []. They point to the notes at the end of the story. The notes are meant to show what the participants did right or wrong, what should have been done prior to the event or what could have been done better.
-----------------------------------------------------------------------------------------------------------------
"It wasn't a voice I recognized" Mike said to Dirk who was standing nervously in the doorway.
"You mean you don't think it's an employee then?"
"I know practically everyonehere and the voice just doesn't fit, no-one has an accent like that here"
"Well, then it does point to a "call home" program then. I looked at where the email came from and it was Amy's."
"Amy? She's been here forever."
"Well like you just said it doesn't seem like an inside job and Amy doesn't sound like a man either"
"Oh, yes, you're right. So he want's ten thousand dollars or he starts using the accounts."
"Well we can't close all the accounts and issue new one's. That would tell everyone we were compromised"
"Can you find this 'Al'? I saw a program on TV once about how a spy was caught. Can we do that?"
"I don't think so. I need to look at Amy's computer but finding the information I would need is going to be difficult. I could really use some log files but I took a quick look around and where there are any logs they overwrite themselves when they fill up which sometimes seems to be overnight on some machines." [16]
"So you're saying that without logs you may never know where all this came from?"
"Basically, yes. They are one of those things that always seem to get the low priority. I'm guessing the previous contractors had better things to do than to bother with them."
"So we don't have a lot of choice then? We are going to have to pay" Mike said glumly.
"Give me some time, maybe I'll get lucky. Maybe Al is a little sloppy. When does he want the money?"
"I don't know, he said he'd contact me soon"
"Let me look around. Call me when you know more, ok?"
"OK, I have some calls to make, get back to me if you find anything"
"Will do."
Gary asked Bill Steel to call an emergency board meeting for 2:00pm today. He needed the 'ok' from the board to move to "investigative" pace. He had explained to the IRT that while the probability was high that he would find the outside locations that the attack had been initiated from the chances are that they were zombies being used by Al to mask his true location and identity. The recommendation would be that the company pay Al and move on. It irked Gary a little but he knew that the chances of successfully finding and being able to prosecute were minimal and that calling in the FBI would affect the company badly since the theft would become public once they were called in. [17]
Arriving back at his office Gary decided to take a look and see who's computer in Cincinnatti was carrying out the scan. A quick check of his DHCP logs indicated that on the day and time of the scan the computer belonged to the manager of the Cincinnatti office, Dan Ereg. "Interesting." thought Gary not doubting that Dan wasn't Al but beginning to wonder how pervasive this attack was. "I do hope he hasn't been hopping from box to box for weeks," he thought, "that's going to be a real pain not to mention the fact that I'm going to have to explain why I didn't notice it". [18]
Dirk sat forlornly in front of Amy's computer. He'd opened the event logs to find nothing. "This really sucks," he thought, "those contractors are useless.... they didn't even turn on auditing. Jesus, all they had to do was set the domain policy.... Less profit in doing it right I guess". Amy's voice brought him back from his angry thoughts.
"You don't look very pleased" she said, "Is it something I've done?"
"No, I don't think so. I'm peeved at the old contractors. I'm trying to find something that should be easy had they done their job correctly.... But they didn't".
"What are you looking for, maybe I can help"
Dirk laughed, "I don't think you could help me, I don't know what it is I'm looking for myself".
"That doesn't make much sense," Amy replied, "How do you even know you should be looking for something then?"
"Ahhh... It's a long story. Maybe I can tell you some other time, when I get this all cleared up".
"Oh please, you've certainly got my curiosity up" she smiled. The double meaning went right over Dirk's head.......
Dirk went back to looking around her computer. He was lost and he knew it. "This is bloody silly," he thought, "Thirty thousand files or more and no idea where to start. If I'm not careful it's going to look like Amy did it and I can't believe that." He wandered aimlessly around Amy's computer for another twenty five minutes and was ready to give up when he opened the task manager. Looking slowly down the list it struck him as odd that mstask.exe was running. "What's this?" he thought, "The scheduler is running". he opened the task scheduler and there was a single entry named "Daily Backup" set to start at 6:00pm daily. "Hmmm, it ran last night at 6:00 and it is due to run again at 6:00. What's it doing?". He quickly checked the properties. What he found made him sit up suddenly. The task was scheduled to run a program from the system32 folder called "msbackup.exe". "Got it." he thought, "Microsoft's backup program is "ntbackup.exe". This must be a trojan. There's an IP address amongst all the switches. I wonder where that is?". He quickly made a note of the address and returned to his own workstation. He "pinged" the IP address and got a response. "Cool," he thought, "It's up. Now what?" [19]
Gary looked up as the technician knocked on his door carrying a copy of Mike Panoff's imaged drive and a manifest detailing what had occurred and where the other drives were currently located. He thanked the technician, filed the manifest after checking it to make sure things were in order and placed the drive in his machine. He went straight to the scheduled task which he found to be called "Daily Backup". "No reason for that whatsoever" he thought as he looked to see what the executable was. "Now that's an interesting command line", he thought. "msbackup -l 80 -e cmd.exe..... that looks all too like Netcat... running as system too because of the scheduler, ouch, nice one." He made the appropriate notes and moved to the event logs. There he found new processes being started just after the portscan had taken place. "VNC was run not long before the access to the financial database.", he thought, "OK, so he's even making himself a desktop to work from. Lets take a look at the VNC executable". He opened Explorer and went to the path shown in the event log. Nothing. "Ok, let's try the recycle bin?" he thought as he switched to it and found it immediately. "Great, the deleted time was not long after the data was stolen." He further noted that there were a couple instances of the VNC executable over the previous few days. "Hmm... He downloads a new version each time. I hope he only uses Dan's machine as his first stop internally". He sat back to think for a minute. "Why was he portscanning for port 80? He has to know he has Netcat on Mike's box....". It took a full three minutes before the possible answer came to him. "He can't guarantee which computers will be left on after work..... ****, he has several machines..... I bet there are 18 machines total..... I need that portscan log again...." [20] He called the network admin and asked that all 18 machines on the list be checked for the scheduled job and was surprised when, just 45 minutes later, the admin was standing at his door with log files of his technicians activity on each machine as Gary had requested.
"You were spot on.", he said, "Every box has the same scheduled job. Busy little beaver that hacker friend of yours eh?"
"Jeez.... He wanted to guarantee access didn't he?"
"So what do you want me to do. I can't pull all the boxes, I don't have replacements."
"Yeah, I know.... tough one.... The board meeting is in 5 minutes so I can't say right now. We've recommended no action so we can move faster but the board has to ratify that. Until then I'm supposed to work as if we were going to court with this stuff. Then again, if we have good evidence on Mike's box and we pick up the box in Cincy, we should be good in a court, anything else we have should just be additional nails in the coffin so just documenting the other machines should be good. It's something I'd have to check with the legal beagles if the board wants us to go after him but we should know in an hour or so. Let's just wait till then."
"Ok, he's a pretty bad hacker though isn't he? He's leaving clues all over the network."
"Yeah, I would have expected better, even the log files were intact. Hopefully he won't be any more careful with the box or boxes he uses to enter the network"
Dirk was now uncertain where to go from here. He'd found the attacker's computer in the internet, it was up and running and he was somehow controlling Amy's machine to do whatever he was doing to get to the financial data. "I need a plan," he thought "this can't be the first time something like this has happened, someone has to have seen this before....." Finally, the "light bulb" came on.... "Google" he thought, "I'm a genius....". He scrambled over to his workstation, opened IE and went to Google. He typed "incident response methodology" and clicked "search". "Oh Hell. 201,000 hits....." He added the word "windows" and re-searched. "That's better" he thought, "cut that down to nearly 42,000 hits. ****, I'm going to be reading all night...." Despite the seemingly overwhelming task Dirk felt a kind of warmth. He wasn't alone, he wasn't the first victim and there was a lot of information out there about how he should proceed. "Damn, I probably should have done this earlier.", he thought. [21]
"You have got to be shitting me", Gary said, "Do they know what that will do?"
"Look Gary, that's the decision of the board. They want you to go as far as possible and then they will call the FBI." Bill said.
"You explained that our chances of finding this ******* is close to zero didn't you?"
"Yeah, but the feeling is that if we give the ten grand he'll be back anyway. Eventually he'll drop the information somewhere and then we're screwed regardless. They feel it is better to face the issue head on, tell the investors about the leak and do everything to minimize the damage. The PR people are already working on the issue. Sorry, but you get to stay on the slow track"
"Hell, I have eighteen boxes I have to drag off the network already.... Should we just close up shop now?"
"Eighteen, how?"
"It'll take me too long to explain but he has eighteen boxes on this network alone he can connect to and control, never mind the other offices."
"Nice, very nice..... Can we handle that?"
"I dunno, listen, I have a thought, what do you think? I'll just leave the boxes up, I already took down Mike's but that should be ok, he expects boxes to be turned off sometimes. If I leave the other boxes alone and let things happen as they will can we still hold up in a court? I want to just put a monitor on all of them and see if he comes back. If we maintain a proper evidence trail on the boxes we have can we allow the other boxes to be tainted and still win on the off chance we ever get this **** into a court?"
"If your solid on the evidence trail of the boxes you have right now and can show his activity from your monitors I'll argue it with any defense attorney but I need a rock solid place to start and I'll need good data to back up our decision. Can you provide that for me?"
"I think so, but you're the legal beagle.... Do you trust me is the question?"
"Er.... Oh, to hell with it.... Go for it... Make sure you are letter perfect on the procedure with everything you do. I need everything documented and I need you to be able to show reasoning. You do that and I'll back you and argue it in any court if we can get it there"
"Thanks, I appreciate that. I'll set up the monitors now, the jobs will kick off in, um, one hour, twenty three minutes.... I'll be ready"
"Good, and good luck... to us all"
"Yeah, we need some, bye." Gary said as he put the phone down. [22]
"Now this really sucks" he thought. "I really don't like this.... I'd rather close all this down now....."
-----------------------------------------------------------------------------------------------------
The Notes:-
[16] Dirk is beginning to realize that the log files on a network are his eyes. When log files don't exist or overwrite themselves when "full" he is blind. All manner of things could have gone on throughout his network minutes before the log began overwriting itself and he would have no way of knowing it.
[17] No matter how good a sleuth you think you are you must realize that once the trail leaves your network it also leaves your logging systems. Once this has occurred the trail will most probably "dead end" fairly quickly. Your task is also to operate in the best interests of your company rather than follow your desire for revenge against your "violator". The IRT's recommendation to the board must be realistic and take into account the various aspects of the whole and their repercussions.
[18] Whether you have the best logging systems in the world or not you can't expect to find every little thing. Even on relatively small networks the traffic volume can be huge and diverse. A good attacker will try to utilize "normal" traffic patterns to mask his malicious activity. Accept the fact that network traffic is extremely complex and unless you packet capture everything on the network you are going to have holes in your logs that may allow malicious traffic to slip by. Do your job, if you have thought about this situation previously and tried to make sure that you log the "right" things you should still be able to piece a trail together. It may not be complete but it will tell you were 'Al' has been which makes cleanup a lot easier.
[19] Bad move Dirk. He might as well have called Al and told him he had found one of his tools. You shouldn't make any direct contact with remote machines at any time during the investigation until you have collected all the information you can. Once you have that information use a dial up connection to another ISP or go and use your home computer to look at the remote machine. You have to be very careful wherever you take a "peek" from. Al is in the process of committing a crime. One that could cost your company a lot of money and more importantly put him in jail with a nice cell-mate called Bubba. He's going to be more than a little suspicious about probes against his machine(s). You really don't want to force his hand and have him release the information for example when you were trying to keep the compromise quiet.
[20] While there isn't always a good reason for things appearing in logs there are often sane ones. Gary nearly bypassed the reason for the portscan and may have become sidetracked by more "exciting" tasks. It's important to look at each event that is relevant to the attack and try to determine why it occurred. Gary got lucky, the thought occurred to him after the fact and he didn't dismiss the question. Had he done so he wouldn't have found all the holes in his network. It helps to try placing yourself in the position of the attacker and work backwards. Why would I portscan this subnet from Cincinnatti trying to find my own Netcat? That question may come up with an answer much quicker than "Why would he be scanning me internally"? The difference is subtle but if you can place yourself in a position where the attackers "problems" become your's you may speed up the process.
[21] No matter what is happening. No matter how stressful it might be, you aren't alone and you aren't the first to have been cracked and have the company's data and it's reputation on the line. No matter how well prepared you are you will most probably be ill-prepared for something that turns up. Let's face it, few of us are full time network security analysts with daily excursions into the world of forensic investigation. Most of us are network admins who rotate ten different hats every day. But that's ok, Google will help. It's a mantra repeated hourly in the computer security world, "If in doubt, Google it" or "Google is your friend". Learn it and live it. Again, it is better to spend time finding out what you _need_ to know to be successful than to ruin your chances of ever being successful.
[22] Unless it's your company and your money things aren't always going to go your way. You might understand how low the chances of finding the attacker are going to be but it is the board's decision as to how they run the company and what may serve it's interests best. You give the best advice you can, you are clear about your abilities and your inabilities and you let them make their decision.... It's their business and that's why they get the "big bucks". It's important that they know your weaknesses as well as your strengths. It may seem foreign to be telling your bosses that you "suck" at something but it is information they need to know. Suggest that they hire a contractor to help you in your areas of deficiency. The important thing is that you reach the truth. Unless they are utter idiots they will appreciate your proactive approach to _their_ problem and your understanding of your own deficiencies. It might sound like "falling on your sword" for the good of the company but if they don't understand and appreciate your committment I can assure you there are better places for your talents.
