ATale of Two Logfiles (Final)
Subtitle: How Proper Procedure and Comprehensive Logging make an Administrators job easier.
This is a story. It's fictional and not necessarily factually/technically correct in all cases but I am using it to demonstrate two things that are very important to an administrator, the procedure and the logs. Both go hand in hand in the event of a compromise and both must be in place prior to the event itself. The proper planning prior to the event will speed up the investigation and save time and therefore money in the "clean-up" and mitigation of the breach.
Throughout the story you will find numbers in []. They point to the notes at the end of the story. The notes are meant to show what the participants did right or wrong, what should have been done prior to the event or what could have been done better.
-----------------------------------------------------------------------------------------------------------------
During his mammoth reading task Dirk determined that he had found a reverse telnet connection by using Netcat to make the connection out through the firewall on port 80 and then spawning a command prompt that could be controlled from the remote computer. He had further decided that what he had seen in the transaction logs was a form of SQL injection that was used to enumerate the tables and their content until an administrative login name and password could be returned. From there he concluded that the rest was trivial. He looked at his watch. It was now 9:30pm. He walked down to Mike's office and found him still there on the phone. He waited for several minutes while Mike finished his call.
"You know, I think we have it. I won't go into the details but I know how he got in, what he did when he was here and thus how he stole the information. I also think I have the IP address of his computer. Did you hear from him again yet?"
"No, and to be honest I don't think we will before tomorrow"
"Then maybe we have time to call the authorities and have them deal with him."
"You think so? Er.... what does this IP address thing give us?"
"It's the unique address of the computer on the internet that he carries out the attack from".
Mike paused for some while. "You think we can pull it off?"
"It wouldn't be up to us but we need to give the FBI as much time as we can. I'd say it's now or never. I have the phone number for the local office right here."
Mike thought again and finally, resignedly said "No. Sorry, this stays in-house."
"It's your company Mike..... Your decision"
Dirk was a little angry as he walked back to his office. He wanted a piece of this thief that had come into his network and made him look bad. "Well, he's not coming back in." he thought as he diverted over to Amy's office switched on her computer and deleted the offending files and the scheduled task. "There, screw you... *******".
Earlier Gary had set up the monitoring for any internal traffic directed at port 80 and had sat back to think about how he was going to lock out the access when he needed to. He knew he had eighteen machines here and one other in the Cincinnatti office. How would he find out how many other boxes there were? His mind was wandering and the two thought processes collided. "Oh Duh...." he thought, "Why not just do an 'Al'? Scan the subnets for port 80 after six in each time zone. They'll show up on my monitor but I'll be able to recognize mine by the source address. Perfect". He was left with one other problem. How to determine if there were more than one access point. "The firewall logs are going to show me that." he thought as he picked up the phone to Cincinnatti. He spoke to the admin there and requested the command line for the scheduled job on Dan Ereg's machine. A few minutes later the phone rang.
"Gary speaking"
"It's Tim in Cincinnatti. I have your command line."
"I just need the IP address it connects to."
"No IP address, it's pointed at a domain name, al.attacker.com"
"Ok, he probably wants to be able to move, thanks"
After putting the phone down he went to the Secure Logging System and filtered the previous month's logs for the IP address he had resolved through nslookup. There they all were. "Damn, this guy sucks. He's so predictable. Always port 80, always 6:00pm in the time zone and only three machines. Well that will make the cleanup easier" he thought. Just to be sure Gary then filtered the logs for the previous month against the internal IP addresses and looked at the transactions immediately after six pm each night. He found they all pointed to the same address, al.attacker.com except one, the very first. "Noooo... It can't be.", he thought "He can't be this frigging dumb.... Can he?" He quickly ran a Whois against the IP and found it was a common high speed ISP. "C'mon now.", he thought, "Let this be my lucky day". He opened google and entered the IP address. "Oh Baby, twelve hits". They were mostly abuse reports from different locations. "Sweet, a pattern, this guy has been practicing from home and got himself noticed before". The eighth one down really caught his eye. It was an email to a list which, (as many do), contained the headers. There was an email address too. Better yet the email address showed as a real name, <Joshua Albin>. "Well what have we here.... It's 'Al'..." Gary laughed out loud as he reached for the company phone book. He quickly looked up the number for the President of the Board and dialled it.
"The President's office, Julie speaking, how may I help you?"
"Er, yeah, hi, it's Gary Cunez, Corporate Computer Security Manager. I really need to talk to the president, like, er...., right now, is he available?"
"One moment please" as Julie placed him on hold.
A few moments later he was talking to the President.
"Gary, This is Bill Smoltz, the president, what do you have for me?"
"Sir, I'm pretty sure I have 'Al'. His name is Joshua Albin. With some assistance from his ISP, HighSpeedAtHome.com, I think we can give the authorities an address."
"How did you manage that. I thought these people did everything they could to hide themselves?"
"I won't go into detail but he is pretty sloppy and made a single, rather large mistake. It took only a couple of minutes of digging which showed a pattern of abusive behaviour and, more importantly an email address."
"So what do you want me to do?"
"Let's bring in the FBI. With this amount of information they could have him in hours and he would be out of our hair. Not to mention the fact that if he's only asking for ten thousand he's probably running the same trick against others. We can help ourselves and others."
"It's a bit late Gary. Two hours ago a company statement was Fedex'ed overnight to all our investors stating that their accounts were frozen without each transaction being accompanied by a code issued with the statement and that the company would not deal with blackmailers or other fraudulent activities now or in the future."
"Sir, that's ok. How do you think the investors would feel if they received a second statement within twenty four hours telling them to relax, we caught him? I have him Sir, I know it."
"You do have a point.", Bill hesitated, "Ok, you call the FBI and give them what you have. I have some more calls to make now. I'll get back to you"
"Thank you Sir. I really feel good about this"
Early the next morning Agent Hicks sat for two hours with Gary as he went over the evidence he had collected. Gary finished up by showing him the first IP address, the ISP, the Google results indicating a pattern from the IP address and finally the email.
"I think you have him there Gary my man", Agent Hicks smiled, "You are even luckier than you think. I've worked with HighSpeedAtHome before, several times. They are great record keepers like yourself. This may only take a phone call or two to start my little ball rolling." He grinned broadly.
"Phew, I'm glad. It's a gut feeling that this is the guy but everything points to the owner of that IP being a bad kid in cyberspace."
"I can't argue with you there. Can I use your phone?"
"Yeah"
Agent Hicks made a couple of calls. "Ok, I know who I need to talk to now," he said, "Let's see what HighSpeedAtHome have for us" as he dialed the number he just noted down.
"Yeah, Hi, Marvin Brenner please"
"Yes, I'll hold"
Hicks waited.
"Yeah, Marvin Brenner? Hi, This is Agent Hicks of the FBI, you should have been informed that I would be calling"
"Good. I need to know if there have been connections between the following two IP addresses in the last month. Can you do that?..... Good". Agent Hicks read off Gary's Cincinnatti IP address and that of Joshua Albin. A minute or so went by and he began listening again.
"Good.... Ok, can you confirm that there were no other connection attempts to the second address I gave you just before or after that connection was made". Another long pause ensued.
"Ok, the second IP address had two attempted connections on port 1433 five minutes after the connection in question. Can you look at the source address of the port 1433 attempt and tell me if this was a single instance or part of a scan?"
"Ok, so you are saying that three weeks ago today at 6:00pm almost exactly the first IP address connected to the second on port 80 and that apart from a confirmed portscan for an SQL server from an unrelated source which touched the second IP address there were no other connections in or out for more than an hour.... Great, I want you to hold the logs because there will be a subpoena for them before you finish work. One final thing, a yes or no question, is the subscriber's name Albin....." another pause, "William Albin. Great, I'll be seeing you soon, Bye.
Agent Hicks hung up and turned to Gary.
"Nice work.... It's 'Al' all right, no question. The dumbass made the first connection go to his home. I have to go, I'll be in touch"
It was 5:30 pm and Joshua was bored. He'd spent most of the day trying to work out how he was going to get the money without giving himself away and still wasn't any better off. Since leaving school two years before and deciding college wasn't for him he had wandered, unsuccessfully, from job to job. His current employment was the graveyard shift cleaning a large hotel's kitchen and he hated that too. The doorbell rang downstairs. "Screw it,", he thought, "the old man can get it.... he needs the exercise". He heard the door open and some muffled conversation.
"Josh, Can you come here a minute" William Albin called up the stairs
"What now dad, I'm busy"
"It's a girl to see you son"
"Er, just a minute"
Agent Hicks nodded thanks to William for not alerting his son. Joshua appeared round the corner and was confronted by two "suits" and two badges.
"FBI Joshua. Please be calm, we just need to ask you some questions". Before Joshua could speak Agent Hicks reminded him of his right to silence and a lawyer. Joshua's only comment was to his father, "I wasn't going to take the money Dad, I was just showing them how vulnerable they are.... Dad...."
Agent Hicks colleague took Joshua to the waiting car while Agent Hicks went upstairs and removed a computer and a laptop from Joshua's room.
Gary took the phone call from Agent Hicks.
"We have him and his computers. No doubt it was him, he partially confessed to his father when we arrested him. I'm going to be busy for a while, I'll get back in touch when I have what I need. Just make sure that your evidence stays clean, ok"
"Yeah, no problem, it stays where it is.... and thanks"
"No problem, you made our job real easy, bye.
Gary called the President again and informed him of the developments. The second statement was already prepared and was sent out to all the investors.
An hour later in Seattle Dirk and Amy were just ordering dinner while he tries to explain the technicalities of how the attacker was causing her machine to connect to his and then have him control it. He felt pretty good about the whole thing. He had found the compromise, worked out how the attacker was doing it and had even found the attackers computer. Yep, he was "The Boy" and he was going to leverage that all he possibly could with Amy tonight.
Mike sat in his office at TFCU. "It's getting late he thought" as he looked at the clock in the system tray of his computer. He noticed it change from 5:59 to 6:00pm. What he didn't notice was the flicker of the drive light under the desk as his computer sent a DNS request for al.attacker.com followed by a SYN........
-----------------------------------------------------------------------------------------------------
The Synopsis
Subsequent investigation and questioning of Joshua proved that the initial breach took place through a malicious script disguised as an enticing spam message sent to select members, (usually high ranking company officers), that insinuated itself into the local security zone of the machine and downloaded Netcat disguised as msbackup.exe and created the scheduled job for 6:00pm each evening to set up the reverse connection through port 80 on the firewall thus making it hard to detect and even harder to stop. From there Joshua had used other tools to elevate his privileges and compromise numerous machines throughout his "conquered" networks. It was then simply an issue of determining where the financial information was kept and how to get the required authentication information or use other methods such as SQL injection to get the data he required.
-----------------------------------------------------------------------------------------------------
The Lessons
Dirk's a nice kid and not a bad admin, but he was utterly unprepared. It wasn't his fault. We've all seen it, whether it be a new job or simply going to help a friend. The network that was never built with a cohesive plan in mind. It takes months to learn it and maintaining it at the same time means things simply "go by the wayside". Dirk found himself in a stressful situation which he had no experience with and, worse yet, had no plan to try to address the problem. People say that "information is power" and it is true. Logfiles are information. Therefore logfiles _are_ power. Even if you are unprepared for a situation a comprehensive logging system can allow you to "muddle through" without the nicely documented plan your boss would like to see that will still probably contain "holes". The holes can be filled by the information in the logfiles. As hard as Dirk tried his efforts were stymied by the lack of information. That lack of information coupled with a lack of understanding of how the attacker works led him to believe that he had a clean network, that he knew what had occurred and where the attacker was attacking from. The reality is he was left with a dirty network, he had no way of knowing that other machines were compromised, he didn't understand that Joshua wasn't attacking him from his own computer and he didn't know that Amy's machine was a "jump off" point inside his own network, (though in this case Joshua did use her machine to get the data he needed, logs may have shown him a different picture).
Gary has an advantage over Dirk. His company employs him to do the security. He's been there for a while and he's built his security architecture himself. He monitors the network, he logs it heavily and he wrote his procedures while he had time to research them. He also watches what the potential attackers can do and adjusts his systems accordingly. So when the "bad thing" happens he can move in, confident that unless the attack is extraordinary he has a good chance of piecing it all together. Gary ended up with a clean network, sufficient information to put Joshua in jail for three years and a handsome pay raise, (ok, I lied a little ;)).
There is no reason why the "average" admin can't accomplish what Gary did. It starts with looking at each part of the network and each project you take on from a security standpoint. Simple questions like "Can I log these transactions somewhere?", "How could this be exploited?", "Can I know who connects to this and when", at the start and both implementing and documenting everything you can goes a long way to helping you when the "proverbial" hits the fan. Computers are cheap and drive space is cheaper. An old PC with a nice big 80 Gig drive is "chump change" today. Couple that with a CD writer to archive the logs to at less than $0.50 per CD and your ability to log your system properly is acheived in less than $2-300. What's the problem?
"Google it" is a mantra amongst security aware admins, and it's a good one. I would propose another mantra. "If it communicates, log it". I wouldn't want to be blind in my real life, why would I want to be blind in my cyber life?
Finally, ("Thank god" yells the crowd), think about this beforehand. You have time during your commute, over lunch or even in the shower. When things start coming together use any other spare time to create your "procedure". It might not be perfect but it will make the stress managable and it might, with good logs, help you do what you need to do.... Which is better than running around like a headless chicken while the sky falls......
For reference this is basically how I log things and this is my procedure
