A Tale of Two Log Files, (Part 1)
Subtitle: How Proper Procedure and Comprehensive Logging make an Administrators job easier.
This is a story. It's fictional and not necessarily factually/technically correct in all cases but I am using it to demonstrate two things that are very important to an administrator in the event of a compromise, the procedure and the logs. Both go hand in hand in the event of a compromise and both must be in place prior to the event itself. The proper planning prior to the event will speed up the investigation and save time and therefore money in the "clean-up" and mitigation of the breach. It's a lot to do with the mindset and approach whether you are prepared or not and that is what I will try to show during the story.
Throughout the story you will find numbers in []. They point to the notes at the end of the story. The notes are meant to show what the participants did right or wrong, what should have been done prior to the event or what could have been done better.
-----------------------------------------------------------------------------------------------------------------
Nestled in a small industrial park outside Seattle, Tree Fellers Credit Union, (TFCU), is a small financial institution of some 150 employees, Dirk Gardner had recently started there as the company IT manager after many years of IT functions being outsourced to a series of contractors. He was still in the process of gathering and understanding the convoluted implementations and systems the string of contractors had left him with. Returning to his office, morning coffee in hand, he almost bumped into Amy Walker loaded down with yesterday's mail, they exchanged pleasantries and went about their business. Amy was the receptionist at TFCU's headquarters facility and had been working for the company almost since the day it had opened for business, he liked Amy, after all, she was attractive and single.
Halfway across the country in Kansas City, Gary Cunez drove up to the central data processing facility of Scales, Kohl & Brenner, an international investment banking organization with offices throughout the USA. As he pulled into his parking space Mike Panoff, the personnel manager of the facility was getting out of his car.
"Whoo hoo, lookout folks, it's the security geek" Mike said in his usual joking fashion.
"Be afraid", retorted Gary, "Be very afraid.... I'm watching you today" he laughed.
Both Dirk and Gary were expecting a quiet Friday followed by a lazy summer weekend. They both had plans that would change.
Dirk began checking his email, sipping his coffee as he went. It was all pretty usual with the customary a pile of spam, "Damn them" he thought, "You update the filters one day and the very next day they find a way around them". One message caught his eye, not only for the time it was sent but the fact that it was from the CEO. "What was the CEO doing sending me emails at 2:40am?" he thought. Opening it up he realized that all was not well at TFCU. It simply read:-
"I'll be contacting you soon..... It will be in your best interest to cooperate. Wait for a message from Al."
This terse note was followed by a two column list of credit card numbers and social security numbers. Dirk opened TFCU's financial application, waited for the connection and authentication to the SQL server and typed in the first credit card number. "Oh ****" he thought as the number was found and the social security number for the card matched the number next to it on the list. After randomly picking three other numbers from the list and trying them it was all becoming alarmingly clear. "This is not good" he though as his mind raced through the myriad of things he should do, and must do, trying to determine an ordered and potentially "employment-saving" course of action. "Just to be sure", he thought to himself, "Let's look at the email headers". His heart sunk as the headers confirmed that the email had indeed originated from a private address within his own network. [1]
Back in Kansas City, Gary was collecting his voicemail. It was the usual stuff from the night technicians about occurences on the network until he reached one with a voice he didn't recognize.
"Hi, you don't know me, just call me Al. I got your number from Active Directory... You gotta love all the details in there huh? Can't talk for long but I just want you to know that I'm a little short on cash and I'm considering taking advantage of some of your customer's accounts. Before I do maybe we can come to some arrangement.... Check your email.... Bye now".
Gary listened three times before he determined that he really had heard what he thought he had heard. Like Dirk he had a sinking feeling that his job may have just gone on the line and as a first step he saved the voicemail. Looking through his surprisingly short list of emails in his inbox he knew which one the voicemail referred to. It was from his own account at around 4:00am. It said the same as the email Dirk had received just minutes before in Seattle but had account details that matched those of current Scales, Kohl & Brenner customers from all over the world. "Well", Gary thought to himself, "I guess this is where I find out if I'm worth what I'm paid.", as he reached for the policy manual he had worked so hard on two years ago and had been testing and updating regularly ever since. [2]
This time Dirk nearly knocked Amy flat as he rounded the corner near the CEO's office. Apologizing hurriedly, he burst into the CEO's office..
"Er, I need to talk you you right now", Dirk said as he nodded apologetically at the CFO sat across the desk from the CEO.
"Can it wait, this is pretty important"
"No, I'm afraid not. This is very important, it's a security issue"
"Well let's make it quick, I have another meeting in 20 minutes"
Dirk explained the email he had received, how it had come from the CEO himself and that he had confirmed that the information in it was genuine and the source of the email was from within the company network.
"So where do we go from here", the CEO asked, "We can't bring everything down. What are you going to do?"
"Right now..... I don't know", replied Dirk. "I know that doesn't sound good but I need time to think this through"
"Isn't there a procedure for this?"
"I really don't know", Dirk nervously replied. "The previous contractors never really did anything with a firm plan in mind so I am unsure what resources I have to hand just yet. That's part of what I need to think about".
"Look, if customer information is in this "Al's" hands we need to move quickly." interjected the CFO.
"What do you suggest?" Dirk retorted
"Well... I don't know, that's your job"
"Yes, and that's what I'm going to try to do and my first task, in the absence of a procedure, will be to develop one as I go. It's not something I can rush" [3]
Meanwhile Gary was putting the phone down on his last phone call to the Incident Response Team, (IRT). He was happy that he had only had to call one backup person since all the rest were available. The incident meeting was set for 10:00am so he was happy for the hour he would have to prepare. He knew that access to the custom produced system that managed the customer information was unreachable from outside the firewall. This left him with two initial avenues. Either it's an "inside job" or the private network is penetrated in some other way. In either case it would be difficult to pin this one down. He checks his logs and systems daily and nothing had peeked his curiosity recently so his gut feeling was to lean towards an "inside job". A feeling he quickly brushed aside. [4] His starting point was obvious. Since the data had to have come from the custom application's server at some point it would be the right place to look first.
Knowing that the information had to have been stolen prior to 4:00am when he received the email he began by looking in the application log for the period prior to that. It took but a few minutes to find a successful login from a person he knew to be a day staff at 8:30pm the previous day. A quick phone call confirmed what he thought.
"Hello, Mike Panoff"
"Mike, it's Gary, I told you I'd be watching you. What time did you leave last night?"
"Wow, you weren't joking." Mike laughed
"No, actually, I'm not. What time did you leave?"
"Er... It was about 5:30 I guess. Why?"
"Did you log off and shut down the machine?"
"Er... I logged off but I leave it switched on and just turn off my monitor. Why, what's happened?"
"Have you been working on it this morning?"
"Yeah, I.. er... checked my mail and that's it... I had a meeting right after I got in"
"Good. Do me a favor. Leave it exactly as it is. Don't touch it, log out, turn it off or anything. If your boss wonders why you aren't working have her call me, ok?"
"Ok, no problem. Sounds serious, what's up?"
"Ahhh, probably nothing. Hard to tell at the moment. I just need to look around for a minute. Usually this stuff is just me not properly understanding what my systems are telling me", Gary said. He thought about faking a laugh but decided he may not carry it off and give the game away, so he didn't. [5]
Dirk sat in his office trying to decide the appropriate course of action. Obviously he needed to work out where the attack came from but without a clear picture of what had been done in the past it seemed insurmountable. Was the attack from inside or out? "Is the firewall effective?", "I know the transaction logs for the SQL server are in order", "Is there anyone inside the company with a "beef" that would play games?", were all questions that went through his mind. After ten minutes of fighting the desire to "hit the keyboard" he realized he was getting nowhere.
"I'm starting from scratch" he thought..... "So where do I start? Let's start at the beginning.. test the firewall, what does it allow and to where" [6]
Dirk picks up the phone apprehensively, "This isn't going to go down well", he thought as he dialed the CEO's extension.
"Mike, I have to go home for an hour. In order to find out if this was an inside or outside "job". I need to scan the firewall from the outside. I need to know what the firewall will allow and what it won't. I've looked at the rules the contractors put in there but they are confused... Different people have allowed and denied things in the past that may conflict.... I have to find out what is really allowed and denied from outside. Until I know that I really can't be sure where I need to look next."
"Dirk, what do you mean by that?"
"Well, I'm trying to narrow down the possibilities, I'm trying to eliminate the easy ones first. If I can eliminate some things quickly then I can concentrate on those things that are more probable. In that way I might be able to conclude this more quickly and give you the answers you need"
"Dirk, I really don't like the idea of you leaving right now, this is a very big problem and I need my IT staff here to help me."
"Mike, you need to trust me. I can't easily, properly and accurately scan your network from the inside. I need to be outside, my home is fifteen minutes away and I can do the job properly from there. The cost in time is small compared to me trying to guess at this point."
"Ok, make sure your cellphone is on."
"No problem, you have my number, right?"
"Yes"
"OK, I'll be back in an hour or two" [7]
Part 2 is in the works..... Don't you just love flights taking you to vacations..... Hopefully there'll be a network I can access when I get there.... ;)
-----------------------------------------------------------------------------------------------------
The Notes:-
[1] Dirk made his first good decision, he confirmed that the compromised information was, in fact, genuine. Sometimes determining whether or not there is cause for concern is much more difficult and must be done carefully so as not to disturb or lose any information that might be critical later on in the investigation. That said it is important to confirm that there actually has been a compromise. The procedures and actions you carry out during the investigation are time consuming and entails involving a lot of people with limited computer knowledge. At a minimum it will cost money in time and effort and it could cause a lot of harm to an organizations reputation to be telling people outside the organization of a compromise and having to rescind it later.
[2] Discovering a breach in your security is a stressful event that causes a rush of feelings and thoughts. The combination of feelings and thoughts is a bad thing because mistakes can easily be made. Accepting the fact "up front" that there will come a day that, one way or another, a compromise will take place on a network you are responsible for allows you to plan thoroughly. The absence of worry over your employment status, the unhurried ability to ask "what if's" and the time to test and practice with the tools you will be using is invaluable. In our story Dirk had been left with the traditional mess a string of contractors leave and a lack of documentation or policies. He is "flying by the seat of his pants". Gary, on the other hand, has a network that he has implemented the security mechanisms for himself, tested them and finally written/tested the policies for.
[3] Dirk is on a roll. Despite pressure from the top two executives in his company he understands that he is in a "bad place" that will only get worse if he doesn't think carefully about his actions before he carries them out. In any investigation you will come across something that may not have been planned for in your procedures. In order to properly deal with it you need to determine a course of action that is appropriate, that fit's in with the procedures already in place and that you document as you go in order to be able to place it as an addendum to the policy for future reference, you need to "pause". This takes time and thought. If the CEO comes back and demands legal action against the attacker then these actions must be sound enough to stand up in court. It's an old adage but it works - Stop, Think, Act. Better to spend an hour thinking about the possible repercussions of your actions than a week trying to find your way out of the mess you made by them.
[4] It is a human tendency to have a "gut" feeling about something and to seek evidence to prove that it is so. This is the wrong way to conduct any investigation. An investigation seeks the truth rather than evidence that might uphold a theory. All avenues must be looked at. Gary has made a subconcious decision to disregard a direct attack through the firewall in preference of an internal attack or one that originates outside the firewall using an internal resource as the "stepping off point". While he has set aside the potential for a direct attack he has not succumbed to the "gut feeling". He has prioritized his investigation but may still go back and investigate a direct attack. There is a very fine line between the "prioritization" and the "gut feeling" but it is one that should be considered while determining where to start. You must maintain "the big picture" and remember that even though you are following a course of action that currently excludes the possibility that something else occurred you absolutely must not do anything that would preclude you from discovering the evidence that lies in those things you have discarded as improbable or impossible.
[5] Your investigation is going to raise the curiosity of everyone you "touch". Frankly, it's an exciting thing for them and when they are excited they are going to talk about it, to anyone and everyone. They don't have any facts and they probably have no idea about what they are talking about or the possible repercussions of telling people what they "know". They may also not be particularly discriminatory about who they tell which could make the problem even bigger, (they may be telling an inside attacker that s/he has raised the suspicions of the IT staff). The rule is: Tell nobody outside the IRT what is going on. You may have to bring people "into the loop" as time goes on but that should be done formally and with the permission if the IRT itself. Keep it all simple. You "don't know", "it's just a normal audit of systems"..... Use your imagination but have the answers to such questions formulated before you call people to request information. Yes, it takes more time, but in the end it is the best policy.
[6] It's very easy to "hit the keyboard" and start your investigation without a policy in place. It's equally easy to find something that goes outside the parameters set down in your existing policy and carry on regardless on that keyboard. Don't, it's a bad idea that will come back to haunt you. TFCU is lucky, Dirk is smart. He thinks before he acts. He starts with things that will move the investigation forward quickly, (narrow down the potential issues), without detrimentally affecting any data, systems or, more importantly, his investigation.
[7] Having thought carefully about his course of action Dirk is now leaving his chosen trade. He is, unwittingly, entering "people management". The company's management will be equally, if not more stressed out than you are. It's their money and reputation you are dealing with now. You know what you need to do. You probably have an idea of how much time it might take, but you lied, (let's be honest, lying is ok sometimes). You started the conversation with a set paramater, (an hour). By the end of the conversation you have exceeded the original expectation, (an hour or two), and have given yourself the time to do what you need to do. During the conversation you demonstrated the benefits to your course of action and you delineated the problems to him/her if you don't follow your course. You "bought" the acceptance of your course of action then you "gave yourself a chance". It's not necessarily your strength, but it's a fine skill to learn. It won't be as quick and easy as the story suggests but if it is something that gives you the "comfort level" to make decisions and move forward then it is something you need to learn to "manipulate".
