A Tale of Two Logfiles (Part II)
I really apologize for the long period between "episodes". It was due to two factors:-
1. Radisson Hotels and Orlando International Airport in Florida are somewhat retarded. No wireless at the airport, and a choice between internet over the TV or 89 cents a minute to dialup the internet from the laptop. Neither of which was accepable or useful..... :mad:
2. My sweetie and my friends kept me sufficiently busy to make a quick wardrive impossible. When I wrote my note last night I was at a friends house for a BBQ, was a tad drunk and was being yelled at by my other sweetie, (our hostess), for working on the computer when I only see her once a year...... :D
So.... To Part II
Title: A Tale of Two Log Files (Part II)
Subtitle: How Proper Procedure and Comprehensive Logging make an Administrators job easier.
This is a story. It's fictional and not necessarily factually/technically correct in all cases but I am using it to demonstrate two things that are very important to an administrator in the event of a compromise, the procedure and the logs. Both go hand in hand in the event of a compromise and both must be in place prior to the event itself. The proper planning prior to the event will speed up the investigation and save time and therefore money in the "clean-up" and mitigation of the breach. It's a lot to do with the mindset and approach whether you are prepared or not and that is what I will try to show during the story.
Throughout the story you will find numbers in []. They point to the notes at the end of the story. The notes are meant to show what the participants did right or wrong, what should have been done prior to the event or what could have been done better.
This one was written with no technical reference available.... on a plane..... just ignore the inaccuracies.....
--------------------------------------------------------------------------------------------------
Following his procedure Gary had carefully carried out the remote portion of his investigation of Mike Panoff's computer and was somewhat surprised to see that everything was as it should be. He called network administration and requested a new computer imaged for Mike Panoff's department be brought to Mike's office, picked up his forensic CD carrying his "toolkit" and made his way to Mike's office.
"What's up then Gary", Mike said as he entered.
"Ahhh... Not much... There's nothing wrong with your computer from my point of view but I noticed the CPU fan seems to be failing. I've ordered you a new box and it'll be here in the hour. I'm doing the netadmins a favor and running the diagnostics for them"
"But what was it that made you look at it in the first place?"
"Oh, that...." Gary laughed, "Dumbass here misread a log file and transposed a couple of numbers. Those logs turn you cross-eyed after the first ten minutes" [8]
Gary and Mike carried on chatting as Gary mapped a network drive and began running through his repertoire of tricks sending the output away to the newly mapped drive. He didn't have to look to his procedure book to determine what he should be doing next which may have raised Mike's suspicion's because he had built the CD from the procedure manual and had practiced with it. He had batch files written for the more complex switch requirements of some tools so that he wouldn't have to remember the commands or refer to a manual for just this situation. All he needed to know was that he needed to run every tool on the CD, with the batch file if it was available and the location to send the resulting data. [9]
An hour and fourty seven minutes after leaving, Dirk was again parking his car in TFCU's parking lot. His spirits were up from the initial low of the day and he had promised himself to have a "cold one" for Fyodor next time he had the chance. The NMap scans he ran against his sixteen IP public subnet had shown that there was no direct exposure to the public network. Small victory, but a victory all the same. He dropped into the CEO's office on the way back to his own.
"Well, I have some news.... It wasn't directly from the outside. I can't find a way in from the outside"
"Er... But that's not good news then. Doesn't that mean that "Al" is an employee?"
"It could, but that is yet to be seen"
"I'm not sure I understand.... If it isn't from outside then it must be from inside. Who's computer did the email come from?"
"I don't know right now, I haven't checked. But I'm telling you that right now we don't want to be accusing anyone of anything".
"I'd like to know who owns that computer.... and I'd like to know pretty soon".
"Mike, listen to me. There are a lot of ways that someone within the network messed up a machine without knowing it and now someone outside controls their computer." [10]
"You just told me that there was no access from the outside, so that's not possible"
"Mike, it is.... It's like.... er... ET calling home.... I don't remember the proper term but I read about it a few months ago. It's a way around the firewalls" [11]
"So you are saying that even though no-one can get in from outside, the firewall we payed that contractor almost $3000 for a few years back is useless?"
"No, no... not at all. I have to let some traffic flow freely from the inside to the out or you won't get your email, other staff won't be able to use the web etc. etc.. So if Al got a program inside here that calls home through the... um, web interface that I have to have open then there isn't a damned thing I can do to stop it." [12]
Gary leant back in his chair as he addressed the meeting. The entire team was assembled and he had outlined the initial evidence that indicated a compromise had taken place.
"I just completed the data collection from a user's computer that may have been used to steal the data. The computer has been switched off and removed from the user's office and he has been given a replacement. I took a quick look at the data I collected and there is one thing that jumps out at me that seems a little odd but I need to get the drive imaged and secured before I can look deeper into that."
"Can you be more clear" said Bill Steel, the legal representative on the team, "I mean do we have a suspect or are there other possibilities"
"Well, at this point I'm not going to point a finger anywhere. The odd thing on that box is a scheduled backup for 6:00pm nightly. It's odd because by policy all user data is forced up to the file servers to keep it safe so there is no reason to run backups on the client workstations. However, until I can access a copy of the drive I can't determine what is really there and why.
The meeting continued as the parties moved, step by step, through the series of questions they need to have answers to so that they can determine the course of action the company was going to take in this instance. The sticking point was "Al". Having not had subsequent contact with him the team had no way of knowing what he was going to request. Clearly it was money but to a banking institution the amount would be of great interest. The meeting adjourned with no initial recommendation which would be postponed until after "Al" had made his demands clear. This bothered Gary somewhat in that until the recommendations were formulated and accepted by the board he had to treat everything as if it would have to be presented in a court of law and this would slow him down. He'd made that point at the meeting to ensure everyone was aware. [13]
Dirk sat staring blankly at his monitor. "Oh dear.... Why did it have to be her box, why couldn't it have been that moron in Sales? I wouldn't mind chasing him down and getting his ass nailed" he thought. After leaving Mike's office he had decided to start by looking at the computer that sent the email last night. Having checked his records the IP and DNS name of the machine indicated it was Amy's computer. [14] His mind gyrated as he tried to determine how to approach her and what to do when he got there when suddenly he thought, "Hah, it can't be Amy..... Yes!.... She has no access the the database..... perfect...and I can prove it.... The server's log files will prove that." He scoured the log files for the previous week searching for anything that would point to Amy accessing the server. He was disappointed until he saw the time of the access, 10:27pm six days ago. There wasn't a successful login but there had been two attempts. "Two tries" he thought, "Someone didn't want to trip the automatic lockout. It couldn't have been her, she would have used her password". Poking around a little more he came to the transaction logs for that date. Scrolling down through the endless text he noticed some odd entries. "What on earth is that?" he thought, "Never seen entries like that..."
SELECT username FROM users WHERE MID$(username, 3) = "m" AND 1 = 1;
There were lots of them, rotating the numbers and the letters in the "where" statement. After looking at the entries for a while he thought he could see what was going on. "Someone is doing something to the administrator name in the user tables", he thought, "But what's the 'AND 1=1' all about?" He was familiar with basic SQL queries but his practical knowledge was limited. Looking further down through the log he could see where the queries that extracted account information from the customer tables. "Well, for right now I can see where the leak took place but where the queries originated from is anyone's guess. I'll make backups of these logs and then see if I can find where the queries came from." he thought. [15]
Gary moved across to his Secured Logging Systems analysis console while the image of Mike Panoff's drive was being completed, documented and secured. "No point in sitting on my hands", he thought, "If I'm lucky the internal IDS logs might show something of interest on Mike's box". He filtered the output to show only alerts from the internal sensors and started looking through the logs. While it isn't unusual for workstations to trigger portscan alerts one caught Gary's eye.
Portscan from 192.168.70.153 Ended: Time: 12 seconds, Hosts: 18, TCP: 18, UDP: 0
"What's the Cincinnatti office doing kicking off a portscan? It must have only just exceeded the threshold", he thought as he opened the portscan log itself. "The time is right and if Mike's box is one of the target boxes then things may be coming together". He looked down to the appropriate time and date and found the offending IP address. There it was, Mike's IP address had been scanned from Cincinnatti about 30 minutes before the login to the server took place. "Something isn't right... " he thought, "the scan is against port 80, Mike's box didn't have 80 open. I'm going to need that image to follow this track. First let's see who our new "potential perp from Cincy' might be, maybe we have ourselves a winner." he thought reaching for the phone. His hand didn't get to it before it rang. He picked it up expecting to hear that the techs were done with the drive imaging procedure and that he could get it back.
"Hello, Gary speaking"
"Tell your board that the price is ten thousand dollars. I'm not a greedy lad and ten grand isn't much to your bosses, they could probably each pay it from their own pockets and not miss it".
Gary grabbed for a pencil after the initial shock of hearing "Al's" voice again. He needed to get the converstion down verbatim if he could but he already knew that he wouldn't be able to do that. "What I can scribble down is what they are going to have to get" he thought as he checked his watch and noted down the time.
"Ten grand is a nice chunk of change, I dunno what they are going to say about that" Gary replied
"Bullshit, and you know it. The information I have here is worth way more than ten grand.... Tell them to be smart.... and keep your eye on your email"...
The click of the phone found Gary still writing frantically to try to document the converstion precisely. He'd done a pretty good job, "a word here a word there" he thought as he finished up and reached for the phone again. "We need another IRT meeting, and fast" he thought as he began dialing familiar numbers
-----------------------------------------------------------------------------------------------------
The Notes:-
[8] Here's Gary lying again, but it's all in a good cause. He's come up with a story for why the box needs replacement and why it came to his attention in the first place. They don't have to even be good stories for most users because they trust that you know what you are doing. Admitting to making a mistake goes a long way to having people believe you. It doesn't matter that you didn't make a mistake, the user is likely to empathize and accept your story more readily.
[9] This is one of the great benefits of being prepared. Gary doesn't have to concentrate so hard on what he is doing to the exclusion of all else. His tools are all there, organized, easy to use and they follow the policy laid down. If the policy requires that certain tools be run in a certain order place them in folders on the CD called "1st", "2nd" etc. The more you can do beforehand to make your task easy in the real event then the less stressed and more effective you will be. A further side benefit is demonstrated in the story, Gary can go about his difficult task while making it seem easy, stress free and routine without raising people's suspicions. Furthermore, he can complete any documentation of the tools run and in what order from the time/date stamps on the files he is creating.
[10] Kudos again to Dirk. His boss is hell bent on finding a culprit quickly. He's made a decision, based on a lack of information, and he now wants action. Dirk has, quite rightly, pointed out that there are alternatives that are equally probable and that rash action should not be taken. He also didn't accuse anyone of doing something maliciously, he used the term "messed up" that implys an accident rather than using a more purjorative term such as "downloaded something" which implies a more deliberate act. This helps to keep the stress level of the major stakeholder's down a little and can keep them from interfering in the wrong way.
[11] You don't have to be a cracker to be effective in security. But you do need to keep up with what a cracker can and can't do. You don't even need to remember the details, just that it can be done. Without that knowledge it would have been easy for Dirk to also conclude that this is an inside job. Keeping up with the knowledge means you don't have to discover things for yourself. It also allows you to be creative in your thinking, as a cracker would, in the ways these weaknesses can be exploited.
[12] Dirk thought about how he would phrase technical details so as not to complicate the issue with jargon unless it is absolutely necessary, (which is usually only at the conclusion of an investigation). This just isn't the time to be throwing around port numbers and protocol names to stressed executives that have no idea what you are talking about. The odds are they will ask for clarification which costs you more time and there is a high probability that they will inadvertently misrepresent what you said to others thus confusing the issue and starting the rumor mill turning.
[13] Gary is on top of things as usual. It is critically important that the IRT understand that the investigation must proceed from the beginning at a "litigeous" pace, meaning that the technicians are expected to do everything as if it will be presented in a court of law. This is much slower than moving at an "investigative" pace which lacks much of the documentation and evidence preservation required by a court of law. It is also of critical importance to ensure that the board understands that when you will move from the higher requirement to the lower the evidence will be tainted. It must be stressed and understood by both the IRT and the board of directors that the two methodologies are exclusive and once the litigious methodology is departed from there is no going back with any hope of a successful prosecution.
[14] Regardless of the fact that there are people you like, people you dislike, people you trust and people you don't there is one single rule in an investigation that you must follow. You distrust and dislike them all equally, it's as simple as that. If you don't you will allow preconceptions to cloud the process and possibly, unwittingly, allow them to move you away from the "truth".
[15] Dirk's lack of preparation and knowledge is beginning to show through. He's "playing" with a log file that may contain evidence but he isn't documenting it and he didn't make copies before he looked at them. If Mike choses to try to prosecute the perpetrator in the future the chances are high that he will never get the case to court. Now he's found his evidence he is making the appropriate copies, but this is too late. The copies should be made first, preferably to "write once" media such as CD-R if a litigious course is to be taken and then they should be searched and manipulated from the read only copy of the backup.
