Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: A Learning Experience.......

  1. #1
    AO Ancient: Team Leader
    Join Date
    Oct 2002
    Posts
    5,197

    A Learning Experience.......

    This is a story of an overworked Admin who, in the course of a seemingly normal day managed to bring down and then successfully troubleshoot an Exchange Server for 350 users. It's security related in a couple of ways but in general it is a story that can happen to anyone, that shows that something seemingly unrelated and irrelevant which is immediately dismissed, may not be found for weeks if you don't get lucky or if you don't examine symptoms fully. It's a true story that happened today.... to me.....

    Precursor: The SurfControl server is a web filter. It packet sniffs the network looking for certain protocols, (primarily HTTP, though it is capable of numerous others by default and can have user defined protocols set up too). It compares the requests in the protocol to it's database of allowed/disallowed sites and if it finds a match in the "denied" rules it sends an RST to the remote server and spoofs a response from it to the local client with an "Access Denied" page. I did, in the past have this monitoring SMTP traffic since it's logs and reporting are rather quick and efficient. I turned it off a long time ago because the database became unweildy with all the additional traffic and now use other systems that aren't quite as "pretty" to log email transactions.

    The Story:

    Tiger sat down with the usual morning coffee to look through the nights emails. It's Friday and he's off to Toronto for the weekend to avoid the damned "Dream Cruise" that noisily takes place 600 yards from his house this weekend so he was looking for an easy day with no pressing problems. He knew that he was having an issue with the FTP server and that would need dealing with but other than that the day should be trouble free. Having finished the email he moved on to nights security logs when the phone rang. It was the VP of HR saying that whenever she tried to print this particular PDF she got nothing. Tiger told her to email it over and he'd print it for her. The PDF arrived and it was in Adobe 6.0 format and it wouldn't print.... "Pooh" thought Tiger, "Now I have to d/l the update"...

    While the update was downloading Tiger suddenly remembered that he hadn't checked the SurfControl servers to see if there were any new webmail servers that people were trying to get to that were going unblocked. As he is looking throught the rather long list of newly attempted, unblocked but clearly mail related servers an email arrives. It's the FTP vendor. Just after that the "Download Complete" window pops up for the Adobe reader. At approximately the same time the log search for all the activity from an "interesting" IP address comes up too..... Tiger is a talented chap, (good looking too), so he carries on multi-tasking like a politician telling lies during election season. Having scanned the webmail servers for possible issues Tiger gleefully copies and pastes the entire list into the denied column and presses "Apply".... "Another little avenue of pleasue cut off for the wasters" he thought to himself.

    Fast forward 20 minutes...... The Senior Network Admin calls Tiger.... "Do you have a problem with email? Because mine is getting really squirrelly". "Nope" replies Tiger clicking "Send/Receive" and getting no "waiting" message. "It must be you" he says confidently. Five minutes later the same call comes from the Help Desk. Tiger tries the same "troubleshooting technique" and confidently points out that it must be her..... But now he's getting that nasty feeling that all is not right in paradise. What's worse is that he is well aware that these funny little "niggles" on a friday afternoon usually mean that the weekend is a write-off before you start.....

    Help desk calls back... She has a number of people calling with email problems.... Another employee calls from a user's office... The user can't see the Exchange server.. It pings but the mail store seems to unavailable. Tiger's brain leaps into "Troubleshoot Mode". Question one is "What has been changed?". A quick skim of recent memories in the brain and a few calls indicate nothing that would affect client to Exchange Server mail store has been changed... "Oh pooh..... This is one of 'those problems' that kill a weekend" he thought.

    Tiger quickly pulled up the performance monitor on the Exchange server.... Checking the bandwidth use, mail sent/received and a few other parameters everything seemed normal. Reaching deeper into the toolbox he pulled out Ethereal, (this server does allow Outlook Web Access from the web), and filters is for all traffic to and from the server..... lots of local traffic, nothing from the outside.... The phone is still ringing and Tiger can see Toronto getting further and further away

    Standing at the server looking glumly at performance monitors Tiger's "high tech" admin suggests we virus scan it..... Tiger's initial reaction is "BS, no virus can get past the firewall to this server"..... But then the little light bulb goes off in his head...... "Yep, do it... but don't scan the M: drive", knowing full well that the "high tech kid" is already well aware of that.... It's called leadership isn't it?????

    The lightbulb: About two years ago we got an infection of Elkern inside the network. We have "eliminated" it about three or four times to date.... but it lives out there somewhere, on a laptop or a workstation hidden in a dark corner of a wet basement somewhere. When it comes it somehow manages to get onto one server or another. On the bright side it's symptoms are a basic messing up of the server function but after a virus scan and clean, (it's always cleanable), and a restart the server returns to normal. We track it backwards, clean any workstation that also got affected but we always reach a dead end as to the source.... Easy you say, put AV on every box.... Can't - non-profit, can't afford it.... Easy, go to every box, install AV, scan, uninstall.... Tried it.... Apparently we can't find the particular corner of the wet basement it is hidden in.... <sigh>

    Tiger stalks back to his office, fires up a command prompt and telnets to the SMTP interface of the Exchange server because he notices that he hasn't received a single listserver message in over an hour now..... it functions perfectly and he sends himself a manual message which is received immediately. "Odd" he thinks.... He Terminal Services to the DMZ mailserver and tries the same.... "Connection terminated by server".... At this point tiger dismisses it as a symptom of the Exchange server issues but files it near the top of the pile of "Really odd" because SMTP is usually a reliable service and the virus results are starting to come in...... Yep, it's that bloody Elkern again.... Tiger calls the Senior network admin telling her to check the virus logs of all the data servers because usually when Elkern is activated it gets into the user's server folders too..... but the real time AV hasn't warned us of anything.... ??????

    A little later the Exchange server is cleaned, restarted and seems to be running fine.... The second AV check is running and it seems to be clean so far..... But still we have only 60-70 logins from the user base of over 350..... but we know they are trying.... because the phone keeps ringing.... At this point, (2:45pm), you have no idea how far Toronto seems to be from Detroit..... This is a mess....

    Tiger sits at his workstation and tries to determine which particular little "niggle" about what he has seen in the last two hours bothers him the most...... He decides that he really hates the DMZ mail server not being able to send inbound mail..... "OK", he thinks. "lets find out why"..... Out comes ethereal again but this time he filters for the DMZ server and the Exchange server and lets it run for a couple of minutes..... Not a single packet..... No problem, the DMZ server has probably decided that the Exchange is down and is queuing the mail for future delivery. A quick terminal services session to the DMZ mail server, a telnet session to the SMTP port of the Exchange server, a rejection by the Exchange server and a look at the Ethereal dump solved the problem......

    The ethereal dump showed a normal SYN, SYN/ACK, ACK sequence between the two servers on port 25, (SMTP). But, injected right at the end of that sequence was a message, purportedly from the Exchange server that said......

    "Access denied by Surfcontrol"

    WTF?????? This is an SMTP connection not an HTTP???? A quick look through the large list of servers Tiger had placed on the "denied" list earlier in the day showed that he hadn't noticed his own mail server on that list. Tiger quickly deleted it from the denied list and, whistling nervously to himself took a stroll down to the server room.... By the time he got there the list of logged on users had increased from 60-70 to over 200.... <phew> A couple of phone calls to "known down" users and yet another from a particularly irritating user that was checking every 10 minutes but hadn't checked in the last three minutes proved that all was well and Toronto was again only a few hours away.......

    While I have yet to determine why the SurfControl server chose to block SMTP access to the exchange server this is a good example of how inattention to detail when dealing with security can have unexpected results across a whole network and that even if the decision seemed to be valid you can't always trust the software you use to do things in the way you want them to.

    As a final note.... The reason that SurfControl would allow perfectly good access to 60+ machines throughout the whole incident yet block the others is an architecture issue. Those 60+ machines are "architecturally" closer to the Exchange server than the Surfcontrol box thus they could complete their transactions prior to the SurfControl box interfering.... The other boxes weren't, including the DMZ mail server since the Surfcontrol box is as close to the DMZ as I can put it so I can actually block the "illegal" web access.....

    I learned something today..... I hope others can from my screw-up.....

    And writing all this down made me think more deeply about what occurred and why and will help me not make similar mistakes again.... He says..... Hopefully.....
    Don\'t SYN us.... We\'ll SYN you.....
    \"A nation that draws too broad a difference between its scholars and its warriors will have its thinking done by cowards, and its fighting done by fools.\" - Thucydides

  2. #2
    the beign of authority kurt_der_koenig's Avatar
    Join Date
    Jan 2004
    Location
    Pa
    Posts
    567
    brovo!

    Tiger is a talented chap, (good looking too)
    --lol

  3. #3
    Senior Member gore's Avatar
    Join Date
    Oct 2002
    Location
    Michigan
    Posts
    7,177
    Man, I live 20 minutes from you, you had server problems, and you didn't come near me? When a server screws up, all it needs is a little BOFH love.

  4. #4
    AO Ancient: Team Leader
    Join Date
    Oct 2002
    Posts
    5,197
    Kurt:

    (good looking too)
    Just trust me on this one.... ok?

    Gore:

    you had server problems, and you didn't come near me
    Ahh... so... I had a choice.... While watching Toronto disappear into the distance I was supposed to think to myself.... "Well, I can fix this stuff, or..... I can call AO and see if Gore is about to help me fix it....."

    Sorry mate.... I have this "thing".... I told my boss years ago.... If there is a problem I can't find a way to fix then you will receive my resignation..... Stupid statement looking back on it..... But maybe not.... It keeps me on my toes.....
    Don\'t SYN us.... We\'ll SYN you.....
    \"A nation that draws too broad a difference between its scholars and its warriors will have its thinking done by cowards, and its fighting done by fools.\" - Thucydides

  5. #5
    Senior Member
    Join Date
    Oct 2002
    Posts
    4,055
    If there is a problem I can't find a way to fix then you will receive my resignation.....
    While perhap's you shouldn't have made the statement (I wouldn't have atleast) it does in fact keep you on your toes and sort of "forces" you to be up and about on updates, patches, etc and making sure everything is running right. Good story by the way, very interesting.
    Space For Rent.. =]

  6. #6
    AO Part Timer
    Join Date
    Feb 2003
    Posts
    331
    Sorry mate.... I have this "thing".... I told my boss years ago.... If there is a problem I can't find a way to fix then you will receive my resignation..... Stupid statement looking back on it..... But maybe not.... It keeps me on my toes.....
    This is what seperates those who have the passion, and those who just have the degree.

    Nice read. Thanks Tiger

  7. #7
    AO Security for Non-Geeks tonybradley's Avatar
    Join Date
    Aug 2002
    Posts
    830

    Smile

    I live 20 minutes from you, you had server problems, and you didn't come near me?
    I think I am even closer to him and I didn't get a call either.

    Great job and I am glad it didn't screw up your weekend. Get a pint in Toronto for me too!

  8. #8
    Senior Member gore's Avatar
    Join Date
    Oct 2002
    Location
    Michigan
    Posts
    7,177
    I know, the bastard. If I found out what company he worked at I'm sure I could make the day more interesting. Servers love Etherkills. Then hook the chair into a mains outlet...Ohhh yea. A little BOFH love goes a long way.


    EDIT:

    Does anyone else find it funny that Tony and Tiger liver near me... Heh, Tony the Tiger. Hahaha.

  9. #9
    Senior Member SirSub's Avatar
    Join Date
    May 2003
    Location
    Groom Lake, Nevada
    Posts
    148
    Don't forget the part about them being goreate!
    It is impossible to make anything foolproof because fools are so ingenious. - Murphy

  10. #10
    AO Ancient: Team Leader
    Join Date
    Oct 2002
    Posts
    5,197
    Gore: Tony really only lives around the corner from me..... Probably no more than 5 minutes.... So for speed's sake he would probably have been first call if I had the urge to go outside my company.... I have competent staff to help me as well.... I know, I employed them....

    As for giving away my identity so you can make my day "more interesting"..... On yer bike son.... I'm trying to make the process of making a living easier not harder.... And don't forget, I clearly am very good at reaching your goal all on my little lonesome....

    Dopey:

    This is what seperates those who have the passion, and those who just have the degree.
    I never thought of it quite like that before, but now you mention it....
    Don\'t SYN us.... We\'ll SYN you.....
    \"A nation that draws too broad a difference between its scholars and its warriors will have its thinking done by cowards, and its fighting done by fools.\" - Thucydides

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •