A Learning Experience.......
This is a story of an overworked Admin who, in the course of a seemingly normal day managed to bring down and then successfully troubleshoot an Exchange Server for 350 users. It's security related in a couple of ways but in general it is a story that can happen to anyone, that shows that something seemingly unrelated and irrelevant which is immediately dismissed, may not be found for weeks if you don't get lucky or if you don't examine symptoms fully. It's a true story that happened today.... to me.....
Precursor: The SurfControl server is a web filter. It packet sniffs the network looking for certain protocols, (primarily HTTP, though it is capable of numerous others by default and can have user defined protocols set up too). It compares the requests in the protocol to it's database of allowed/disallowed sites and if it finds a match in the "denied" rules it sends an RST to the remote server and spoofs a response from it to the local client with an "Access Denied" page. I did, in the past have this monitoring SMTP traffic since it's logs and reporting are rather quick and efficient. I turned it off a long time ago because the database became unweildy with all the additional traffic and now use other systems that aren't quite as "pretty" to log email transactions.
The Story:
Tiger sat down with the usual morning coffee to look through the nights emails. It's Friday and he's off to Toronto for the weekend to avoid the damned "Dream Cruise" that noisily takes place 600 yards from his house this weekend so he was looking for an easy day with no pressing problems. He knew that he was having an issue with the FTP server and that would need dealing with but other than that the day should be trouble free. Having finished the email he moved on to nights security logs when the phone rang. It was the VP of HR saying that whenever she tried to print this particular PDF she got nothing. Tiger told her to email it over and he'd print it for her. The PDF arrived and it was in Adobe 6.0 format and it wouldn't print.... "Pooh" thought Tiger, "Now I have to d/l the update"...
While the update was downloading Tiger suddenly remembered that he hadn't checked the SurfControl servers to see if there were any new webmail servers that people were trying to get to that were going unblocked. As he is looking throught the rather long list of newly attempted, unblocked but clearly mail related servers an email arrives. It's the FTP vendor. Just after that the "Download Complete" window pops up for the Adobe reader. At approximately the same time the log search for all the activity from an "interesting" IP address comes up too..... Tiger is a talented chap, (good looking too), so he carries on multi-tasking like a politician telling lies during election season. Having scanned the webmail servers for possible issues Tiger gleefully copies and pastes the entire list into the denied column and presses "Apply".... "Another little avenue of pleasue cut off for the wasters" he thought to himself.
Fast forward 20 minutes...... The Senior Network Admin calls Tiger.... "Do you have a problem with email? Because mine is getting really squirrelly". "Nope" replies Tiger clicking "Send/Receive" and getting no "waiting" message. "It must be you" he says confidently. Five minutes later the same call comes from the Help Desk. Tiger tries the same "troubleshooting technique" and confidently points out that it must be her..... But now he's getting that nasty feeling that all is not right in paradise. What's worse is that he is well aware that these funny little "niggles" on a friday afternoon usually mean that the weekend is a write-off before you start..... :eek:
Help desk calls back... She has a number of people calling with email problems.... Another employee calls from a user's office... The user can't see the Exchange server.. It pings but the mail store seems to unavailable. Tiger's brain leaps into "Troubleshoot Mode". Question one is "What has been changed?". A quick skim of recent memories in the brain and a few calls indicate nothing that would affect client to Exchange Server mail store has been changed... "Oh pooh..... This is one of 'those problems' that kill a weekend" he thought.
Tiger quickly pulled up the performance monitor on the Exchange server.... Checking the bandwidth use, mail sent/received and a few other parameters everything seemed normal. Reaching deeper into the toolbox he pulled out Ethereal, (this server does allow Outlook Web Access from the web), and filters is for all traffic to and from the server..... lots of local traffic, nothing from the outside.... The phone is still ringing and Tiger can see Toronto getting further and further away
Standing at the server looking glumly at performance monitors Tiger's "high tech" admin suggests we virus scan it..... Tiger's initial reaction is "BS, no virus can get past the firewall to this server"..... But then the little light bulb goes off in his head...... "Yep, do it... but don't scan the M: drive", knowing full well that the "high tech kid" is already well aware of that.... It's called leadership isn't it????? :rolleyes:
The lightbulb: About two years ago we got an infection of Elkern inside the network. We have "eliminated" it about three or four times to date.... but it lives out there somewhere, on a laptop or a workstation hidden in a dark corner of a wet basement somewhere. When it comes it somehow manages to get onto one server or another. On the bright side it's symptoms are a basic messing up of the server function but after a virus scan and clean, (it's always cleanable), and a restart the server returns to normal. We track it backwards, clean any workstation that also got affected but we always reach a dead end as to the source.... Easy you say, put AV on every box.... Can't - non-profit, can't afford it.... Easy, go to every box, install AV, scan, uninstall.... Tried it.... Apparently we can't find the particular corner of the wet basement it is hidden in.... <sigh>
Tiger stalks back to his office, fires up a command prompt and telnets to the SMTP interface of the Exchange server because he notices that he hasn't received a single listserver message in over an hour now..... it functions perfectly and he sends himself a manual message which is received immediately. "Odd" he thinks.... He Terminal Services to the DMZ mailserver and tries the same.... "Connection terminated by server".... At this point tiger dismisses it as a symptom of the Exchange server issues but files it near the top of the pile of "Really odd" because SMTP is usually a reliable service and the virus results are starting to come in...... Yep, it's that bloody Elkern again.... Tiger calls the Senior network admin telling her to check the virus logs of all the data servers because usually when Elkern is activated it gets into the user's server folders too..... but the real time AV hasn't warned us of anything.... ??????
A little later the Exchange server is cleaned, restarted and seems to be running fine.... The second AV check is running and it seems to be clean so far..... But still we have only 60-70 logins from the user base of over 350..... but we know they are trying.... because the phone keeps ringing.... At this point, (2:45pm), you have no idea how far Toronto seems to be from Detroit..... This is a mess....
Tiger sits at his workstation and tries to determine which particular little "niggle" about what he has seen in the last two hours bothers him the most...... He decides that he really hates the DMZ mail server not being able to send inbound mail..... "OK", he thinks. "lets find out why"..... Out comes ethereal again but this time he filters for the DMZ server and the Exchange server and lets it run for a couple of minutes..... Not a single packet..... No problem, the DMZ server has probably decided that the Exchange is down and is queuing the mail for future delivery. A quick terminal services session to the DMZ mail server, a telnet session to the SMTP port of the Exchange server, a rejection by the Exchange server and a look at the Ethereal dump solved the problem......
The ethereal dump showed a normal SYN, SYN/ACK, ACK sequence between the two servers on port 25, (SMTP). But, injected right at the end of that sequence was a message, purportedly from the Exchange server that said......
"Access denied by Surfcontrol"
WTF?????? This is an SMTP connection not an HTTP???? A quick look through the large list of servers Tiger had placed on the "denied" list earlier in the day showed that he hadn't noticed his own mail server on that list. Tiger quickly deleted it from the denied list and, whistling nervously to himself took a stroll down to the server room.... By the time he got there the list of logged on users had increased from 60-70 to over 200.... <phew> A couple of phone calls to "known down" users and yet another from a particularly irritating user that was checking every 10 minutes but hadn't checked in the last three minutes proved that all was well and Toronto was again only a few hours away....... :D
While I have yet to determine why the SurfControl server chose to block SMTP access to the exchange server this is a good example of how inattention to detail when dealing with security can have unexpected results across a whole network and that even if the decision seemed to be valid you can't always trust the software you use to do things in the way you want them to.
As a final note.... The reason that SurfControl would allow perfectly good access to 60+ machines throughout the whole incident yet block the others is an architecture issue. Those 60+ machines are "architecturally" closer to the Exchange server than the Surfcontrol box thus they could complete their transactions prior to the SurfControl box interfering.... The other boxes weren't, including the DMZ mail server since the Surfcontrol box is as close to the DMZ as I can put it so I can actually block the "illegal" web access.....
I learned something today..... I hope others can from my screw-up.....
And writing all this down made me think more deeply about what occurred and why and will help me not make similar mistakes again.... He says..... Hopefully.....
