Hacked network...

Had a big networking outfit come in and put in a new server at our company (one server, 12 desktops). The decision was made migrate W2K Server from the old server to the new one, including the old ISA firewall (this last one was not my idea!). The server also functions as a proxy & a gateway (it's got two nics). I'm the in-house support guy.

Things were ok for a couple of weeks and then we had a few problems with the server. Various .exe files issued memory address errors and a couple of printers would go offline, making themselves unavailable to the desktops. My first reaction was "hardware".

We had the networking company come back out, they explained that the ISA 2000 firewall was old tech and the server had been hacked. They cleaned things up a bit and installed a Cisco Pix firewall. It was VERY expensive, and my sense is they should've put us in the Pix to begin with.

Now we got hit with both Netsky and Bagle viruses on two machines. I shook those down using RAV's online virus scan, downloaded Symantec's virus removal tools and thought I had everything cleaned up (also scanned the server). But today, I was still getting odd emails from our network that had .cpl attachments, typically viral, but in this case null (0 bytes) files.

Any thoughts on how to proceed? I'm thinking of running RAV's online scan on everything come Saturday when it's slow. This networking outfit put us in CA's network antivirus software over a year ago, and apparently put it on the new server, but I'm guessing it's not set up right, so I'll probably wade into that too. I hate to call these guys because it's $2000 everytime they send a couple of guys out for half-a-day on short notice, and they don't tell us anything. I'd like to at least be grounded enough in our problem that I can doublecheck their work.

Should I be checking for files on the server like netcat or some other remote access stuff that an AV program would miss? I'm familiar with ethereal though it would be some work, could I put that on the network and track down these rogues?

Thanks.

...our finance guy is got an email from himself today saying "Thanks for letting us use your hardware". He didn't send it apparently and he deleted it before I got a chance to read the headers.

Hahahahahahahahahaha! Now THAT is rubbing it in!

Anyway, I used to have problems like this too and having to clean my cousin's machine all the time because hes not a computer dude. I'd have to clean up the trojan infected mess he made fur alle toggen.

I found a great way to get rid of that problem though when I installed this great anti virii here

For a netowrk your size, I would recommend taking all machines off the network and cleaning each one individualy. Its a hassel but what it probably happening is that you clean one machine and then the virus has already spread to another. so its just a big circle of chase the virus.

so take everyone off the network. Get the tools on a disk and clean EVERY SINGLE MACHINE then hoook them back up to the netowrk as you clean them. Make sure they ALL have the most recent viris def.'s and be sure to keep them updated with all OS updates. this includes your firewalls firmware.

You have got to be shitting. Do you know how plausible it is to take that many machines off the network without users, the boss and the CEO killing you? You can't do that. If something happens while the machines are offline, you're ****ed. My idea may not be popular or super plausible but it would work. You could start cleaning boxes while the SUSE machines were up and running. When you get one cleaned, put it to the side. They could all use SUSE for servers and they won't get any worms.

its 12 workstations and a server. with 2 people going at it it would take less then 2 hours of complete downtime. do the server first, and do it after hours. its not that big of a deal. easily doable.

I'd have to agree with XTC. A documented plan as to how you are going to go through each machine will make it faster.

I'd also suggest some of the following:

- do a check for any unusual programs (keyloggers) as well as any physical "things" that shouldn't be there (physical keyloggers)
- change passwords after they've been cleaned and before you put them back on the network
- you might want to consider looking at having another AV scanner tool as a double-check (perhaps after you've cleaned them initially, connect to the network using safe-mode and visit TrendMicro's Housecall)
- change the password for the PIX as well
- install an IDS perhaps?

It does sound like you need something to detect attacks on the network. Where is the mail server for your company? If you run it, is it possible that the email message was just someone messing around? Granted it was from the finance guy to himself but is there a chance of an open relay?

Quote:

---

My idea may not be popular or super plausible but it would work.

---

Lastly, Gore, if you're not going to suggest something that is useful, please don't suggest it. Seriously. While SUSE is wonderful and all, if someone is a Windows admin, it doesn't help them and can make things worse. For whatever reason, his company has chosen Windows and he has to support it since that is the company choice (right or wrong). Installing SUSE isn't the answer since it could open up more issues if he's not familar with it and what they may be. Particuarly since we don't know what the server is used for. (Print server? file server? SQL database? Web? Mail?)

Gore, you have to think outside the "SuSE's the best, install it on everything that understands binary" box. I know you love SuSE, and I don't blame you, but it's just not the best solution in all cases.

The business case for taking all the machines off the network is good, especially if weekend work is available.

The business case for installing SuSE on all PCs isn't great when you build in either Crossover Office or training for OpenOffice. Think also about MS Access apps that are often used by small offices - they aren't going to transfer that nicely.

The business case for installing SuSE on servers is either bad or terrible, depending on what's on the servers. I don't know how well apache run .asp pages, for example. Maybe it does, but I'm sure you can see the possible problems.

The deal is here that it's a real company with real needs. It's not just a fantasy world where SuSE can be plugged in to make everything better.

Hell, I'd love to replace all my Windows installations with SuSE, but I don't have the time (or inclination) to train the users.

Oh, as an aside, one thing I _would_ replace with Linux is the firewall. Smoothwall's nice and it'll work just fine for such a small setup (assuming you don't have insane amounts of data going in and out). Even better, you don't need to know much about Linux.

www.smoothwall.org is the site for the GPL version.

Quote:

---

Oh, as an aside, one thing I _would_ replace with Linux is the firewall.

---

You'd replace a hardware firewall like Pix with a software firewall!? Why?

...I'm well-versed in Smoothwall, using it on a network of my own. He-heh, even built it my own self (I had to do something with those ol' Pentium I's). But the Pix is what we got. Reckon I'll have to dig out the password and check it out. I have yet to have a good look at it.

Thinkm this is what I'm going to do:

1) run HijackThis on everything (I'm well-versed in that one, too) to look for anything unusual like keyloggers (lawdy, I dread what the server's log file will look like...).

2) run RAV's online AV scan (I've found that one to be the best out there).

3) do this on Saturday when it's slow and I'm able to take anything offline without incurring the usual wrath of my fellows.

Thoughts?

Thought: You're mostly concentrating on removing the concequenses of the hack, you're not really looking for the origin of it. You might get lucky and find some trojan, but that trojan had to come from somewhere. I would go as MSMittens suggested and install an IDS to track intrusion attempts on your network.

You have a single problem..... You don't know what it is you are looking for at this point.... All the trojan scanners, AV app, etc. etc. etc. won't help you if the code is altered or home grown.

Hijack this might work but it suffers from a major failing.... It only shows the processes not the threads running within them. Process Explorer from SysInternals is a better choice but you need to know what are normal threads within key processes and what are abnormal which is quite a feat unless you know what you are looking for in the first place.... So that's something of a dead end too.

If these kids really did "use your hardware" then there isn't anything there you can trust. Clearly, downing and re-imaging all the boxes, while thoroughly recommended in this situation, doesn't seem to be a viable solution either....

Which leads to Ms. M. and Guus' solution coupled with doing what you can to move the "obvious" stuff and a _very_ strict ingress and egress policy on the Pix... I would start by blocking _all_ inbound and _all_ outbound traffic through the Pix. The open only those ports that are _required_ for business to continue. Log every attempt to come in or go out and look for things that don't look right.... Odd ports, odd times, (leave all the boxes on 25/7.... If you dont then something that wanted to run might not be able to and you will miss it in the long run as you begin to believe that you have successfully cleaned the problem), odd destinations, (I don't know if you can turn on host resolution on the Pix but if you can I would recommend it - then you can look at the destinations that do not resolve and those that do but to odd locations when coupled with the port tried).

Then you need two more things.....

Snort is king.... :cool:.... Fire it up at the gateway and select only the rules that deal with an internal network, (SMB etc.), and rules that are affected by the open ports you need to conduct business, (the firewall logs will alert you to the clearly "bad" traffic), and monitor the snort logs carefully.

Lastly, and as a final "safety net" pop an ethereal box there, (of let it share the snort box, 13 or so computers shouldn't be able to max out the box), and capture all traffic inbound and out so that when you have suspicions about certain connection attempts or internal boxes you have a complete record of what went on.....

Then I'd start looking at the boxes themselves....

Good luck.....

Another point is cost, a well built firewall distro on a crappy or even have way decent box will run a lot less expense than a pix firewall. Its money you could save and invest in a better networked anti virus solution. It's all about the bottom line to a lot of businesses.

...guess I got my work cut out for me. Just took Beagle.BD off one of the service dept's computer.

Don't know snort though I put it on my W2K laptop (Windows-port). I've run ethereal from the same laptop a time or two though I'm on the learning curve there still.

Could I run Knoppix STD live linux cd off one of the workstations? I've used a numver of the live-cd's out there: Insert, Knoppix. Knoppix-STD. Can't say I'm too familiar with snort, but I'm a fast learner typically. Any good tutorials out there? I know the networking company we use is a Windows shop, the one guy I've talked to is useless on linux.

What a great forum...

This might help.... Since I am the author fire away with any questions.... You might want to combine some of the hardware to your laptop for example but it should give you a good start.

...not sure the boss has as much confidence in me on the server as he does this networking outfit. I'd like to try the Knoppix-STD cd on a workstation though...

I would suggest that since the manifestation of the Hack took place not longer after this "networking outfit" graced you with their "upgrade" that the "networking outfit" may be the cause of your problems rather than the solution.... Let's see... They stuck your one and only production server as the gateway to the network..... Did they ever hear of "best practice" let alone common sense..... My response to that suggestion would have been "Pick yer window.... Yer leaving"... You'd have been better off putting a Linksys in.....

Just my thoughts on the issue..... I could be wrong.... but you can quote me if you like......

...the latest virus-laden emails coming in, even though they show one of our email addresses, are actually originating from Cinergy up in Evansville, IN, now. At least according to the headers. I'll keep an eye on any more of these rogue emails. Can the Beagle virus spoof ip addresses in the headers?

Yeah, Tiger Shark, I'm not too hot on that fancy-pants networking outfit either...

...it was set up like that on the old server, they had an ISA firewall, for what IT'S worth, configured on it, and it worked ok. Not my choice, I like a lot of layers in there. Make 'em work for it I say. :)

No, viruses can't spoof the IP's in the headers from the point where the valid mailservers in the trail begin. By that I mean that the virus could add bogus headers before it sends the message but each mailserver in the trail will add the additional headers on their own so you should see:-

Your mailserver stating it received it from IP X
Mailserver X stating it received it from IP Y (assuming the ISP doesn't block the originating IP which some do).

That would, generally, be it unless you or the sending domain has forwarding servers built into their systems. The virus could add a dummy header after mailserver X, but it doesn't really do much good and isn't something that is worth coding since it will be seen through quite quickly.

Quote:

---

Originally posted here by brokencrow
...the latest virus-laden emails coming in, even though they show one of our email addresses, are actually originating from Cinergy up in Evansville, IN, now. \

---

Thats not too far from me.... I do take donations, I can drive down there and melt down their server. just a nominal fee. besides I'm bored anyway, I can be the vigalante of spam

online virus scans are great, but you NEED an anti virus installed on each computer. Id still strongley suggest you take evey one of those machines off line and clean them well, or the problem WILL happen again.

and I wouldnt completey ignore the idea of gettin that linksys, lol.

Hey there,

I am showing this to my class this evening (right now) and we have some suggestions/questions to add on:

- a question of documentation: did the "networking firm" provide you with any as to how they configured items and setup the network/systems
- do you have a listing of firewall rules and have you ensured that the rules are adequate
- are your employees trained on what not to press
- have you determined the source of the "hack" (inside job? employee?)
- have you checked event viewer for any unusual activities
- what did you mean by migrating the "old server to new server"
- any idea why only the finance guy was targetted for the "email" (name recognition, senior officer, internet site, etc.)

We do believe, however, if the installs by "networking firm" didn't adequately protect your network, you might want to consider getting a refund on some of the support (or lack thereof). Anyways, food for thought.

Quote:

---

Originally posted here by MsMittens
You'd replace a hardware firewall like Pix with a software firewall!? Why?

---

This is purely personal preference. I guess, like gore I'm a little partisan at times. The reason I became turned on to smoothwall in the first place was cost. As they already have pix there's no reason to migrate, unless the maintenance is over costly.

As far as security goes... I have no reason to recommend smoothwall over pix. I'm not going to start yabbering on about peer review of code and all that 'cos you've heard it all before and I dare say the Cisco ppl aren't novices at security.

I will say that the only hardware fireall I used was a pain in the arse to configure, whereas smoothwall's a cinch. Again, that's a personal thing.

...looks like OUR network is no longer the source of any of the rogue emails (my fingers are crossed as I type this). The employees get confused because they see an email address that looks like one of ours, but a quick check of the headers yields a different IP address at this point. I've got our users aware of the problem with spoofed addresses that are apparently generated by the Bagle virus, and they are well aware of the dangers of email attachments. Tomorrow I'll go through all the computers, beginning with an online scan, then I'll begin straightening out any AV configuration problems we have.

MsMittens, no, we have little documentation from the networking company as to how they configure things. Part of that problem is that the there's little tech saavy among the employees, so they fail to ask for it. We do have a list of passwords and email settings, but beyond that, there's not much. I'm only there parttime, doing desktop support and training, some networking. I did have a chance to look at eventviewer, but not for long. Not sure of the firewall rules, haven't looked at that. Don't know where the hack came from or what it's exact nature was (the networking outfit handled that), but I do know that the Bagle virus does a splendid job of spoofing, even pulling user names and putting them in as domains. Quite clever. This company recently replaced their old Compaq ML330 with a brand new HP server, but migrated W2K SBS Server over to the new hardware. And it wasn't only the finance guy getting 'targeted' emails. I was even receiving them to my Yahoo acc't. This virus must pull stuff from the address book. At this point, I think I'm dealing with a virus problem only, but I will keep an eye on it.

I've talked to the owners before about security in the past, having used an online port scan to check out our vulnerabilities (the ISA firewall on the old server looked like Swiss cheese!). I'm very high on Smoothwall, using it on my own network, but they're in the Pix now, and that's ok. They're having me go in and cannablize some of the old hardware to upgrade the remaining Win98 machines, and as I do that, I'll go thru the network more. I've been able to add some improvements such as rewiring some of the jacks and replacing nics over the last year. I've been a bit of a jack-of-all-trades for them, and been especially helpful with some of the proprietary software that has to be installed and updated.

p.s.-- I'll probably install SME Server on the old Compaq and use it for backups in another building on the site. That'll be fun.

...just got a call the server took a dump. Apparently they got a blue screen and the server dumped the memory. They got a call into the networking company...I'm outta here. You can have that Winders junk.

ROFLMAO...

It had nothing whatsoever to do with rogue code trotting merrily around the network did it????

Hope this 'network outfit' are also a 'server outfit'...... :rolleyes:

...I typically end up as the scapegoat because I toggled some email settings on a workstation, or I used the server to view a webpage. No one there will t-o-u-c-h the server, such is the attitude. People get strange about tech stuff...real strange. Reminds me of a car mechanic I knew who commented on his job: he'd put on a new muffler, and the client would come back the next day complaining the mechanic had screwed up his brakes. :(

LOL... You noticed that phenomenon too....

My absolute favorite is when you go and help someone with a problem on a workstation on a 50 node network and then, for the next six weeks you brought down their T1, caused drive failure of their data drives on the only server, made all the backup tapes go bad and the office cat had kittens and they look like you...... :eek:

It's life... I have a good friend that wants me to come and redo his whole company's network, (about 35 workstations), I told him would come and look and see what he needs, then I'll do it for him and document it. I will then train him on how to deal with the day to day stuff bit I will _not_ be making this a second career... There will be no calling me every day to fix this or change that and that if it begins I will simply divorce myself from the whole thing.... He knows me and says fine.... So I pushed my luck and told him that if he wants me to do it then what I say in my initial recommendations goes, no shortcuts.... He says fine, price is no object.... So I reminded him I'm not cheap and he says "yes, I know $150/hour... No problem"... So I went back to the start of the conversation.... "OK, but I'm not looking for a second job.... You really understand that?".... I guess I might have myself a well paying side job for a few weeks and he's paying me for the initial recommendation too..... ;)

If you are in a position to make them pay _and_ to lay down the law..... Don't hesitate to, otherwise you end up as someone's "dawg"....

...and cleaned both Netsky and Beagle out of one of the owners' computers. Everyone else looks clean. Our DSL provider lost a router on Friday and they're still having problems with the RBC clients (the PPPoE crowd is back online apparently), so we were offline anyway. I had downloaded Symantec's tools for those viruses last week and copied the exe's to each computer and ran them. I'd say we're good to go, but I know better...

Broken..... Always expect the unexpected..... but then I guess it won't really be unexpected.... ;)

...is to be expected at this point. What a mess. I was in yesterday trying to get the internet back up. Our DSL provider is Alltel. They lost a router on Friday, told us to pull the plug on our ol' Orchid modem for four hours or more, still no internet. Rebooted the server, still no internet.

Called Alltel's tech support and they had me plug up my laptop straight to the modem. The internet worked fine. But hooked back up to our firewall, nada. So I bypassed the firewall and went straight to the second nic on the server (like it used to be), nada. I hooked the PIX back up and tried accessing it, no luck. I don't know PIX, haven't RTFM'ed yet. Pretty much gave up, then tried a linux boot disk on the server. Don't think it had the driver for the gigabyte nic on the HP server, so that didn't work either.

A couple of strange things though. One, shutting down the server, it had an application error in vcagent.exe and said "the memory could not be written". We've had a number of these app errors. Two, unexpectedly had the W2K background for the desktop disappear right before my eyes. That was weird. Display Properties was open when I came in and the background was set to the bmp, I changed it, scrolled back down and it wasn't there. That bmp was named "bobrand" which is not the default name. Maybe that one is nothing, I dunno.

I also went thru the eventviewer for a bit. Is there a way for me to copy the eventviewer to a spreadsheet so we got some documentation? Thanks.

Yes.... Manage your computer... select event logs right click the log - Save as - name it what you want and change the file type to csv.... Bingo....

Quote:

---

cleaned both Netsky and Beagle out of one of the owners' computers. Everyone else looks clean.

---

What a shock. :D This seems to be the rule rather than the exception. It's also generally the IT person's fault that the boss got these.

....networking co's MCSE was in yesterday, turned out Alltel was the problem (surprised? I'm not...) as they couldn't push an IP past Ye Olde Orchid modem. The MCSE was in the PIX, config'ing it, but couldn't pull an outside IP. Wished I'd had a Smoothie ready to go. We went back & forth with Alltel, they finally rolled us over from an RBE authentication to PPPoE and the net came back up about 5 p.m.

Pointed out a couple of the server issues, but got little response other than a quick check of the eventviewer. It's tough to deal with issues that aren't right in front of you, and this stuff comes in layers. We got the networking layer handled, got my eye out for ass't virii, and we'll see where she takes us...