DC Problems

This has been one doozy of a network problem, and I'm not sure I can even give you guys enough useful info for you to be able to throw ideas at me, but I'm gonna take a whack at it anyway...my boss and I are both stumped on this one so far, and I have a bad feeling this is some deeply rooted Active Directory problem, as Tiger has been warning me would inevitably come sooner or later given my past AD catastrophies....That said...

Yesterday, the DC just stopped working right. This is with NO changes made to it. It was just running, untouched, doing its thing, and just up and stopped. It kicked all clients off the Internet and stopped allowing clients to connect to it remotely via terminal services. Also, it's been preventing network connections to our internal database.

Now how this happened without any configuration settings ever being changed is beyond me. The last time I had problems, it was an issue with the two DCs talking to each other, so I removed the other DC and gave all roles to this one. For the past couple of months it has worked fine as the sole DC on the network, and it's been smooth sailing.

Error logs showed some KCC and RSVP errors for a while, and after trying some fixes those have disappeared. Now the event logs look clean. It looked like it was still trying to replicate to another DC out there that no longer existed, and what's really weird is that at some point it was actually trying to replicate to itself! So I went through AD and further cleaned out references to the now gone other DC and the one mysterious reference to itself.

Checked the network card and drivers, all is in order and passes all diagnostics.

To look the DC over, you'd give it a clean bill of health. Everything looks clean and in good order.

Nonetheless, no client machine can connect to the Internet (and it is a DNS issue surely -- the router looks to be working just fine), the DC keeps kicking off client connections to itself via terminal services, and the connection to the database keeps breaking, seemingly at random times.

For now we're trying to promote one of the other servers as a DC and DNS server as a temporary fix until we figure this out, and I'm about to try a Windows repair on it. Other than though, any ideas at all?

I would be inclined to check the server for virii, and possibly spy/malware. Also, check the system drive for free space and make sure that the server is not set to disable itself if the log file is full.

Bring them both down and wait 10 minutes. Then boot the one that was in charge first. Wait 10 minutes then boot the other. And go set your clients for alternate DNS servers. One primary and one backup so that when the first one is down they can use the second one to get access to address translations.

None of that's working...

Perhaps it's a hardware issue? I'm about to start playing with our switches if see if we can pinpoint something...I'll report back...

This is just getting crazier and crazier. All the hardware seems in check. Even swapped switches, still no use. The network just randomly goes down every few minutes (~15min) or so. If you're connected to the DC remotely when this happens, the remote connection freezes for a couple of minutes and then you're kicked. Then Internet and intranet go down.

So we finally caved in and brought in an outside networking expert to take a look at it. Well, even he's stumped! (So at least I don't feel totally incompetant now, and the boss got to see that even the superior expert is confused -- at least that's good news.) So, now I'm bringing the DC up to date on all the latest patches (the Windows repair rolled it back), and then our networking expert is going to call Microsoft for further assistance.

So, we'll see where this goes...in the meantime, we've got a bunch of employees with nothing to do other than clean their desks.

If you were to deliberately take the system off-line, how would you do it, so that it wouldn't be apparent ?

Have you been 'hacked' ?

If you take the system off-line, does the internal set up still function ?
Without let or hinderence ?

When was it noticed ?
Do you have logs PRIOR to then ?

If possible, a 'new' machine, fresh install, to get you away ?

How complete are your back-ups ?
Can you veriffy the integrity of the back ups ?

Do you have ANY disgruntled EX-employees ?

just some thoughts.

Anytime I have seen a network mysteriously go down it's been a DOS attack. Either from the internet or from some mallware inside on a users PC. Last 2 cases were a worm attacking from the internet that would take about 10 minutes to flood the main router and in another case, some one who installed some marketing "tool" that turned out to be cranking out 1000s of emails thus making the DNS servers unreachable.

Where is the server and the workstations getting their time from. If there is too large a difference between the DC and the workstations then the network can drop because the DC will start rejecting the connections due to the time discrepancy.

Seen it once when I messed up a servers time.... a large number of clients that had probably used it as the login server just dropped right off the network.

Quote:

---

he network just randomly goes down every few minutes (~15min) or so.

---

Do you have a backup battery that is failing knocking off your hardware? I've had that problem on a set of switches before. It didn't bring down my whole network though... just a couple of segments.

What do the system logs say? They've got to be giving you *some* information.

Are you physcially in front of the hardware when it goes down? Do you notice anyting strange? Unusual blinking lights? Power cycle of your switches? etc?

I am ALMOST tempted to try this....

Quote:

---

Seen it once when I messed up a servers time....

---

Road... Go for it... it's easily fixed.... just change the time on a DC that doesn't provide time to the workstations by 1 hour and wait for the phone to start ringing.... Change it back and the phone will stop ringing... The (L)users won't call you back to thank you.... ;)

Quote:

---

Originally posted here by RoadClosed
I am ALMOST tempted to try this....

---

I've had a problem authenticating workstations when the time on the workstations and the server don't match. I've never seen them get KICKED off the network though after they've been authenticated... I'm not saying it doesn't work. I've just never seen it and I don't dare try it.

Well, I'm starting to suspect the spyware suspicion may be correct. To test that theory, we started this morning by only connecting the servers and the three most critical computers to the network. Sure enough, they've done fine with no problems whatsoever. So now I'm running scans on all computers then adding one at a time to the network, watching a while for problems, then connecting the next. So far we now have several connected with no problems. It also may be a hardware issue with our using two switches, so we're on one switch now then will add the second when I fill up the first, and test to see if that's the problem. So it looks like we're on our way to finding the culprit...Should be interesting to see if we indeed do have a rogue computer DOSing the network.

Ok, so now the network's slowing down mysteriously. I've added a few computers, and so far it's been doing ok. Just added a second switch after filling up the first one (leaving the second bare for now to see if it works well or not), and now things are starting to slow down. It's a new switch though. So could it be a network card somewhere slowing things down, or perhaps the switch?

Have you run Ethereal or some other network analyzer on the network?

Are the switches both rated the same [10 / 100 Mbps / Gig ]

Even new kit can fail, if kit works from the box, then it will probably work away for years, it's those first few days that cause the problems.

Are the switches manageable?
Are there any other protocols running wild on the network besides TCP/IP? (Such as IPX, etc)

New switches can fail, can be setup improperly, can overheat.

Kinda sounds like a broadcast storm to me. But I'm not there to scan it.

Tiger Shark is right, get a protocol analyzer on it WITHOUT the 2nd switch attached. Get used to what you are seeing, then add the 2nd switch.

Divide and Conquer.

(Always easier said than done when giving 3rd party advice)

:)

try running EatherApe on your network (comes with the 'audit' CD so you don't have to install *nix). it will render a graphical image in real time of all the protocols all the computers on the network are using and the amount of data being sent.

screen shot

http://etherape.sourceforge.net/images/

Well, I've switched out switches and so far so good, so I dunno...

Guess I'm gonna have to go read up on that Ethereal tutorial huh?

So I switched that second swith out with one of our older switches (try saying that three times fast), and the slowdown immediately stopped. Been adding computers one by one to that switch since yesterday, and so far everything's still going just fine.

As I've added each computer one at a time slowly in hopes of catching the rogue, I have made some discoveries. I just now discovered that the front desk receptionist's computer had quite a few trojans on it. Also, I had network problems soon as I reconnected one of the conference room computers, so now I'm looking at both of those as possible network troublemakers.

Now I've narrowed the problem down further...

The problem wasn't that computer after all. Here's the background:

This room in question is a large conference room. We have a US Robitcs wireless AP upstairs. Employees often use wireless laptops downstairs in this room, so the wireless signal gets somewhat weak. With the last few months, I bought an identical USR AP and configured it as a repeater (just upgraded firmware on both AP and repeater last week, BTW). I've been running the repeater out of my office, but someone suggested moving it directly into that conference room would be even better for signal strength.

Of course, there's only one RJ-45 port in that room, and a computer is already connected to it. So I took an old Netgear 4-port hub and set up the repeater and the box to share the same wall jack. I did this within the last few days.

So I got the idea, maybe it's not the computer, but the hub. So I disconnected the hub and reconnected the computer directly into the wall. Sure enough, no more problems. So it's the hub! Of course, I wanted to further test this before tossing hardware, so I took it upstairs and plugged it directly into the switch with the repeater connected. Sure enough, the problem resurfaced. For argument's sake, I removed the AP from the hub and connected it to another hub, the one it used to be connected to when it was in my office. Well guess what -- the problem resurfaced.

As one last test, I removed the repeater from the hub and plugged it directly into the switch upstairs. Sure enough, everybody got kicked off again, thus confirming my theory.

So somehow it's the USR repeater that's jammed the network all along. Who woulda guessed?! So am I to presume that this little repeater has actually been packet storming the network or something? Anyone else ever run into this kinda thing?

Glad you found the problem!

I'd start learning the Ethereal thingy like Tigershark suggested. It'll surely help later.

Next suggestion, Not to be rude, okay..nevermind that then....Fire that outside network expert. Why an expert doesn't scan the network for a broadcast storm is beyond me.
Unbelievable.
It had me going for broadcast storm, based on how it sounded, which is one of the first things I do in an unknown network, "check out the heartbeat".

That repeater may still be good with a factory reset or firmware update, then use Ethereal to see what it puts out and if good, back into service it goes.

One thing "I" would do, is take that hub out forever. I've seen so many problems with those things I detest looking at them.

Life is so much easier when you finally see the light of day after a problem like this.

Good job!