Flashing Lights on Cisco Catalyst series......ARP flood?

[fraggin]

I have never seen this kind of activity on the switches.
All of the lights on all of the LayerII 100Mb switches were flashing in unison at about 2 blinks per second. That's like 48X8 lights.
The 1Gig backbone switch lights were doing the same thing except it was more like 10 or 15 blinks per second. During this time, we lost network connectivity (due to timeouts) for about two minutes. The Chaos then subsided with techs running around offering educated guesses of WTF was going on....

I turned the light out in the Data Center and Enjoyed the Show.

Unfortunately, there was no realtime scans in place to catch any packets or connections, but I kicked LAN hound off and it found that there was much multicast traffic and a couple of network nodes on the same machines that had the same IP addresses for each node. That was about it. At first, I thought it could be the SSH Xploit since our Firewall runs a Linux kernel, but that is just a partially pheasible guess............Does this sound like an ARP flood?
In my career history, I have never seen such light activity on switches as this morning................

[gunit0072003]

What kind of switch was it (model #)?
what IOS software was it running?
How many users were connected to it?
Was it a stanalone switch or was was it trunked to other switches?
Was it the "blinks" alone that concerned everyone in datacenter or was it fact that users were disconnected from network?

It seems like there was a broadcast strorm that hosed down the switch, however cause could be a number of things.

1) possible known bug in IOS
2) spanning tree was disabled or crapped out causing loop (which is mostly likely scenario, a switched loop network will hose down entire network,,very dangerous)
3) ARP flooding or any malicious code sending broadcast storm ofcourse is also a possible scenario..(disgruntled employees messing around)

cheers

[fraggin]

Catalyst 3550's by Cisco and I Gigabit Catalyst by cisco for backbone switches
All of the 48 ports are connected by GBIC Firewire modules connected point to point, no daisy chain with full duplex 1Gb communication between switches.

Not sure What IOS it was running

About 98% of the ports are full on all of the switches except for one. All users connected.

Trunked to others as stated above

Applications started timing out, some applications "froze", employees could not search databases that we use constantly. The lights caught my attention and then complaints started coming in over the two way radio traffic immediatley thereafter.

Can you extrapolate on the term "switched loop network"........
There are no more than two hops between any location on our lan, so no one has ever configured the spanning tree (This was the excuse that I was given). Is this something that I should start looking at?

Thanks for all the help.......

[gunit0072003]

"Can you extrapolate on the term "switched loop network"........ "

Sorry,,,,,I could have been a bit clearer,,,,,

A switch behaves just like a transparent bridge that uses either IEEE802.1d or the DEC spanning tree algorithm..The function of Spanning tree is to maintain a loop free topology in a switched or bridges network...

Basically in a switched or bridged environment, if you have more than one path to the same destination and they are both forwarding, the packets will proliferate in an endless loop saturating your entire network...It is not like experiencing a loop in a routed environment...In a routed environment, only one packet goes into a loop and only that packet is affected...

In a switched or bridged environment, the packets reproduce infinitely untill you LAN is saturated with broadcast storms,,,this includes your switch and all PCs..servers as well...
Very dangerous and is very ugly......

Spanning tree prevents these loops from ocurring by choosing one path to foward and another to block,,,When the primary link goes down, the blocked path goes into forwarding mode....

By default every time you configure VLANs on Cisco switches, you automatically turn Spanning tree on ,,,it is a safety precaution,,,,However, the CIsco IOS allows you to turn it off for all VLANS except for VLAN 1...Always turned on because VTP traffic rides on VLAN 1 (VTP is another topic not related to this discussion)

Anyway for the comment "no one has ever configured the spanning tree " I hope for your companys sake that that individual was not person responsible for supporting the switches.
Its like driving a car and not knowing where the break peddle is

If you are interested in further reading, I recommend the following book by Radia Perlman,,,,A must have for any network engineer,,,She is an extremely brillant and well known author in the IT industry..

http://www.amazon.com/exec/obidos/tg...er#reader-link

There are also a lot of tutorials on the net on Spanning tree,,I didnt get into details,,,Just wanted to give you basics to help you understand what might have caused the network outage...

Good luck,,,