Slow Lan-- What will be the reason?

Hi guys,

I am having a Gigabit LAN setup with a Cisco Router for connecting to Internet via Leased line & PIX firewall for protection. Having foure L2 switch (Cisco 2950) connected in ring type network (structure cabling with RJ 45 ). Spanning Tree has been enabled at every cisco switch. We are using D-link 1024R unmanaged switch for connecting PCs of end user. We have total 600 PCs connected in network. No VLAN has been created due to unmanaged swtch hasn't got vlan facility. Main purpose of our network in data sharing between Intrane rather than Internet.

But from las 1-2 months our LAN becom slow.

what will be the reason behind it ? how can I check the available bandwith & used bandwith of our network? will it be any virus problem?

Shall I use Layer3 switch to connect all four L2 switches?

..Viruss


Thanks in advance

chances are you have a bunch of infected machines blasting your network and eating up the bandwidth. I would use a decent network monitoring tool (solarwinds is awesome, but expensive, Netowrk Sniffer is also very good) and watch the traffic levels on the routers. Also check your firewall logs and look for fishy traffic. make sure the computers are patched (yea this is a pain for that many computer, but necessary)

chances are with the right tools you will be able to narrow the traffic down to a few IP addresses and just shut them down, clean them, and let them back in.

good luck.

Thanks XTC46..

I will try your valuable suggestions practically. I have tried ethereal, but not able to detect the exact problem. What types of packets should I look to solve the problem? Shall I use any Enterprise version antivirus software to detect network viruses(welchia,nimda)? will You suggest me any good AV software?

... Viruss

You might try pulling one of the noticeably affected machines and scanning it for malware in safe mode. :)

which software shall I use to scan thoroughly? Another question comes in my mind that connecting network in ring type topology will make LAN slow? is there any broadcasting done by unmanage switch ?

sorry for asking such silly questions..

PC-Cillin, Kaspersky, AVG, McAfee will all probably do the job. I am not a fan of Norton, particularly in an environment where you already have a problem of it running slow. As you can get trial versions, you might like to try more than one?

I would also get SpyBot S&D and AdAware SE.

Remember, update everything and run it in safe mode.....................this is why I am suggesting you pull a machine and check that to see if you do have a malware malware problem, and if so, its extent.

It might even be worth putting a local software firewall on the machine and re-connecting it to see who tries to connect, and on what ports.

:)

if you are using ethereal try turning a machine on and not running any programs, if it starts broadcasting more than what it should than you have a problem. Also try going into a command prompt and typing in "netstat -a" without the quotes. that will show you all open ports and what their state is. if you see any strange connections take that machine and scan it with an antivirus and adaware.

make sure all scans are in safemode.

LANs slowdown is a big problem in many parts of the world. This is mainly due to spyware and malware being downloaded into your machines from the Internet. Once a host program is downloaded into your machine, it starts broadcasting data packets at exponentially increasing and alarming rates.
Use some personal firewall and one Antivirus onto your machines. Monitor logs from the firewall and see which machine on your network is initiating the packet attacks etc inside your LAN. Clean those machines afterwards which have been detected to be infected.

Hi

One source of problems has been thoroughly discussed (gauravjulka ,
do you actually read previous posts before posting? I do not want to
be rude, since sometimes it is good to paraphrase, elaborate or add
additional information.).

Quote:

---

But from las 1-2 months our LAN becom slow.

---

I am not sure, what that means. From every node to every node or just from
some boxes to a few other boxes? Is one of the switches damaged? Collisions?
Is one of the NIC's damaged and throwing electronic noise on the cable
(I actually had this recently in my home network)? Cables ok?
Does the spanning tree really eliminate all possible loops? Check your network
topology drawings (Visio does really a nice job of illustrating these kind of issues).

I would try to proceed as follows:
physical layer ok? hardware?
data link layer ok? spanning trees, loops?
...
malicious activity?

Note, besides "ordinary" loops, there is the issue of bridging loops in the STP
environment, which comes from a port that should be blocking, but is forwarding
traffic instead[1]

Cheers.

[1] http://www.cisco.com/warp/public/473/16.html

Sec_ware, I think you have the order to check tings mixed up. In my opinion starting from hardware failure is generally a bad idea becasue in a network this size there could be a problem almost anyware. you want to start with the easy/obvious possibilities first. If the slow down is just in the last month or two it probably is malware (which has been pointed out over and over again) from there I agree take a look at some network diagrams (visio is my fav also) and see where else the problem can be. check your logs for collissions, errors, time outs. things of that nature. THEN if all else fails start testing cables and all that good stuff. Token ring networks are generally slower, and you may want to look into upgrading to a better topology (if time/money permits) most common are star topologies.

Hi XTC46

What you say sounds pretty reasonable to me :) So please ignore my mixed ordering.
I just recalled to have spent hours searching software problems, while actually one
router was partly damaged (due to some loose contact) - this was years ago. I simply
realised that the focus was on malware, and I wanted to add a few more ingrediences.

According to your posts, you would:
- Identify the "affected" segments of the network (could be the whole one)
- Check the type of traffic there (malicious activity?)
- If so, detach identified nodes and clean them.
- If not, check the logs for collisions etc., re-evaluate your network design, ...
- Check the hardware.

Something like that? ;)

Cheers.

Viruss,

I guess I would want to know what exactly "was slowing down", but one of the other possiblities you might look into for the effected systems are:

1) Incorrect DNS. If the workstations are pointing to the wrong DNS it will slow things way down.
2) Any new protocol's added? Look to see if someone added AppleTalk (chatty at best) or IPX
3) Find the point of bottle neck. Is it just slow to log on? Accessing file shares? Everything?
4) Bad cabling. If all these effected systems are on the same piece of backbone cabling..?

Most of these things can be done while you are running a Adaware or Spybot or AV scan.

I have had a bad 3Com switch cause this type of problem as well.

Hey,

As the problem is relatively new, I think you said 1-2 months old. Have any users started streaming audio/radio/video to the desktops??

With 600 users, once one user figures out that they can listen to the radio via computer this news can spread quickly....

It was a little unclear to me.. is your internet traffic slow? Is the intranet traffic slow? or both.

I also ran into a situation where the switch changed from 100 Mbs to 10 Mbs by a sloppy admin. This might be an issue if you do not administer the network devices...

Hope this helps,
F1fan

Thanks to all of you for your valuable suggestions.

Let me clear my network scenario.
1) My data sharing between Intranet is slow. Internet access is not so important as it has only 128 kbps leased line.
2)600 PCs are connected in four switches all over network. There is no IP address scheme followed for this network. i.e. We are using 192.168.X.X/class B & there is no IP segments in our network.
3)There is not a certain area where LAN becomes slow. Some time it becomes very slow or some times it is better.
4)Due to lack of segmentation of network, I can't able to locate exact faulty switch in such huge area.
5) I have tried escan antivirus on client PCs but couldn't find any virus.

Mean while I will check with Ad-aware SE.
Shall I convert existing network in four different IP segmetns, so that there will not be any broadcasting on other network untill any request comes.
Shall I use Layer 3 switch to coneect these four segments? or Linux PC as router will work ofr it?

first off you REALLY need to rething the archetecture of your network. 600 nodes on a network that is unsegment is a hell of a lot. when you say some times its good some times its not it is probably casued by more users being online. In a token ring network, the more people online, the longer each has to wait for their turn and for the data to get to them. I would really consider switching to a star topology ifit is at all possible. if not, atleast segment the parts of the lan, and get ethereal going on that network. set up a few boxes (one for each segment if you can segment it) and see what the traffic is. look for tons of out bound traffic on any one computer (major sign of an infestation) also get a tool like solarwinds if money permits to monitor the load on the switches. 4 switches under very heavy traffic on a what seems to be very disorganized network could easily casue both slow downs and complete crashes. wha could be happening (depending on the design of your network) is that one of the switches is crashing and since the computers are in a ring configuration they have to send untill they hit the break, then back track all the way back around the network.

so, 1) get etheal running no matter what.
2) if possible rethink the network design (yea this is a huge project ofr a network your size, but it WILL help)
3) move from un managed switches to something more advanced if money/trained staff permits.
4) segment that lan and rething you ip scheme.

wow.. great suggestion . thanks to XTC46

I am trying to make four segment of the exisitng network.But let me know that will it possible to create VLAN for these segments by departmental wise ? what will be the basic hardware setup require for it i.e Fast Ethernet card or Gigabit card? (As I told early, I am having four Cisco 2950 switches with Dlink 1024R un-managed switches below it)
Will it possible to create VLAN with unmanaged switches? (Due to money problem, its not possible to change all unmanaged switches at a time)

I am thinking about purchasing a Layer 3 switch (cisco 4503 / 4506 for connecting these four cisco switches for routing purpose.

I request You guys to give me your valuable suggestions for this network..

thanks in advance.. Viruss

I have a few questions that I am surprised have not been asked here...


What version of Windows Server are you using? NT-x? Windows 2000 Server? Windows 2000 Advanced Server? Windows 2003 Server? UNIX? Solaris? I am assuming by the previous posts that it is on a Windows NT managed network as I have never heard of a Token Ring configuration ever working on a 2000 Server managed network. If it is a 2000 or 2003 network, does it have the typical DNS, DHCP, and DC, servers with redundant backup servers?

It sounds to me like your network is not on a domain at all, the IP range for internal Workgroups that you mentioned: 192.168.0.1 to 192.168.0.255 only allows a maximum of 254 connections on a subnet mask of 255.255.255.0, one of those will have to be on the LAN side of the router and another as the default gateway to the WAN, leaving a maximum of possible client connections of 252. How in the world does a network of 600 computers work on a network without DHCP or NAT translators to assign IP addresses? I can't imagine hooking up more than 5 or 10 workstations in a workgroup envroment, much less 600! I have problems with 3 computers on a workgroup at home. I would go nuts trying to keep a 600 node network running with that type of un-secure network architecture.

And last but not least... What version OS are the client workstations on? Are they all legitimate Win2k pro workstations? or are the a mish mash of Legacy system such as (shudder) win95, 98, ME, XP

aeallison: Reread the thread. Viruss has 600 clients in a 192.168.x.x Class B network. That's 192.168.0.0/16 or 192.168.0.0 netmask 255.255.0.0. The tokenring network was just an example to indicate more users -> slower network.


Viruss: As noted by others, I believe 600 nodes in 1 broadcast domain is way too much. You might be plagued by broadcast storms. Windows machines tend to broadcast a lot of crap.. Other things to check are the NetBios node types. This maybe a cause if you have a lot of broadcast traffic. Resolving DNS/WINS might be an issue too. But I highly recommend segmenting your network. It'll make life a whole lot easier...

Other things to look for are speed/duplex mismatches between the switch and the NICs. We set everything. We've had our fair share of users complaining about slow connections when the NIC and the switchport where both set to auto/auto (speed/duplex). Turns out both will try another setting every now and then and then it'll screw up.

SirDice : I have reread the thread as you suggested and (after reading your post) I am beginning to understand better what viruss is trying to accomplish. What I don't understand is how this system/network ever worked reliably. I do understand why he is haveing problems, with network speed.

I agree with SirDice that setting adaptors and switches to auto negotiate is a bad idea. I have had problems with this kind of setup before.

And why allow the entire network internet connectivity? If it is only an ISDN or DSL type connection with a 128k maximum bandwidth. This would be fine for home use ore for a small (less than 50 node) network. I am not trying to imply that this is an impossible configuration, just a badly structured one. My guess... tell me if I am wrong... is that this network started out with only a few nodes and through time it has had more and more added to it that it has long ago outgrown its original purpose. And as for gigabit adaptors and routers?? the connections will never be any faster than the slowest connection on the network. Again correct me if I am in error, the limit is in a 128k WAN connection, isn't this one eighth the bandwidth that is capable over a 1mb connection, why use a 1000mb connection? Spend that money on insuring that all of the clients are using the same OS and configurations that the users ( lusers as Gore would imply :D ) cannot screw with in order to chat or listen to music when they should be working. My experience tells me that out of 600 users at least 1 of those users will think he knows more about computers and networking than the Administrator.

Quote:

---

Originally posted here by aeallison
Again correct me if I am in error, the limit is in a 128k WAN connection, isn't this one eighth the bandwidth that is capable over a 1mb connection, why use a 1000mb connection?

---

He's doesn't seem to be worried about the internet connection, just the intranet portion which is not limite by the WAN connection.

Are your clients talking to a small number of servers in a server farm, or to each other? If you have 600 machines hitting one server this could also be a bottleneck.

I 100% agree that you have too many nodes on that one broadcast lan. Splitting them up is the best way to reduce the traffic load.

Thank you zENGER I stand corrected. Again I am only limited by what I would do in a similar situation.

Quote:

---

Originally posted here by SirDice
aeallison: Reread the thread. Viruss has 600 clients in a 192.168.x.x Class B network. That's 192.168.0.0/16 or 192.168.0.0 netmask 255.255.0.0. The tokenring network was just an example to indicate more users -> slower network.


Viruss: As noted by others, I believe 600 nodes in 1 broadcast domain is way too much. You might be plagued by broadcast storms. Windows machines tend to broadcast a lot of crap.. Other things to check are the NetBios node types. This maybe a cause if you have a lot of broadcast traffic. Resolving DNS/WINS might be an issue too. But I highly recommend segmenting your network. It'll make life a whole lot easier...

Other things to look for are speed/duplex mismatches between the switch and the NICs. We set everything. We've had our fair share of users complaining about slow connections when the NIC and the switchport where both set to auto/auto (speed/duplex). Turns out both will try another setting every now and then and then it'll screw up.

---

SirDice,

as I told early , I am using Dlink unmanaged switch where I can't telnet thse switches for any management task. (give me any idea to do it via any other source/method).

One more question :- I am going to make four segments of existing network. what should I use as a gateway between these four segments ? i.e. I mean to say any Linux PC with five ethernet cards ( this may be funny & i don't know wheter it will work or not) or any Layer 3 switch is required?

You could do it with a Linux PC. But I recommend investing in a couple of good switches. If you can replace the unmanaged switches with managed ones.

Are all those 600 clients in one building? What are the distances? Are the servers in one (server) room?

I would take a 100Mbit switch for the servers. Use 10/100Mbit switches for your clients. Connect all the switches with 1000Mbit (as a backbone). Use managed L3 switches. I believe the price difference between L2 and L3 switches isn't that much these days.

If distance permits you may want to look at a Catalist 4/5/6000 series (Cisco) or similar. With that you can easily segment your network using VLANs. You can also ditch all your other switches (if distance isn't a problem).

Quote:

---

Originally posted here by SirDice
You could do it with a Linux PC. But I recommend investing in a couple of good switches. If you can replace the unmanaged switches with managed ones.

Are all those 600 clients in one building? What are the distances? Are the servers in one (server) room?

I would take a 100Mbit switch for the servers. Use 10/100Mbit switches for your clients. Connect all the switches with 1000Mbit (as a backbone). Use managed L3 switches. I believe the price difference between L2 and L3 switches isn't that much these days.

If distance permits you may want to look at a Catalist 4/5/6000 series (Cisco) or similar. With that you can easily segment your network using VLANs. You can also ditch all your other switches (if distance isn't a problem).

---

thanks for the reply.

actually all these 600 PCs are not in one building. The distance betwwen switch 1 to switch 4 is more than 200 meter. So i can not convert the network in star topology. OS of these PCs are mixed type i.e. Win 98, win 2k, win xp, linux.
Basically we have three servers. Web server, DNS server, proxy server.

Quote:

---

actually all these 600 PCs are not in one building. The distance betwwen switch 1 to switch 4 is more than 200 meter. So i can not convert the network in star topology. OS of these PCs are mixed type i.e. Win 98, win 2k, win xp, linux.

---

"And may The Lord have mercy on your soul"

;)

You can still convert them to multiple star topoologies linked with fibre.

I agree with the above, create a smaller star topology in each building. then link those networks together. This is good for several reasons. 1) you can monitor each are seperatly 2) you will see a huge speed increase becasue there iwll be far less traffic on each segment. 3) the new topology alone is just faster. 4) when something breaks it will be easier to track down.

What I would do is create seperate subnets for each building. have a managed switch on each floor of every building and have all that leading back to one high end router/switch.