Slow Network needs Smarter Switch?

Hope everyone had a good weekend and enjoyed the holiday as well.

Now down to business...

The network I am in charge of has been growing a lot over the past year. We have 55 employees...where just a year ago it was around 30. We have over 200 nodes that are all connected into unmanaged/webmanaged switches which are all 100Mb or 1Gb. We have 75+ computers, 60+ Xbox 360s, 25+ PS3s, and 60+ VOIP Phones.

Recently as we have been getting our newest game title ready we have increased the amount of traffic and amount of data going over our network considerably. In doing this the performance over the past month has been dropping off and now I am trying to figure out the best course of action so as we grow further this won't happen in the future.

I wanted to get a good poll going of ideas to see what would really help out in making our network performance shine. Would Layer 3 switches help out? Or would a better topology be the better/cheaper answer? What other things can really cause bottlenecks on a network? Are there good tools that will help me discover what areas really need improvement?

As always thanks in advance for taking the time to respond to me and my dilemma, if you need more information just ask and I will do my best to provide it.

Is all data/voice on one flat network?
No VLANs? One subnet/broadcast domain?

Have you looked into packetshapers to give priority to the important data, and less bandwith to the not-so-important data?

Cheers:

What is the current sustained backplane bandwidth on your existing layer2 device? One advantage of the layer3 switch approach is the ability to VLAN and route traffic on the same backplane while utilizing Quality of Service (QoS) mechanisms such as Cisco's CAR, NBAR or Traffic Shaping.

Yes everything is on the same flat network. Servers occupy XXX.XXX.2.X, while printers, phones, computers, consoles (except the PS3s) occupy XXX.XXX.5.XXX, PS3s are XXX.XXX.4.XXX. The phones are on seperate switches, but they still are connected to a main switch where the internet comes in.

This in my opinion is a network that just grew too fast and wasn't planned out as well as it should. My job now is to find out why our app server is so slow and I have a feeling it is the network and the amount of data being pushed over and over through the dumb switches.

No VLANs are in place and very little segregation is in palce. The uplink from each switch is 1 Gb and I am not sure if we are really exceeding bandwidth as much as pushing so many requests from all these nodes simulataneously.

Check the big four on your server, CPU, Memory, Network, Hard disk. Could simply be that your server needs a second network card and another gigabit link into the network.

Could be the problem we have - the network is fine but the server hard drive isn't fast enough to keep up. 320Mbps hard drive, Gigabit ethernet.

We have over 200 nodes too, probably close to 300 now. Network is the least of our problems, however I suspect ours gets entirely different usage patterns to yours.

Yes ...server hardware is also something to look at...

MLF

The strange thing is the computer seems to be running fine. I am going to run a full hardware diag. plus do a full system update/bios update. I will let everyone know the results.

I also took a look at the application running on the server (Perforce), but according to the technician everything checked out, except that people were making very large requests over the network, but that is to be expected since we are making a game. Only over the last month has speed become an issue.

The HD setup is in raid 5 with 10K rpm 150GB SCSI drives so not sure if that is a bottleneck. Also when I tried to increase the ram (which I thought would help) I found out that Windows 2003 server doesn't support more than 4GB unless you change some sort of registry entry so it can take the more ram. Would this still be something I should do to increase server performance.

Thanks for the suggestions guys...keep them coming.

well, even a raid of scsi drives has a max throughput of 320 Mpbs because of the SCSI controller card's limits.

Window 2003>Performance Monitor

when you say slowness...performance dropping off.....exactly what is slow...running an application...browsing the network\internet...printing????

http://www.windowsnetworking.com/art...e_Monitor.html

What else is running on the server?

MLF

It is slow over the network for one. The server itself seems to be slow to respond but only when there are lots of people at work (which I expected would cause some performance decreases).

It functions as an App server, DNS, DHCP, WINS, file, printer and a DC. Yes it does a lot, but it has been doing this for a while. I still am led to believe that a better design for our network would do some good stuff for performance on this server as well as the other main File and Email server on our network.

I'd say the main two issues I am trying to resolve are the slow response times from the Application and the lag in response I get from the File server (mainly it just hangs when you try to access it and then after a minute will be very quick as if it was waiting in line on the network).

Hope this helps and I am going to see if I can use the performance monitor to identify the problem, but I am not the best understanding what is overloading and what is within normal functionality, guess I'll just have to try it and learn it.

Here is a basic segment that I was monitoring. Lots of spikes from the HD, but should I be looking at anything specifically that would shed more light on where the problem lies...or perhaps I am staring straight at it and not knowing it.

On the graph I uploaded it shows the Avg. Disk Queue Length in green, Pages/sec of memory in dark blue, packets/sec in light blue/teal, processor in red and bytes total/sec in pink.

I don't know if you can make it all out, but perhaps this will give some insight.

Ok so just over the last few minutes every part of the performance monitor was peaking, except the processor. It seems the server is just running at full load constantly.

Just a moment after I typed the top portion the monitor started to calm down, but for a good few minutes everything was maxing out. Packets, bytes and other related network monitoring are still maxing out. The HD, Processor, memory are all more or less back to normal again.

get the shared printer off the server...share it from a workstation.

What type of applications...???

Do you have an AV scanning the server?
disable and see if there is a difference?


MLF

yeah, things we have considered to reduce access loads:
1) moving shared printers
2) moving desktop icons etc to local pc
3) moving start menu to local pc
4) reducing logs
5) removing AV

Unfortunately for us none of those is an option right now, but maybe for you.
If none of those helps/is possible its time to get a second server to share the load.

yeah...another thing sure to chew resources is roaming profiles....

are you using them???

MLF

So one area I have already begun replacing is all the Hubs in our office with Switches. This should help out since I was detecting lots of RX and TX errors from people who have a lot of nodes at their desks and were forced to use Hubs from a while back.

Another thing I am going to do is use a Layer 3 switch to give priority to the main server.

I don't believe the server has AV on it, but I'll double check. I should find a way to move the print server off their as well, but it isn't used very much anyways. What logs do you speak of? I think that would also help, but not sure what logs are taking place on this server. I don't believe there are any roaming profiles, but I'll do a quick double check to be sure.

It is too bad no one here is a Perforce expert, but I am learning it as I go. Do you think if I could somehow offload some of the load onto a secondary network card in the server that it might help? Unfortunately these servers were all set up as is long before I got here.

Replacing those hubs should be priority one. You mentioned your network traffic has increased recently and those hubs are allowing a lot of collisions.

Once you have your switches in place configure your network cards to run in full duplex. This should make a huge difference.

Good luck.

Thanks for confirming my thoughts. These hubs were causing a lot of collisions. Now that we are putting the switches in place things should get a bit better.

So are you saying setting the NICs to 100Mb/Full Duplex mode instead of auto is a better method? What about the other option of setting it to auto negotiate/1000Mb.

If your hardware is capable of 1Gb/s speeds configure it as such. Hardcoding the nic and switch to full duplex is recommended as this essentially doubles your bandwidth and if you can do so you should with the speed at 1Gb/s. You could leave it to auto-negotiate although this is typically frowned upon for reasons I am unsure but apparently having nics operating at varying duplex is a bad thing.

Ignoring the details replacing those hubs irregardless of your nics configuration will prove a drastic improvement.

The 1Gb speed is on an autonegotiate setting. The full duplex setting says 100 Mb. Why isn't there a 1Gb full duplex mode?

I also am going to update all the drivers and firmware on the server that has been a sluggish performer. It could be that it was just network issues, but better get everything out of the way.

Would setting the VOIP phones to a VLan help out? Or would it be just as good to set them to low priority in QoS? Once again thanks for all the help and if you can think of anything else that can really streamline a network let me know.

Does anyone have a recommended switch that is good for large amounts of traffic/data and works well under heavy loads. Currently we have a mixed bag of 3com superstack switches, Netgear smart switch, Dell unmanaged/webmanaged switch, and some PoE switches for the VOIP phones. Perhaps someone knows of a good way to set these up for best performance.

I would definately look into tweaking the app....

from your performance monitor something is definely thrashing your harddrives

either set up a seperate array for the application (usually done with sql apps)

system and regular data are mirrored and the sql is in its own RAID array...on a seperate controller

Ask the vendor....they must have some recommended guidelines....if not...threaten to change applications...that usually starts the wheels rolling.

MLF

From Wikipedia,

Quote:

---

Autonegotiation is a requirement for using 1000BASE-T[2] according to the standard. Several device drivers will allow you to force 1000 Mbps full duplex to eliminate autonegotiation issues.

---

I wasn't aware of that, good to know.

well, we use 3com super stack over here.

One trick is to reduce the number you have stacked (if any) for example we have 5 switches stacked in one rack here. This means that roughly 1/3 of our network is coming off a single Gigabit line from the core switch, which makes no sense given that one room of 12 pcs has a gigabit line all of its own. Again, the whole "grown not planned" syndrome.

Short version: if you have too many pcs per backbone line that could be a problem.

It might be worth checking for firmware updates for all your switches. New firmware rarely hurts. Unless the install goes wrong and turns your hardware into a paperweight.

Thanks for the heads up guys. I realized that since half our company is in a new section of our office that has only 1 Gb line...that may be a problem. Unfortunatly the other Gb line is for the phones so I am not sure what I can do about that. I may have to get a wiring guy in here to run a few more lines from our new office section so we can have at least 3 or more Gb lines running from this new part of our office to our server room.

I am also looking to find new drivers for the network cards so that perhaps I can set them to full duplex 1000Mb mode. Instead of auto/auto negotiate or 100Mb full duplex.

By the way we also have a bunch of Super stacks over here and they are also stacked on top of each other. I'll try out the new firmware too. Thanks again for all the ideas and possible problem areas.