Printable View
Hi Folks,
I have a Cisco Cat3550 router, and 5 subnets, one of which is a rather large 172.16.0.0/16 subnet. Of late, I've seen this backbone router sending out ARP floods on this 65,000+ hosts network, effectively flooding it. Anyone have an idea why? Is this normal, and what purpose would it serve?
Thanks a lot for your input,
_Scim_
When you say ARP floods what kind of transmissions are they?
Well, I used Ethereal to capture traffic, and the router was broadcasting ARP requests incrementally for the entire Network like
Who has 172.16.131.1 tell 172.16.0.253
Who has 172.16.131.2 tell 172.16.0.253
Who has 172.16.131.3 tell 172.16.0.253
...
_Scim_
Sounds like it is updating it's own tables. What protocols do you have running on the router?
it supposed to arp on that network (ethernet perhaps?) to find mac addresses of the hosts on that subnet. but "fllooding" the subnet.... its odd
router only does that when its a "new ip" and its mac isnt on router' arp table.
Only have RIPv1 running on the router. Regarding updating its arp tables, it wouldn't go about polling every ip on that net for an arp response would it? I would've thought building the Mac/port table would be just as in L2 switches, as and when frames are received on that port.
Cacosapo, I didn't get what you meant by 'router only does that [flooding] when its a "new ip" and its mac isnt on router' arp table.'
Thanks for the responses
Scim
I've seen this happen when router memory is going bad and it has a tough time keeping track of whats going on, it's possible that it's answering queries from another device, in other words, something is going through the subnet an IP at a time and asking the router where each ip is this will cause the router to arp for an answer and of course send it on off. What other stuff you got going on in ethereal there? a good dump would be handy
Scimitar,
As you know arp's only come , and are used in a layer 2 environment. Maybe you need so segment up your network more. Maybe install some more routers and definitely use more divided subnets.
As a understand you have effectively /16 subnets without being segmented by layer3 devices ? IMHO a cisco 3550 ( http://www.cisco.com/en/US/products/hw/switches/ps646/ )
is a switch ( a gigabit-root-switch ), which would explain you have that big subnets get flooded.
Also if you really have that big subnets your bandwiths will get slow due to requests like for.ex. ARP.
You can stop ARP-requests or layer2 - protocols by dividing networks with a layer3 device, most namely routers. This can also be achieved with implementing VLAN's if you have only a switched environment, but you definetely need routers. Did you implemented VLAN's ? etc etc ...
Maybe a scheme of your networks would help us to solve your problem. More input on your backbone might be helpful too.
Cheers
Shrekkie does have a point, and after reading it I noticed I mistook your catalyst 3500 for a 3800 series router... eh... go finger. Either way, the 3500 is capable of layer 3 routing between Vlans, if your stuck in a budget rut you may want to look at using that feature to cut the traffic. I would in either case, follow shrekkies advice and segment up a bit.
If your subnets really need to be as large as they seem I assume you have other switches in your topology that may be misconfigured or need a configuration change as well. The incrimental requests definately point to something odd (a scan going on somewhere) as switches typically learn pathing by examining traffic and not asking whats where.
Could it be that some device is looking for something? whether it be someone running printer software that's scanning the subnet for a printer, or even something so odd as a box running as a honeynet (these will scan around to see what is and isnt in use)? Typically in brodcast storms and arp cache updates the requests are not so sequential.
- back -
Sorry, I couldn't get back to this thread earlier. Needed to give some attention to my academics over the end of the week.
Regarding the router flooding, its not doing that regularly, but at what would seem random intervals of time. Maybe I can script ethereal to run at certain intervals, parse the results for ARPs from the router, and record/graph it. Or would anyone have an idea about a tool that does just that?
Regarding the network scheme, I've a Cisco Cat3550, which provides routing between 4 different networks, 196.1.64.0/24, 196.1.65.0/24, 196.1.67.0/24 and 172.16.0.0/16. No Vlans are used, as the network has grown adhoc. I do realize the network has not only to be segmented further, but also undergo a major redesign, but that'd be the senior admin's decision. Me, I'm just the graduate student trying to make money and get experience .. and loving it.
I'll see what I can find with periodic network sniffing. Again, thanx a lot for your feedback.
Scim
The C - class subnets are just fine and will not have much flooding. But it's crazy to have a /16 subnet unsegmented ! How many real hosts are in this subnet anyway ?Quote:
Regarding the network scheme, I've a Cisco Cat3550, which provides routing between 4 different networks, 196.1.64.0/24, 196.1.65.0/24, 196.1.67.0/24 and 172.16.0.0/16.
If you really have that much hosts in such subnet, then periodic sniffing won't help you cause it'll just be normal everyday request-traffic. I believe the problem is obvious :o
Anyway , good luck,
'Amen.Quote:
But it's crazy to have a /16 subnet unsegmented ! How many real hosts are in this subnet anyway ?
And the number of actuals hosts on that is less than 400.. yup, 400 out of 65,534. Of course, there's also the other security issue of using routable, non-private addresses such as 196.1.x.x. But then, the network's really grown ad hoc, and rite now, I'm supposed to just fix, not mess around. Anyway, the idea at the moment of the higher-ups, is to model the entire network in OPNET [the college got a multiple user license, and are itching to use it], and based on the simulation results, recommend changes, model these changes.. and finally, implement. Its slow, but why'd I complain about getting free opnet classes? ;-) Right, I guess thats enough banter from my end... back to work.
Scim
Okay, there's 3Com Network Inspector running on one of the machines in the 196.1.67.0 Network, that every 12 hours discovers/updates the entire network diagram. For the 172.16.0.0/16 Net, it pings each and every possible IP on that net. Hence the router sending incessant ARPs.
However, one more thing, ethereal shows the source IP for these arps as 196.1.64.253 [which is the default gw for the 196.1.64.0 net], instead of 196.1.67.253 [the default gw for 196.1.67.0 Net]. The machine running 3Com NI is on the 196.1.67.0 network. Shouldn't the source hence be 196.1.67.253 ? If I'm not being clear, please let me know.
To recap the college network, 4 different networks - 172.16.0.0/16, 196.1.65.0/24, 196.1.64.0/24, and 196.1.67.0/24.
Thanx
Scim
Here is what I would do.
1) Scrap everything and start over with a solid design that contains a bunch of class C segments. Use a router and a switch (I assume a router in each building) at each location and have all of those routers feed into your 3550. This will:
a) Remove the horrible issue you have with a /16 mungo range.
b) Provide a logical design for management and growth.
c) Cut down on tons of ARP traffic on the *entire* WAN. You'll still see normal updates but they will be localized.
While you are at it, for God sake man, use RFC 1918 and ditch the damn public address schemes! Using public addresses on a private network can cause all sorts of DNS fun if you have the right circumstances. Things like yahoo resolving to an internal server, etc.
I would use VLANs only if necessary. While they certainly are useful, many times jr. techs seem to dismiss the idea that certain ports on a switch/router do different things and unless you are hardass about documentation, you're bound to run into someone throwing a host on the wrong segment when patching them in.
Seems like you have a Cisco shop. May I suggest getting CiscoWorks to manage your environment.
Hi all,
most of the things have already been said , so far about arp storming and ad hoc growth , and 1918 and 196..... , honestly thatz how most of the times networks grow.
I have seen many many networks that ways... here are a few things u should look at immediately..
1 Ciscoworks might sound expensive proposal, try www.solarwinds.net , download 1 copy of Engineers edition and that will last for 1 month atleast...( enough time to resolve current things).
2 As horse stated , restructing from scratch, might not be practically viable, but you surley
can choose a small area, rectify it ( work on its archt. carrefully) identify the traffic flow, the best designed network is were you know how the traffic flows.
Try and use hierarchy for designing your network, believe me you will be able to sleep w/o thinking of your network :-).
regards.