Ping Response from Cisco router

I'm configuting a PtP T1 connection with two Cisco 1841 modular routers.


Been having a lot of CRC erros, Telco says its not their problem, but I have cisco, and a buddy of mine that works for another ISP saying that my equipment is fine and its a hardware problem with the T1.

Phone company doesn't even know how to work the test equipment. THey get errors when they do a test a certain way, but don't know if its valid or not cause they've never used the equipment before.

Anyway, I can ping and what not across the T1, just I lose a lot of packets, and get a lot of CRC errors as stated above..The other day I got Alarms on both sides of the T1 and couldn't do anything. Came in today to attempt to troubleshoot it, and the errors gone on one side, but still on the other. I'm about to go the other side of the T1 and see what I can do, however I have one question.

When I do a ping from the router with no alarms to the other router I get the following.


grouter#ping 10.20.1.2
Sending 5, 100-byte ICMP echos to 10.20.1.2, timeout is 2 seconds:
.U.U.
Success rate 0 percent.



What does the u mean? I've never seen that before...tried some google searches but didn't come up with much.

Thanks in advance

U stands for a destination unreachable pdu was received. Take a look here for more answers http://www.cisco.com/warp/public/63/...raceroute.html

Early symptoms, from my experience, are that this problem is actually in the network between your devices. Meaning... blame the telco company.

Ahhh.... You've reached the point of no return...

The telco doesn't know how to troubleshoot it so they will pass it back to you as your problem and you don't know what to tell them because you don't know what they don't know... been here... ;)

First... Get on the line with their tech, (not the clerk you report the issue to), and have him describe the circuit to you. Draw it and annotate the drawing with the same names/designations he uses... then you can talk his language... The have him indicate which of the pieces of equipment he has described to you are available test points - this becomes important later when you realize that he doesn't know how to properly troubleshoot. Then have him confirm error free connectivity from the furthest test point to it's opposite end - the smartjack... stress the "error free... Have him run all zero's and then run all ones if his test suite can do it because these are the most stressful on the equipment. I'm blocking on the name of the standard test they do right now but it often sucks... It'll almost certainly be the first one he tries and it is a "battery" of tests... sort of... It really doesn't _test_ the line and the pieces of equipment. Make sure that he is thorough in his troubleshooting technique. They will often go through a pre-rehearsed set of actions with no thought to the symptoms themselves...

That should start you off... Yell if you need more help... In fact, make notes of everything he says and does and post back if it is unresolvbed and I'll see if we can't find the issue from here... Unfortunately, I learned what I know about this through several weeks of different issues... It's very frustrating because they always want to put it back on you.

have you ran any debug commands?
are you using static routes?

try debug ip packet and see what you get

Heres the way its setup at the moment.



Router --------------------- Router
^T1 line

I have static routes setup between the routers for my different networks at each end. I am able to ping from both routers to each other on both the T1 interfaces and LAN interfaces.

I do a ping from each end while logged into the router pinging to the other router. I ping 1000 times, and get about 93% success rate.

when I do a show interface serial0/0/0 I see about 90-120 input errors and the exact same number of CRC erros.

However, I only get CRC erros on one side of the T1, not on the other.

I've swapped out CSU/DSU cards in the router, changed clock settings, but it doesn't work :(


The test equipment the telco used was T-bird equipment. They run 0's 1's and a mix of it...if they set it to Terminate they got all kinds of errors, but they said they're not sure if thats right due to the fac that T-bird isn't meant to terminate.

So I talked with the Supervisor over the Technicians, and they manage to bring in some Frame Relay equipment so they can actually terminate the link.(this is the stuff they don't know anything about). Running certain tests they get abotu 20 CRC errors after 15 minutes. Running another kind of test they get about 25errors a sec on the T1.

But once again they're not really sure if its valid caue they've never used this equipment.


They told me straight up they don't really know what else to do. I showed them my errors on the Routers.

I'm frustrated beyond all belief.

They're talking to Techs at the NOC now trying to see if they know anything about this.


Just in case your wondering what I want to do with this T1, this is it:

We can't get DSL at one of our offices, so we have DSL where we can get it then the PTP T1 from the DSL to the Office.....my plan is to route the Office internet over the T1 to the DSL.

I've talked to cisco and they say its a everyday solution for a lot of companys. The telco thinks its crazy idea and that it would never work.

I would like to add that it does work, I can ping from the officer, over the T1 and out to any websites, its just the connection doesn't last long, pings are good about 60ms, but if I try to open any connections with websites they just timeout....Downloading files from file servers download at <2.2KB/s before they time out.

But I dont' know, I just think they don't know what eht F**$ they're doing and maybe trying to blame it on that? But I told them "look, I can't even get a clean ping across your line to the other side."


So they're still looking into it, but I'm about to go explode I think.


I guess this is more of a rant than anything. I've tried everything I know, and cisco says its telco problem, as we did numerous tests. I can show you router configs if you want.

Quote:

---

Originally posted here by patrande1
try debug ip packet and see what you get

---

Carefull with this one... I've managed to flood myself off routers this way.. If there's lot's of traffic you'll get sh*tloads of data.. enough to kill your remote connection...


Cheyenne: Not really a solution but I'd be looking for a different telco right about now.. One that does have enough technical knowledge..

Ok... Number One... Your solution is perfect... I run all my offices over PTP T1's and route them out to the internet via one of two other T1's. So tell them to hush... They are throwing things like that in to muddy the waters.

The T'Birds are the best test box they have... The newer ones suck because they can only run preset tests... T'birds can run anything they like, (user patterns)... The issue is most likely with a piece of equipment, (repeater or whatever), on the line itself. The trouble is that they don't know what they are doing from what you say... You need to keep on them... Every time they try to close the ticket don't let them... You've changed all your stuff on the routers etc so you know it's not you. Plus, they are testing smartjack to smartjack I hope, looped too... Thus, _none_ of your equipment is involved yet they still see errors... It's their problem... The only trouble is _you_ have to make them fix it. Have them loop up each piece of equipment one at a time and run the tests then go to the other end and do the same. This will tell them the segment of the circuit that is causing the errors and they can have it investigated.

One final thing... When they do fix it they will run a final test... Then the silly bastards will leave a piece of equipment looped... Your Cisco facing the loop with tell you the circuit is looped... I have managed to down the loop using the Cisco itself once but the command to do so eludes me right now - it's under the int serX/X IIRC and it's a remote loopback command. It works sometimes but not all the time but it may save you a few hours waiting for them to fix it themselves.

Good luck... I know how frustrating this can be - It took me 7 weeks... Yes seven... to get one circuit right once because they didn't have T'birds and they didn't know what they were doing. I used the cisco to send a certain pattern and showed them the issue... You'll never guess what one tech said.....

"Maybe that a pattern the circuit isn't supposed to pass"

I blew up... "It's Fsking ONES and ZEROS for christs sake once you get past two of them then the minimum pattern is set"... I threw him out and asked for another tech... :D

Just wanted to mention one thing (TS is right on), just type ping at the prompt and you'll get alot more options, one you should use is to change the size of the packet. Set it >> than the MTU so that it forces fragmentation. Doing this is usually a pretty good way to 'stress' the circuit and point out any lingering issues and give you a clear indication if the circuit really is running clean... (this would explain why your pings might look ok but HTTP bombs since the responses over HTTP are much larger than the typical 56 byte ping packet )...

One thing stood out in a previous message...

The test equipment the telco used was T-bird equipment. They run 0's 1's and a mix of it...if they set it to Terminate they got all kinds of errors, but they said they're not sure if thats right due to the fac that T-bird isn't meant to terminate.

The Tberd can terminate a circuit - seen this many times - the reason it gets higher errors is possibly a timing setting mixup in the provider csu or in the segments along the way - the circuit should/must provide timing to the cpe (router) equipment

One other possibility to investigate are the T1 settings in the smart jacks at both ends - maybe one is set for T1 B8ZS coding an another AMI coding - search the web for the manufactorer and model - most used by US providers are adtran - they have a 9 pin dte connection for terminal access - ps - providers don'tlike you connecting to their equipment.

As far as stress testing a T1 - the tberd should have a setting (depending on model) for a 3 in 24 test (3 ones in 24 bits, ones are used by the circuit to pass and keep clock)

Basically - if you cant get a tberd to run clean on the circuit end to end or thru a loop the circuits bad.

Perhaps if you post your configuration as it will help. In addition if its a true point-to-point T1 and not a frame-relay T1 then the errors are coming from your gear. If its a P-to-P you can insert a loopback plug in the smartjack at one end and see the loop on the other CSU. Are you sure your subnet masks are correct on each end?

I'll get my current configs and post them for you guys.

IT is a true PtP T1...no Frame Relay, its quite a long run actually.

According to the techs, The T1 goes from Copper to Fiber, back onto copper, then onto a T1 span before terminating. Its about a 20 mile run.


On the DSL side of the T1 I have no alarms and the line status is up...however now on the office side I now have alarms, and the line is down. I tried no shutdown, disabling the interface, re-enabling, power cycling the router etc, bu the line is staying down on the office end.

I've checked the timings....theres two options on the cisco routers. Line and Internal. I have it set on line at both ends right. I tried it with internal clock source the other day but no luck...Both sides are running B8zs, and esf.

About 5 days ago, I could ping across the line (still got the CRC errors, and connection timeouts). I went in a few days ago and both sides were down. I had changed nothing with the configs, I came in the other day to troubleshoot it, see if it was anything wrong on my side, and as I said above the DSL side no longer had alarms and the T1 was up on that side, but was still down on the Office side.


My basic configuration is this.

T1 Interface at the DSL location:
ip address 10.20.1.1 255.255.255.252
encapsulation ppp
Clock source is line
b8zs and esf

T1 Interface at office location:
ip address 10.20.1.2 255.255.255.252
encapsulation ppp
clock source is line
b8zs, and esf
(I'll get a complete running-config output for you later)

They've ran tests from their offices to both routers going both ways, and they say they don't see any errors(when they see errors, is when they set the T-birds to terminate at the office T1). They've looped the smart jacks, and what not. when they just set the T-Birds to view packets coming in(bridge mode?) they don't see errors, as I stated in my previous posts, when they set the T-bird to terminate they got about 1500+ errors after a minute run.

Now with the current alarms on the office router they can loop the smart jack but not to the CSU, I've checked everything on that router, made sure I didn't have loopback set, or didn't have the interface administratively down, but its all correct. I made no changes to it, left one night came back the next day and it was down.

I may be having a complete brain fart but I have my encapsulations set to HDLC(?).

The rest is ESF, B8ZS, timing = line.

How many timeslots do you have... For a _full_ T1 both ends should be set to 24. For a _normal_ fractional T1, (386), both ends need to be set to 6.

Do you have independent CSU/DSU's or are they WIC cards in the router? I'm guessing WIC because of the way you have written... I have a few CSDU/DSU's but hate playing with them.. Mine are Verilink so I can look stuff up on them if you need me too...

Right now I'm leaning towards this being your problem until we have made sure everything is good...

[EDIT]

Oh... I forgot... The configuration of the circuit you describe is a problem.... They work fine but when you have to call the provider they cause no end of trouble... See, you're running _two_ T1's... One on either side of the fibre... But, most likely, you have only been given the circuit ID of _one_ of the two T1's. What muddies the waters even more is that the tech often doesn't know of the other T1... he assumes it just runs off to fibre and someone else deals with the fibre... So he works on the last test point before the fibre when the error is on the other side of the fibre... This is where things can all get screwed up...

[/EDIT]

I did have encapsulation set to HDLC, but still got errors, and had a couple people tell me to change it to PPP, so thats what its at right now.

Even with it having HDLC still got errors across the T1.

I have WIC CSU/DSU's that fit in the router.

I'll check the timeslots, pretty sure they're both set to 24....any idea on what command it is to check the number of timeslots available?

I believe its "show service-module" or perhaps "show controller"

Just to let you all know, this issue was resolved.

This T1 is about 18 miles long and they had some Cable pair problems on the repeaters.

So I can get out to the internet now across the T1, works great...however heres another problem I've run into now some.

Alot of sites load, (google, yahoo, newegg etc) however some sites hang.

For example going to www.hotmail.com it says finding site...site found, then it redirects to a login page, but hangs there, like I'll never see the login page, just the Header up top.

MSN does this somewhat as well...other sites that do this is mapquest.

The whole site just hangs, and won't load on the computer.

Any ideas as to what that could be?

Does it just happen on HTTPS sites or others too? (Not sure if map quest has a secure section or not)

non https sites don't load sometimes as well.

mapquest.com wouldn't load.
A few other sites as well wouldn't load.

Leaning toward maybe a MTU problem maybe?

Could be! Did you alter it when you was trying to solve your earlier problem?

I've had problems similar in the past, One had me confused for days! It turned out to be a few 'shite' ACL's.
Obviously http traffic can be spread over different ports to enable faster load times - turned out the person before me had blocked a few of these ports accidently - which resulted in a lot of time out errors with some sites.

Is it effecting everyone in the office or just a few W/S's?

I've resolved the issue.

It was a MTU issue on the ethernet ports of the routers.


Setting

ip tcp adjust-mss 1360 on the routers allows all websites to load now.


Thanks your help.

Why wouldn't you just put it back to the default, 1472(?)?

I have had this issue with routers....lowering the MTU helps with having to constantly reset the ISP modems as well.

MLF

I'll probably set it to 1492 and play with it...I just set it at 1360 now just to get it to work....I'll probably fine tune it later more.



*edit*
I don't think I answered your question correctly tigershark.

I belive you asked why did I just not leave the MTU at the default setting on the routers?


If thats what your asking, a MTU of 1492 was set up atm dialer that goes to the DSL.

That setting was changed by the telco company when they were tweaking the T1 line and DSL settings.

I belive that setting and the fact that all traffic is going over 2 switches and 2-3 routers depending on location screwed up that part of network...setting the MTU lower solved that problem for me.