crash course...

Ok, I know everyone here has prolly had something like this happen to them. It's Murphy's law... It seems that everytime my boss desides to take a trip, I get presented with a situation I have no idea how to handle. But I somehow work through it pretty well... I'm feeling pretty good about what I did and how I handled it... :)

Sorry if I bore you... I had to tell someone and none of my friends care nor know what the hell a router is...
They know beer pong...

On monday I recieved a call saying that one of my branches was down. (I work in a bank) I'm like... ok, the frame is prolly down. I'll call telco. and have the ciruit checked.

I call them and they confirm that indeed the frame is down. So, I tell them to work offline and go about my usual morning routines... changing backup tapes, checking logs, returning phone calls, usual stuff...

2 hours later it is still not up, so I call telco back. They say it is now up.
It wasn't. So I drove out to the site and see no rx/tx LEDs flashing on the router. The LAN was working just fine. I'm convinced that its the telco., as was everyone else in my department.

My boss already left for a business trip in FL. He is the only one who knows routers where I work. We are a small team. Only 3 people. One is MIS, My boss is IT and I'm a little of both. I am an "Informations System Assistant" still in training. I'm going to school for IT and MIS. (Networking, programming, and tyring to learn security on my own.)

Well, I get the telco out there to make sure that it was indeed me and not them. It was me... The lines were just fine. I couldn't figure out what the hell was wrong. The router logs had alarms all over them. I called Motorola and they say that the line was most likely bouncing.
I knew it wasn't because the telco just left.

I hooked up a modem to the router and had support dial in.
Their response is... "Whoa, what the hell is that? Your T1 port should not say its an ISDN interface... I've never seen that before." They get a couple of other techs to take a look. Noboby knew what was wrong. They tell me it is either hardware or the configuration is corrupt.

My theory... there was a power failure that caused the frame to go down then the electric in the building went out. When the frame goes down, it kicks in to ISDN. During the switch, the power was cut and it corrupted the config.

Right now I'm taking a cisco class and I have the concepts down, but I have no idea how to troubleshoot a router let alone configure a router... but I learned really fast. Well, in about 16 hours with a lot of coffee!

To make a long story short... if that is possible....
I hurried to my office and grabbed any docs I could find, a laptop, coffee, and other testing equipment.

I grabbed a different router for a new site we are putting in. Printed every single configuration. Then printed the config for the router that doesn't work. It all looks the same though. Line by line except for the IPs and a couple of other settings that shouldn't have made a difference.

I was stressing... worried that I wasn't going to be able to fix it...

So I download the scripts for both the routers and back them up. I import the script from the router that isn't working to the one that is working. Now I have 2 routers that don't work... I switch it back. Then I import the settings for the router that does work.

Ok, I'm back online but can't get out of the LAN. Can't even see the router because the ip settings were wrong (I was connected directly to it via serial). So, with about 5 or 6 more tries I finally get them back up about 15min before they open for business again the next day.

I prolly looked like I was robbing the bank because I was the only one there running around all night with a flashlight, pc equipment and bags that I carried everything in there with. All the computer equipment is near the safe... so I can only imagine how it looks from the outside.

I ended up working for almost 36hrs with only a few food/smoke breaks... I was so proud of myself that I figured it all out and was pumped. I feel like I can tackle any thing thrown my way now. I just have to bust my ass.

Last time he went on vacation... the exchange server crashed and I had to rebuild it and restore all the data that was lost. It feels so good when you can go in not knowing anything about it and sucessfully getting it all back to working order.

I can't wait to find out what is going to happen next time he is away... I'm going to start prepping now. I can always use the overtime... :)

What a way to learn:)
then again things we learn the hard way tend to stay with us longer
Glad you got it up and running
Regards
Mike

Hehehehe
Nice story man
Good Job
:)

Heh, nice job.

On a serious side-note: did you inform the local police department / security officer that you were working that late? If not: I'm kinda surprised no-one reported you. Sounds like a serious security thread to me.

Congrats, on fixing the prob. This is the best way to learn, against the clock and all by yourself
Last time I worked overnight on a bank LAN / WAN we had to tell the local cops just so they didnt respond to all the "I think the Banks being done over " calls from all the concerned citizens :(

No, I didn't even think of calling the police to let them know. It is in a not so nice area too... banks are always getting robbed around there. So far we haven't been hit, but I still look over my back when I'm in the parking lot.

I told everyone in the bank who should have known I was there and they didn't even think of calling them.

About a year ago, we were going through a conversion. I didn't have the alarm codes for one of the branches and set off the alarm. The security co. called and asked the the usual secret phrase and I was supposed to respond with the pass phrase.

They were like, "This is the Comptrollers Office, is your cash in balance?" I was like... "Huh? What?", them, "Sir, is everything alright?" I'm like "No everything isn't alright I can't get the alarm to go off, I can't get this damn PC to log in the network and I need more coffee!"

The cops showed up very shortly after. I wasn't dressed in my usual suit with business cards and etc. I was just in shorts and a polo shirt. They didn't even question me. They just said be more careful. I got a call with the correct codes to shut it off and the cop sat in the parking lot for a while after that.

The same night at a different branch I had a similar experience. I had a big white van that I was moving new equiment in and old equipment out and a cop was sitting right up the parking lot watching us, but never said a word to us. That was strange too. I guess they wait for the alarms.

If I saw two guys with a white van moving computers out of a bank with duffel bags *for the cords and etc.) I would have def. said something to them.

I'll make sure to keep that in mind next time.