Intresting Network Issue

[bAgZ]

Hi everyone i seem to have this pretty weird problem and i am not too sure why this
would happen. Here goes, the current network is dived in two parts with trusted
LAN and DMZ zone, then access to Internet through two different gateways or
routers. Now everything runs just fine until such time when one of the gateways dies,
then i start experiencing the following and this is only happening on machines in a
DMZ and main firewall separating two networks.

All machines in the DMZ are Linux or BSD and every time the default gateway is not
functioning i i seem to loose the ability to access any services on the machines using
non-functioning gateway. In other words, just to explain this further, if the server
with IP lets say 10.0.0.25 uses gateway 10.0.0.1(and this gateway then stops
functioning) and i access this internally from range 192.168.1.0 through the firewall
i have a problem ssh-ing or using pop3 or any other service even though the machine
responds perfectly fine to pings, obviously because i don't have to access Internet to
access my servers in the DMZ zone. There is no error it's just as if all the traffic
disappears somewhere. So i am wondering if anyone has experienced this.

Here is a little diagram of what the setup is. Look if am not too clear on this let me
know and i will give more information. Found nothing of this sort of thing on google.
I am looking into fail-over setup where machines can pick gateways automatically
but thats another topic all together. Thanks in advance.

LAN -----Firewall------DMZ---- twogateways-----internet

[SirDice]

Add a static route to your 192.168.1.0 network (on your machines in the DMZ)..

As for fail over routing.. That depends on the type of routers used as gateways..
Usually you define a virtual IP address.. Both gateways listen on that virtual address..
The default gateway will point to that virtual IP address..

And to see what happens fire up tcpdump on one of the machines in the DMZ.... I'll bet you see lots of ICMP redirects.. The static route should help prevent those too..

[bAgZ]

Thanks for that i will try today. As for the fail-over i was just wondering if i was to write
a little script to find out if the gateways is live or not and than just change default
gateway to it something quick like that?
The other option i am considering is downloading all the kernel patches for detecting
a dead gateway. I know Windows supports this but you need to patch 2.4 and 2.6
kernels for Linux as for the BSD i will have to look it up.
Also for load balance and fail over i looked up some Cisco protocols
like BGP what do you think of that as an option. Thanks a lot for the feedback.

[SirDice]

As for the fail-over i was just wondering if i was to write a little script to find out if the gateways is live or not and than just change default gateway to it something quick like that?

Simple, yet effective solution..

Also for load balance and fail over i looked up some Cisco protocols like BGP what do you think of that as an option.

I believe BGP would be overkill for your situation..

What's the type of gateways you use?

[bAgZ]

Yeah i also think BGP is an overkill actually i don't think it would work in
this scenario.
I have two routers one is a Cisco 2600 where the leased line is connected and
the other is Cisco 805 for an ADSL. They are both small lines compareing to
anything similar in Europe or US. Being stuck in Africa with Telkom is a disaster.
But thats a basic setup nothing massive. I just want to make a simple switch in
case one of the gateways goes dead.

[SirDice]

I just want to make a simple switch in case one of the gateways goes dead.

Have a look at HSRP. Both the 805 and the 2600 should support that..

Hot Standby Router Protocol (HSRP): Frequently Asked Questions

[bAgZ]

Well while we are on the issue of fail-over, i am building a freebsd failover setup for my firewall.
so far this is what i have in mind. Use freevrrpd (http://www.us.bsdshell.net).

Your thoughts on this sort of setup? Have you ever tried anything like this?
My aim is really that everything happens automatically without human intervenion
in case i am not around.

[SirDice]

Use freevrrpd (http://www.us.bsdshell.net).

http://www.b0l.org/?idcategory=3&idsection=1
http://www.freshports.org/net/freevrrpd/

My aim is really that everything happens automatically without human intervenion in case i am not around.

HSRP and VRRP are both designed to work "automagicly" (if setup correctly)..

Never tried freeVRRPd though.. Looks like it should cooperate with Cisco's VRRP too..

[bAgZ]

Sorry one more question about those static routes and loosing a gateway.
I added static routes but then when i telnet to my pop3 which is in a DMZ and
has no active gatway i see its connecting, then it says connected but you cannot
send any commands to the machine itself.

When i run ethereal i see that arp runs broadcast who has that IP example 10.0.1.34.
It seems that arp gets messed up. Then when i flush the ARP table on a machine
in a DMZ i can connect to it without any problems even if the gateway is down. Why would ARP get messed up i thought that NAT breaks but NAT runs fine.

[SirDice]

Look at your routing tables on the DMZ machines.. netstat -rn Look for routes with the D flag.. These are added because of ICMP redirects and are probably the cause of your problems..
And ARP: arp -an

How's your firewall setup? Using NAT? Is it bridged? Statefull filters?

Can you post a tcpdump output from one of your DMZ machines? A plain tcpdump -i eth0 -n -e will do.. You can obfuscate the IPs if you want.. Use some thing like a.a.a.1 for 192.168.0.1; a.a.a.2 for 192.168.0.2; b.b.b.1 for 10.0.0.1 etc.. So we know a.a.a.1 is on the same subnet as a.a.a.2 and b.b.b.1 is not.. The actual IPs are irrelevant but we have to know how the subnets are setup...

For the sake of simplicity I'll use one gateway:

[LAN; 192.168.0.x/24]---------{1}[firewall]{1}----[DMZ; 10.0.0.x/24]-------{2}[gateway]

MachineA= 192.168.0.200
MachineDMZ= 10.0.0.100
Firewall LAN interface= 192.168.0.1
Firewall DMZ interface= 10.0.0.1
Gateway DMZ interface= 10.0.0.2


On MachineDMZ:
route add default 10.0.0.2

On MachineA:
route add default 192.168.0.1


What happens when MachineA connects to MachineDMZ?

MachineA: Arp who has 192.168.0.1? Tell 192.168.0.200
MachineA: send SYN to MachineDMZ
MachineDMZ: recieves SYN
MachineDMZ: Arp who has 10.0.0.2? Tell 10.0.0.100
MachineDMZ: Send SYN/ACK to MachineA
Here's where things go wrong..
Instead of sending the SYN/ACK back to the firewall it'll send it to the gateway. MachineDMZ doesn't know where 192.168.0.x/24 is so it sends it to it's default gateway..

The gateway will receive the SYN/ACK but doesn't know where to send it, so in turn it'll send it to it's default gateway. This is probably your ISP.. Hence no connection...

If the gateway knows the 192.168.0.0 network it will send an ICMP redirect to MachineDMZ telling it it can find the 192.168.0.0 network via the firewall..
MachineDMZ will then add a dynamic route (the ones with the D flag in the netstat -rn output).

By adding a static route on MachineDMZ route add -net 192.168.0.0/24 10.0.0.1 you prevent this whole dynamic thing making sure MachineDMZ knows the way back to 192.168.0.0/24 with or without a default gateway..

Hope this clears things up a bit... If not.. just ask