how to use two gateways for different two NIC.

Latest response

Hi,

we are having one server and it is having two NIC card. we are assigned two different sub-net IP's on both the NIC. on first IP is 192.168.0.0/27 and on second NIC IP is 192.168.0.32/27.

for first card gateway is 192.168.0.30 and for second card gateway is 192.168.0.62 . both NIC connected to our core switch and Gateway as provided above on two different ports. now we are running ospf on our switch and network.

we are able to see the route is populated in our network however we are unable to ping 192.168.0.32/27 network when we are able to ping 192.168.0.0/27 network.

when we are checking on server route -n output. We are getting entry of both network and default gateway is 192.168.0.30 only.

what our understanding that one more gateway need to made or some static route entry need to be made.

Os is Red hat enterprise Linux 6.4

please suggest. thanks in advance...

Responses

There a few different ways to address this.
Here is the support doc for RHEL 5
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Deployment_Guide/s1-networkscripts-static-routes.html

A method that I use:

# egrep 'GATEWAY|DEFROUTE' ifcfg-bond0
GATEWAY=192.168.0.1
DEFROUTE=yes

# cat route-bond1
ADDRESS0=10.98.92.36
NETMASK0=0.0.0.255
GATEWAY0=10.10.0.1
ADDRESS1=10.98.81.64
NETMASK1=0.0.0.255
GATEWAY1=10.10.0.1
ADDRESS2=10.98.60.87
NETMASK2=0.0.0.255
GATEWAY2=10.10.0.1

Hi James,

configuration mentioned above is for Bond. however in our environment we are not using bond. It's simple ETH2 and ETh3. any suggestion!!

Thank you

The configuration is the same for bond and eth. Unfortunately, I believe you will need to implement your network routes as Jaime has suggested (below). I am not aware of a more simple method to make traffic stay on the intended interface (I, too, have had a similar issue with multi-homed boxes - but, my non-public interface was purposefully on a non-routable segment)

In general, if you're looking for behavior akin to Solaris's ability to support per-interface default routes, then you need to use iproute2's policy-based routing extensions. Absent these policy-routing rules, packets that come in on the "wrong" interface get treated as martian and dropped.

Back in late 2010, when the customer I worked for was migrating their automated provisioning hosts from multi-homed Solaris-based systems to multi-homed EL5-based systems, I had to figure this out for their build-automation team. It was an exercise in unlearning the Solaris Way and taking on the Linux Way.

Any way, if you're looking at "multiple default gateway" style routing, you'll want to look up the various tutorials on 'iproute2' and 'policy-based routing'. I'd point you at what I wrote up, but, it's got a heavy dose of Solaris prejudice in it.

Please do, i can 'survive' a bit of Solaris superiority charms :D

I dislike policy routing on Linux and strongly encourage you to think of a better way to do this. Ideally, work out what you want to be accessed via each interface and create routes like James has suggested above.

If you really do need policy routing, it's documented here:

How can I route network traffic such that the packets go out via the same interface they came in?
https://access.redhat.com/knowledge/solutions/19596

Note that you now have to add a routing table entry to every ip route command you run, as all your routes will be in tables other than the main routing table. This is somewhat non-obvious and will make your system more difficult to troubleshoot, especially if someone comes along who doesn't know about policy routing and doesn't use the iproute2 tools.

You won't be able to use the "legacy" network tools like ifconfig and route anymore.

Hi

Thanks for your reply. However we also not want to go in complicated mode. We have two network one for service and second one for O&M. we just want to achieve that both network gateway will be separated. both network will be communicate through own NIC only. We had done james method however no luck..

Opinions vary. I detest setting up a ton of static routes unless there's a clear need to force traffic along a given path. If my network is complex enough that I want lots of explicit routing, I should probably be running a routing service to handle it - preferably by communicating with upstream routers and letting those routers tell my system "send those packets to me".

Policy-routing is only as difficult as you decide to make it. If you're only looking to emulate Solaris-style multiple defaultroute type of functionality - that is, you're simply looking to ensure path-symmetry (return packets the path they came in on) - the setup is pretty straight forward. There's only a couple additional configuration files:
* an (optional) /etc/hosts style policy-naming file, /etc/iproute2/rt_tables
* an /etc/sysconfig/network-scripts/route- file (or /etc/sysconfig/network-scripts/route-. if I'm doing tagged interfaces)
* an /etc/sysconfig/network-scripts/rule- file (or /etc/sysconfig/network-scripts/rule-.) if I'm doing tagged interfaces)

The route- files need two lines (below is probably pedantic/overkill at any rate):
table ${TABLENAME} to ${NETBASE}/${CIDR} dev ${DEV}
table ${TABLENAME} to default via ${GATEWAY} dev ${DEV}

The rule- file only needs one line (again, being pedantic):
from ${NETBASE}/${CIDR} table ${TABLENAME} priority ${NUM}

Whether one is better than the other probably depends on what you're looking to handle and how many routes you're looking to manage. If you've only got a couple static routes to declare, then use static routes. If you've got several (or there's no guarantee that remote systems' packets will come in on a predictable interface - the problem that I had to deal with when opting for policy-routing), use policy-routes. If you've got a high number of routes to enumerate, make use of a routing-daemon to pull routes from upstream routers.

Hello

I wonder if we have a slight misunderstanding of the problem here. To help us understand please check this:

1] we are having one server and it is having two NIC card.
2] we are assigned two different sub-net IP's on both the NIC. ?
2.1] Do you mean one NIC has one subnet and the other NIC has a different subnet? {I guess this is the case}
2.2] or both NICs have two subnets and those two subnets are the same?

3] on first IP is 192.168.0.0/27 and on second NIC IP is 192.168.0.32/27. {you mean subnet on 1st card is 192.168.0.0/27 and the 2nd card has 192.168.0.32/27 subnet}

4] for first card gateway is 192.168.0.30 and for second card gateway is 192.168.0.62 {that makes me think my guess for 2.1 above is correct}

Is this what you have for 1st Nic ?
The 192.168.0.0/27 subnet
IPADDR=192.168.0.1
NETMASK=255.255.255.224
GATEWAY=192.168.0.30

Is this what you have for 2nd Nic ?
The 192.168.0.32/27 subnet
IPADDR=192.168.0.33
NETMASK=255.255.255.224
GATEWAY=192.168.0.62

EDIT: I had the format incorrect

RHEL stores route information in per-interface files (not globally)
Do you have any route /etc/sysconfig/network-scripts/route-interface files?

What is the output of ip route?

Updated Static Route section in Beta RHEL6 Deployment Guide is here:

Thank you

I should have also mentioned it is not good to have GATEWAY in both interface configs as I posted above. As explained in the guide, the last one to be read gets used. See "Configuring The Default Gateway"

Though, if you set "GATEWAY" or "GATEWAYDEV" in your /etc/sysconfig/network file, you don't have to worry about "last declare wins" situations.

Agreed. I wonder if this feeling is stronger in those who learnt networking before, or separate to, learning Linux? If you see the text at the link posted you will notice this "Specifying the gateway globally has certain advantages in static networking environments, especially if more than one network interface is present". That was my attempt at being diplomatic.

Hi Stephen,

Yes the configuration is same as per point 2.1. below is the ip Route information

192.168.0.32/27 dev eth3 proto scope link src 192.168.0.38
192.168.0.0/27 via 192.168.0.30 dev eth2
192.168.0.0/27 dev eth2 proto kernal scope link src 192.168.0.6
169.254.0.0/16 dev eth2 scope link matric 1002
169.254.0.0/16 dev eth3 scope link matrics 1003

Destination gateway genmask flgs matric ref use iface
192.168.0.32 * 255.255.255.224 U 0 0 0 eth3

192.168.0.0 192.168.0.30 255.255.255.224 UG 0 0 0 eth2

192.168.0.0 * 255.255.255.224 U 0 0 0 eth2
Link-local * 255.255.0.0 U 1002 0 0 eth2
Link-local * 255.255.0.0 U 1003 0 0 eth3

we have /etc/sysconfig/network-scripts/route-eth2 and /etc/sysconfig/network-scripts/route-eth3 file.

Thanks

Hello

The reason the others are talking about complex things is they have assumed you want packets to be able to come in on one interface {EDIT: My mistake:} and go out the same interface.

But you did not say that is what you want to do, so I read your statement as just wanting to be able to ping from one system to hosts in two different subnets. Am I wrong?

That should be simple without any static routes. You would only need static routes if you wanted to ping some subnet not connected to the system.

Keep in mind, the ECHO REPLY needs to find its way back, can you ping the system described from the other side? Do their routing tables show a route back to the system described?

Sorry, the article Jamie linked to is for when you want traffic to go back out the same interface it came in on, I was thinking of outgoing routes and incoming routes being different. That is controlled by rp_filter, see: https://access.redhat.com/solutions/53031

the rhel 6 docummentation for the route-interface method previously mentioned.

I've used this method a number of times.

Steven, thanks for posting that. It does seem better (anyone wanting to see the difference, go to paragraph 9.5, and 9.5.1 and 9.5.2 of that link Stephen Wadeley posted above)

RHEL 6.6 beta example
Quoted material from link -> Thanks to Stephen Wadeley:

 The following is an example of a 
route-interface file using the 
network/netmask directives format.
 The default gateway is 192.168.0.1 
but a leased line or WAN connection 
is available at 192.168.0.10. The two 
static routes are for 
reaching the 10.10.10.0/24 and 
172.16.1.0/24 
networks:

ADDRESS0=10.10.10.0
NETMASK0=255.255.255.0
GATEWAY0=192.168.0.10
ADDRESS1=172.16.1.10
NETMASK1=255.255.255.0
GATEWAY1=192.168.0.10

Guys,

Thank you so much for all your support.

Problem get resolved and it is working okay by one default gateway.

We only need to add a static route towards the outside network on NIC from which our preferred sub-net need to communicate and entries are persistent.