How to connect two network interfaces on the same subnet?

Solution Unverified - Updated 2020-08-14T06:02:05+00:00 -

Environment

Red Hat Enterprise Linux 5
Red Hat Enterprise Linux 6
Red Hat Enterprise Linux 7
Multiple network interfaces, each with an IP address in the same subnet

Issue

How to connect two network interfaces on the same subnet?

In our environment, there are three bonding devices connected with the same segment.

+---------------------+
|       Linux         |
|   .168      .169    | 
+-----+--------+------+
      │        │
+-----+--------+------+
|       Switch        |
+---------+-----------+
          │
+---------+-----------+
|         .1          |
|       Gateway       |
+---------------------+

We have captured packet and found that packet should be transmitted from bond0 was actually transmitted from bond1
Also, it was confirmed that the transmit port staggered even if there was no bonding setting.

Resolution

Is is usually not a good idea to connect two interfaces using the same subnet on the system, as you may run into the issues presented in the following two pages:

The recommended solution is to use a single interface or a single bond/team interface, then add the other IP addresses to that interface in the form of an alias IP address. If fault tolerance and/or load balancing is required at the network interface level, then an alias on a bond or team should be used. Please note that teaming is not supported in Red Hat Enterprise Linux 6 or earlier. If fault tolerance and/or load balancing is not needed at the interface level, then you can add the alias IP address to a single interface.

The following page provides steps on how to add an alias IP address: How to assign alias IP addresses to a network card (NIC)?.

The following page provides steps on how to set up a bond interface: How do I configure a bonding device on Red Hat Enterprise Linux (RHEL)?

The following pages provide documentation on teaming:

Red Hat Enterprise Linux 7: Red Hat Enterprise Linux 7 Networking Guide Chapter 8. Configure Network Teaming
Red Hat Enterprise Linux 8: Red Hat Enterprise Linux 8 Configuring and managing networking Chapter 17. Configuring network teaming

If your environment necessitates a multi-interface same-subnet solution and the above alias interface approach does not work for your environment, you may perform the steps in the Resolution section of the following two pages to avoid the most common issues when setting up multiple interfaces on the same subnet:

Root Cause

When there are 2 interfaces on the same subnet there is no assurance as to which interface will be used to transmit traffic and the machine will accept traffic for either IP on either interface.
This is because in Linux the IP address belongs to the host and is not associated with the interface.
If you ping with -I DEV, attempting to use a given interface, there is no guarantee the reply packet (if there even is one) will come back to the same interface, so pings done with -I DEV may not work.

Diagnostic Steps

Setup system with 2 interfaces on the same subnet.
Ping a target and capture packets with tcpdump.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

13 Comments

ko Newbie 15 points

21 July 2011 6:53 PM

kontron_crms

Is there a typo in step 2) ??

Should these lines be identical?

# ip route add 10.64.208.0/24 dev eth0 src 10.64.208.180 table t1



    # ip route add 10.64.208.0/24 dev eth0 src 10.64.208.208 table t1

or shouldn't the second 'ip route add' look like this...

# ip route add 10.64.208.0/24 dev eth1 src 10.64.208.208 table t2

2. Add routing table to t1, t2 :

# ip route add 10.64.208.0/24 dev eth0 src 10.64.208.180 table t1
# ip route add table t1 default via 10.64.208.254 dev eth0
# ip route show table t1
10.64.208.0 dev eth0  scope link  src 10.64.208.180
default via 10.64.208.254 dev eth0



    # ip route add 10.64.208.0/24 dev eth0 src 10.64.208.208 table t1
    # ip route add table t2 default via 10.64.208.254 dev eth1
    # ip route show table t2
    10.64.208.0 dev eth1  scope link  src 10.64.208.208
    default via 10.64.208.254 dev eth0

VA Community Member 86 points

10 May 2018 2:58 PM

Vasily Averin

this article still have wrong device name for default routing in t2 table

# ip route add 10.64.208.0/24 dev eth1 src 10.64.208.208 table t2
# ip route add table t2 default via 10.64.208.254 dev eth1
# ip route show table t2
10.64.208.0 dev eth1  scope link  src 10.64.208.208
default via 10.64.208.254 dev eth0     <<<<<< dev eth1 should be here

ko Newbie 15 points

21 July 2011 8:39 PM

kontron_crms

There is also anotehr typo...

As for 4):

# grep arp_filter /etc/sysconfig 
net.ipv4.conf.all.arp_filter = 1

should say

# grep arp_filter /etc/sysctl.conf 
net.ipv4.conf.all.arp_filter = 1

rh Red Hat Guru 1692 points

25 July 2011 5:11 PM

rhn-support-rudiyar

Thanks for your comments. I've updated the article with the corrections.

ko Newbie 15 points

2 August 2011 9:09 PM

kontron_crms

This script automatically implements the changes and creation of static routes described above, in case anyone wants to use it.

All you need to provide is the subnet that you want to effect.

#!/bin/sh

# fixFlux.sh - Assigns fixed routes for each NIC

# residing on the same subnet to resolve

# the ARP FlUX problem

# See https://access.redhat.com/kb/docs/DOC-59096

# Date: 7/21/2011

SCRIPT_REVISION=1.0.1

SCRIPT_LOG=/dev/nul

# Author: Ken Banks

# Company: Kontron America.

# CHANGE LOG:

# 7/21/2011 1.0.1 Original code.

# 8/23/2011 1.0.2 Fix a bug with 'undo' to re-enble NetworkManager

# and another bug that allowed multiple 'no's to be

# added to NM_CONTROLLED= if the script was run more than once.

SYSCTL=/etc/sysctl.conf

RT_TABLES=/etc/iproute2/rt_tables

function usage() {

echo "usage: $0 subnet <undo>"

echo " version $SCRIPT_REVISION"

echo " : This script will solve the Arp Flux for multiple NICs"

echo " : residing on the same subnet, as described in this document:"

echo " : https://access.redhat.com/kb/docs/DOC-59096"

echo

echo " e.g. $0 192.168.1 - Configures a static route for ever IC on 192.168.1.0"

echo " $0 192.168.1 undo - Reverts the changes caused above and re-instates NetworkManager"

}

function log_message() {

echo -e "$*"

echo -e "$*" >> $SCRIPT_LOG

}

function compareValues() {

awk "BEGIN { if ( $1 $2 $3 ) print \"TRUE\" }"

}

function addRoute() {

NICNAME=$1

TABLENUM=$2

TABLENAME=$3

SUBNET=$4

DEFAULTGW=$5

NICIP=$6

FirstOctect=`echo $SUBNET | awk -F. '{ print $1 }'`

ip route show | grep $6 > /dev/nul

if [ "$?" != "0" ] ; then

echo ip route add $SUBNET dev $NICNAME src $NICIP table $TABLENAME

ip route add $SUBNET dev $NICNAME src $NICIP table $TABLENAME

echo ip route add table $TABLENAME default via $DEFAULTGW dev $NICNAME

ip route add table $TABLENAME default via $DEFAULTGW dev $NICNAME

#echo ip route show table $TABLENAME

#ip route show table $TABLENAME

ETHROUTE=/etc/sysconfig/network-scripts/route-$NICNAME

echo $FirstOctect.0.0.0/8 dev $NICNAME src $NICIP table $TABLENAME > $ETHROUTE

echo default via $DEFAULTGW dev $NICNAME table $TABLENAME >> $ETHROUTE

echo "Modified: $ETHROUTE"

ETHRULE=/etc/sysconfig/network-scripts/rule-$NICNAME

echo table $TABLENAME from $NICIP > $ETHRULE

echo "Modified: $ETHRULE"

}

function disableNetworkManager() {

service NetworkManager status | grep stopped >/dev/nul

if [ "$?" != "0" ] ; then

service NetworkManager stop > /dev/nul

echo "NetworkManager service is stopped."

chkconfig NetworkManager --list | grep :on >/dev/nul

if [ "$?" == "0" ] ; then

chkconfig NetworkManager off > /dev/nul

echo "NetworkManager service is disabled."

ETHSCRIPT=/etc/sysconfig/network-scripts/ifcfg-$1

TMPSCRIPT=/tmp/tmpcfg-$1

grep -i NM_CONTROLLED=[\"yes\"] $ETHSCRIPT | awk -F. '{ print $1 }' > /dev/nul

if [ "$?" == "0" ] ; then

cp $ETHSCRIPT $TMPSCRIPT

cat $TMPSCRIPT | sed 's/NM_CONTROLLED=["yes"].*/NM_CONTROLLED=no/' > $ETHSCRIPT

echo "Modified: $ETHSCRIPT"

}

function processNIC() {

# inet 192.168.6.99/24 brd 192.168.6.255 scope global eth0show table

# inet 192.168.6.60/24 brd 192.168.6.255 scope global eth1

export NICNAME=$1

export TABLENUM=$2

export TABLENAME=$3

export SUBNET=$4

export DEFAULTGW=`ip route | grep default | awk '{ print $3 }'`

export NICDATA=`ifconfig -a $NICNAME | grep Bcast`

NICIP=`echo $NICDATA | awk '{ print $2 }' | awk -F: '{ print $2 }'`

if [ "$NICIP" == "" ] ; then

echo "ERROR: No IP for this NIC: $NICNAME"

exit

echo "====PROCESS NIC==== $NICDATA"

echo "==== $NICNAME ===="

grep $TABLENAME $RT_TABLES > /dev/nul

if [ "$?" != "0" ] ; then

echo "Creating route table: $TABLENUM $TABLENAME"

echo "$TABLENUM $TABLENAME" >> $RT_TABLES

else

echo "Route table already exists: $TABLENUM $TABLENAME"

addRoute $NICNAME $TABLENUM $TABLENAME $SUBNET $DEFAULTGW $NICIP

disableNetworkManager $NICNAME

echo

}

function delete_route() {

ip route delete $1 dev $2

}

function undo_routes() {

SUBNET=$1

echo 0 > /proc/sys/net/ipv4/conf/default/arp_filter

echo 0 > /proc/sys/net/ipv4/conf/all/arp_filter

cp $RT_TABLES /tmp/rt_tables

grep -v "^.00 t[0-9]*" /tmp/rt_tables > $RT_TABLES

rm -f /etc/sysconfig/network-scripts/r[ou][ul][te]*

for i in `ip route show | grep src | awk '{ print $1"-"$3 }'` ; do

delete_route `echo $i | sed 's/-/ /'`

done

grep net.ipv4.conf.all.arp_filter $SYSCTL > /dev/nul

if [ "$?" == "0" ] ; then

grep "arp_filter[ =]*0" $SYSCTL > /dev/nul

if [ "$?" != "0" ] ; then

cp $SYSCTL /tmp/tmpctl.conf

cat /tmp/tmpctl.conf | sed 's/arp_filter[ =]*./arp_filter = 0/' >> $SYSCTL

echo "Modified: $SYSCTL"

for device in `ip addr | grep $SUBNET | awk '{ print $7 }'` ; do

ETHSCRIPT=/etc/sysconfig/network-scripts/ifcfg-$device

TMPSCRIPT=/tmp/tmpcfg-$device

grep -i NM_CONTROLLED=[\"no\"] $ETHSCRIPT | awk -F. '{ print $1 }' > /dev/nul

if [ "$?" == "0" ] ; then

cp $ETHSCRIPT $TMPSCRIPT

cat $TMPSCRIPT | sed 's/NM_CONTROLLED=[\"no\"].*/NM_CONTROLLED=yes/' > $ETHSCRIPT

echo "Modified: $ETHSCRIPT"

done

chkconfig NetworkManager on

service NetworkManager start

}

function main() {

[ "$1" == "" ] && usage && exit

SUBNET=$1

[ "$2" == "undo" ] && undo_routes $SUBNET && exit

echo "# $0 $*"

log_message "=========================================================================="

log_message "=============================BEGIN ROUTE CREATION========================="

log_message "=========================================================================="

table=1

for device in `ip addr | grep $SUBNET | awk '{ print $7 }'` ; do

processNIC $device $((100 * $table)) t$table $SUBNET

table=$((1 + $table))

done

echo 1 > /proc/sys/net/ipv4/conf/default/arp_filter

echo "set /proc/sys/net/ipv4/conf/default/arp_filter = 1"

echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter

echo "set /proc/sys/net/ipv4/conf/all/arp_filter = 1"

grep net.ipv4.conf.all.arp_filter $SYSCTL > /dev/nul

if [ "$?" != "0" ] ; then

echo "net.ipv4.conf.all.arp_filter = 1" >> $SYSCTL

echo "Modified: $SYSCTL"

else

grep "arp_filter[ =]*1" $SYSCTL > /dev/nul

if [ "$?" != "0" ] ; then

cp $SYSCTL /tmp/tmpctl.conf

cat /tmp/tmpctl.conf | sed 's/arp_filter[ =]*./arp_filter = 1/' >> $SYSCTL

echo "Modified: $SYSCTL"

service network restart

log_message "=========================================================================="

log_message "==========================ROUTES HAVE BEEN CREATED========================"

log_message "=========================================================================="

echo "Contents of file: $RT_TABLES"

cat $RT_TABLES

echo "Command output from: ip route show"

ip route show

}

main $*

�� Newbie 15 points

19 May 2018 2:39 AM

星亚徐

the prefix was 24 at the beginning, but became 8 in the next step,way ? and I need to run "ip route add default via 10.64.208.254 dev eth1 metric 101" to make this routes persistent.

�� Newbie 15 points

19 May 2018 2:39 AM

星亚徐

the prefix was 24 at the beginning, but became 8 in the next step,way ? and I need to run "ip route add default via 10.64.208.254 dev eth1 metric 101" to make this routes persistent.

RG Newbie 7 points

7 November 2018 10:42 PM

Roger Gray

The section with dot point "Add rules to apply traffic to the routing tables". Shows how to add an "ip rule" but does not have the correct command for displaying the newly added rule.

eg//

# ip rule add table t1 from 10.64.208.180       <- OK
# ip rule add table t2 from 10.64.208.208       <- OK

# ip route show                                                         <- Does not help.  Where is the rule just added?
10.64.208.0/24 dev eth0  proto kernel  scope link  src 10.64.208.180
10.64.208.0/24 dev eth1  proto kernel  scope link  src 10.64.208.208
169.254.0.0/16 dev eth1  scope link
default via 10.64.208.254 dev eth0

After adding a "rule", it would be better to run a "ip rule show" rather than "ip route show" eg//

# ip rule show
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

# ip rule add table em1 from 172.16.13.149

# ip rule show
0:      from all lookup local
32765:  from 172.16.13.149 lookup em1   <- Here is the "rule" added via "ip rule add"
32766:  from all lookup main
32767:  from all lookup default

MH Newbie 15 points

15 November 2019 1:21 PM

Matthias Helmling

What would be the equivalent nmcli commands on el8?

MB Red Hat Community Member 21 points

14 June 2020 10:19 PM

Matthew Brown

Hello,

I have re-written the Resolution section substantially. As a result, many of the previous comments may no longer make sense as they applied to the steps that were in the Resolution section before the change. The new Resolution section provides links to existing content in the knowledge base that provide more implementation details and example commands.

JW Active Contributor 120 points

31 December 2020 7:24 PM

Jonathon Wright

The root cause should be more specific: "When there are 2 interfaces on the same subnet..."

Clarify same subnet. Does it mean that the IPs have the same class C (e.g x.x.2.1 and x.x.2.100) octet, or is it when both IPs have the same gateway, or something else?

Red Hat Guru 10629 points

4 January 2021 11:14 PM

Jamie Bainbridge

It means when two IP addresses have a subnet mask such that the addresses share the same network address and broadcast address. For example:

192.168.0.1/24 and 192.168.0.254/24 are in the same subnet because of the /24 subnet mask
192.168.0.1/25 and 192.168.0.254/25 are not in the same subnet because the /25 subnet mask

With classless VLSM being so widespread these days, it's uncommon to refer to subnets in terms of classful ranges like "Class C" anymore.

zz Newbie 5 points

20 February 2021 1:49 PM

zhenxing zhang

I do need two connections with same subnet, because I use two separated Infiniband connections.

Here are my solution.

Background:

Kernel version     : Linux 3.10.0-1127.13.1.el7.x86_64
Nmcli tool version : 1.18.4-3.el7
IB0 info           : NUMA0, 192.168.12.180/16
IB1 info           : NUMA1, 192.168.12.181/16

Step1: Set IP Route and Rule

IB1="192.168.12.180"
IB2="192.168.12.181"

echo "1 ibnuma0" >> /etc/iproute2/rt_tables
echo "2 ibnuma1" >> /etc/iproute2/rt_tables
ip route add 192.168.0.0/16 dev ib0 src ${IB1} table ibnuma0
ip route add 192.168.0.0/16 dev ib1 src ${IB2} table ibnuma1

ip rule add from ${IB1}/32 table ibnuma0
ip rule add to ${IB1}/32 table ibnuma0
ip rule add from ${IB2}/32 table ibnuma1
ip rule add to ${IB2}/32 table ibnuma1

This method fails after reboot, but you can use "rc.local".

Replacing rc.local in systemd Linux systems

I cannot use nmcli to achieve the same result, can anyone tell me? I try nmcli connection modify ib1 ipv4.routes "192.168.0.0/16 src=192.168.12.122 table=2" and edit /etc/sysconfig/network-scripts/route-ib* file directly.

Step2: Set ARP

sysctl net.ipv4.conf.{all,default,ib0,ib1}.rp_filter=2
sysctl net.ipv4.conf.{all,default,ib0,ib1}.arp_ignore=2
sysctl net.ipv4.conf.{all,default,ib0,ib1}.arp_announce=2
sysctl net.ipv4.conf.{all,default,ib0,ib1}.arp_filter=1

Why is my server responding to arp requests from the wrong interface on Red Hat Enterprise Linux ?

When RHEL has multiple IPs configured, only one is reachable from a remote network.

Log in to Your Red Hat Account

Red Hat Account

Customer Portal

Select Your Language

Infrastructure and Management

Cloud Computing

Storage

Runtimes

Integration and Automation

Services

Tools

Red Hat Insights

Red Hat Product Security Center

Security Updates

Resources

Customer Portal Community

Customer Events

Stories

How to connect two network interfaces on the same subnet?

Environment

Issue

Resolution

Root Cause

Diagnostic Steps

13 Comments