How to connect two network interfaces on the same subnet?
Environment
- Red Hat Enterprise Linux 5
- Red Hat Enterprise Linux 6
- Red Hat Enterprise Linux 7
- Multiple network interfaces, each with an IP address in the same subnet
Issue
- How to connect two network interfaces on the same subnet?
- In our environment, there are three bonding devices connected with the same segment.
- We have captured packet and found that packet should be transmitted from bond0 was actually transmitted from bond1
- Also, it was confirmed that the transmit port staggered even if there was no bonding setting.
+---------------------+
| Linux |
| .168 .169 |
+-----+--------+------+
│ │
+-----+--------+------+
| Switch |
+---------+-----------+
│
+---------+-----------+
| .1 |
| Gateway |
+---------------------+
Resolution
Add routing tables and rules binding source IP address for each route, and add those as default gateway for each network interface.
Assuming this networking enviroment :
+------------------------------------------+
| Linux |
| eth0 eth1 |
| 10.64.208.180 10.64.208.208 |
+------------------------------------------+
# ip addr show
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
inet 10.64.208.180/24 brd 10.65.211.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
inet 10.64.208.208/24 brd 10.65.211.255 scope global eth1
-
Add new routing tables in
/etc/iproute2/rt_tables
# cat /etc/iproute2/rt_tables 100 t1 101 t2
-
Add routes to those routing tables
# ip route add 10.64.208.0/24 dev eth0 src 10.64.208.180 table t1 # ip route add table t1 default via 10.64.208.254 dev eth0 # ip route show table t1 10.64.208.0 dev eth0 scope link src 10.64.208.180 default via 10.64.208.254 dev eth0 # ip route add 10.64.208.0/24 dev eth1 src 10.64.208.208 table t2 # ip route add table t2 default via 10.64.208.254 dev eth1 # ip route show table t2 10.64.208.0 dev eth1 scope link src 10.64.208.208 default via 10.64.208.254 dev eth1
-
Add rules to apply traffic to the routing tables
# ip rule add table t1 from 10.64.208.180 # ip rule add table t2 from 10.64.208.208 # ip route show 10.64.208.0/24 dev eth0 proto kernel scope link src 10.64.208.180 10.64.208.0/24 dev eth1 proto kernel scope link src 10.64.208.208 169.254.0.0/16 dev eth1 scope link default via 10.64.208.254 dev eth0
-
Set interfaces ready for receiving ARP replies
# sysctl net.ipv4.conf.default.arp_filter=1
-
Checking ping with
-I IPADDR
# ping -I 10.64.208.180 DSTADDR
-
To make this routes persistent following configuration files have to be changed
-
For network addresses and routes:
# cat /etc/sysconfig/network-scripts/ifcfg-eth* # ifcfg-eth0 DEVICE=eth0 BOOTPROTO=none ONBOOT=yes NETMASK=255.0.0.0 IPADDR=10.64.208.180 GATEWAY=10.64.208.254 TYPE=Ethernet # ifcfg-eth1 DEVICE=eth1 BOOTPROTO=none ONBOOT=yes NETMASK=255.0.0.0 IPADDR=10.64.208.208 GATEWAY=10.64.208.254 TYPE=Ethernet # cat /etc/sysconfig/network-scripts/route-eth* # route-eth0 10.0.0.0/8 dev eth0 src 10.64.208.180 table t1 default via 10.64.208.254 dev eth0 table t1 # route-eth1 10.0.0.0/8 dev eth1 src 10.64.208.208 table t2 default via 10.64.208.254 dev eth1 table t2
-
For routing rules:
# cat /etc/sysconfig/network-scripts/rule-eth* # rule-eth0 table t1 from 10.64.208.180 # rule-eth1 table t2 from 10.64.208.208
-
For receiving ARP replies:
# grep arp_filter /etc/sysctl.conf net.ipv4.conf.all.arp_filter = 1 net.ipv4.conf.default.arp_filter = 1
-
For sending ARP:
# grep /etc/sysctl.conf net.ipv4.conf.all.arp_announce = 2 net.ipv4.conf.default.arp_announce = 2
-
Note: Refer to /usr/share/doc/kernel-doc-<version>/Documentation/networking/ip-sysctl.txt
for more information about these settings.
Root Cause
- When there are 2 interfaces on the same subnet there is no assurance as to which interface will be used to transmit traffic and the machine will accept traffic for either IP on either interface.
- This is because in Linux the IP address belongs to the host and is not associated with the interface.
- If you ping with
-I DEV
, attempting to use a given interface, there is no guarantee the reply packet (if there even is one) will come back to the same interface, so pings done with-I DEV
may not work.
Diagnostic Steps
- Setup system with 2 interfaces on the same subnet.
- Ping a target and capture packets with
tcpdump
.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
8 Comments
Is there a typo in step 2) ??
Should these lines be identical?
or shouldn't the second 'ip route add' look like this...
2. Add routing table to t1, t2 :
.
this article still have wrong device name for default routing in t2 table
There is also anotehr typo...
As for 4):
should say
Thanks for your comments. I've updated the article with the corrections.
This script automatically implements the changes and creation of static routes described above, in case anyone wants to use it.
All you need to provide is the subnet that you want to effect.
#!/bin/sh
# fixFlux.sh - Assigns fixed routes for each NIC
# residing on the same subnet to resolve
# the ARP FlUX problem
# See https://access.redhat.com/kb/docs/DOC-59096
#
# Date: 7/21/2011
SCRIPT_REVISION=1.0.1
SCRIPT_LOG=/dev/nul
# Author: Ken Banks
# Company: Kontron America.
#
# CHANGE LOG:
# 7/21/2011 1.0.1 Original code.
# 8/23/2011 1.0.2 Fix a bug with 'undo' to re-enble NetworkManager
# and another bug that allowed multiple 'no's to be
# added to NM_CONTROLLED= if the script was run more than once.
SYSCTL=/etc/sysctl.conf
RT_TABLES=/etc/iproute2/rt_tables
function usage() {
echo "usage: $0 subnet <undo>"
echo " version $SCRIPT_REVISION"
echo " : This script will solve the Arp Flux for multiple NICs"
echo " : residing on the same subnet, as described in this document:"
echo " : https://access.redhat.com/kb/docs/DOC-59096"
echo
echo " e.g. $0 192.168.1 - Configures a static route for ever IC on 192.168.1.0"
echo " $0 192.168.1 undo - Reverts the changes caused above and re-instates NetworkManager"
}
function log_message() {
echo -e "$*"
echo -e "$*" >> $SCRIPT_LOG
}
function compareValues() {
awk "BEGIN { if ( $1 $2 $3 ) print \"TRUE\" }"
}
function addRoute() {
NICNAME=$1
TABLENUM=$2
TABLENAME=$3
SUBNET=$4
DEFAULTGW=$5
NICIP=$6
FirstOctect=`echo $SUBNET | awk -F. '{ print $1 }'`
ip route show | grep $6 > /dev/nul
if [ "$?" != "0" ] ; then
echo ip route add $SUBNET dev $NICNAME src $NICIP table $TABLENAME
ip route add $SUBNET dev $NICNAME src $NICIP table $TABLENAME
echo ip route add table $TABLENAME default via $DEFAULTGW dev $NICNAME
ip route add table $TABLENAME default via $DEFAULTGW dev $NICNAME
#echo ip route show table $TABLENAME
#ip route show table $TABLENAME
fi
ETHROUTE=/etc/sysconfig/network-scripts/route-$NICNAME
echo $FirstOctect.0.0.0/8 dev $NICNAME src $NICIP table $TABLENAME > $ETHROUTE
echo default via $DEFAULTGW dev $NICNAME table $TABLENAME >> $ETHROUTE
echo "Modified: $ETHROUTE"
ETHRULE=/etc/sysconfig/network-scripts/rule-$NICNAME
echo table $TABLENAME from $NICIP > $ETHRULE
echo "Modified: $ETHRULE"
}
function disableNetworkManager() {
service NetworkManager status | grep stopped >/dev/nul
if [ "$?" != "0" ] ; then
service NetworkManager stop > /dev/nul
fi
echo "NetworkManager service is stopped."
chkconfig NetworkManager --list | grep :on >/dev/nul
if [ "$?" == "0" ] ; then
chkconfig NetworkManager off > /dev/nul
fi
echo "NetworkManager service is disabled."
ETHSCRIPT=/etc/sysconfig/network-scripts/ifcfg-$1
TMPSCRIPT=/tmp/tmpcfg-$1
grep -i NM_CONTROLLED=[\"yes\"] $ETHSCRIPT | awk -F. '{ print $1 }' > /dev/nul
if [ "$?" == "0" ] ; then
cp $ETHSCRIPT $TMPSCRIPT
cat $TMPSCRIPT | sed 's/NM_CONTROLLED=["yes"].*/NM_CONTROLLED=no/' > $ETHSCRIPT
echo "Modified: $ETHSCRIPT"
fi
}
function processNIC() {
# inet 192.168.6.99/24 brd 192.168.6.255 scope global eth0show table
# inet 192.168.6.60/24 brd 192.168.6.255 scope global eth1
export NICNAME=$1
export TABLENUM=$2
export TABLENAME=$3
export SUBNET=$4
export DEFAULTGW=`ip route | grep default | awk '{ print $3 }'`
export NICDATA=`ifconfig -a $NICNAME | grep Bcast`
NICIP=`echo $NICDATA | awk '{ print $2 }' | awk -F: '{ print $2 }'`
if [ "$NICIP" == "" ] ; then
echo "ERROR: No IP for this NIC: $NICNAME"
exit
fi
echo "====PROCESS NIC==== $NICDATA"
echo "==== $NICNAME ===="
grep $TABLENAME $RT_TABLES > /dev/nul
if [ "$?" != "0" ] ; then
echo "Creating route table: $TABLENUM $TABLENAME"
echo "$TABLENUM $TABLENAME" >> $RT_TABLES
else
echo "Route table already exists: $TABLENUM $TABLENAME"
fi
addRoute $NICNAME $TABLENUM $TABLENAME $SUBNET $DEFAULTGW $NICIP
disableNetworkManager $NICNAME
echo
}
function delete_route() {
ip route delete $1 dev $2
}
function undo_routes() {
SUBNET=$1
echo 0 > /proc/sys/net/ipv4/conf/default/arp_filter
echo 0 > /proc/sys/net/ipv4/conf/all/arp_filter
cp $RT_TABLES /tmp/rt_tables
grep -v "^.00 t[0-9]*" /tmp/rt_tables > $RT_TABLES
rm -f /etc/sysconfig/network-scripts/r[ou][ul][te]*
for i in `ip route show | grep src | awk '{ print $1"-"$3 }'` ; do
delete_route `echo $i | sed 's/-/ /'`
done
grep net.ipv4.conf.all.arp_filter $SYSCTL > /dev/nul
if [ "$?" == "0" ] ; then
grep "arp_filter[ =]*0" $SYSCTL > /dev/nul
if [ "$?" != "0" ] ; then
cp $SYSCTL /tmp/tmpctl.conf
cat /tmp/tmpctl.conf | sed 's/arp_filter[ =]*./arp_filter = 0/' >> $SYSCTL
echo "Modified: $SYSCTL"
fi
fi
for device in `ip addr | grep $SUBNET | awk '{ print $7 }'` ; do
ETHSCRIPT=/etc/sysconfig/network-scripts/ifcfg-$device
TMPSCRIPT=/tmp/tmpcfg-$device
grep -i NM_CONTROLLED=[\"no\"] $ETHSCRIPT | awk -F. '{ print $1 }' > /dev/nul
if [ "$?" == "0" ] ; then
cp $ETHSCRIPT $TMPSCRIPT
cat $TMPSCRIPT | sed 's/NM_CONTROLLED=[\"no\"].*/NM_CONTROLLED=yes/' > $ETHSCRIPT
echo "Modified: $ETHSCRIPT"
fi
done
chkconfig NetworkManager on
service NetworkManager start
}
function main() {
[ "$1" == "" ] && usage && exit
SUBNET=$1
[ "$2" == "undo" ] && undo_routes $SUBNET && exit
echo "# $0 $*"
log_message "=========================================================================="
log_message "=============================BEGIN ROUTE CREATION========================="
log_message "=========================================================================="
table=1
for device in `ip addr | grep $SUBNET | awk '{ print $7 }'` ; do
processNIC $device $((100 * $table)) t$table $SUBNET
table=$((1 + $table))
done
echo 1 > /proc/sys/net/ipv4/conf/default/arp_filter
echo "set /proc/sys/net/ipv4/conf/default/arp_filter = 1"
echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter
echo "set /proc/sys/net/ipv4/conf/all/arp_filter = 1"
grep net.ipv4.conf.all.arp_filter $SYSCTL > /dev/nul
if [ "$?" != "0" ] ; then
echo "net.ipv4.conf.all.arp_filter = 1" >> $SYSCTL
echo "Modified: $SYSCTL"
else
grep "arp_filter[ =]*1" $SYSCTL > /dev/nul
if [ "$?" != "0" ] ; then
cp $SYSCTL /tmp/tmpctl.conf
cat /tmp/tmpctl.conf | sed 's/arp_filter[ =]*./arp_filter = 1/' >> $SYSCTL
echo "Modified: $SYSCTL"
fi
fi
service network restart
log_message "=========================================================================="
log_message "==========================ROUTES HAVE BEEN CREATED========================"
log_message "=========================================================================="
echo "Contents of file: $RT_TABLES"
cat $RT_TABLES
echo "Command output from: ip route show"
ip route show
}
main $*
the prefix was 24 at the beginning, but became 8 in the next step,way ? and I need to run "ip route add default via 10.64.208.254 dev eth1 metric 101" to make this routes persistent.
the prefix was 24 at the beginning, but became 8 in the next step,way ? and I need to run "ip route add default via 10.64.208.254 dev eth1 metric 101" to make this routes persistent.
The section with dot point "Add rules to apply traffic to the routing tables". Shows how to add an "ip rule" but does not have the correct command for displaying the newly added rule.
eg//
After adding a "rule", it would be better to run a "ip rule show" rather than "ip route show" eg//