KVM HOST node net.ipv4.ip_forward=0 and bridge and GUEST node with net.ipv4.ip_forward=1 even sysctl setting to "No Forwarninig(0)"

Latest response

Dear Community,

Recently I prepared a HOST on RHEL7.8 and 3 GUEST with also RHEL7.8.

The network interconnection from HOST and GUEST is made using a bridge interface without usage of the native default virtb0 interface created during installation of HOST server.

On the GUEST the network interfaces defined are using vnet

[HOST]
$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens3f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:22 brd ff:ff:ff:ff:ff:ff
3: ens3f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:22 brd ff:ff:ff:ff:ff:ff
4: ens3f2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:24 brd ff:ff:ff:ff:ff:ff
5: ens3f3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:24 brd ff:ff:ff:ff:ff:ff
6: ens3f4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond2 state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:26 brd ff:ff:ff:ff:ff:ff
7: ens3f5: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond2 state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:26 brd ff:ff:ff:ff:ff:ff
8: enp1s0f4u4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 2e:34:cc:78:08:aa brd ff:ff:ff:ff:ff:ff
9: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:22 brd ff:ff:ff:ff:ff:ff
11: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:24 brd ff:ff:ff:ff:ff:ff
12: bond2: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue master br2 state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:26 brd ff:ff:ff:ff:ff:ff
20: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:2f:f7:d2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
21: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:2f:f7:d2 brd ff:ff:ff:ff:ff:ff
30: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:22 brd ff:ff:ff:ff:ff:ff
    inet <IP1> brd 172.16.150.255 scope global br0
       valid_lft forever preferred_lft forever
31: br2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:26 brd ff:ff:ff:ff:ff:ff
    inet <IP2> brd 192.168.26.63 scope global br2
       valid_lft forever preferred_lft forever
32: bond1.823@bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1823 state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:24 brd ff:ff:ff:ff:ff:ff
33: br1823: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:24 brd ff:ff:ff:ff:ff:ff
    inet <IP3> brd 192.168.23.255 scope global br1823
       valid_lft forever preferred_lft forever
34: bond1.824@bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1824 state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:24 brd ff:ff:ff:ff:ff:ff
35: br1824: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:24 brd ff:ff:ff:ff:ff:ff
36: bond1.827@bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1827 state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:24 brd ff:ff:ff:ff:ff:ff
37: br1827: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 16:1e:b1:80:00:24 brd ff:ff:ff:ff:ff:ff
    inet <IP4> brd 192.168.25.255 scope global br1827
       valid_lft forever preferred_lft forever
50: vnet6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:51:17:b7 brd ff:ff:ff:ff:ff:ff
51: vnet7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br1823 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:a5:7c:54 brd ff:ff:ff:ff:ff:ff
52: vnet8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br1824 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:94:63:4b brd ff:ff:ff:ff:ff:ff
53: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:f0:fa:d5 brd ff:ff:ff:ff:ff:ff
54: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br1823 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:82:ce:d6 brd ff:ff:ff:ff:ff:ff
55: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br1824 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:4f:c8:e9 brd ff:ff:ff:ff:ff:ff
56: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:79:2b:da brd ff:ff:ff:ff:ff:ff
57: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br1823 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:81:c4:c2 brd ff:ff:ff:ff:ff:ff
58: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br1824 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:e9:23:cd brd ff:ff:ff:ff:ff:ff
[GUEST]
net.ipv4.ip_forward = 1
$ grep  net.ipv4.ip_forward /etc/sysctl.conf
net.ipv4.ip_forward = 0

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 52:54:00:51:17:b7 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:a5:7c:54 brd ff:ff:ff:ff:ff:ff
    inet <IP2> brd 192.168.23.255 scope global eth1
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 52:54:00:94:63:4b brd ff:ff:ff:ff:ff:ff
5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:9e:19:ec brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
6: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:9e:19:ec brd ff:ff:ff:ff:ff:ff

I have now two interface disabled manually. But they work ok.

The virbr0 even not in the configuration appears on the GUEST.

Based on all this information I would request the following help from community:

1) Once using bridge to share network between HOST and GUEST, do I need to setup n the HOST IP_Forwarding as 1(enabled)?
OBS: Even actually setup on HOST as ip_forward as 0 I do not see problems and documentation has no requirement for this as 1

2) What or where is the GUEST setting up ip_forward as 1 since sysctl OS config is disabled?
I am assuming this is caused by default virb0. How can I disable it on GUEST since I do not intend to use NAT as well I did not create any configuration for it in GUEST XML setup?

Please send your comments on these doubts above.

Responses

1) Once using bridge to share network between HOST and GUEST, do I need to setup n the HOST IP_Forwarding as 1(enabled)?

No. ip_forward is the tunable to enable IP routing and NAT. The bridge used as your br0 forwards frames like a Layer 2 switch, it does not route or NAT.

2) What or where is the GUEST setting up ip_forward as 1 since sysctl OS config is disabled?

By the presence of virbr0 inside the guest, it looks like you've also installed libvirtd in there. This is not required in the guest.

When libvirtd is setup with a NAT network like virbr0 then it will add firewall rules to NAT to guests inside the network and enable ip_forward so that the NAT works. That's likely why the guest has ip_forward=1.

Hi Jamie,

I detected an issue in my Kickstart from HOST installation where it remaining installing:

@virtualization-client @virtualization-platform

This was the reason for the libvirt under GUEST. Once removed all worked fine. So now sysctl is now properly setting up ip_forwarding and as 0(no forwarding).

Regarding ip_forwarding on HOST I get confuse why once libvirt there is installed with default virb0 interface it did not setup ip_forwarding like in the GUEST.

Your explanation about the bridge make sense and then it is working as expected, or no NAT as virtb0 with IP Table rules.

Thanks for your help, very helpful.

Excellent, it's always good to get to the bottom of something :)

I am also not sure why ip_forward=0 on the host. Is there something under /etc/sysctl.d/ setting it?

If you're not using the virbr0 then you could remove it from the host too:

virsh net-list --all
# the name for the virbr0 network should be "default"
virsh net-destroy default
virsh net-undefine default

I do this on my KVM systems, so that the br0 is managed by NetworkManager. libvirt doesn't manage any networks at all, it just tells VMs to make use of the existing br0.

Hi Jamie,

I know the "newest" RHEL7 releases it is recommended to use NM as network manager but by my past experience as well some issues I had to manage Kickstart configuration over NM I decide to use network-scripts, so I can control all low level details.

#HOST##network  --bootproto=dhcp --device=bond1  --noipv4 --noipv6 --no-activate --onboot=yes --bondslaves=ens3f2,ens3f3 --bondopts=downdelay=0,miimon=100,mode=balance-alb,updelay=0
#HOST##network  --bootproto=dhcp --noipv4 --noipv6 --no-activate --onboot=yes --device=bond1 --vlanid=823 --interfacename=bond1.823
#HOST##network  --bootproto=dhcp --noipv4 --noipv6 --no-activate --onboot=yes --device=bond1 --vlanid=827 --interfacename=bond1.827
#HOST##network  --bootproto=dhcp --noipv4 --noipv6 --no-activate --onboot=yes --device=bond1 --vlanid=824 --interfacename=bond1.824
#HOST##network  --bootproto=static --ip=#IP_INTRA# --netmask=#MASK_INTRA# --noipv6 --no-activate --onboot=yes --interfacename=br1823 --device=br1823 --nodefroute --bridgeslaves=bond1.823 --bridgeopts=priority=32768,stp=no
#HOST##network  --bootproto=static --ip=#IP_RMS# --netmask=#MASK_RMS# --gateway=#RMS_GW# --noipv6 --no-activate --onboot=yes --interfacename=br1827 --device=br1827 --nodefroute --bridgeslaves=bond1.827 --bridgeopts=priority=32768,stp=no
#HOST##network  --bootproto=dhcp --noipv4 --noipv6 --no-activate --onboot=yes --interfacename=br1824 --nodefroute --bridgeslaves=bond1.824 --device=br1824 --bridgeopts=priority=32768,stp=no
#HOST##
#HOST##network  --bootproto=dhcp --device=bond2 --noipv4 --noipv6 --no-activate --onboot=yes --bondslaves=ens3f4,ens3f5 --bondopts=mode=802.3ad,lacp_rate=fast,miimon=100,xmit_hash_policy=layer2+3
#HOST##network  --bootproto=static --ip=192.168.26.#IPSUFFIX# --netmask=255.255.255.0 --device=br2 --no-activate --onboot=yes --bridgeslaves=bond2 --mtu=9000 --nodefroute --bridgeopts=priority=32768,stp=no

I could not make the network Kickstart to manage network as I want, just as commented above.

Your advise to remove default network is something I was planning and now I am pretty sure that I should do it.

Once more thanks for the support on this topic!!!

Glad to help however I can :)

I'm not sure about Kickstart either sorry, but we're very happy to look into it further via a support case if you like.