Intermittent packet loss in the namespaces of OSP HA when pinging Neutron router IPs
Environment
Red Hat Enterprise Linux OpenStack Platform 6.0
Red Hat Enterprise Linux OpenStack Platform 7.0
Red Hat OpenStack Platform 8.0
Issue
In an high availability (HA) environment, intermittent packet loss occurs when pinging to namespace's external interface (router IP). This doesn't seem to be a physical problem. Issue does not happen with instances that are directly connected in the external network. This issue happens across different tenants.
Ping to namespace external interface (router IP)
--- 172.16.0.10 ping statistics ---
200 packets transmitted, 187 received, 6% packet loss, time 40409ms
rtt min/avg/max/mdev = 0.234/1.803/13.303/2.489 ms, ipg/ewma 203.060/1.364 ms
Ping to namespace external interface (Floating IP)
--- 172.16.0.50 ping statistics ---
200 packets transmitted, 165 received, +20 errors, 17% packet loss, time 39934ms
rtt min/avg/max/mdev = 2.602/3.945/7.688/0.523 ms, pipe 16, ipg/ewma 200.678/3.842 ms
Resolution
Neutron on one of the controllers (controller2) seems to be misconfigured:
IP address of the controller node is 10.0.0.6
ip a ls dev eth2
3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1550 qdisc mq state UP qlen 1000
link/ether 11:11:11:11:11 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.6/24 brd 10.0.0.255 scope global eth2
valid_lft forever preferred_lft forever
inet6 fe80::66 scope link
valid_lft forever preferred_lft forever
But local_ip in ovs_neutron_plugin.ini contains 10.0.0.7
grep local_ip * -R | grep -v '#'
plugins/openvswitch/ovs_neutron_plugin.ini:local_ip=10.0.0.7
Due to the fact that local_ip is wrong, VXLAN tunnels can't not form between controller2 and the other controllers. controller2's keepalived needs to see VRRP keepalived messages from its neighbors over VXLAN tunnels. Otherwise, it will assume that it lost contact to the MASTER and will transition from BACKUP to MASTER role. As a consequence, tenant external and internal gateway IPs appear twice on controller1 and controller2, which explains the packet loss when pinging external gateway IPs from the outside.
First, modify local_ip to 10.0.0.6
grep local_ip * -R | grep -v '#'
plugins/openvswitch/ovs_neutron_plugin.ini:local_ip=10.0.0.6
Afterwards, bring the node into standby mode and bring it back to active with pacemaker:
pcs cluster standby pcmk-controller2
pcs cluster unstandby pcmk-controller2
Root Cause
keepalived provides L3 high availability for neutron's L3 agent. keepalives transition over the HA network.
For more details, please check upstream documentation: https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP
Due to the fact that local_ip was wrong, VXLAN tunnels did not form between controller2 and the other controllers. controller2's keepalived needs to see VRRP keepalived messages from its neighbors over VXLAN tunnels on the HA network. Otherwise, it will assume that it lost contact to the MASTER and will transition from BACKUP to MASTER role. As a consequence, one can observe that external facing bridges (qg-...) and internal facing bridges (qr-...) for all tenants shared the same IP address on controller1 and controller2., whereas controller3 behaved correctly and did not share the same IP address. Only one node should be active at any given time as a L3 router in any given tenant. This split brain scenario explains the observed intermittent packet loss.
Diagnostic Steps
First, make sure that the underlying physical network is o.k.: verify interface counters and run a MTR in order to make sure that the issue is not elsewhere in the network.
Verify that the interface is part of br-ex
# ovs-vsctl list-ports br-ex
eth0
Verify that the interface is up
# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: on (auto)
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
Verify interface error counters for the interface, the bridge and in the tenant's namespace
# ip -s link ls dev eth0
4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
link/ether 00:00:00:00:01 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
38289033908 373689339 0 0 0 27343265
TX: bytes packets errors dropped carrier collsns
103630677437 138884305 0 0 0 0
# ip -s link ls dev br-ex
11: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether 00:00:00:00:01 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
20777934960 330157120 0 164306 0 0
TX: bytes packets errors dropped carrier collsns
83430 1963 0 0 0 0
# ip netns exec qrouter-aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa ip -s link ls
257: qg-00000000-01: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether 00:00:00:00:02 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
124183203 1758290 0 34 0 0
TX: bytes packets errors dropped carrier collsns
110 1 0 0 0 0
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
RX: bytes packets errors dropped overrun mcast
0 0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
0 0 0 0 0 0
240: ha-00000000-10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether 00:00:00:00:00:03 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
815063 14957 0 47 0 0
TX: bytes packets errors dropped carrier collsns
846 9 0 0 0 0
248: qr-00000000-20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether 00:00:00:00:00:04 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
2029197 34365 0 101 0 0
TX: bytes packets errors dropped carrier collsns
298 3 0 0 0 0
Verify if the issue is somewhere else in the network or if it is at the last hop.
In this case, MTR identifies the issue as being a problem with the last hop. Hops with 100% packet loss are configured not to send IP unreachables and thus can be ignored during analysis. What is important in the following output is that we see 0% packet loss elsewhere in the network but 10% packet loss for the last hop.
$ mtr -n -r 172.16.0.10
Start: Fri Feb 12 16:00:34 2016
HOST: localhost Loss% Snt Last Avg Best Wrst StDev
1.|-- 192.168.0.10 0.0% 10 0.2 0.2 0.1 0.3 0.0
2.|-- 192.168.1.1 0.0% 10 0.6 0.6 0.5 0.8 0.0
3.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
4.|-- 10.10.1.101 0.0% 10 2.3 2.6 2.2 4.0 0.5
5.|-- 10.10.2.150 0.0% 10 3.7 13.1 2.0 50.1 15.0
6.|-- 10.10.3.51 0.0% 10 4.6 4.6 2.2 13.6 3.3
7.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
8.|-- 10.10.81.3 0.0% 10 2.9 3.0 2.2 3.6 0.0
9.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
10.|-- 10.10.254.254 0.0% 10 15.5 4.8 3.2 15.5 3.8
11.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
12.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
13.|-- 172.16.0.10 10.0% 10 2.7 3.5 2.7 4.7 0.4
Focusing on IP address configuration in tenant namespace shows duplicate IPs for L3 on controller1 and controller2. controller3 behaves correctly.
Also, controller2 cannot ping controller1 and controller3 within the namespace (on network 169.254.192.0/18)
controller1
ip netns exec qrouter-aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa ifconfig -a
ha-00000000-10: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.192.1 netmask 255.255.192.0 broadcast 169.254.255.255
inet6 fe80::3 prefixlen 64 scopeid 0x20<link>
ether 00:00:00:00:00:03 txqueuelen 0 (Ethernet)
RX packets 5096 bytes 347716 (339.5 KiB)
RX errors 0 dropped 89 overruns 0 frame 0
TX packets 618875 bytes 29706450 (28.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
(...)
qg-00000000-01: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.16.0.10 netmask 255.255.248.0 broadcast 0.0.0.0
inet6 fe80::2 prefixlen 64 scopeid 0x20<link>
ether 00:00:00:00:02 txqueuelen 0 (Ethernet)
RX packets 302702570 bytes 19996121350 (18.6 GiB)
RX errors 0 dropped 172556 overruns 0 frame 0
TX packets 13697858 bytes 13265858397 (12.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-00000000-20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.1 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::3 prefixlen 64 scopeid 0x20<link>
ether 00:00:00:00:00:04 txqueuelen 0 (Ethernet)
RX packets 13235872 bytes 13244125286 (12.3 GiB)
RX errors 0 dropped 24 overruns 0 frame 0
TX packets 10814422 bytes 1667874147 (1.5 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
controller2
ip netns exec qrouter-aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa ifconfig -a
ha-11111111-01: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.192.3 netmask 255.255.192.0 broadcast 169.254.255.255
inet6 fe80::10 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:99:5b:f0 txqueuelen 0 (Ethernet)
RX packets 8232 bytes 559049 (545.9 KiB)
RX errors 0 dropped 231 overruns 0 frame 0
TX packets 621261 bytes 29820954 (28.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
(...)
qg-00000000-01: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.16.0.10 netmask 255.255.248.0 broadcast 0.0.0.0
inet6 fe80::2 prefixlen 64 scopeid 0x20<link>
ether 00:00:00:00:02 txqueuelen 0 (Ethernet)
RX packets 292779125 bytes 18409783683 (17.1 GiB)
RX errors 0 dropped 1677040 overruns 0 frame 0
TX packets 1751236 bytes 109440906 (104.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-00000000-20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.1 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::3 prefixlen 64 scopeid 0x20<link>
ether 00:00:00:00:00:04 txqueuelen 0 (Ethernet)
RX packets 7810 bytes 527886 (515.5 KiB)
RX errors 0 dropped 95 overruns 0 frame 0
TX packets 790052 bytes 46850188 (44.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
controller3
ip netns exec qrouter-aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa ifconfig -a
ha-2222222-01: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.192.2 netmask 255.255.192.0 broadcast 169.254.255.255
inet6 fe80::50 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:85:c5:5d txqueuelen 0 (Ethernet)
RX packets 632355 bytes 30631801 (29.2 MiB)
RX errors 0 dropped 43 overruns 0 frame 0
TX packets 21 bytes 1494 (1.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
(...)
qg-00000000-01: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 00:00:00:00:02 txqueuelen 0 (Ethernet)
RX packets 288826788 bytes 18262780164 (17.0 GiB)
RX errors 0 dropped 176150995 overruns 0 frame 0
TX packets 2 bytes 220 (220.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qr-00000000-20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 00:00:00:00:00:04 txqueuelen 0 (Ethernet)
RX packets 1661723 bytes 94516487 (90.1 MiB)
RX errors 0 dropped 64 overruns 0 frame 0
TX packets 2 bytes 188 (188.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
For further analysis, keepalived configuration can be identified and verified as follows:
ps auxwww
root 5555 0.0 0.0 111640 1320 ? Ss Feb03 0:22 keepalived -P -f /var/lib/neutron/ha_confs/99999999-9999-9999-9999-999999999999/keepalived.conf -p /var/lib/neutron/ha_confs/99999999-9999-9999-9999-999999999999.pid -r /var/lib/neutron/ha_confs/99999999-9999-9999-9999-999999999999.pid-vrrp
root 5556 0.0 0.0 111640 1320 ? Ss Feb03 0:22 keepalived -P -f /var/lib/neutron/ha_confs/99999999-9999-9999-9999-999999998888/keepalived.conf -p /var/lib/neutron/ha_confs/99999999-9999-9999-9999-999999998888.pid -r /var/lib/neutron/ha_confs/99999999-9999-9999-9999-999999998888.pid-vrrp
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments