Instances loosing connectivity to external network randomly
Issue
- Our vms across compute nodes are losing ip address. Its not able to renew the DHCP after the lease expires.
- Our investigation show that the DHCP request are carried out fine by "tap", "qbr" and "qvo" interfaces but then its not going out of Compute nodes physical interface.
- The only way to fix this is to restart "opensvwitch-agent" and restart network on the Compute nodes. After this the vm gets Ip from DHCP servers.
- Our production vms are all getting IP address from Openstack DHCP server and now they are randomly loosing network connectivity causing random outages
- Instances are loosing ip.addresses when the DHCP lease expires. These ip.addresses are not renewed.
- Envrionment has 8 compute nodes and issue randomly happens on ALL instances on RANDOM compute nodes out of those 8 computes.
- The issue seems to be in the following region:
---------- Last known DHCP request position
|
v
[ethX]--[tapX]--[qvbX]--[qvoX]---[br-int]--[phy-br-bond-X]--[bondX]
| |
| |
'----------------'
|
'--------Unverified space
- In the unverified_space We tried adding a snooper interface on the compute node to the "br-int" and "br-bond-d" using How to use tcpdump on Openvswitch interfaces ? which leads to Segmentation Fault on ovs-vswitchd:
2015-11-25T18:45:51.749Z|00003|daemon(monitor)|EMER|fork child died before signaling startup (killed (Segmentation fault))
- Openvswitch commands like adding an interface hang on the compute node.
Environment
- Openstack 5 (Icehouse) on RHEL 7 in HA.
- Environment has 8 compute Nodes.
- Using VLAN (no VXLAN or GRE)
+------[ COMPUTE HOST ]------+ +----[ NEUTRON HOST ]----+
|+----------+ +----------+| | qdhcpXXX qdhcpYYY |
|| INSTANCE | | INSTANCE || | +--------+ +--------+ |
|| | | || | | dnsmasq| | dnsmasq| |
|+--[eth0]--+ +--[eth0]--+| | | DHCP | | DHCP | |
| | | | | |+------+| |+------+| |
|+--[tapX]--+ +--[tapX]--+| | || nsX || || nsX || |
|| qbrX | | qbrX || | |+------+| |+------+| |
|+--[qvbX]--+ +--[qvbX]--+| | +--------+ +--------+ |
| | | | | |
| +---+ +---+ | | +-[tapX]------[tapY]-+ |
| | | | | | br-int | |
| +--[qvoX]---[qvoY]--+ | | +----[int-br-eth1]---+ |
| | br-int | | | | |
| +---[int-br-eth1]---+ | | | |
| | | | | |
| +---[phy-br-bond-d]-+ | | +--[phy-br-eth1]--+ |
| | br-bond-d | | | | br-eth1 | |
| +------[bond-d]-----+ | | +------[eth1]-----+ |
| | | | | |
| +---------+---+ | | | |
| | | | | | |
+---[eno1]--------[enod2]-----+ +--------[eth1]--[eth0]--+
| +-------------- + |
+-----| INTERNAL VLAN |----+
+---------------+
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.
