Instances loosing connectivity to external network randomly
Issue
- Our vms across compute nodes are losing ip address. Its not able to renew the DHCP after the lease expires.
- Our investigation show that the DHCP request are carried out fine by "tap", "qbr" and "qvo" interfaces but then its not going out of Compute nodes physical interface.
- The only way to fix this is to restart "opensvwitch-agent" and restart network on the Compute nodes. After this the vm gets Ip from DHCP servers.
- Our production vms are all getting IP address from Openstack DHCP server and now they are randomly loosing network connectivity causing random outages
- Instances are loosing ip.addresses when the DHCP lease expires. These ip.addresses are not renewed.
- Envrionment has 8 compute nodes and issue randomly happens on ALL instances on RANDOM compute nodes out of those 8 computes.
- The issue seems to be in the following region:
---------- Last known DHCP request position
|
v
[ethX]--[tapX]--[qvbX]--[qvoX]---[br-int]--[phy-br-bond-X]--[bondX]
| |
| |
'----------------'
|
'--------Unverified space
- In the unverified_space We tried adding a snooper interface on the compute node to the "br-int" and "br-bond-d" using How to use tcpdump on Openvswitch interfaces ? which leads to Segmentation Fault on ovs-vswitchd:
2015-11-25T18:45:51.749Z|00003|daemon(monitor)|EMER|fork child died before signaling startup (killed (Segmentation fault))
- Openvswitch commands like adding an interface hang on the compute node.
Environment
- Openstack 5 (Icehouse) on RHEL 7 in HA.
- Environment has 8 compute Nodes.
- Using VLAN (no VXLAN or GRE)
+------[ COMPUTE HOST ]------+ +----[ NEUTRON HOST ]----+
|+----------+ +----------+| | qdhcpXXX qdhcpYYY |
|| INSTANCE | | INSTANCE || | +--------+ +--------+ |
|| | | || | | dnsmasq| | dnsmasq| |
|+--[eth0]--+ +--[eth0]--+| | | DHCP | | DHCP | |
| | | | | |+------+| |+------+| |
|+--[tapX]--+ +--[tapX]--+| | || nsX || || nsX || |
|| qbrX | | qbrX || | |+------+| |+------+| |
|+--[qvbX]--+ +--[qvbX]--+| | +--------+ +--------+ |
| | | | | |
| +---+ +---+ | | +-[tapX]------[tapY]-+ |
| | | | | | br-int | |
| +--[qvoX]---[qvoY]--+ | | +----[int-br-eth1]---+ |
| | br-int | | | | |
| +---[int-br-eth1]---+ | | | |
| | | | | |
| +---[phy-br-bond-d]-+ | | +--[phy-br-eth1]--+ |
| | br-bond-d | | | | br-eth1 | |
| +------[bond-d]-----+ | | +------[eth1]-----+ |
| | | | | |
| +---------+---+ | | | |
| | | | | | |
+---[eno1]--------[enod2]-----+ +--------[eth1]--[eth0]--+
| +-------------- + |
+-----| INTERNAL VLAN |----+
+---------------+
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Close
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.