Instances loosing connectivity to external network randomly

Solution Verified - Updated -

Issue

  • Our vms across compute nodes are losing ip address. Its not able to renew the DHCP after the lease expires.
  • Our investigation show that the DHCP request are carried out fine by "tap", "qbr" and "qvo" interfaces but then its not going out of Compute nodes physical interface.
  • The only way to fix this is to restart "opensvwitch-agent" and restart network on the Compute nodes. After this the vm gets Ip from DHCP servers.
  • Our production vms are all getting IP address from Openstack DHCP server and now they are randomly loosing network connectivity causing random outages
  • Instances are loosing ip.addresses when the DHCP lease expires. These ip.addresses are not renewed.
  • Envrionment has 8 compute nodes and issue randomly happens on ALL instances on RANDOM compute nodes out of those 8 computes.
  • The issue seems to be in the following region:

---------- Last known DHCP request position | v [ethX]--[tapX]--[qvbX]--[qvoX]---[br-int]--[phy-br-bond-X]--[bondX] | | | | '----------------' | '--------Unverified space
2015-11-25T18:45:51.749Z|00003|daemon(monitor)|EMER|fork child died before signaling startup (killed (Segmentation fault))
  • Openvswitch commands like adding an interface hang on the compute node.

Environment

  • Openstack 5 (Icehouse) on RHEL 7 in HA.
  • Environment has 8 compute Nodes.
  • Using VLAN (no VXLAN or GRE)

+------[ COMPUTE HOST ]------+ +----[ NEUTRON HOST ]----+ |+----------+ +----------+| | qdhcpXXX qdhcpYYY | || INSTANCE | | INSTANCE || | +--------+ +--------+ | || | | || | | dnsmasq| | dnsmasq| | |+--[eth0]--+ +--[eth0]--+| | | DHCP | | DHCP | | | | | | | |+------+| |+------+| | |+--[tapX]--+ +--[tapX]--+| | || nsX || || nsX || | || qbrX | | qbrX || | |+------+| |+------+| | |+--[qvbX]--+ +--[qvbX]--+| | +--------+ +--------+ | | | | | | | | +---+ +---+ | | +-[tapX]------[tapY]-+ | | | | | | | br-int | | | +--[qvoX]---[qvoY]--+ | | +----[int-br-eth1]---+ | | | br-int | | | | | | +---[int-br-eth1]---+ | | | | | | | | | | | +---[phy-br-bond-d]-+ | | +--[phy-br-eth1]--+ | | | br-bond-d | | | | br-eth1 | | | +------[bond-d]-----+ | | +------[eth1]-----+ | | | | | | | | +---------+---+ | | | | | | | | | | | +---[eno1]--------[enod2]-----+ +--------[eth1]--[eth0]--+ | +-------------- + | +-----| INTERNAL VLAN |----+ +---------------+

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In