[RHEV] My RHEV-H hypervisor suddenly lost connectivity. I see a new breth0 interface is now the default route for the host, how can I recover?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Virtualization 2.2
  • Both Red Hat Entperise Linux and Red Hat Enterprise Hypervisor hosts.

Issue

  • My RHEV-H suddenly lost connectivity. I can't ping my host from another server in the same network.
  • I see a new interface 'breth0' that has the same IP as 'rhevm' and networking isn't working anymore.

Resolution

A ful fix for this issue is currently being worked on within the following bugzilla :

Bug 754331- breth0 co-exist with rhevm  bridge

https://bugzilla.redhat.com/show_bug.cgi?id=754331

The following workaround can be applied to each host where the breth0 device is present after installation :

  • Place the host into maintenance mode via the RHEV-M GUI.
  • Edit the following file to disable the breth0 device at boot :

    ONBOOT=no
    
  • If this is a RHEV-H host then persist the file with the following command :

    # persist /etc/sysconfig/network-scripts/ifcfg-breth0
    
  • Restart the networking service on the host and confirm that the breth0 device is not active and that the default route is via the rhevm device :

    # service network restart
    # ifconfig
    # route
    

Root Cause

The following bugzilla details the root cause of this issue :

Bug 754331- breth0 co-exist with rhevm  bridge

https://bugzilla.redhat.com/show_bug.cgi?id=754331

In short, the inteface should be deleted after the system starts, but due to a bug in the ovirt-node package, vdsm renames the interface on reboot and cleanup fails.

Diagnostic Steps

  • Check the following commands:

    # route -n
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    10.10.0.0       0.0.0.0         255.255.255.0   U     0      0        0 breth0
    10.10.0.0       0.0.0.0         255.255.255.0   U     0      0        0 rhevm
    169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 rhevm
    0.0.0.0         10.10.0.254     0.0.0.0         UG    0      0        0 breth0
    
    
    # ifconfig
    
    breth0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
              inet addr:10.10.0.31  Bcast:10.10.0.255  Mask:255.255.255.0
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:13083 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:0 (0.0 b)  TX bytes:549486 (536.6 KiB)
    
    ...
    
    
    rhevm     Link encap:Ethernet  HWaddr 5C:F3:FC:79:BE:FC  
              inet addr:10.10.0.31  Bcast:10.10.0.255  Mask:255.255.255.0
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:48465 errors:0 dropped:0 overruns:0 frame:0
              TX packets:58 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:2722079 (2.5 MiB)  TX bytes:4066 (3.9 KiB)
    

Notice that both interfaces have the same IP and route. This is wrong and should be corrected by disabling the breth0 interface.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.