Network connection lost on Red Hat Enterprise Linux guest running on VMware ESX 5.1

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 6
  • VMware ESX 5.1

Issue

  • Virtual machines running in VMWARE vSphere were network unresponsive. Message log contained entries of ethX interface being hung. A restart of the network service helped for about 30 seconds at which time the VM was not accessible again from the network.
  • Network failures utilizing either vmxnet3 or e1000 NIC. All attempts of driver removal and reattachment or rebooting of the server, result in network failure with messages such as the following found in /var/log/messages:

    kernel: NETDEV WATCHDOG: eth0 (vmxnet3): transmit queue 0 timed out
    kernel: eth0: tx hang
    kernel: eth0: resettting
    kernel: eth0: intr type 3, mode 0, 5 vectors allocated
    kernel: Failed to activate dev eth0: error 1
    

Resolution

  • For the current workaround we highly recommend that the vendor of your VMware setup be engaged to determine the appropriate path forward for resolution.

Diagnostic Steps

  • Search for the existence of entries similar to that listed below in VMware host logs :

    2013-08-06T03:29:30.928Z| vcpu-0| I120: VMXNET3 user: failed to activate Ethernet1, status: 0xbad0001
    2013-08-06T15:36:58.086Z| vcpu-2| I120: VMXNET3 user: failed to activate Ethernet1, status: 0xbad0001 
    2013-08-06T15:39:03.986Z| vcpu-3| I120: VMXNET3 user: failed to activate Ethernet1, status: 0xbad0001 
    2013-08-06T15:40:51.743Z| vcpu-3| I120: VMXNET3 user: failed to activate Ethernet1, status: 0xbad0001 
    2013-08-06T15:55:05.494Z| vcpu-2| I120: VMXNET3 user: failed to activate Ethernet1, status: 0xbad0001
    
  • Messages indicating port flapping similar to those listed below may be observed:

    2013-08-07T03:26:42.110Z cpu1:13840)NetPort: 2747: resuming traffic on DV port 268 
    2013-08-07T03:26:42.110Z cpu1:13840)NetPort: 1380: enabled port 0x6000022 with mac de:ad:be:ef:de:ad
    2013-08-07T03:28:29.927Z cpu9:288181)NetPort: 1574: disabled port 0x600002c 
    2013-08-07T03:28:30.342Z cpu9:288181)NetPort: 2747: resuming traffic on DV port 715 
    2013-08-07T03:28:30.343Z cpu9:288181)NetPort: 1380: enabled port 0x600002c with mac de:ad:be:ef:de:ad
    2013-08-07T03:28:30.563Z cpu9:288181)NetPort: 1574: disabled port 0x600002c 
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments