Interface name changes after update to RHEL 7.3 and network does not start

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 7.3
  • systemd
  • VMware guest

Issue

  • After updating to RHEL 7.3, NIC names have changed from names such as eno16777728 to ens192.
  • The network does not come up on boot after a yum update to RHEL 7.3.
  • A network interface previously had a device name enoX where X > 16383 and now has an ensY style name.

It was found that VMware firmware/bios reports meaningless data for the device names causing inconsistent, changing or missing network device names after reboot. An update to systemd in RHEL 7.3 was made to provide consistent device names but can result in a final network device name change if the virtual guest already had the long device name enoXXXXX.

This is further explained on the Red Hat TAM Blog Red Hat Enterprise Linux 7.3: Achieving persistent and consistent network interface naming in VMware environments.

Resolution

Affected are systems that

  • are running on a VMware hypervisor
  • and are Red Hat Enterprise Linux 7.0, 7.1, or 7.2 virtual guests upgrading to Red Hat Enterprise Linux 7.3
  • and are using the vmxnet3 driver

This issue can be avoided by updating installation repositories to include the systemd-219-30.el7_3.6 or later package. With this newer systemd release, affected interfaces are identified upon the package install and a udev rule is automatically generated so the renaming issue is entirely avoided. Please see RHBA-2016:2698 - Bug Fix Advisory. Note this update will not fix systems which have already been updated to 7.3. Note that the DVD ISO images will not be updated to include this errata update as most virtual environments are using some other network repository for installs that are updated to include errata updates.

The instructions which follow assume the OS is already using the changed named. Please see the Root Cause section of this article for steps to identify what the new name will be before the change takes place

Update the network configuration to reflect the interface name change:

  • Find the new interface name with the ip command. In the example below, the current interface name is ens192:

    [root@rhel73 ~]# ip link
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT 
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
        link/ether 00:0c:29:ad:aa:5a brd ff:ff:ff:ff:ff:ff
    

If not using NetworkManager:

The ifcfg files in the /etc/sysconfig/network-scripts/ directory can be modified directly. The DEVICE parameter value should be updated to match the new interface name.

If using NetworkManager:

  • Find the name of the existing network configuration profile using nmcli; under the NAME column. In the example below, the profile name is eno16780032:

    [root@dhcp181-180 ~]# nmcli connection
    NAME         UUID                                  TYPE            DEVICE 
    eno16780032  9b7334c3-9d6f-424a-9a53-9127d245ef75  802-3-ethernet  --     
    
  • Update the profile to point to the new interface name. In the example below, the profile eno16780032 is updated to apply to the interface named ens192. After, the connection is brought up so the profile will be applied to the interface:

    # nmcli connection modify eno16780032 connection.interface-name ens192
    # nmcli connection up eno16780032
    
  • Optionally, the profile name can be updated as well:

    # nmcli connection modify eno16780032 connection.id ens192
    

Root Cause

  • This issue is a result of an update in systemd-219-21.el7 and later to provide persistent device names across reboots caused by incorrect and inconsistent device names reported by the VMware firmware/bios. From the errata:

    "Previously, some network devices had unacceptably long names. This was due to certain firmware reporting meaningless data, such as the device's `onboard index` value, which the kernel passed to user-space. This resulted in problems with maximum name length, especially with VLANs. With this update, *systemd* rejects unacceptably long names and falls back to a different naming scheme. As a result, long network device names will no longer appear."
    
  • The slot name of an interface can be seen before the system is updated/restarted so that any interface configurations can be adjusted in advance. Locate the ID_NET_NAME_SLOT property with udevadm. In the example below, the slot-based name for the device eno16780032 is ens192:

    # udevadm info /sys/class/net/eno16780032
    P: /devices/pci0000:00/0000:00:16.0/0000:0b:00.0/net/eno16780032
    E: DEVPATH=/devices/pci0000:00/0000:00:16.0/0000:0b:00.0/net/eno16780032
    E: ID_BUS=pci
    E: ID_MODEL_FROM_DATABASE=VMXNET3 Ethernet Controller
    E: ID_MODEL_ID=0x07b0
    E: ID_NET_DRIVER=vmxnet3
    E: ID_NET_LABEL_ONBOARD=enEthernet0
    E: ID_NET_NAME_MAC=enx005056a570ba
    E: ID_NET_NAME_ONBOARD=eno16780032
    E: ID_NET_NAME_PATH=enp11s0
    E: ID_NET_NAME_SLOT=ens192
    E: ID_OUI_FROM_DATABASE=VMware, Inc.
    E: ID_PATH=pci-0000:0b:00.0
    E: ID_PATH_TAG=pci-0000_0b_00_0
    E: ID_PCI_CLASS_FROM_DATABASE=Network controller
    E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
    E: ID_VENDOR_FROM_DATABASE=VMware
    E: ID_VENDOR_ID=0x15ad
    E: IFINDEX=2
    E: INTERFACE=eno16780032
    E: SUBSYSTEM=net
    E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/eno16780032
    E: TAGS=:systemd:
    E: USEC_INITIALIZED=28864
    
  • Also be aware that any application or configuration which specifically refers to the old interface name will need to be updated to refer to the new interface name. Please contact the appropriate application vendor for any questions regarding this point.

  • The updated systemd-219-30.el7_3.6 package includes the following RPM %POST script which locates any problematic enoX style interface names and creates a persistent udev rule so their names will not unexpectedly change upon reboot:

    %triggerin -- systemd < 219-21
    . /etc/sysconfig/network-scripts/network-functions
    
    RULES_FILE="/etc/udev/rules.d/90-eno-fix.rules"
    DRACUT_CONFIG="/etc/dracut.conf.d/90-eno-fix.conf"
    
    NEED_REBUILD=
    WROTE_MSG=
    
    for i in /sys/class/net/eno* ; do
        DEVICE=${i##*/}
    
        [[ "$DEVICE" =~ eno[0-9]+(d[0-9]+)?$ ]] || continue
        [ "$(echo $DEVICE | sed -e 's/eno\([0-9]\+\).*/\1/')" -lt "16383" ] && continue
    
        HWADDR=$(get_hwaddr $DEVICE | tr '[:upper:]' '[:lower:]')
        [ -z "$HWADDR" ] && continue
    
        if [ -z "$WROTE_MSG" ]; then
           echo "# This file was automatically generated on systemd update" > "$RULES_FILE"
           WROTE_MSG=yes
        fi
    
        echo "SUBSYSTEM==\"net\", ACTION==\"add\", DRIVERS==\"?*\", ATTR{address}==\"$HWADDR\", NAME=\"$DEVICE\"" >> "$RULES_FILE"
        NEED_REBUILD=yes
    done
    
    if [ -n "$NEED_REBUILD" ]; then
        echo "install_items+=\" $RULES_FILE \"" > "$DRACUT_CONFIG"
        dracut -f
    fi
    

Related links:
- https://www.redhat.com/en/about/blog/red-hat-enterprise-linux-73-achieving-persistent-and-consistent-network-interface-naming-vmware-environments
- https://bugzilla.redhat.com/show_bug.cgi?id=1391944
- https://communities.vmware.com/thread/478439?start=0&tstart=0
- https://github.com/systemd/systemd/commit/6c1e69f9456d022f14dd00737126cfa4d9cca10c

Diagnostic Steps

  • Confirm the host is using systemd-219-21.el7 or later:

    # rpm -q systemd
    
  • Confirm the existing network configuration profiles refer to eno style names which are no longer in use on the host. The connection.interface-name property is important here:

    # nmcli connection show <profile name> | grep connection -m 3
    connection.id:                          eno33559296
    connection.uuid:                        e7f57f65-ac60-455a-a6eb-9a530733512c
    connection.interface-name:              eno33559296
    
  • If not using NetworkManager, verify the DEVICE parameter value in the ifcfg files of the /etc/sysconfig/network-scripts/ directory.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.