Latest overcloud image fails to configure the network, reverting to previous versions of the image work fine

Solution In Progress - Updated -

Issue

  • After updating to the latest minor update adding new overcloud nodes with the current image fails to configure network on the device. Reverting back to the previous version of the image does configure the networking on the new nodes being added.

  • Package version list:

overcloud-full-latest-16.1-x86_64.tar -> /usr/share/rhosp-director-images/overcloud-full-16.1-20210430.3.el8ost.x86_64.tar

(undercloud) [stack@director01 ~]$ sudo rpm -qa | grep image
rhosp-director-images-16.1-20210430.3.el8ost.noarch
rhosp-director-images-16.1-20210310.1.el8ost.noarch
octavia-amphora-image-x86_64-16.1-20210322.1.el8ost.noarch
rhosp-director-images-ipa-x86_64-16.1-20210310.1.el8ost.noarch
rhosp-director-images-x86_64-16.1-20210322.1.el8ost.noarch
octavia-amphora-image-x86_64-16.1-20210430.3.el8ost.noarch
diskimage-builder-3.0.1-0.20200609103442.68017d0.el8ost.noarch
rhosp-director-images-x86_64-16.1-20210310.1.el8ost.noarch
rhosp-director-images-ipa-x86_64-16.1-20210430.3.el8ost.noarch
ansible-role-tripleo-modify-image-1.2.1-1.20201114004656.1dffa21.el8ost.noarch
rhosp-director-images-ipa-x86_64-16.1-20210322.1.el8ost.noarch
rhosp-director-images-ipa-16.1-20210430.3.el8ost.noarch
genisoimage-1.1.11-39.el8.x86_64
openstack-tripleo-image-elements-10.6.2-1.20201113215051.7dc0fa1.el8ost.noarch
octavia-amphora-image-x86_64-16.1-20210310.1.el8ost.noarch
rhosp-director-images-x86_64-16.1-20210430.3.el8ost.noarch
rhosp-director-images-ipa-16.1-20210310.1.el8ost.noarch
  • The following script is being used to customize the overcloud-full image:
virt-customize -a /home/stack/images/overcloud-full-gpu.qcow2 --copy-in '/home/stack/templates/custom_scripts/rhosp_image_customize.sh:/var/tmp' --run-command 'chmod +x /var/tmp/rhosp_image_customize.sh' --run-command '/var/tmp/rhosp_image_customize.sh' --run-command 'rm -f /var/tmp/rhosp_image_customize.sh'

rhosp_image_customize.sh:

#!/bin/bash

if ! grep -i "blacklist nouveau" /etc/modprobe.d/blacklist-nouveau.conf
then
  /usr/bin/echo 'blacklist nouveau' > /etc/modprobe.d/blacklist-nouveau.conf
fi

if ! rpm -q NVIDIA-vGPU-rhel-8.2_4.18.0_193.28.1-460.32.04.x86_64
then
  yum install -y http://10.10.10.10/pub/export/mplex/metroplex/Library/custom/Custom_Products/nvidia/Packages/n/NVIDIA-vGPU-rhel-8.2_4.18.0_193.28.1-460.32.04.x86_64.rpm
fi

if ! [ -f /etc/modprobe.d/vfio.conf ] /etc/modprobe.d/vfio.conf
then
  /usr/bin/echo 'options vfio-pci ids=10de:1bb3,10de:1eb8' > /etc/modprobe.d/vfio.conf
fi

if [ $(rpm -q --queryformat '%{VERSION}' dracut) -lt 049 ] || [ $(rpm -q --queryformat '%{VERSION}' dracut) -eq 049 ] && [ $(rpm -q --queryformat '%{RELEASE}' dracut | grep -Po '^[^.]+') -lt 92 ]
then
  echo 'omit_drivers+="nvidia"' > /etc/dracut.conf.d/nvidia.conf
  dracut -f --regenerate-all
fi

if ! [ -f /etc/yum.repos.d/upstream.repo ]
then
cat << EOF > /etc/yum.repos.d/upstream.repo

(REPO STUFF)

EOF
fi
  • Deployment fails with:
...
        "[2021/07/21 04:19:21 PM] [INFO] interface eno3 is up, trying with ip command",
        "[2021/07/21 04:19:21 PM] [INFO] running ifdown on interface: ens7f1",
        "[2021/07/21 04:19:21 PM] [INFO] interface ens7f1 is up, trying with ip command",
        "[2021/07/21 04:19:21 PM] [INFO] running ifdown on interface: ens7f0",
        "[2021/07/21 04:19:21 PM] [INFO] interface ens7f0 is up, trying with ip command",
...
        "[2021/07/21 04:22:08 PM] [ERROR] Failure(s) occurred when applying configuration",
        "[2021/07/21 04:22:08 PM] [ERROR] stdout: ERROR     : [/etc/sysconfig/network-scripts/ifup-eth] Device vlan130 does not seem to be present, delaying initialization.",
        ", stderr: WARN      : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.",
        "WARN      : [ifup] 'network-scripts' will be removed in one of the next major releases of RHEL.",
        "WARN      : [ifup] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.",
        "A dependency job for openvswitch.service failed. See 'journalctl -xe' for details.",
        "2021-07-21T20:19:49Z|00002|fatal_signal|WARN|terminating with signal 14 (Alarm clock)",
        "/etc/sysconfig/network-scripts/ifup-ovs: line 74:  6470 Alarm clock             ovs-vsctl -t ${TIMEOUT} -- --if-exists del-port \"$OVS_BRIDGE\" \"$DEVICE\" -- add-port \"$OVS_BRIDGE\" \"$DEVICE\" $OVS_OPTIONS -- set Interface \"$DEVICE\" type=internal ${OVS_EXTRA+-- $OVS_EXTRA}",
        "",
        "[2021/07/21 04:22:08 PM] [ERROR] stdout: ERROR     : [/etc/sysconfig/network-scripts/ifup-eth] Device vlan110 does not seem to be present, delaying initialization.",
        ", stderr: WARN      : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.",
        "WARN      : [ifup] 'network-scripts' will be removed in one of the next major releases of RHEL.",
        "WARN      : [ifup] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.",
        "A dependency job for openvswitch.service failed. See 'journalctl -xe' for details.",
        "2021-07-21T20:20:12Z|00002|fatal_signal|WARN|terminating with signal 14 (Alarm clock)",
        "/etc/sysconfig/network-scripts/ifup-ovs: line 74:  7306 Alarm clock             ovs-vsctl -t ${TIMEOUT} -- --if-exists del-port \"$OVS_BRIDGE\" \"$DEVICE\" -- add-port \"$OVS_BRIDGE\" \"$DEVICE\" $OVS_OPTIONS -- set Interface \"$DEVICE\" type=internal ${OVS_EXTRA+-- $OVS_EXTRA}",
        "",
        "[2021/07/21 04:22:08 PM] [ERROR] stdout: ERROR     : [/etc/sysconfig/network-scripts/ifup-eth] Device vlan190 does not seem to be present, delaying initialization.",
        ", stderr: WARN      : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.",
        "WARN      : [ifup] 'network-scripts' will be removed in one of the next major releases of RHEL.",
        "WARN      : [ifup] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.",
        "A dependency job for openvswitch.service failed. See 'journalctl -xe' for details.",
        "2021-07-21T20:20:23Z|00002|fatal_signal|WARN|terminating with signal 14 (Alarm clock)",
        "/etc/sysconfig/network-scripts/ifup-ovs: line 74:  7589 Alarm clock             ovs-vsctl -t ${TIMEOUT} -- --if-exists del-port \"$OVS_BRIDGE\" \"$DEVICE\" -- add-port \"$OVS_BRIDGE\" \"$DEVICE\" $OVS_OPTIONS -- set Interface \"$DEVICE\" type=internal ${OVS_EXTRA+-- $OVS_EXTRA}",
        "",
...

Environment

  • Red Hat OpenStack Platform 16.1 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content