Latest overcloud image fails to configure the network, reverting to previous versions of the image work fine
Issue
-
After updating to the latest minor update adding new overcloud nodes with the current image fails to configure network on the device. Reverting back to the previous version of the image does configure the networking on the new nodes being added.
-
Package version list:
overcloud-full-latest-16.1-x86_64.tar -> /usr/share/rhosp-director-images/overcloud-full-16.1-20210430.3.el8ost.x86_64.tar
(undercloud) [stack@director01 ~]$ sudo rpm -qa | grep image
rhosp-director-images-16.1-20210430.3.el8ost.noarch
rhosp-director-images-16.1-20210310.1.el8ost.noarch
octavia-amphora-image-x86_64-16.1-20210322.1.el8ost.noarch
rhosp-director-images-ipa-x86_64-16.1-20210310.1.el8ost.noarch
rhosp-director-images-x86_64-16.1-20210322.1.el8ost.noarch
octavia-amphora-image-x86_64-16.1-20210430.3.el8ost.noarch
diskimage-builder-3.0.1-0.20200609103442.68017d0.el8ost.noarch
rhosp-director-images-x86_64-16.1-20210310.1.el8ost.noarch
rhosp-director-images-ipa-x86_64-16.1-20210430.3.el8ost.noarch
ansible-role-tripleo-modify-image-1.2.1-1.20201114004656.1dffa21.el8ost.noarch
rhosp-director-images-ipa-x86_64-16.1-20210322.1.el8ost.noarch
rhosp-director-images-ipa-16.1-20210430.3.el8ost.noarch
genisoimage-1.1.11-39.el8.x86_64
openstack-tripleo-image-elements-10.6.2-1.20201113215051.7dc0fa1.el8ost.noarch
octavia-amphora-image-x86_64-16.1-20210310.1.el8ost.noarch
rhosp-director-images-x86_64-16.1-20210430.3.el8ost.noarch
rhosp-director-images-ipa-16.1-20210310.1.el8ost.noarch
- The following script is being used to customize the overcloud-full image:
virt-customize -a /home/stack/images/overcloud-full-gpu.qcow2 --copy-in '/home/stack/templates/custom_scripts/rhosp_image_customize.sh:/var/tmp' --run-command 'chmod +x /var/tmp/rhosp_image_customize.sh' --run-command '/var/tmp/rhosp_image_customize.sh' --run-command 'rm -f /var/tmp/rhosp_image_customize.sh'
rhosp_image_customize.sh:
#!/bin/bash
if ! grep -i "blacklist nouveau" /etc/modprobe.d/blacklist-nouveau.conf
then
/usr/bin/echo 'blacklist nouveau' > /etc/modprobe.d/blacklist-nouveau.conf
fi
if ! rpm -q NVIDIA-vGPU-rhel-8.2_4.18.0_193.28.1-460.32.04.x86_64
then
yum install -y http://10.10.10.10/pub/export/mplex/metroplex/Library/custom/Custom_Products/nvidia/Packages/n/NVIDIA-vGPU-rhel-8.2_4.18.0_193.28.1-460.32.04.x86_64.rpm
fi
if ! [ -f /etc/modprobe.d/vfio.conf ] /etc/modprobe.d/vfio.conf
then
/usr/bin/echo 'options vfio-pci ids=10de:1bb3,10de:1eb8' > /etc/modprobe.d/vfio.conf
fi
if [ $(rpm -q --queryformat '%{VERSION}' dracut) -lt 049 ] || [ $(rpm -q --queryformat '%{VERSION}' dracut) -eq 049 ] && [ $(rpm -q --queryformat '%{RELEASE}' dracut | grep -Po '^[^.]+') -lt 92 ]
then
echo 'omit_drivers+="nvidia"' > /etc/dracut.conf.d/nvidia.conf
dracut -f --regenerate-all
fi
if ! [ -f /etc/yum.repos.d/upstream.repo ]
then
cat << EOF > /etc/yum.repos.d/upstream.repo
(REPO STUFF)
EOF
fi
- Deployment fails with:
...
"[2021/07/21 04:19:21 PM] [INFO] interface eno3 is up, trying with ip command",
"[2021/07/21 04:19:21 PM] [INFO] running ifdown on interface: ens7f1",
"[2021/07/21 04:19:21 PM] [INFO] interface ens7f1 is up, trying with ip command",
"[2021/07/21 04:19:21 PM] [INFO] running ifdown on interface: ens7f0",
"[2021/07/21 04:19:21 PM] [INFO] interface ens7f0 is up, trying with ip command",
...
"[2021/07/21 04:22:08 PM] [ERROR] Failure(s) occurred when applying configuration",
"[2021/07/21 04:22:08 PM] [ERROR] stdout: ERROR : [/etc/sysconfig/network-scripts/ifup-eth] Device vlan130 does not seem to be present, delaying initialization.",
", stderr: WARN : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.",
"WARN : [ifup] 'network-scripts' will be removed in one of the next major releases of RHEL.",
"WARN : [ifup] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.",
"A dependency job for openvswitch.service failed. See 'journalctl -xe' for details.",
"2021-07-21T20:19:49Z|00002|fatal_signal|WARN|terminating with signal 14 (Alarm clock)",
"/etc/sysconfig/network-scripts/ifup-ovs: line 74: 6470 Alarm clock ovs-vsctl -t ${TIMEOUT} -- --if-exists del-port \"$OVS_BRIDGE\" \"$DEVICE\" -- add-port \"$OVS_BRIDGE\" \"$DEVICE\" $OVS_OPTIONS -- set Interface \"$DEVICE\" type=internal ${OVS_EXTRA+-- $OVS_EXTRA}",
"",
"[2021/07/21 04:22:08 PM] [ERROR] stdout: ERROR : [/etc/sysconfig/network-scripts/ifup-eth] Device vlan110 does not seem to be present, delaying initialization.",
", stderr: WARN : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.",
"WARN : [ifup] 'network-scripts' will be removed in one of the next major releases of RHEL.",
"WARN : [ifup] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.",
"A dependency job for openvswitch.service failed. See 'journalctl -xe' for details.",
"2021-07-21T20:20:12Z|00002|fatal_signal|WARN|terminating with signal 14 (Alarm clock)",
"/etc/sysconfig/network-scripts/ifup-ovs: line 74: 7306 Alarm clock ovs-vsctl -t ${TIMEOUT} -- --if-exists del-port \"$OVS_BRIDGE\" \"$DEVICE\" -- add-port \"$OVS_BRIDGE\" \"$DEVICE\" $OVS_OPTIONS -- set Interface \"$DEVICE\" type=internal ${OVS_EXTRA+-- $OVS_EXTRA}",
"",
"[2021/07/21 04:22:08 PM] [ERROR] stdout: ERROR : [/etc/sysconfig/network-scripts/ifup-eth] Device vlan190 does not seem to be present, delaying initialization.",
", stderr: WARN : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.",
"WARN : [ifup] 'network-scripts' will be removed in one of the next major releases of RHEL.",
"WARN : [ifup] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.",
"A dependency job for openvswitch.service failed. See 'journalctl -xe' for details.",
"2021-07-21T20:20:23Z|00002|fatal_signal|WARN|terminating with signal 14 (Alarm clock)",
"/etc/sysconfig/network-scripts/ifup-ovs: line 74: 7589 Alarm clock ovs-vsctl -t ${TIMEOUT} -- --if-exists del-port \"$OVS_BRIDGE\" \"$DEVICE\" -- add-port \"$OVS_BRIDGE\" \"$DEVICE\" $OVS_OPTIONS -- set Interface \"$DEVICE\" type=internal ${OVS_EXTRA+-- $OVS_EXTRA}",
"",
...
Environment
- Red Hat OpenStack Platform 16.1 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.