Unable to deploy an overcloud node when using the software raid feature
Issue
-
We are unable to deploy an overcloud node when using the software raid feature.
-
We configured the following in
/var/lib/config-data/puppet-generated/ironic/etc/ironic/ironic.confandrestart ironic_conductor:
enabled_raid_interfaces=agent,idrac,no-raid
- We then ran the following script to configure the raid interface,
root_devicehint,logical_disksand cleaning steps for each nodes:
# cat soft_raid.sh
nodes="control-0-rhosp16 control-1-rhosp16 control-2-rhosp16 compute-0-rhosp16 compute-1-rhosp16"
for node in $nodes; do
openstack baremetal node set --raid-interface agent $node
openstack baremetal node manage $node
openstack baremetal node set $node --property root_device='{"name": "/dev/md127"}'
echo '{
"logical_disks": [
{
"size_gb": "MAX",
"raid_level": "1",
"controller": "software",
"is_root_volume": true,
"disk_type": "ssd"
}
]
}' | openstack baremetal node set $node --target-raid-config -
echo '[{
"interface": "raid",
"step": "delete_configuration"
},
{
"interface": "deploy",
"step": "erase_devices_metadata"
},
{
"interface": "raid",
"step": "create_configuration"
}]' | openstack baremetal node clean $node --clean-steps -
done
for node in $nodes; do
openstack overcloud node provide $node
done
- We tagged each nodes into their respective profiles using the following scripts:
# cat tag_hosts.sh
nodes="control-0-rhosp16 control-1-rhosp16 control-2-rhosp16 compute-0-rhosp16 compute-1-rhosp16"
boot_mode=bios
for p in $nodes; do
if [[ $p =~ control ]]; then
openstack baremetal node set --property capabilities="profile:control,boot_mode:${boot_mode}" $p
else
openstack baremetal node set --property capabilities="profile:compute,boot_mode:${boot_mode}" $p
fi
done
- We generate whole disk images using the following script:
# cat gen_image.sh
export DIB_LOCAL_IMAGE=./rhel-guest-image-8.4-1244.x86_64.qcow2
export REG_METHOD=portal
export REG_USER=<username>
export REG_PASSWORD=<password>
export REG_RELEASE="8.4"
export REG_POOL_ID=8a85f9a17af8e616017b0d59c6476885
export REG_REPOS="rhel-8-for-x86_64-baseos-eus-rpms \
rhel-8-for-x86_64-appstream-eus-rpms \
rhel-8-for-x86_64-highavailability-eus-rpms \
ansible-2.9-for-rhel-8-x86_64-rpms \
fast-datapath-for-rhel-8-x86_64-rpms \
openstack-16.1-for-rhel-8-x86_64-rpms"
export DIB_BLOCK_DEVICE_CONFIG='''
- local_loop:
name: image0
- partitioning:
base: image0
label: mbr
partitions:
- name: root
flags: [ boot,primary ]
size: 40G
- lvm:
name: lvm
base: [ root ]
pvs:
- name: pv
base: root
options: [ "--force" ]
vgs:
- name: vg
base: [ "pv" ]
options: [ "--force" ]
lvs:
- name: lv_root
base: vg
extents: 23%VG
- name: lv_tmp
base: vg
extents: 4%VG
- name: lv_var
base: vg
extents: 45%VG
- name: lv_log
base: vg
extents: 23%VG
- name: lv_audit
base: vg
extents: 4%VG
- name: lv_home
base: vg
extents: 1%VG
- mkfs:
name: fs_root
base: lv_root
type: xfs
label: "img-rootfs"
mount:
mount_point: /
fstab:
options: "rw,relatime"
fsck-passno: 1
- mkfs:
name: fs_tmp
base: lv_tmp
type: xfs
mount:
mount_point: /tmp
fstab:
options: "rw,nosuid,nodev,noexec,relatime"
fsck-passno: 2
- mkfs:
name: fs_var
base: lv_var
type: xfs
mount:
mount_point: /var
fstab:
options: "rw,relatime"
fsck-passno: 2
- mkfs:
name: fs_log
base: lv_log
type: xfs
mount:
mount_point: /var/log
fstab:
options: "rw,relatime"
fsck-passno: 3
- mkfs:
name: fs_audit
base: lv_audit
type: xfs
mount:
mount_point: /var/log/audit
fstab:
options: "rw,relatime"
fsck-passno: 4
- mkfs:
name: fs_home
base: lv_home
type: xfs
mount:
mount_point: /home
fstab:
options: "rw,nodev,relatime"
fsck-passno: 2
'''
openstack overcloud image build \
--image-name overcloud-hardened-full \
--config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-python3.yaml \
--config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-rhel8.yaml
- We upload the generated image to glance:
# cat upload_image.sh
if [ -e overcloud-hardened-full.qcow2 ]; then
mv overcloud-hardened-full.qcow2 overcloud-full.qcow2
fi
openstack overcloud image upload --image-path /home/stack/images --whole-disk
- The overcloud deployment will always faill with "No valid hosts found" and after uncompressing the deployment logs from
/var/log/containers/ironic/deploy/*.tar.gz, the following errors are seen injournal:
Dec 10 19:40:57 host-192-0-2-21 ironic-python-agent[631]: 2021-12-10 19:40:57.181 631 ERROR ironic_python_agent.extensions.image [-] Installing GRUB2 boot loader to device /dev/md127 failed with Unexpected error while running command.
Command: mount -o bind /dev /tmp/tmpbxx9z1aj/dev
Exit code: 32
Stdout: ''
Stderr: 'mount: /tmp/tmpbxx9z1aj/dev: mount point does not exist.\n'.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Dec 10 19:40:57 host-192-0-2-21 ironic-python-agent[631]: 2021-12-10 19:40:57.208 631 WARNING ironic_python_agent.extensions.image [-] Unable to umount vfat partitions. Error: Unexpected error while running command.
Command: chroot /tmp/tmpbxx9z1aj /bin/sh -c "umount -a -t vfat"
Exit code: 127
Stdout: ''
Stderr: "chroot: failed to run command '/bin/sh': No such file or directory\n": oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Dec 10 19:40:59 host-192-0-2-21 ironic-python-agent[631]: 2021-12-10 19:40:59.299 631 WARNING ironic_python_agent.extensions.image [-] Unable to umount /tmp/tmpbxx9z1aj/dev. Error: Unexpected error while running command.
Command: umount /tmp/tmpbxx9z1aj/dev
Exit code: 32
Stdout: ''
Stderr: 'umount: /tmp/tmpbxx9z1aj/dev: no mount point specified.\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Dec 10 19:41:01 host-192-0-2-21 ironic-python-agent[631]: 2021-12-10 19:41:01.894 631 WARNING ironic_python_agent.extensions.image [-] Unable to umount /tmp/tmpbxx9z1aj/proc. Error: Unexpected error while running command.
Command: umount /tmp/tmpbxx9z1aj/proc
Exit code: 32
Stdout: ''
Stderr: 'umount: /tmp/tmpbxx9z1aj/proc: no mount point specified.\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Dec 10 19:41:03 host-192-0-2-21 ironic-python-agent[631]: 2021-12-10 19:41:03.310 631 WARNING ironic_python_agent.extensions.image [-] Unable to umount /tmp/tmpbxx9z1aj/run. Error: Unexpected error while running command.
Command: umount /tmp/tmpbxx9z1aj/run
Exit code: 32
Stdout: ''
Stderr: 'umount: /tmp/tmpbxx9z1aj/run: no mount point specified.\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Dec 10 19:41:04 host-192-0-2-21 dhclient[821]: DHCPDISCOVER on ens8 to 255.255.255.255 port 67 interval 7 (xid=0xb0b07a7e)
Dec 10 19:41:05 host-192-0-2-21 ironic-python-agent[631]: 2021-12-10 19:41:05.188 631 WARNING ironic_python_agent.extensions.image [-] Unable to umount /tmp/tmpbxx9z1aj/sys. Error: Unexpected error while running command.
Command: umount /tmp/tmpbxx9z1aj/sys
Exit code: 32
Stdout: ''
Stderr: 'umount: /tmp/tmpbxx9z1aj/sys: no mount point specified.\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Dec 10 19:41:05 host-192-0-2-21 ironic-python-agent[631]: 2021-12-10 19:41:05.202 631 ERROR ironic_python_agent.extensions.image [-] Error setting up bootloader. Error Command execution failed: Installing GRUB2 boot loader to device /dev/md127 failed with Unexpected error while running command.
Command: mount -o bind /dev /tmp/tmpbxx9z1aj/dev
Exit code: 32
Stdout: ''
Stderr: 'mount: /tmp/tmpbxx9z1aj/dev: mount point does not exist.\n'.: ironic_python_agent.errors.CommandExecutionError: Command execution failed: Installing GRUB2 boot loader to device /dev/md127 failed with Unexpected error while running command.
Dec 10 19:41:05 host-192-0-2-21 ironic-python-agent[631]: 2021-12-10 19:41:05.222 631 ERROR root [-] Command execution error: Command execution failed: Installing GRUB2 boot loader to device /dev/md127 failed with Unexpected error while running command.
Command: mount -o bind /dev /tmp/tmpbxx9z1aj/dev
Exit code: 32
Stdout: ''
Stderr: 'mount: /tmp/tmpbxx9z1aj/dev: mount point does not exist.\n'.: ironic_python_agent.errors.CommandExecutionError: Command execution failed: Installing GRUB2 boot loader to device /dev/md127 failed with Unexpected error while running command.
Command: mount -o bind /dev /tmp/tmpbxx9z1aj/dev
Exit code: 32
Stdout: ''
Stderr: 'mount: /tmp/tmpbxx9z1aj/dev: mount point does not exist.\n'.
2021-12-10 19:41:05.222 631 ERROR root Traceback (most recent call last):
2021-12-10 19:41:05.222 631 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/image.py", line 488, in _install_grub2
2021-12-10 19:41:05.222 631 ERROR root _mount_for_chroot(path)
2021-12-10 19:41:05.222 631 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/image.py", line 697, in _mount_for_chroot
2021-12-10 19:41:05.222 631 ERROR root utils.execute('mount', '-o', 'bind', fs, path + fs)
2021-12-10 19:41:05.222 631 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/utils.py", line 82, in execute
2021-12-10 19:41:05.222 631 ERROR root return ironic_utils.execute(*cmd, **kwargs)
2021-12-10 19:41:05.222 631 ERROR root File "/usr/lib/python3.6/site-packages/ironic_lib/utils.py", line 99, in execute
2021-12-10 19:41:05.222 631 ERROR root result = processutils.execute(*cmd, **kwargs)
2021-12-10 19:41:05.222 631 ERROR root File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 431, in execute
2021-12-10 19:41:05.222 631 ERROR root cmd=sanitized_cmd)
2021-12-10 19:41:05.222 631 ERROR root oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-12-10 19:41:05.222 631 ERROR root Command: mount -o bind /dev /tmp/tmpbxx9z1aj/dev
2021-12-10 19:41:05.222 631 ERROR root Exit code: 32
2021-12-10 19:41:05.222 631 ERROR root Stdout: ''
2021-12-10 19:41:05.222 631 ERROR root Stderr: 'mount: /tmp/tmpbxx9z1aj/dev: mount point does not exist.\n'
2021-12-10 19:41:05.222 631 ERROR root
2021-12-10 19:41:05.222 631 ERROR root During handling of the above exception, another exception occurred:
2021-12-10 19:41:05.222 631 ERROR root
2021-12-10 19:41:05.222 631 ERROR root Traceback (most recent call last):
2021-12-10 19:41:05.222 631 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/base.py", line 256, in execute_command
2021-12-10 19:41:05.222 631 ERROR root result = ext.execute(command_part, **kwargs)
2021-12-10 19:41:05.222 631 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/base.py", line 208, in execute
2021-12-10 19:41:05.222 631 ERROR root return cmd(**kwargs)
2021-12-10 19:41:05.222 631 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/base.py", line 326, in wrapper
2021-12-10 19:41:05.222 631 ERROR root result = func(self, **command_params)
2021-12-10 19:41:05.222 631 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/image.py", line 947, in install_bootloader
2021-12-10 19:41:05.222 631 ERROR root prep_boot_part_uuid=prep_boot_part_uuid)
2021-12-10 19:41:05.222 631 ERROR root File "/usr/lib/python3.6/site-packages/ironic_python_agent/extensions/image.py", line 613, in _install_grub2
2021-12-10 19:41:05.222 631 ERROR root raise errors.CommandExecutionError(error_msg)
2021-12-10 19:41:05.222 631 ERROR root ironic_python_agent.errors.CommandExecutionError: Command execution failed: Installing GRUB2 boot loader to device /dev/md127 failed with Unexpected error while running command.
2021-12-10 19:41:05.222 631 ERROR root Command: mount -o bind /dev /tmp/tmpbxx9z1aj/dev
2021-12-10 19:41:05.222 631 ERROR root Exit code: 32
2021-12-10 19:41:05.222 631 ERROR root Stdout: ''
2021-12-10 19:41:05.222 631 ERROR root Stderr: 'mount: /tmp/tmpbxx9z1aj/dev: mount point does not exist.\n'.
2021-12-10 19:41:05.222 631 ERROR root
Environment
- Red Hat OpenStack Platform 16.2 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.
