Baremetal APIs don't work after updating RHOSP16 to 16.2.4 or 16.1.9 when iRMC driver is registered.
Environment
- Red Hat OpenStack Platform 16.1
- Red Hat OpenStack Platform 16.2
- iRMC driver is used as the power management driver
Issue
- Baremetal APIs don't work after updating RHOSP16 to 16.2.4
$ openstack overcloud node import nodes.json
Waiting for messages on queue 'tripleo' with no timeout.
Resource temporarily unavailable, please retry. (HTTP 503)
Exception registering nodes: Resource temporarily unavailable, please retry. (HTTP 503)
$
The following error messages are in /var/log/containers/ironic/ironic-conductor.log in the director node.
2022-12-13 14:05:05.993 8 ERROR oslo_service.service Traceback (most recent call last):
2022-12-13 14:05:05.993 8 ERROR oslo_service.service File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 810, in run_service
2022-12-13 14:05:05.993 8 ERROR oslo_service.service service.start()
2022-12-13 14:05:05.993 8 ERROR oslo_service.service File "/usr/lib/python3.6/site-packages/ironic/common/rpc_service.py", line 63, in start
2022-12-13 14:05:05.993 8 ERROR oslo_service.service self.manager.init_host(admin_context)
2022-12-13 14:05:05.993 8 ERROR oslo_service.service File "/usr/lib/python3.6/site-packages/ironic/conductor/base_manager.py", line 154, in init_host
2022-12-13 14:05:05.993 8 ERROR oslo_service.service hardware_types = driver_factory.hardware_types()
2022-12-13 14:05:05.993 8 ERROR oslo_service.service File "/usr/lib/python3.6/site-packages/ironic/common/driver_factory.py", line 247, in hardware_types
2022-12-13 14:05:05.993 8 ERROR oslo_service.service return _get_all_drivers(HardwareTypesFactory())
2022-12-13 14:05:05.993 8 ERROR oslo_service.service File "/usr/lib/python3.6/site-packages/ironic/common/driver_factory.py", line 317, in __init__
2022-12-13 14:05:05.993 8 ERROR oslo_service.service self.__class__._init_extension_manager()
2022-12-13 14:05:05.993 8 ERROR oslo_service.service File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2022-12-13 14:05:05.993 8 ERROR oslo_service.service return f(*args, **kwargs)
2022-12-13 14:05:05.993 8 ERROR oslo_service.service File "/usr/lib/python3.6/site-packages/ironic/common/driver_factory.py", line 387, in _init_extension_manager
2022-12-13 14:05:05.993 8 ERROR oslo_service.service on_missing_entrypoints_callback=missing_callback))
2022-12-13 14:05:05.993 8 ERROR oslo_service.service File "/usr/lib/python3.6/site-packages/stevedore/named.py", line 81, in __init__
2022-12-13 14:05:05.993 8 ERROR oslo_service.service verify_requirements)
2022-12-13 14:05:05.993 8 ERROR oslo_service.service File "/usr/lib/python3.6/site-packages/stevedore/extension.py", line 203, in _load_plugins
2022-12-13 14:05:05.993 8 ERROR oslo_service.service self._on_load_failure_callback(self, ep, err)
2022-12-13 14:05:05.993 8 ERROR oslo_service.service File "/usr/lib/python3.6/site-packages/ironic/common/driver_factory.py", line 372, in _catch_driver_not_found
2022-12-13 14:05:05.993 8 ERROR oslo_service.service raise exception.DriverLoadError(driver=ep.name, reason=exc)
2022-12-13 14:05:05.993 8 ERROR oslo_service.service ironic.common.exception.DriverLoadError: Driver, hardware type or interface irmc could not be loaded. Reason: No module named 'packaging'.
Resolution
Not fixed in RHOSP16.1 and RHOSP16.2 yet. The failure is tracked in Bug 2152623 (for RHOSP16.1) and Bug 2129136 (for RHOSP16.2).
Workaround
Before installing/updating to 16.2.4 or 16.1.9, do the following steps.
If the overcloud nodes don't use iRMC driver
- Remove irmc from in undercloud.conf. For example,
enabled_hardware_types = ipmi
-
Do openstack undercloud install
-
Confirm irmc driver isn't registered
(undercloud) [stack@localhost ~]$ openstack baremetal driver list
+---------------------+-----------------------+
| Supported driver(s) | Active host(s) |
+---------------------+-----------------------+
| ipmi | localhost.localdomain |
+---------------------+-----------------------+
(undercloud) [stack@localhost ~]$
- Follow the install or upgrade procedure
If the overcloud nodes use iRMC driver
- Edit containers-prepare-parameter.yaml not to update ironic-conductor container.
For RHOSP16.1
parameter_defaults:
ContainerImagePrepare:
- push_destination: true
excludes:
- ironic-conductor
set:
name_prefix: openstack-
name_suffix: ''
namespace: registry.redhat.io/rhosp-rhel8
neutron_driver: ovn
rhel_containers: false
tag: '16.1'
tag_from_label: '{version}-{release}'
- push_destination: true
includes:
- ironic-conductor
set:
name_prefix: openstack-
name_suffix: ''
namespace: registry.redhat.io/rhosp-rhel8
neutron_driver: ovn
rhel_containers: false
tag: '16.1.8'
tag_from_label: '{version}-{release}'
For RHOSP16.2
parameter_defaults:
ContainerImagePrepare:
- push_destination: true
excludes:
- ironic-conductor
set:
name_prefix: openstack-
name_suffix: ''
namespace: registry.redhat.io/rhosp-rhel8
neutron_driver: ovn
rhel_containers: false
tag: '16.2'
tag_from_label: '{version}-{release}'
- push_destination: true
includes:
- ironic-conductor
set:
name_prefix: openstack-
name_suffix: ''
namespace: registry.redhat.io/rhosp-rhel8
neutron_driver: ovn
rhel_containers: false
tag: '16.2.3'
tag_from_label: '{version}-{release}'
- Follow the install or upgrade procedure
Root Cause
The latest ironic-conductor container requires 'packaging' python package, but it's not included.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.