Baremetal APIs don't work after updating RHOSP16 to 16.2.4 or 16.1.9 when iRMC driver is registered.

Solution In Progress - Updated -

Environment

  • Red Hat OpenStack Platform 16.1
  • Red Hat OpenStack Platform 16.2
  • iRMC driver is used as the power management driver

Issue

  • Baremetal APIs don't work after updating RHOSP16 to 16.2.4

Resolution

Not fixed in RHOSP16.1 and RHOSP16.2 yet. The failure is tracked in Bug 2152623 (for RHOSP16.1) and Bug 2129136 (for RHOSP16.2).

Workaround

Before installing/updating to 16.2.4 or 16.1.9, follow these steps.

If the overcloud nodes don't use iRMC driver

  1. Remove irmc from in undercloud.conf on the director node. For example, replace it with ipmi.
enabled_hardware_types = ipmi

  1. Execute openstack undercloud install on the director node.

  2. Confirm irmc driver isn't registered

(undercloud) [stack@localhost ~]$ openstack baremetal driver list
+---------------------+-----------------------+
| Supported driver(s) | Active host(s)        |
+---------------------+-----------------------+
| ipmi                | localhost.localdomain |
+---------------------+-----------------------+
(undercloud) [stack@localhost ~]$
  1. Follow the install or upgrade procedure
    For RHOSP16.1, please follow these steps and these as well
    For RHOSP16.2, please follow these steps and thse steps as well

If the overcloud nodes use iRMC driver

  1. Edit containers-prepare-parameter.yaml not to update ironic-conductor container.

For RHOSP16.1

parameter_defaults:
  ContainerImagePrepare:
  - push_destination: true
    excludes:
      - ironic-conductor
    set:
      name_prefix: openstack-
      name_suffix: ''
      namespace: registry.redhat.io/rhosp-rhel8
      neutron_driver: ovn
      rhel_containers: false
      tag: '16.1'
    tag_from_label: '{version}-{release}'
  - push_destination: true
    includes:
      - ironic-conductor
    set:
      name_prefix: openstack-
      name_suffix: ''
      namespace: registry.redhat.io/rhosp-rhel8
      neutron_driver: ovn
      rhel_containers: false
      tag: '16.1.8'
    tag_from_label: '{version}-{release}'

For RHOSP16.2

parameter_defaults:
  ContainerImagePrepare:
  - push_destination: true
    excludes:
      - ironic-conductor
    set:
      name_prefix: openstack-
      name_suffix: ''
      namespace: registry.redhat.io/rhosp-rhel8
      neutron_driver: ovn
      rhel_containers: false
      tag: '16.2'
    tag_from_label: '{version}-{release}'
  - push_destination: true
    includes:
      - ironic-conductor
    set:
      name_prefix: openstack-
      name_suffix: ''
      namespace: registry.redhat.io/rhosp-rhel8
      neutron_driver: ovn
      rhel_containers: false
      tag: '16.2.3'
    tag_from_label: '{version}-{release}'
  1. Follow the install or upgrade procedure.
    For RHOSP16.1, please follow these steps and these as well.
    For RHOSP16.2, please follow these steps and these as well.

Root Cause

The reason of the issue is that packages newer than 13.0.8-2.20220929054719.d31cae6 ironic-conductor container requires packaging python package, but said dependece is not being included nor installed by default.

Diagnostic Steps

  • Baremetal APIs don't work after updating RHOSP16 to 16.2.4, Resource temporarily unavailable is output as below.
  $ openstack overcloud node import nodes.json
  Waiting for messages on queue 'tripleo' with no timeout.

  Resource temporarily unavailable, please retry. (HTTP 503)
  Exception registering nodes: Resource temporarily unavailable, please retry. (HTTP 503)
  $

The following error messages can be seen in /var/log/containers/ironic/ironic-conductor.log on the director node.

  2022-12-13 14:05:05.993 8 ERROR oslo_service.service Traceback (most recent call last):
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service   File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 810, in run_service
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service     service.start()
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service   File "/usr/lib/python3.6/site-packages/ironic/common/rpc_service.py", line 63, in start
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service     self.manager.init_host(admin_context)
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service   File "/usr/lib/python3.6/site-packages/ironic/conductor/base_manager.py", line 154, in init_host
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service     hardware_types = driver_factory.hardware_types()
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service   File "/usr/lib/python3.6/site-packages/ironic/common/driver_factory.py", line 247, in hardware_types
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service     return _get_all_drivers(HardwareTypesFactory())
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service   File "/usr/lib/python3.6/site-packages/ironic/common/driver_factory.py", line 317, in __init__
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service     self.__class__._init_extension_manager()
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service     return f(*args, **kwargs)
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service   File "/usr/lib/python3.6/site-packages/ironic/common/driver_factory.py", line 387, in _init_extension_manager
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service     on_missing_entrypoints_callback=missing_callback))
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service   File "/usr/lib/python3.6/site-packages/stevedore/named.py", line 81, in __init__
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service     verify_requirements)
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service   File "/usr/lib/python3.6/site-packages/stevedore/extension.py", line 203, in _load_plugins
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service     self._on_load_failure_callback(self, ep, err)
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service   File "/usr/lib/python3.6/site-packages/ironic/common/driver_factory.py", line 372, in _catch_driver_not_found
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service     raise exception.DriverLoadError(driver=ep.name, reason=exc)
  2022-12-13 14:05:05.993 8 ERROR oslo_service.service ironic.common.exception.DriverLoadError: Driver, hardware type or interface irmc could not be loaded. Reason: No module named 'packaging'.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments