FFU 13 to 16.1 fails with error 'Cannot find config file: /etc/puppet/hiera.yaml' when starting up containers

Solution Verified - Updated -

Environment

  • Red Hat Openstack Platform (RHOSP) 16.1

Issue

  • Containers fail to start during Undercloud/Overcloud FFU from OSP13 to OSP16.1
  • The error found in the containers is: stderr F Cannot find config file: /etc/puppet/hiera.yaml

Resolution

  • Follow the following steps to get the right DNF module, packages and solve the issue:

    • Disable the current container-tools DNF stream and enable container-tools:2.0:

        sudo dnf -y module disable container-tools:rhel8
  sudo dnf -y module enable container-tools:2.0

    • Run distro-sync to synchronize the modules content:

        sudo dnf distro-sync

    • Reinstall openstack-selinux:

        sudo dnf reinstall openstack-selinux

    • Create /.autorelabel and reboot

        sudo touch /.autorelabel
  sudo reboot

    • Relaunch the failed command, either openstack undercloud upgrade or openstack overcloud upgrade run --limit .

Root Cause

When upgrading from OSP13 to OSP16.1 it is suggested to set up the right container-tools:2.0 DNF stream 6.2. Enabling repositories for the undercloud. However, if this step is skipped the user might end up performing the upgrade using container-tools:rhel8, which provides a different Podman and containers-selinux package version than container-tools:2.0.
This situation will make the upgrade fail with the following error when trying to bring up the new OSP16.1 containers:

stderr F Cannot find config file: /etc/puppet/hiera.yaml

Diagnostic Steps

  • Most of my containers failed to start
[root@undercloud-0 stdouts]# podman ps --all                                                                                                                        [76/1982]
CONTAINER ID  IMAGE                                                                                                  COMMAND               CREATED      STATUS               
   PORTS  NAMES
daef301f70eb  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-conductor:16.1_20200428.1       /usr/bin/bootstra...  6 hours ago  Exited (1) 6 hours ag
o         nova_db_sync
5374da1bd3fc  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-api:16.1_20200428.1             /usr/bin/bootstra...  6 hours ago  Exited (1) 6 hours ag
o         nova_api_ensure_default_cell
a13c05e326f0  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-placement-api:16.1_20200428.1        /usr/bin/bootstra...  6 hours ago  Exited (1) 6 hours ag
o         placement_api_db_sync
49d24235f731  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-api:16.1_20200428.1             /usr/bin/bootstra...  6 hours ago  Exited (1) 6 hours ag
o         nova_api_map_cell0
8b378374f33c  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-placement-api:16.1_20200428.1        /usr/bin/bootstra...  6 hours ago  Exited (1) 6 hours ag
o         placement_api_db_extract_data_from_nova_api
8a740f842bca  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-api:16.1_20200428.1             /usr/bin/bootstra...  6 hours ago  Exited (1) 6 hours ag
o         nova_api_db_sync
240a00abd8b1  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-ironic-inspector:16.1_20200428.1     true                  6 hours ago  Exited (0) 6 hours ag
o         ironic_inspector_get_ipa
a731b33c15d1  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-ironic-inspector:16.1_20200428.1     /usr/bin/bootstra...  6 hours ago  Exited (1) 6 hours ag
o         ironic_inspector_db_sync
fc916fa48d04  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp15-openstack-nova-api:20200115.1                  /usr/bin/bootstra...  6 hours ago  Exited (1) 6 hours ag
o         nova_api_online_data_migrations_stein
5847bc213029  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-keystone:16.1_20200428.1             kolla_start           6 hours ago  Up 6 hours ago       
          keystone
7c620a465131  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-iscsid:16.1_20200428.1               kolla_start           6 hours ago  Up 6 hours ago       
          iscsid
1e774c12b251  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp15-openstack-ironic-api:20200115.1                /usr/bin/bootstra...  6 hours ago  Exited (1) 6 hours ag
o         ironic_online_data_migrations_stein
c96cec68abd1  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp15-openstack-ironic-inspector:20200115.1          /usr/bin/bootstra...  6 hours ago  Exited (1) 6 hours ag
o         ironic_inspector_db_sync_stein
3e3d5948dcbc  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp15-openstack-ironic-api:20200115.1                /usr/bin/bootstra...  6 hours ago  Exited (1) 6 hours ag
o         ironic_db_sync_stein
3d13a9ea28a0  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-ironic-api:16.1_20200428.1           /usr/bin/bootstra...  6 hours ago  Exited (1) 6 hours ag
o         ironic_db_sync
4ac3a3602ecd  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-account:16.1_20200428.1        chown -R swift: /...  6 hours ago  Exited (0) 6 hours ag
o         swift_setup_srv
660ac674bfe1  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-object:16.1_20200428.1         /bin/bash -c sed ...  6 hours ago  Exited (0) 6 hours ag
o         swift_rsync_fix
bb5b2754bafb  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp15-openstack-nova-conductor:20200115.1            /bin/bash -c if /...  6 hours ago  Exited (0) 6 hours ag
o         nova_db_sync_stein
  • When checking the log of some of the failed containers in /var/log/containers/stdouts I find:
2020-04-29T10:42:11.995417386-04:00 stderr F Cannot find config file: /etc/puppet/hiera.yaml

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments