Deployment with pre-provisioned nodes fails at step 2 with mysql_wait_bundle related errors

Issue

Deployment in pre-provisioned nodes fails at "Wait for containers to start for step 2 using paunch" in controller node with errors related to podman.

The most important snippet of the error is:

Error executing ['podman', 'run', '--name', 'mysql_wait_bundle', '--label', 'config_id=tripleo_step2', '--label', 'container_name=mysql_wait_bundle',...
...
Error: Failed to apply catalog: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

The most relevant parts of the FATAL line in the ansible log are:

2024-03-11 06:56:17,056 p=194813 u=mistral n=ansible | 2024-03-11 06:56:17.055616 | 005056a8-5afe-3982-619d-00000000553f |      FATAL | Wait for containers to start for step 2 using paunch | osp-controller-0 | error={"ansible_job_id": "825645371258.258538",
"attempts": 6, "changed": false, "finished": 1, "msg": "Paunch failed with config_id tripleo_step2",
"rc": 1, "stderr": "Error executing ['podman', 'container', 'exists', 'mysql_wait_bundle']: returned 1
Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mysql_wait_bundle', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id
Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mysql_wait_bundle', '--format', '{{.Names}}']\"
Error executing ['podman', 'run', '--name', 'mysql_wait_bundle', '--label', 'config_id=tripleo_step2', '--label', 'container_name=mysql_wait_bundle', '--label', 'managed_by=tripleo-Controller', '--label', 'config_data={\"command\": [\"/container_puppet_apply.sh\", \"2\", \"file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user\", \"include tripleo::profile::pacemaker::database::mysql_bundle\", \"\"], \"detach\": false, \"environment\": {\"TRIPLEO_DEPLOY_IDENTIFIER\": \"1710152976\", \"TRIPLEO_CONFIG_HASH\": \"7e1109fb74c4e620928b4e96ba5b8a54\"}, \"image\": \"director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-mariadb:16.2\", \"ipc\": \"host\", \"net\": \"host\", \"start_order\": 0, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/var/lib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro\", \"/etc/puppet:/tmp/puppet-etc:ro\", \"/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro\", \"/var/lib/mysql:/var/lib/mysql:rw,z\", \"/var/lib/config-data/puppet-generated/mysql/root:/root:rw\"]}', '--conmon-pidfile=/var/run/mysql_wait_bundle.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/mysql_wait_bundle.log', '--env=TRIPLEO_CONFIG_HASH=7e1109fb74c4e620928b4e96ba5b8a54', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1710152976', '--net=host', '--ipc=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/var/lib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro', '--volume=/etc/puppet:/tmp/puppet-etc:ro', '--volume=/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro', '--volume=/var/lib/mysql:/var/lib/mysql:rw,z', '--volume=/var/lib/config-data/puppet-generated/mysql/root:/root:rw', 'director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-mariadb:16.2', '/container_puppet_apply.sh', '2', 'file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user', 'include tripleo::profile::pacemaker::database::mysql_bundle', '']: returned 1
Error running ['podman', 'run', '--name', 'mysql_wait_bundle', '--label', 'config_id=tripleo_step2', '--label', 'container_name=mysql_wait_bundle', '--label', 'managed_by=tripleo-Controller', '--label', 'config_data={\"command\": [\"/container_puppet_apply.sh\", \"2\", \"file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user\", \"include tripleo::profile::pacemaker::database::mysql_bundle\", \"\"], \"detach\": false, \"environment\": {\"TRIPLEO_DEPLOY_IDENTIFIER\": \"1710152976\", \"TRIPLEO_CONFIG_HASH\": \"7e1109fb74c4e620928b4e96ba5b8a54\"}, \"image\": \"director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-mariadb:16.2\", \"ipc\": \"host\", \"net\": \"host\", \"start_order\": 0, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/var/lib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro\", \"/etc/puppet:/tmp/puppet-etc:ro\", \"/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro\", \"/var/lib/mysql:/var/lib/mysql:rw,z\", \"/var/lib/config-data/puppet-generated/mysql/root:/root:rw\"]}', '--conmon-pidfile=/var/run/mysql_wait_bundle.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/mysql_wait_bundle.log', '--env=TRIPLEO_CONFIG_HASH=7e1109fb74c4e620928b4e96ba5b8a54', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1710152976', '--net=host', '--ipc=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/var/lib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro', '--volume=/etc/puppet:/tmp/puppet-etc:ro', '--volume=/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro', '--volume=/var/lib/mysql:/var/lib/mysql:rw,z', '--volume=/var/lib/config-data/puppet-generated/mysql/root:/root:rw', 'director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-mariadb:16.2', '/container_puppet_apply.sh', '2', 'file,file_line,concat,augeas,galera_ready,mysql_database,mysql_grant,mysql_user', 'include tripleo::profile::pacemaker::database::mysql_bundle', '']. [1]
...
Error: Failed to apply catalog: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
...
Error executing ['podman', 'container', 'exists', 'nova_api_init_logs']: returned 1
Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_api_init_logs', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id
Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_api_init_logs', '--format', '{{.Names}}']\"
Error executing ['podman', 'container', 'exists', 'nova_conductor_init_log']: returned 1
Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_conductor_init_log', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id
Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_conductor_init_log', '--format', '{{.Names}}']\"
Error executing ['podman', 'container', 'exists', 'rabbitmq_wait_bundle']: returned 1
Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=rabbitmq_wait_bundle', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id
Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=rabbitmq_wait_bundle', '--format', '{{.Names}}']\"
...