New Build puppet stuck looping

Issue

• Created the network / vip's / and created 3 controllers / 2 compute hosts with no problems.

• On the overcloud hosts we can see the podman images have been download

• When we run the "openstack overcloud deploy" it's stuck "WAITING | Wait for puppet host configuration to finish" to the 3 controller hosts to finally fail with this:

2025-06-30 12:46:18,611 p=284062 u=stack n=ansible | 2025-06-30 12:46:18.611281 | 52540008-08af-20a6-b8db-000000009ef2 |      FATAL | Wait for puppet host configuration to finish | overcloud-controller-0 | error={"ansible_job_id": "983448111829.89657", "attempts": 102, "changed": true, "cmd": "set -o pipefail; puppet apply  --modulepath=/etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --detailed-exitcodes --summarize --color=false   /var/lib/tripleo-config/puppet_step_config.pp 2>&1 | logger -s -t puppet-user", "delta": "0:17:01.809647", "end": "2025-06-30 12:46:13.251621", "failed_when_result": true, "finished": 1, "msg": "non-zero return code", "rc": 4, "results_file": "/tmp/.ansible_async/983448111829.89657", "start": "2025-06-30 12:29:11.441974", "started": 1, "stderr": "<13>Jun 30 12:29:11 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
<13>Jun 30 12:29:13 puppet-user:    (file: /etc/puppet/hiera.yaml)
<13>Jun 30 12:29:13 puppet-user: Warning: Undefined variable '::deploy_config_name';
<13>Jun 30 12:29:13 puppet-user:    (file & line not available)
<13>Jun 30 12:29:13 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.10/deprecated_language.html
<13>Jun 30 12:29:13 puppet-user:    (file & line not available)
<13>Jun 30 12:29:13 puppet-user: Warning: Unknown variable: '::deployment_type'. (file: /etc/puppet/modules/tripleo/manifests/profile/base/database/mysql/client.pp, line: 89, column: 8)
<13>Jun 30 12:29:13 puppet-user: error: Could not connect to controller: Connection refused
<13>Jun 30 12:29:13 puppet-user: Warning: Unknown variable: '::deployment_type'. (file: /etc/puppet/modules/tripleo/manifests/packages.pp, line: 39, column: 69)
<13>Jun 30 12:29:13 puppet-user: Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 0.27 seconds
<13>Jun 30 12:46:13 puppet-user: Notice: /Stage[main]/Pacemaker::Corosync/Exec[Create Cluster tripleo_cluster]/returns: Warning: Unable to read the known-hosts file: No such file or directory: '/var/lib/pcsd/known-hosts'
<13>Jun 30 12:46:13 puppet-user: Notice: /Stage[main]/Pacemaker::Corosync/Exec[Create Cluster tripleo_cluster]/returns: Error: Hosts 'overcloud-controller-0', 'overcloud-controller-1', 'overcloud-controller-2' are not known to pcs, try to authenticate the hosts using 'pcs host auth overcloud-controller-0 overcloud-controller-1 overcloud-controller-2' command
<13>Jun 30 12:46:13 puppet-user: Notice: /Stage[main]/Pacemaker::Corosync/Exec[Create Cluster tripleo_cluster]/returns: Error: None of hosts is known to pcs.
<13>Jun 30 12:46:13 puppet-user: Notice: /Stage[main]/Pacemaker::Corosync/Exec[Create Cluster tripleo_cluster]/returns: Error: Errors have occurred, therefore pcs is unable to continue
<13>Jun 30 12:46:13 puppet-user: Error: '/sbin/pcs cluster setup tripleo_cluster overcloud-controller-0 addr=10.10.10.127 overcloud-controller-1 addr=10.10.10.25 overcloud-controller-2 addr=10.10.10.186 totem token=10000 ' returned 1 instead of one of [0]
<13>Jun 30 12:46:13 puppet-user: Error: /Stage[main]/Pacemaker::Corosync/Exec[Create Cluster tripleo_cluster]/returns: change from 'notrun' to ['0'] failed: '/sbin/pcs cluster setup tripleo_cluster overcloud-controller-0 addr=10.10.10.127 overcloud-controller-1 addr=10.10.10.25 overcloud-controller-2 addr=10.10.10.186 totem token=10000 ' returned 1 instead of one of [0]
<13>Jun 30 12:46:13 puppet-user: Notice: /Stage[main]/Pacemaker::Corosync/Exec[Start Cluster tripleo_cluster]: Dependency Exec[Create Cluster tripleo_cluster] has failures: true
<13>Jun 30 12:46:13 puppet-user: Warning: /Stage[main]/Pacemaker::Corosync/Exec[Start Cluster tripleo_cluster]: Skipping because of failed dependencies
<13>Jun 30 12:46:13 puppet-user: Warning: /Stage[main]/Pacemaker::Service/Service[corosync]: Skipping because of failed dependencies
<13>Jun 30 12:46:13 puppet-user: Warning: /Stage[main]/Pacemaker::Service/Service[pacemaker]: Skipping because of failed dependencies
<13>Jun 30 12:46:13 puppet-user: Warning: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]: Skipping because of failed dependencies
<13>Jun 30 12:46:13 puppet-user: Warning: /Stage[main]/Pacemaker::Stonith/Pacemaker::Property[Disable STONITH]/Pcmk_property[property--stonith-enabled]: Skipping because of failed dependencies
<13>Jun 30 12:46:13 puppet-user: Notice: Applied catalog in 1019.28 seconds
<13>Jun 30 12:46:13 puppet-user: Application:
<13>Jun 30 12:46:13 puppet-user:    Initial environment: production
<13>Jun 30 12:46:13 puppet-user:    Converged environment: production
<13>Jun 30 12:46:13 puppet-user:          Run mode: user
<13>Jun 30 12:46:13 puppet-user: Changes:
<13>Jun 30 12:46:13 puppet-user: Events:
<13>Jun 30 12:46:13 puppet-user:           Failure: 1
<13>Jun 30 12:46:13 puppet-user:             Total: 1
<13>Jun 30 12:46:13 puppet-user: Resources:
<13>Jun 30 12:46:13 puppet-user:            Failed: 1
<13>Jun 30 12:46:13 puppet-user:       Out of sync: 1
<13>Jun 30 12:46:13 puppet-user:           Skipped: 5
<13>Jun 30 12:46:13 puppet-user:             Total: 26
<13>Jun 30 12:46:13 puppet-user: Time:
<13>Jun 30 12:46:13 puppet-user:          Schedule: 0.00
<13>Jun 30 12:46:13 puppet-user:         File line: 0.00
<13>Jun 30 12:46:13 puppet-user:           Package: 0.00
<13>Jun 30 12:46:13 puppet-user:              File: 0.00
<13>Jun 30 12:46:13 puppet-user:              User: 0.00
<13>Jun 30 12:46:13 puppet-user:            Augeas: 0.01
<13>Jun 30 12:46:13 puppet-user:           Service: 0.01