FFU 13->16.2 overcloud upgrade run getting stuck on task 'Wait for puppet host configuration to finish' on final controller

Issue

• In this environment 2 of the 3 controllers were upgraded to 16.2. On the final controller (003) it's failing with:

2022-07-05 13:18:21,844 p=573364 u=mistral n=ansible | 2022-07-05 13:18:21.844470 | 525400c7-f74c-f527-c2f6-000000003323 |      FATAL | Wait for puppet host configuration to finish | ctr-003 | error={"ansible_job_id": "273882328474.77112", "attempts": 1200, "changed": false, "failed_when_result": true, "finished": 0, "started": 1}

After hitting this failure pcs status on this controller showed:

WARNINGS:
No stonith devices and stonith-enabled is not false
...
Node List:
  * Node ctr-001 (1): UNCLEAN (offline)
  * Node ctr-002 (2): UNCLEAN (offline)
  * Online: [ ctr-003 (3) ]

However the other controllers did not have that warning and also do not see this controller:

Node List:
  * Online: [ ctr-001 (1) ctr-002 (2) ]
  * GuestOnline: [ galera-bundle-0@ctr-001 galera-bundle-1@ctr-002 rabbitmq-bundle-0@ctr-001 rabbitmq-bundle-1@ctr-002 redis-bundle-0@ctr-001 redis-bundle-1@ctr-002 ]