OpenStack stack update fails with "CREATE_FAILED resources.WorkflowTasks_Step5_Execution: Failure caused by error in tasks: nova_compute_discovery_workflow"

Issue

You are blacklisting nodes.

You are trying to do a stack update or a scale out and it fails with something like this:

overcloud.AllNodesDeploySteps.WorkflowTasks_Step5_Execution:
resource_type: OS::TripleO::WorkflowSteps
physical_resource_id: d85c8341-f416-4a81-9ac4-4de632c34533
status: UPDATE_FAILED
status_reason: |
...
deploy_config [task_ex_id=30029fec-741d-4fbd-a687-fef2631f25da] -> Timeout for heat deployment 'create_admin'
[action_ex_id=46e721e9-6ec8-4c3e-8ac4-e8e2387b14d5, idx=0]: Timeout for heat deployment 'create_admin'
send_message [task_ex_id=ca35b95b-4720-48dd-b706-f6e2d5d7e96b] -> Workflow failed due to message status
[wf_ex_id=d0d076f9-5749-4fef-b596-48cc2339ef28, idx=0]: Workflow failed due to message status

In /var/log/heat/heat-engine.log you see errors like these:

/var/log/mistral/engine.log:2022-02-03 15:50:10.181 2017 INFO workflow_trace [req-5b324349-45b9-48ca-9fc7-90c5e5b17dc5 dca64e7bc2854f1a94650535ec1cc234 d82d1bda58d34220a985219f3bae65ac - - -] Task 'create_admin_via_nova' (d5262c9d-b148-45df-a513-b3efcf9c882c) [RUNNING -> ERROR, msg=Failure caused by error in tasks: create_admin
/var/log/mistral/engine.log:2022-02-03 15:50:10.533 2017 INFO workflow_trace [req-5b324349-45b9-48ca-9fc7-90c5e5b17dc5 dca64e7bc2854f1a94650535ec1cc234 d82d1bda58d34220a985219f3bae65ac - - -] Workflow 'tripleo.access.v1.enable_ssh_admin' [RUNNING -> ERROR, msg=Failure caused by error in tasks: create_admin_via_nova
/var/log/mistral/engine.log:2022-02-03 15:50:10.658 2017 INFO workflow_trace [req-5b324349-45b9-48ca-9fc7-90c5e5b17dc5 dca64e7bc2854f1a94650535ec1cc234 d82d1bda58d34220a985219f3bae65ac - - -] Task 'enable_ssh_admin' (72d90bed-fe2f-4d66-9607-047b6a0c3e3e) [RUNNING -> ERROR, msg=Failure caused by error in tasks: create_admin_via_nova
/var/log/mistral/engine.log:2022-02-03 15:50:11.685 2017 INFO workflow_trace [req-5b324349-45b9-48ca-9fc7-90c5e5b17dc5 dca64e7bc2854f1a94650535ec1cc234 d82d1bda58d34220a985219f3bae65ac - - -] Workflow 'tripleo.nova.v1.cellv2_discovery' [RUNNING -> ERROR, msg=Failure caused by error in tasks: enable_ssh_admin
/var/log/mistral/engine.log:2022-02-03 15:50:11.731 2017 INFO workflow_trace [req-5b324349-45b9-48ca-9fc7-90c5e5b17dc5 dca64e7bc2854f1a94650535ec1cc234 d82d1bda58d34220a985219f3bae65ac - - -] Task 'nova_compute_discovery_workflow' (f568cfe0-c847-422f-8e2b-6f32f2586855) [RUNNING -> ERROR, msg=Failure caused by error in tasks: enable_ssh_admin
/var/log/mistral/engine.log:2022-02-03 15:53:14.129 2017 INFO workflow_trace [req-5b324349-45b9-48ca-9fc7-90c5e5b17dc5 dca64e7bc2854f1a94650535ec1cc234 d82d1bda58d34220a985219f3bae65ac - - -] Workflow 'tripleo.cirp03osp.workflow_tasks.step5' [RUNNING -> ERROR, msg=Failure caused by error in tasks: nova_compute_discovery_workflow