Heat stack update hangs and reaches timeout during openstack overcloud deploy

Issue

• openstack overcloud deploy hangs while updating the Heat stack, and ends up with timeout.

  $ openstack overcloud deploy ....
    :
  2023-01-01 00:34:08Z [overcloud-ComputeLeaf2-xxxxxxxxxxxx-3-xxxxxxxxxxxx.ManagementPort]: UPDATE_IN_PROGRESS  state changed
  2023-01-01 00:34:10Z [overcloud-ComputeLeaf2-xxxxxxxxxxxx-4-xxxxxxxxxxxx.TenantPort]: UPDATE_IN_PROGRESS  state changed
  2023-01-01 00:41:02Z [overcloud-Compute-xxxxxxxxxxxx.8.Compute]: CREATE_COMPLETE  state changed
  2023-01-01 00:41:05Z [overcloud-Compute-xxxxxxxxxxxx-6-xxxxxxxxxxxx.InternalApiPort]: UPDATE_COMPLETE  state changed
  2023-01-01 00:41:07Z [overcloud-ComputeLeaf2-xxxxxxxxxxxx-0-xxxxxxxxxxxx.NetworkConfig]: UPDATE_COMPLETE  state changed
  
   ==> Updating Heat stack hangs, and reaches timeout.
  
  2023-01-01 04:28:18Z [ContrailController]: UPDATE_FAILED  UPDATE aborted (Task update from ResourceGroup "ContrailController" [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] Stack "overcloud" [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] Timed out)
  2023-01-01 04:28:18Z [overcloud-ContrailController-xxxxxxxxxxxx]: UPDATE_FAILED  Stack UPDATE cancelled
  2023-01-01 04:28:18Z [ComputeLeaf2]: UPDATE_FAILED  UPDATE aborted (Task update from ResourceGroup "ComputeLeaf2" [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] Stack "overcloud" [xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] Timed out)
  2023-01-01 04:28:18Z [overcloud]: UPDATE_FAILED  Timed out
  2023-01-01 04:28:18Z [overcloud]: UPDATE_FAILED  Timed out
    :
  

• openstack stack failures list <OVERCLOUD_NAME> --long shows a lot of failed resources, and all of the failed resources shows Stack UPDATE cancelled or Stack CREATE cancelled.

  $ openstack stack failures list overcloud --long
  overcloud.Compute.1.PreNetworkConfig.DockerContrailVrouterAgentDpdkImageNormalize:  
    resource_type: https://10.0.0.1:13808/v1/AUTH_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/overcloud/user-files/home/stack/overcloud/templates/tools/contrail/contrail-image-normalize.yaml
    physical_resource_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    status: UPDATE_FAILED
    status_reason: |   
      resources.DockerContrailVrouterAgentDpdkImageNormalize: Stack UPDATE cancelled
  overcloud.Compute.1.ComputeExtraConfigPre.ContrailNodeInit:            
    resource_type: https://10.0.0.1:13808/v1/AUTH_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/overcloud/user-files/home/stack/overcloud/templates/extraconfig/pre_deploy/contrail/contrail-node-init.yaml
    physical_resource_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    status: UPDATE_FAILED  
    status_reason: |                           
      resources.ContrailNodeInit: Stack UPDATE cancelled
  overcloud.Compute.0.PreNetworkConfig.DockerContrailVrouterPluginInitImageNormalize:  
    resource_type: https://10.0.0.1:13808/v1/AUTH_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/overcloud/user-files/home/stack/overcloud/templates/tools/contrail/contrail-image-normalize.yaml
    physical_resource_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    status: UPDATE_FAILED
    status_reason: |      
      resources.DockerContrailVrouterPluginInitImageNormalize: Stack UPDATE cancelled
  overcloud.Compute.0.ComputeExtraConfigPre.ContrailNodeInit:            
    resource_type: https://10.0.0.1:13808/v1/AUTH_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/overcloud/user-files/home/stack/overcloud/templates/extraconfig/pre_deploy/contrail/contrail-node-init.yaml
    physical_resource_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    status: UPDATE_FAILED            
    status_reason: |
      resources.ContrailNodeInit: Stack UPDATE cancelled
  overcloud.Compute.2.ComputeExtraConfigPre.ContrailNodeInit:            
    resource_type: https://10.0.0.1:13808/v1/AUTH_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/overcloud/user-files/home/stack/overcloud/templates/extraconfig/pre_deploy/contrail/contrail-node-init.yaml
    physical_resource_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    status: UPDATE_FAILED
    status_reason: |     
      resources.ContrailNodeInit: Stack UPDATE cancelled
          :