Overcloud controllers not getting the TLS cert file

Issue

• The /etc/pki/tls/private/overcloud_endpoint.pem is not bein created even though we added the relevant files from documentation for SSL.

• Overcloud deployment fails with the following errors :

# openstack stack failures list overcloud
overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.1:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: b515c6a9-4ba4-4f62-b57d-5a756de417b7
  status: CREATE_FAILED
  status_reason: |
    Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
            "2020-05-22 19:37:59,709 INFO: 54435 -- Removing container: docker-puppet-neutron",
            "2020-05-22 19:37:59,846 INFO: 54435 -- Finished processing puppet configs for neutron",
            "2020-05-22 19:37:59,846 ERROR: 54432 -- ERROR configuring haproxy"
        ]
    }
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/0487ddd1-f485-4275-9d17-d064d162cdbd_playbook.retry

    PLAY RECAP *********************************************************************
    localhost                  : ok=32   changed=18   unreachable=0    failed=1

    (truncated, view all with --long)
  deploy_stderr: |

overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: fd177583-1bac-4952-9d76-7891766b271a
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
            "2020-05-22 19:37:59,591 INFO: 55109 -- Finished processing puppet configs for neutron",
            "2020-05-22 19:37:59,605 INFO: 55111 -- Finished processing puppet configs for heat_api_cfn",
            "2020-05-22 19:37:59,606 ERROR: 55106 -- ERROR configuring haproxy"
        ]
    }
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/390d4959-a4b1-4197-94ea-df9d1618a5dc_playbook.retry

    PLAY RECAP *********************************************************************
    localhost                  : ok=33   changed=19   unreachable=0    failed=1

    (truncated, view all with --long)
  deploy_stderr: |

overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.2:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: ea18b9e1-71ce-40b6-a72c-39cc5ea85b66
  status: CREATE_FAILED
  status_reason: |
    Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
            "2020-05-22 19:37:39,601 INFO: 36139 -- Removing container: docker-puppet-neutron",
            "2020-05-22 19:37:39,659 INFO: 36139 -- Finished processing puppet configs for neutron",
            "2020-05-22 19:37:39,659 ERROR: 36134 -- ERROR configuring haproxy"
        ]
    }
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/b56d8605-c0c8-46ac-954d-bffb32c08d53_playbook.retry

    PLAY RECAP *********************************************************************
    localhost                  : ok=32   changed=18   unreachable=0    failed=1

    (truncated, view all with --long)
  deploy_stderr: |

• Why don't we see NodeTLSData being deployed as the below shows:

# openstack stack event list --nested-depth 10 overcloud | grep -i tls | grep -v NodeTLSCAData
2020-05-22 14:20:17Z [overcloud.ControllerServiceChain.ServiceChain.53.HAProxyBase.LoadbalancerServiceBase.HAProxyPublicTLS]: CREATE_IN_PROGRESS  state changed
2020-05-22 14:20:17Z [overcloud.ControllerServiceChain.ServiceChain.53.HAProxyBase.LoadbalancerServiceBase.HAProxyPublicTLS]: CREATE_COMPLETE  state changed
2020-05-22 14:20:19Z [overcloud.ControllerServiceChain.ServiceChain.53.HAProxyBase.LoadbalancerServiceBase.HAProxyInternalTLS]: CREATE_IN_PROGRESS  state changed
2020-05-22 14:20:19Z [overcloud.ControllerServiceChain.ServiceChain.53.HAProxyBase.LoadbalancerServiceBase.HAProxyInternalTLS]: CREATE_COMPLETE  state changed
2020-05-22 14:20:22Z [overcloud.ControllerServiceChain.ServiceChain.48.GlanceApiPuppetBase.TLSProxyBase]: CREATE_IN_PROGRESS  state changed
2020-05-22 14:20:22Z [overcloud.ControllerServiceChain.ServiceChain.48.GlanceApiPuppetBase.TLSProxyBase]: CREATE_COMPLETE  state changed
2020-05-22 14:21:08Z [overcloud.ControllerServiceChain.ServiceChain.87.NeutronBase.TLSProxyBase]: CREATE_IN_PROGRESS  state changed
2020-05-22 14:21:08Z [overcloud.ControllerServiceChain.ServiceChain.87.NeutronBase.TLSProxyBase]: CREATE_COMPLETE  state changed
2020-05-22 14:21:34Z [overcloud.ControllerServiceChain.ServiceChain.107.NovaMetadataBase.TLSProxyBase]: CREATE_IN_PROGRESS  state changed
2020-05-22 14:21:35Z [overcloud.ControllerServiceChain.ServiceChain.107.NovaMetadataBase.TLSProxyBase]: CREATE_COMPLETE  state changed
2020-05-22 14:22:15Z [overcloud.ControllerServiceChain.ServiceChain.137.SwiftProxyBase.TLSProxyBase]: CREATE_IN_PROGRESS  state changed
2020-05-22 14:22:15Z [overcloud.ControllerServiceChain.ServiceChain.137.SwiftProxyBase.TLSProxyBase]: CREATE_COMPLETE  state changed

• Both SSL key and certificate are present in the stack as per below and they are sure matching (did openssl x509 -noout -modulus -in server.crt | openssl md5 and compared with qeuavalent command for key and they match) also certificate is still valid:

# openstack stack environment show overcloud | grep -i ssl
  DeployedSSLCertificatePath: /etc/pki/tls/private/overcloud_endpoint.pem
  SSLCertificate: '-----BEGIN CERTIFICATE-----
  SSLIntermediateCertificate: ''
  SSLKey: '-----BEGIN RSA PRIVATE KEY-----
  SSLRootCertificate: '-----BEGIN CERTIFICATE-----