Why does the overcloud deployment fails with ERROR, msg=Failure caused by error in tasks: enable_ssh_admin ?

Issue

Scenario 1 - Ceph issues

The overcloud deployment fails with the following error:

Overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:
  resource_type: OS::Mistral::ExternalResource
  physical_resource_id: AAAAAAAA-9d6e-4016-ac2e-2bf349bc93e8
  status: CREATE_FAILED
  status_reason: |
    resources.WorkflowTasks_Step2_Execution: ERROR

Mistral engine log shows :
 Workflow 'tripleo.storage.v1.ceph-install' [RUNNING -> ERROR, msg=Failure caused by error in tasks: enable_ssh_admin

  enable_ssh_admin [task_ex_id=e330222b-6a5a-4b8f-9999-3c544444448c] -> Failure caused by error in tasks: create_admin_via_nova

  create_admin_via_nova [task_ex_id=444444b-1c79-4b5b-a7ae-2c8222225414] -> Failure caused by error in tasks: create_admin

  create_admin [task_ex_id=AAAAAAAA-aa6a-426a-8006-c1a74b2eecc0] -> One or more actions had failed.
    [wf_ex_id=1fb859bd-0deb-4c71-b368-AAAAAAAA, idx=0]: Failed to run task [error=Can not evaluate YAQL expression [expression=task(deploy_config).result.deploy_stderr, error=Unknown function "#property#deploy_stderr", data={}], wf=tripleo.deployment.v1.deploy_on_server, task=send_message]:

If ceph nodes journal is inspected is it possible to find the following errors:

Jun 28 10:41:47 cephstorage-0 os-collect-config[6260]: [2018-06-28 10:41:47,680] (heat-config) [INFO] Completed /usr/libexec/heat-config/hooks/ansible
Jun 28 10:41:47 cephstorage-0 os-collect-config[6260]: [2018-06-28 10:41:47,681] (heat-config) [DEBUG] Running heat-config-notify /var/lib/heat-config/deployed/584b14df-15bf-4e1b-9f98-bb3aadd2f79c.json < /var/lib/heat-config/deployed/AAAAAA-15bf-4e1b-BBBB-bb3aadd2f79c.notify.json
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: [2018-06-28 10:41:49,143] (heat-config) [INFO]
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: [2018-06-28 10:41:49,143] (heat-config) [ERROR] Error running heat-config-notify. [1]
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: [2018-06-28 10:41:49,143] (heat-config) [ERROR] [2018-06-28 10:41:49,026] (heat-config-notify) [DEBUG] Signaling to https://10.206.112.241:13808/v1/AUTH_e4f4c08fb273405ca3a4d686f2745ed5/create_admin-655768ec-6d69-48ad-a5c5-AAAAAAAA/c67a29e2-BBBB-455f-a2b4-634fe023df24?temp_url_sig=ABABABABA9e8345b4649882e03bd10a6d74636b&temp_url_expires=1530193265 via PUT
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: Traceback (most recent call last):
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: File "/usr/bin/heat-config-notify", line 163, in 
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: sys.exit(main(sys.argv, sys.stdin))
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: File "/usr/bin/heat-config-notify", line 110, in main
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: headers={'content-type': 'application/json'})
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: File "/usr/lib/python2.7/site-packages/requests/api.py", line 126, in put
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: return request('put', url, data=data, **kwargs)
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: File "/usr/lib/python2.7/site-packages/requests/api.py", line 58, in request
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: return session.request(method=method, url=url, **kwargs)
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in request
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: resp = self.send(prep, **send_kwargs)
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: r = adapter.send(request, **kwargs)
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 512, in send
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: raise SSLError(e, request=request)
Jun 28 10:41:49 cephstorage-0 os-collect-config[6260]: requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)

Scenario 2 - Octavia issues

The overcloud deployment fails with the following error:

openstack stack failures list overcloud --long
overcloud.AllNodesDeploySteps.WorkflowTasks_Step5_Execution:
  resource_type: OS::TripleO::WorkflowSteps
  physical_resource_id: faa96b03-6394-424c-b982-c2c3ed272f19
  status: CREATE_FAILED
  status_reason: |
    resources.WorkflowTasks_Step5_Execution: ERROR

From /var/log/mistral/engine.log:

2019-05-10 17:13:33.679 6437 INFO workflow_trace [req-903f0d0e-15be-46d7-9821-807046234eb1 b049394837db4beb914e7995b2c1a4f9 2c7dff002fdf4ede9f32e8d741804b90 - default default] Workflow 'tripleo.overcloud.workflow_tasks.step5' [RUNNING -> ERROR, msg=Failure caused by error in tasks: octavia_post_workflow

  octavia_post_workflow [task_ex_id=f2672afd-d429-4ff9-92f2-788ace6759ff] -> Failure caused by error in tasks: enable_ssh_admin

  enable_ssh_admin [task_ex_id=88bc8c01-1681-4b64-a10c-97b9ba09c619] -> Failure caused by error in tasks: create_admin_via_nova

  create_admin_via_nova [task_ex_id=4f2cfd0a-2f49-4b4d-a135-0fe742de0b87] -> Failure caused by error in tasks: create_admin

  create_admin [task_ex_id=e1e2317b-cebe-4cf4-8296-f05702d9baef] -> One or more actions had failed.
    [wf_ex_id=8e796bf2-f032-4d5f-b1ac-28890fa09e31, idx=0]: None
    [wf_ex_id=ab1114bc-1ccc-4c62-a0d9-c50e16f82196, idx=1]: None
    [wf_ex_id=c345ea7d-090e-4ff4-b211-405a509bd541, idx=2]: None

    [wf_ex_id=b8edacfe-8754-4123-842e-de22769d79a0, idx=0]: Failure caused by error in tasks: create_admin

  create_admin [task_ex_id=e1e2317b-cebe-4cf4-8296-f05702d9baef] -> One or more actions had failed.
    [wf_ex_id=8e796bf2-f032-4d5f-b1ac-28890fa09e31, idx=0]: None
    [wf_ex_id=ab1114bc-1ccc-4c62-a0d9-c50e16f82196, idx=1]: None
    [wf_ex_id=c345ea7d-090e-4ff4-b211-405a509bd541, idx=2]: None


    [wf_ex_id=d359e87d-59f6-4177-9e3f-5370a06f1aef, idx=0]: Failure caused by error in tasks: create_admin_via_nova

  create_admin_via_nova [task_ex_id=4f2cfd0a-2f49-4b4d-a135-0fe742de0b87] -> Failure caused by error in tasks: create_admin

  create_admin [task_ex_id=e1e2317b-cebe-4cf4-8296-f05702d9baef] -> One or more actions had failed.
    [wf_ex_id=8e796bf2-f032-4d5f-b1ac-28890fa09e31, idx=0]: None
    [wf_ex_id=ab1114bc-1ccc-4c62-a0d9-c50e16f82196, idx=1]: None
    [wf_ex_id=c345ea7d-090e-4ff4-b211-405a509bd541, idx=2]: None

    [wf_ex_id=b8edacfe-8754-4123-842e-de22769d79a0, idx=0]: Failure caused by error in tasks: create_admin

  create_admin [task_ex_id=e1e2317b-cebe-4cf4-8296-f05702d9baef] -> One or more actions had failed.
    [wf_ex_id=8e796bf2-f032-4d5f-b1ac-28890fa09e31, idx=0]: None
    [wf_ex_id=ab1114bc-1ccc-4c62-a0d9-c50e16f82196, idx=1]: None
    [wf_ex_id=c345ea7d-090e-4ff4-b211-405a509bd541, idx=2]: None



    [wf_ex_id=65c746a5-f96b-4211-bfe5-3b7132641699, idx=0]: Failure caused by error in tasks: enable_ssh_admin

  enable_ssh_admin [task_ex_id=88bc8c01-1681-4b64-a10c-97b9ba09c619] -> Failure caused by error in tasks: create_admin_via_nova

  create_admin_via_nova [task_ex_id=4f2cfd0a-2f49-4b4d-a135-0fe742de0b87] -> Failure caused by error in tasks: create_admin

  create_admin [task_ex_id=e1e2317b-cebe-4cf4-8296-f05702d9baef] -> One or more actions had failed.
    [wf_ex_id=8e796bf2-f032-4d5f-b1ac-28890fa09e31, idx=0]: None
    [wf_ex_id=ab1114bc-1ccc-4c62-a0d9-c50e16f82196, idx=1]: None
    [wf_ex_id=c345ea7d-090e-4ff4-b211-405a509bd541, idx=2]: None

    [wf_ex_id=b8edacfe-8754-4123-842e-de22769d79a0, idx=0]: Failure caused by error in tasks: create_admin

  create_admin [task_ex_id=e1e2317b-cebe-4cf4-8296-f05702d9baef] -> One or more actions had failed.
    [wf_ex_id=8e796bf2-f032-4d5f-b1ac-28890fa09e31, idx=0]: None
    [wf_ex_id=ab1114bc-1ccc-4c62-a0d9-c50e16f82196, idx=1]: None
    [wf_ex_id=c345ea7d-090e-4ff4-b211-405a509bd541, idx=2]: None


    [wf_ex_id=d359e87d-59f6-4177-9e3f-5370a06f1aef, idx=0]: Failure caused by error in tasks: create_admin_via_nova

  create_admin_via_nova [task_ex_id=4f2cfd0a-2f49-4b4d-a135-0fe742de0b87] -> Failure caused by error in tasks: create_admin

  create_admin [task_ex_id=e1e2317b-cebe-4cf4-8296-f05702d9baef] -> One or more actions had failed.
    [wf_ex_id=8e796bf2-f032-4d5f-b1ac-28890fa09e31, idx=0]: None
    [wf_ex_id=ab1114bc-1ccc-4c62-a0d9-c50e16f82196, idx=1]: None
    [wf_ex_id=c345ea7d-090e-4ff4-b211-405a509bd541, idx=2]: None

    [wf_ex_id=b8edacfe-8754-4123-842e-de22769d79a0, idx=0]: Failure caused by error in tasks: create_admin

  create_admin [task_ex_id=e1e2317b-cebe-4cf4-8296-f05702d9baef] -> One or more actions had failed.
    [wf_ex_id=8e796bf2-f032-4d5f-b1ac-28890fa09e31, idx=0]: None
    [wf_ex_id=ab1114bc-1ccc-4c62-a0d9-c50e16f82196, idx=1]: None
    [wf_ex_id=c345ea7d-090e-4ff4-b211-405a509bd541, idx=2]: None



] (execution_id=faa96b03-6394-424c-b982-c2c3ed272f19)
2019-05-10 17:13:33.680 6437 DEBUG mistral.services.triggers [req-903f0d0e-15be-46d7-9821-807046234eb1 b049394837db4beb914e7995b2c1a4f9 2c7dff002fdf4ede9f32e8d741804b90 - default default] No JSON object could be decoded on_workflow_complete /usr/lib/python2.7/site-packages/mistral/services/triggers.py:242
2019-05-10 17:13:33.688 6437 DEBUG mistral.services.scheduler [req-903f0d0e-15be-46d7-9821-807046234eb1 b049394837db4beb914e7995b2c1a4f9 2c7dff002fdf4ede9f32e8d741804b90 - default default] Scheduler deleted 1 delayed calls. delete_calls /usr/lib/python2.7/site-packages/mistral/services/scheduler.py:310