Compute scale out failing on OSP13

Issue

• Scaleout operation is failing on OSP13. Currently, it's stuck on WorkflowTasks_Step2_Execution:

2020-07-02 12:12:01Z [atl1-AllNodesDeploySteps-overcloud-ComputeG9Deployment_Step1-scsdb6ttvofd.13]: UPDATE_COMPLETE  state changed
2020-07-02 12:12:02Z [atl1-AllNodesDeploySteps-overcloud-ComputeG9Deployment_Step1-scsdb6ttvofd]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2020-07-02 12:12:02Z [atl1-AllNodesDeploySteps-overcloud.ComputeG9Deployment_Step1]: UPDATE_COMPLETE  state changed
2020-07-02 12:12:03Z [atl1-AllNodesDeploySteps-overcloud.WorkflowTasks_Step2_Execution]: UPDATE_IN_PROGRESS  state changed
2020-07-02 12:12:04Z [atl1-AllNodesDeploySteps-overcloud.WorkflowTasks_Step2_Execution]: UPDATE_COMPLETE  The Resource WorkflowTasks_Step2_Execution requires replacement.
2020-07-02 12:12:05Z [atl1-AllNodesDeploySteps-overcloud.WorkflowTasks_Step2_Execution]: CREATE_IN_PROGRESS  state changed

• From "/var/log/mistral/ceph-install-workflow.log", we can see [3] the following failure:

2020-07-02 15:57:43,321 p=10725 u=mistral |  failed: [10.10.10.116 -> 10.10.10.116] (item={u'name': u'client.radosgw', u'mode': u'0600', u'key': u'AQB4lyleAAAAABAAhg/bzqn5a
IVDKKekelo4fA==', u'caps': {u'mgr': u'allow *', u'mon': u'allow rw', u'osd': u'allow rwx'}}) => {

"msg": "non-zero return code",
"rc": 1,
"start": "2020-07-02 15:57:41.707164",
"stderr": "Error response from daemon: No such container: ceph-create-keys",
"stderr_lines": [
    "Error response from daemon: No such container: ceph-create-keys"
],
"stdout": "",
"stdout_lines": []

[...]
2020-07-02 16:00:01,263 p=10725 u=mistral |  PLAY RECAP *********************************************************************
2020-07-02 16:00:01,264 p=10725 u=mistral |  10.10.10.105               : ok=33   changed=0    unreachable=0    failed=0
2020-07-02 16:00:01,265 p=10725 u=mistral |  10.10.10.106               : ok=33   changed=0    unreachable=0    failed=0
2020-07-02 16:00:01,265 p=10725 u=mistral |  10.10.10.107               : ok=33   changed=0    unreachable=0    failed=0
2020-07-02 16:00:01,265 p=10725 u=mistral |  10.10.10.108               : ok=33   changed=0    unreachable=0    failed=0
2020-07-02 16:00:01,265 p=10725 u=mistral |  10.10.10.109               : ok=33   changed=0    unreachable=0    failed=0
2020-07-02 16:00:01,265 p=10725 u=mistral |  10.10.10.110               : ok=33   changed=0    unreachable=0    failed=0
2020-07-02 16:00:01,265 p=10725 u=mistral |  10.10.10.111               : ok=33   changed=0    unreachable=0    failed=0
2020-07-02 16:00:01,265 p=10725 u=mistral |  10.10.10.112               : ok=33   changed=0    unreachable=0    failed=0
2020-07-02 16:00:01,266 p=10725 u=mistral |  10.10.10.113               : ok=33   changed=0    unreachable=0    failed=0
2020-07-02 16:00:01,266 p=10725 u=mistral |  10.10.10.114               : ok=33   changed=0    unreachable=0    failed=0
2020-07-02 16:00:01,266 p=10725 u=mistral |  10.10.10.115               : ok=33   changed=0    unreachable=0    failed=0
2020-07-02 16:00:01,266 p=10725 u=mistral |  10.10.10.116               : ok=57   changed=4    unreachable=0    failed=1 
2020-07-02 16:00:01,267 p=10725 u=mistral |  10.10.10.118               : ok=33   changed=0    unreachable=0    failed=0
2020-07-02 16:00:01,267 p=10725 u=mistral |  10.10.10.119               : ok=33   changed=0    unreachable=0    failed=0
2020-07-02 16:00:01,267 p=10725 u=mistral |  10.10.10.120               : ok=33   changed=0    unreachable=0    failed=0
2020-07-02 16:00:01,268 p=10725 u=mistral |  10.10.10.121               : ok=33   changed=0    unreachable=0    failed=0
[...]

• And on the compute itself we can see the following error message:

Jul  2 15:52:24 914830-compute034 ansible-command: Invoked with warn=True executable=None _uses_shell=False _raw_params=docker run --rm -d -v /etc/ceph:/etc/ceph:z --name ceph-create-keys --entrypoint=sleep 10.10.10.53:8787/rhceph/rhceph-3-rhel7:3-40 300 removes=None argv=None creates=None chdir=None stdin=None
[...]
Jul  2 15:57:40 914830-compute034 ansible-ceph_key: Invoked with directory_mode=None force=None remote_src=None cluster=ceph import_key=False owner=None follow=False auid=None group=None unsafe_writes=None serole=None content=NOT_LOGGING_PARAMETER secret=AQCneyheAAAAABAADeol8F2T0BVo4IOIIPO1aw== containerized=docker exec  ceph-create-keys  setype=None state=present dest=/etc/ceph selevel=None caps={'mgr': 'allow *', 'mon': 'profile rbd', 'osd': 'profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=vms, profile rbd pool=images'} regexp=None src=None name=client.openstack seuser=None delimiter=None mode=0600 attributes=None backup=None
Jul  2 15:57:40 914830-compute034 dockerd-current: time="2020-07-02T15:57:40.827727085Z" level=error msg="Error setting up exec command in container ceph-create-keys: No such container: ceph-create-keys"
[...]
Jul  2 15:57:41 914830-compute034 dockerd-current: time="2020-07-02T15:57:41.735386274Z" level=error msg="Handler for POST /v1.26/containers/ceph-create-keys/exec returned error: No such container: ceph-create-keys"
Jul  2 15:57:41 914830-compute034 dockerd-current: time="2020-07-02T15:57:41.735631700Z" level=error msg="Handler for POST /v1.26/containers/ceph-create-keys/exec returned error: No such container: ceph-create-keys"
[...]