Ceph storage node scale-out fails with msg: 0 objects found in container: overcloud_ceph_ansible_fetch_dir in mistral logs in Red Hat OpenStack Platform 13

Issue

OSP 13 Ceph storage node scale-out fails with msg: 0 objects found in container: overcloud_ceph_ansible_fetch_dir in mistral logs in Red Hat OpenStack Platform 13

The deployment fails on WorkflowTasks_Step2:

2018-10-31 11:34:40Z [overcloud-AllNodesDeploySteps-2swiux6uq6ug.WorkflowTasks_Step2_Execution]: UPDATE_IN_PROGRESS  state changed<br data-hasqtip="true">2018-10-31 11:34:41Z [overcloud-AllNodesDeploySteps-2swiux6uq6ug.WorkflowTasks_Step2_Execution]: UPDATE_COMPLETE  The Resource WorkflowTasks_Step2_Execution requires replacement.<br data-hasqtip="true">2018-10-31 11:34:41Z [overcloud-AllNodesDeploySteps-2swiux6uq6ug.WorkflowTasks_Step2_Execution]: CREATE_IN_PROGRESS  state changed<br data-hasqtip="true">2018-10-31 11:35:36Z [overcloud-AllNodesDeploySteps-2swiux6uq6ug.WorkflowTasks_Step2_Execution]: CREATE_FAILED

/var/log/mistral/engine.log shows:

2018-10-31 11:35:33.804 27985 INFO workflow_trace [req-92baf38c-bc83-480c-b159-108270f4caa0 b923ac7ec043480abc9beaf2cc0e6b29 d9740120a235481e81a33f7c64ae9c38 - - -] Workflow 'tripleo.storage.v1.ceph-install' [RUNNING -> ERROR, msg=Failure
 caused by error in tasks: restore_fetch_directory

  restore_fetch_directory [task_ex_id=4dbbaf7c-6125-4f44-b239-f308e93c948f] -> {msg: 0 objects found in container: overcloud_ceph_ansible_fetch_dir but one object was expected.}
    [action_ex_id=636a4976-1d07-406c-a10a-1ce3a68b7dcd, idx=0]: {u'msg': u'0 objects found in container: overcloud_ceph_ansible_fetch_dir but one object was expected.'}
] (execution_id=b7c731ec-8c6f-4303-a0a5-631a36bbfeb8)
2018-10-31 11:35:33.841 27985 INFO mistral.engine.engine_server [req-92baf38c-bc83-480c-b159-108270f4caa0 b923ac7ec043480abc9beaf2cc0e6b29 d9740120a235481e81a33f7c64ae9c38 - - -] Received RPC request 'on_action_complete'[action_ex_id=b7c7
31ec-8c6f-4303-a0a5-631a36bbfeb8, result=Result [data=None, error=Failure caused by error in tasks: restore_fetch_directory

A close examination of the swift containers reveals that the container is empty:

(undercloud) [stack@undercloud-1 ~]$ . stackrc
(undercloud) [stack@undercloud-1 ~]$ swift list
swift list __cache__
ov-3kzq2qiept-0-kv7h2ym5rufn-CephStorage-ejlwfdxkhumb
ov-izwnkbj7agf-0-6mepiwde6wun-Controller-we6uw6kelfmy
ov-izwnkbj7agf-1-4ywcwm7m4ejf-Controller-gp4zs6dlrbj4
ov-izwnkbj7agf-2-zfpqwbqracxg-Controller-upwscms2tvd4
ov-vcnuxo7ks3-0-62bry3hflm4b-NovaCompute-tyj6co3m6e6w
overcloud
overcloud-swift-rings
overcloud_ceph_ansible_fetch_dir
overcloud_ceph_ansible_fetch_dir_segments
(undercloud) [stack@undercloud-1 ~]$ swift list overcloud_ceph_ansible_fetch_dir
(undercloud) [stack@undercloud-1 ~]$