ceph storage node based cluster fail to install

Issue

• We started with clean install but the environment with 3 controller and 3 storage node fails to install.

• The deployment fails at step2and the following deployment command is being used:

(undercloud) [stack@undercloud ~]$ openstack overcloud deploy --templates -e /home/stack/templates/node-info.yaml -e /home/stack/templates/overcloud_images.yaml -e /home/stack/templates/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/templates/openstack-tripleo-heat-templates/environments/network-environment.yaml -e /home/stack/templates/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml    -n /home/stack/templates/network_data.yaml   -r /home/stack/templates/roles_data.yaml -e /home/stack/templates/openstack-tripleo-heat-templates/environments/host-config-and-reboot.yaml -e /home/stack/templates/openstack-tripleo-heat-templates/environments/services/neutron-sriov.yaml -e /home/stack/templates/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml

• The overcloud failed to deploy properly and was deleted prior being redeployed.

• OSDs are not coming up at step2with errors similar to this:

2019-06-29 01:35:23,175 p=11290 u=mistral |  FAILED - RETRYING: wait for all osd to be up (4 retries left).
2019-06-29 01:35:34,294 p=11290 u=mistral |  FAILED - RETRYING: wait for all osd to be up (3 retries left).
2019-06-29 01:35:45,404 p=11290 u=mistral |  FAILED - RETRYING: wait for all osd to be up (2 retries left).
2019-06-29 01:35:56,514 p=11290 u=mistral |  FAILED - RETRYING: wait for all osd to be up (1 retries left).
2019-06-29 01:36:07,640 p=11290 u=mistral |  fatal: [10.10.10.7 -> 10.10.10.10]: FAILED! => {"attempts": 60, "changed": false, "cmd": "test \"$(docker exec ceph-mon-overcloud-controller-0 ceph --cluster ceph -s -f json | python -c 'import sys, json; print(json.load(sys.stdin)[\"osdmap\"][\"osdmap\"][\"num_osds\"])')\" -gt 0 && test \"$(docker exec ceph-mon-overcloud-controller-0 ceph --cluster ceph -s -f json | python -c 'import sys, json; print(json.load(sys.stdin)[\"osdmap\"][\"osdmap\"][\"num_osds\"])')\" = \"$(docker exec ceph-mon-overcloud-controller-0 ceph --cluster ceph -s -f json | python -c 'import sys, json; print(json.load(sys.stdin)[\"osdmap\"][\"osdmap\"][\"num_up_osds\"])')\"", "delta": "0:00:00.770688", "end": "2019-06-29 05:36:07.592073", "msg": "non-zero return code", "rc": 1, "start": "2019-06-29 05:36:06.821385", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
2019-06-29 01:36:07,641 p=11290 u=mistral |  NO MORE HOSTS LEFT *************************************************************
2019-06-29 01:36:07,642 p=11290 u=mistral |  PLAY RECAP *********************************************************************
2019-06-29 01:36:07,642 p=11290 u=mistral |  10.10.10.10              : ok=160  changed=20   unreachable=0    failed=0
2019-06-29 01:36:07,642 p=11290 u=mistral |  10.10.10.11              : ok=81   changed=9    unreachable=0    failed=0
2019-06-29 01:36:07,642 p=11290 u=mistral |  10.10.10.7               : ok=79   changed=9    unreachable=0    failed=1
2019-06-29 01:36:07,642 p=11290 u=mistral |  10.10.10.8               : ok=78   changed=9    unreachable=0    failed=0
2019-06-29 01:36:07,643 p=11290 u=mistral |  INSTALLER STATUS ***************************************************************