Overcloud deployment fails on compute SR-IOV with "Failed to get placement service endpoint"

Issue

• Overcloud deployment fails on SR-IOV compute nodes with "ERROR:nova_wait_for_placement_service:Retry - Failed to get placement service endpoint".

2019-06-25 14:28:33Z [0]: CREATE_IN_PROGRESS  state changed
2019-06-25 14:29:50Z [cloud.AllNodesDeploySteps.ControllerDeployment_Step4.0]: SIGNAL_IN_PROGRESS  Signal: deployment cb2f1b32-a228-4fb3-8ae0-f9a3749b2c4e succeeded
2019-06-25 14:29:51Z [cloud.AllNodesDeploySteps.ControllerDeployment_Step4.0]: CREATE_COMPLETE  state changed
2019-06-25 14:29:51Z [cloud.AllNodesDeploySteps.ControllerDeployment_Step4]: CREATE_COMPLETE  Stack CREATE completed successfully
2019-06-25 14:29:51Z [cloud.AllNodesDeploySteps.ControllerDeployment_Step4]: CREATE_COMPLETE  state changed
2019-06-25 14:39:10Z [0]: SIGNAL_IN_PROGRESS  Signal: deployment e109f499-958a-4ff6-a5a9-c2e716adc663 failed (2)
2019-06-25 14:39:11Z [0]: CREATE_FAILED  Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
2019-06-25 14:39:11Z [cloud.AllNodesDeploySteps.ComputesriovDeployment_Step4]: CREATE_FAILED  Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
2019-06-25 14:39:11Z [cloud.AllNodesDeploySteps.ComputesriovDeployment_Step4]: CREATE_FAILED  Error: resources.ComputesriovDeployment_Step4.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2019-06-25 14:39:11Z [cloud.AllNodesDeploySteps]: CREATE_FAILED  Resource CREATE failed: Error: resources.ComputesriovDeployment_Step4.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2019-06-25 14:39:12Z [cloud.AllNodesDeploySteps]: CREATE_FAILED  Error: resources.AllNodesDeploySteps.resources.ComputesriovDeployment_Step4.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2019-06-25 14:39:12Z [cloud]: CREATE_FAILED  Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ComputesriovDeployment_Step4.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2

 Stack cloud CREATE_FAILED 

cloud.AllNodesDeploySteps.ComputesriovDeployment_Step4.1:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: ea260cb4-a7c0-44b0-a65a-dc338a761024
  status: CREATE_FAILED
  status_reason: |
    Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
            "stdout: e0a08a4d7de8e277a2b4e82ecac1eb076387784a34f2ab16adc0529998d9c319", 
            "stdout: 7a5f28fb143ce011b155337c6ec96052dc89c6d87a5c9a18934a405f46344170", 
            "stdout: 402020811deac97b57979736eb17ac75e6378eb74ccc9d2113c283c729b95525"
        ]
    }
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/30687b8f-9f81-476f-9b55-e6e13e7cdfe9_playbook.retry

    PLAY RECAP *********************************************************************
    localhost                  : ok=5    changed=2    unreachable=0    failed=1   

    (truncated, view all with --long)
  deploy_stderr: |

cloud.AllNodesDeploySteps.ComputesriovDeployment_Step4.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: e109f499-958a-4ff6-a5a9-c2e716adc663
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
            "stdout: 0ae4a79a752bf145f571ff13c04e1e0fb581ee889f4a164097176ed0e7f32c4c", 
            "stdout: 8aac6ea0d4fa9977ddcd94d7040c4f10da40034a4c463926118055339fac3615", 
            "stdout: aab5d98e4f73bc4c0e7015da7ab35450d789bb700f76102f3f359ca5fca55c73"
        ]
    }
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/76f513b0-103f-4a11-adb3-61e204dcb36a_playbook.retry

    PLAY RECAP *********************************************************************
    localhost                  : ok=5    changed=2    unreachable=0    failed=1   

    (truncated, view all with --long)
  deploy_stderr: |

Heat Stack create failed.

• In /var/log/messages on compute node.

Jun 25 10:39:08 computesriov-0 os-collect-config: "Digest: sha256:a594fe571b2c08a85f980b282a6aa95546565cc9d2f562f49acad4dccce185aa",
Jun 25 10:39:08 computesriov-0 os-collect-config: "Status: Downloaded newer image for 192.168.0.1:8787/rhosp13/openstack-neutron-sriov-agent:latest",
Jun 25 10:39:08 computesriov-0 os-collect-config: "stdout: 13019427e6cde77d6ea1b131cbfc000f27fd9f1bb4d1571c36e5708073175e29",
Jun 25 10:39:08 computesriov-0 os-collect-config: "Error running ['docker', 'run', '--name', 'nova_wait_for_placement_service', '--label', 'config_id=tripleo_step4', '--label', 'container_name=nova_wait_for_placement_service', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 2, \"image\": \"192.168.0.1:8787/rhosp13/openstack-nova-compute:latest\", \"command\": \"/docker-config-scripts/nova_wait_for_placement_service.py\", \"user\": \"root\", \"volumes\": [\"/var/lib/docker-config-scripts/:/docker-config-scripts/\", \"/var/lib/config-data/puppet-generated/nova_libvirt/etc/nova:/etc/nova:ro\"], \"net\": \"host\", \"detach\": false, \"privileged\": false}', '--net=host', '--privileged=false', '--user=root', '--volume=/var/lib/docker-config-scripts/:/docker-config-scripts/', '--volume=/var/lib/config-data/puppet-generated/nova_libvirt/etc/nova:/etc/nova:ro', '192.168.0.1:8787/rhosp13/openstack-nova-compute:latest', '/docker-config-scripts/nova_wait_for_placement_service.py']. [1]",
Jun 25 10:39:08 computesriov-0 os-collect-config: "stdout: DEBUG:keystoneauth.identity.v3.base:Making authentication request to http://10.0.0.10:5000/v3/auth/tokens",
Jun 25 10:39:08 computesriov-0 os-collect-config: "DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.0.0.10",
Jun 25 10:39:08 computesriov-0 os-collect-config: "ERROR:nova_wait_for_placement_service:Retry - Failed to get placement service endpoint:",
Jun 25 10:39:08 computesriov-0 os-collect-config: "Traceback (most recent call last):",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/docker-config-scripts/nova_wait_for_placement_service.py\", line 68, in <module>",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    name='placement')[0].id",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneclient/v3/services.py\", line 93, in list",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    **kwargs)",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneclient/base.py\", line 75, in func",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    return f(*args, **new_kwargs)",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneclient/base.py\", line 397, in list",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    self.collection_key)",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneclient/base.py\", line 125, in _list",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    resp, body = self.client.get(url, **kwargs)",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 304, in get",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    return self.request(url, 'GET', **kwargs)",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 463, in request",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 189, in request",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    return self.session.request(url, method, **kwargs)",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 573, in request",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    auth_headers = self.get_auth_headers(auth)",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 900, in get_auth_headers",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    return auth.get_headers(self, **kwargs)",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py\", line 95, in get_headers",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    token = self.get_token(session)",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py\", line 88, in get_token",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    return self.get_access(session).auth_token",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    self.auth_ref = self.get_auth_ref(session)",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py\", line 177, in get_auth_ref",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    authenticated=False, log=False, **rkwargs)",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 848, in post",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    return self.request(url, 'POST', **kwargs)",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 698, in request",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    resp = send(**kwargs)",
Jun 25 10:39:08 computesriov-0 os-collect-config: "  File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 772, in _send_request",
Jun 25 10:39:08 computesriov-0 os-collect-config: "    raise exceptions.ConnectFailure(msg)",
Jun 25 10:39:08 computesriov-0 os-collect-config: "ConnectFailure: Unable to establish connection to http://10.0.0.10:5000/v3/auth/tokens: HTTPConnectionPool(host='10.0.0.10', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f84ccbec910>: Failed to establish a new connection: [Errno 111] Connection refused',))",
Jun 25 10:39:08 computesriov-0 os-collect-config: "DEBUG:keystoneauth.identity.v3.base:Making authentication request to http://10.0.0.10:5000/v3/auth/tokens",
Jun 25 10:39:08 computesriov-0 os-collect-config: "DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTP connection (2): 10.0.0.10",
Jun 25 10:39:08 computesriov-0 os-collect-config: "ConnectFailure: Unable to establish connection to http://10.0.0.10:5000/v3/auth/tokens: HTTPConnectionPool(host='10.0.0.10', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f84ccbcd750>: Failed to establish a new connection: [Errno 111] Connection refused',))",
Jun 25 10:39:08 computesriov-0 os-collect-config: "DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTP connection (3): 10.0.0.10",
Jun 25 10:39:08 computesriov-0 os-collect-config: "ConnectFailure: Unable to establish connection to http://10.0.0.10:5000/v3/auth/tokens: HTTPConnectionPool(host='10.0.0.10', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f84ccbcd5d0>: Failed to establish a new connection: [Errno 111] Connection refused',))",
Jun 25 10:39:08 computesriov-0 os-collect-config: "DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTP connection (4): 10.0.0.10",
Jun 25 10:39:08 computesriov-0 os-collect-config: "ConnectFailure: Unable to establish connection to http://10.0.0.10:5000/v3/auth/tokens: HTTPConnectionPool(host='10.0.0.10', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f84ccbcd690>: Failed to establish a new connection: [Errno 111] Connection refused',))",
Jun 25 10:39:08 computesriov-0 os-collect-config: "DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTP connection (5): 10.0.0.10",