overcloud deployment fails with keystone errors

Issue

• The overcloud heat stack deployment errored when adding new compute nodes with errors related to keystone authentication connection timeout.

• heat-engine (/var/log/heat/engine.log) fails to reach nova-api due to issues connecting to keystone with error similar to this:

2019-04-08 07:45:17.095 9335 DEBUG heat.engine.scheduler [req-3a4d179b-61a7-415f-bcc4-c2af4291e6bc - admin - default default] Task update from TemplateResource "PreNetworkConfig" [c9c126b0-8969-4b55-b7b6-163777380f7c] Stack "overcloud-ComputeHCI-jgjtwc7jnhe3-7-xyo4occa7gvw" [46ca0bfc-2dbd-4311-8ef4-31ef4e95c99d] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-04-08 07:45:17.093 9324 INFO heat.engine.resource [req-3a4d179b-61a7-415f-bcc4-c2af4291e6bc - admin - default default] CREATE: ServerUpdateAllowed "NovaCompute" Stack "overcloud-ComputeHCI2-bljwkfgx7e7c-18-opqag7cbwxv5" [e37ba12d-e4e8-45f0-bea0-83a5c6eed67d]
....
2019-04-08 07:45:17.093 9324 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
2019-04-08 07:45:17.093 9324 ERROR heat.engine.resource     raise exceptions.from_response(resp, body, url, method)
2019-04-08 07:45:17.093 9324 ERROR heat.engine.resource ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2019-04-08 07:45:17.093 9324 ERROR heat.engine.resource <class 'keystoneauth1.exceptions.connection.ConnectTimeout'> (HTTP 500) (Request-ID: req-34e0e894-dd79-431e-bdad-0d62663e8afa)

• nova-api (/var/log/nova/nova-api.log) also logs issues reaching to keystone with error similar to this one:

2019-04-08 07:45:17.080 8180 ERROR nova.api.openstack.wsgi [req-000d10ce-1481-43c9-ba9e-10af17a4ea28 - - - - -] Unexpected exception in API method: ConnectTimeout: Request to http://10.15.1.100:9696/v2.0/ports/6e91102f-07e5-4e55-93f6-91087431a469 timed out
2019-04-08 07:45:17.080 8180 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
...
2019-04-08 07:45:17.080 8180 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 763, in _send_request
2019-04-08 07:45:17.080 8180 ERROR nova.api.openstack.wsgi     raise exceptions.ConnectTimeout(msg)
2019-04-08 07:45:17.080 8180 ERROR nova.api.openstack.wsgi ConnectTimeout: Request to http://10.15.1.100:9696/v2.0/ports/6e91102f-07e5-4e55-93f6-91087431a469 timed out
2019-04-08 07:45:17.080 8180 ERROR nova.api.openstack.wsgi 
2019-04-08 07:45:17.089 8180 INFO nova.api.openstack.wsgi [req-000d10ce-1481-43c9-ba9e-10af17a4ea28 - - - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'keystoneauth1.exceptions.connection.ConnectTimeout'>