overcloud deployment fails with keystone errors

Solution In Progress - Updated -

Issue

  • The overcloud heat stack deployment errored when adding new compute nodes with errors related to keystone authentication connection timeout.

  • heat-engine (/var/log/heat/engine.log) fails to reach nova-api due to issues connecting to keystone with error similar to this:

2019-04-08 07:45:17.095 9335 DEBUG heat.engine.scheduler [req-3a4d179b-61a7-415f-bcc4-c2af4291e6bc - admin - default default] Task update from TemplateResource "PreNetworkConfig" [c9c126b0-8969-4b55-b7b6-163777380f7c] Stack "overcloud-ComputeHCI-jgjtwc7jnhe3-7-xyo4occa7gvw" [46ca0bfc-2dbd-4311-8ef4-31ef4e95c99d] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209
2019-04-08 07:45:17.093 9324 INFO heat.engine.resource [req-3a4d179b-61a7-415f-bcc4-c2af4291e6bc - admin - default default] CREATE: ServerUpdateAllowed "NovaCompute" Stack "overcloud-ComputeHCI2-bljwkfgx7e7c-18-opqag7cbwxv5" [e37ba12d-e4e8-45f0-bea0-83a5c6eed67d]
....
2019-04-08 07:45:17.093 9324 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
2019-04-08 07:45:17.093 9324 ERROR heat.engine.resource     raise exceptions.from_response(resp, body, url, method)
2019-04-08 07:45:17.093 9324 ERROR heat.engine.resource ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2019-04-08 07:45:17.093 9324 ERROR heat.engine.resource <class 'keystoneauth1.exceptions.connection.ConnectTimeout'> (HTTP 500) (Request-ID: req-34e0e894-dd79-431e-bdad-0d62663e8afa)
  • nova-api (/var/log/nova/nova-api.log) also logs issues reaching to keystone with error similar to this one:
2019-04-08 07:45:17.080 8180 ERROR nova.api.openstack.wsgi [req-000d10ce-1481-43c9-ba9e-10af17a4ea28 - - - - -] Unexpected exception in API method: ConnectTimeout: Request to http://10.15.1.100:9696/v2.0/ports/6e91102f-07e5-4e55-93f6-91087431a469 timed out
2019-04-08 07:45:17.080 8180 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
...
2019-04-08 07:45:17.080 8180 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 763, in _send_request
2019-04-08 07:45:17.080 8180 ERROR nova.api.openstack.wsgi     raise exceptions.ConnectTimeout(msg)
2019-04-08 07:45:17.080 8180 ERROR nova.api.openstack.wsgi ConnectTimeout: Request to http://10.15.1.100:9696/v2.0/ports/6e91102f-07e5-4e55-93f6-91087431a469 timed out
2019-04-08 07:45:17.080 8180 ERROR nova.api.openstack.wsgi 
2019-04-08 07:45:17.089 8180 INFO nova.api.openstack.wsgi [req-000d10ce-1481-43c9-ba9e-10af17a4ea28 - - - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'keystoneauth1.exceptions.connection.ConnectTimeout'>

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content