After reboot director node, getting command output error in "openstack stack list " or " openstack overcloud status --plan"

Solution In Progress - Updated -

Issue

  • We have successfully deployed OSP16 STACK but after a reboot of the undercloud / director node , we can't get output of many commands.

  • Any openstack CLI call is failing:

(undercloud) [stack@undercloud ~]$ openstack stack list

ERROR: b"<html><body><h1>504 Gateway Time-out</h1>\nThe server didn't respond in time.\n</body></html>\n"
  • Even openstack overcloud status --plan overcloud is failing:
(undercloud) [stack@undercloud ~]$ openstack overcloud status --plan overcloud
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/mistralclient/api/base.py", line 144, in _create
    resp = self.http_client.post(url, data, headers)
  File "/usr/lib/python3.6/site-packages/mistralclient/api/httpclient.py", line 54, in decorator
    resp = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/mistralclient/api/httpclient.py", line 120, in post
    return self.session.post(self.base_url + url, data=body, **options)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1106, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 943, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.GatewayTimeout: Gateway Timeout (HTTP 504)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1058, in take_action
    plan=plan
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 438, in get_deployment_status
    workflow_input=workflow_input
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/base.py", line 39, in start_workflow
    workflow_input=workflow_input
  File "/usr/lib/python3.6/site-packages/mistralclient/api/v2/executions.py", line 61, in create
    return self._create('/executions', data)
  File "/usr/lib/python3.6/site-packages/mistralclient/api/base.py", line 146, in _create
    self._raise_api_exception(ex.response)
  File "/usr/lib/python3.6/site-packages/mistralclient/api/base.py", line 216, in _raise_api_exception
    error_message=error_data)
mistralclient.api.base.APIException: b"<html><body><h1>504 Gateway Time-out</h1>\nThe server didn't respond in time.\n</body></html>\n"
b"<html><body><h1>504 Gateway Time-out</h1>\nThe server didn't respond in time.\n</body></html>\n"
  • Rabbitmq service is not restarting properly:
Jul 29 21:03:52 undercloud systemd[1]: tripleo_rabbitmq.service: Main process exited, code=exited, status=1/FAILURE
Jul 29 21:03:52 undercloud systemd[1]: tripleo_rabbitmq.service: Failed with result 'exit-code'.
Jul 29 21:03:52 undercloud systemd[1]: tripleo_rabbitmq.service: Service RestartSec=100ms expired, scheduling restart.
Jul 29 21:03:52 undercloud systemd[1]: tripleo_rabbitmq.service: Scheduled restart job, restart counter is at 34.

Environment

  • Red Hat OpenStack Platform 16.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content