openstack Unauthorized HTTP 401 on running any commands from controller nodes.

Solution Verified - Updated -

Issue

  • Running client commands like nova, they get a :
Wed Sep 23 15:37:31 CEST 2015 ERROR: openstack Unauthorized (HTTP 401) (Request-ID: req-46c2dccb-806f-468e-89b0-6f7129d6a986)
  • Doing a sequence of 'penStack-service list' approximately 1 out of 6 fail with authorization failed.
  • Restarting all controller nodes possibly solved the issue for now; that is they have not had the issue since they rebooted the controller nodes.
  • one controller node "macb083fee7c1ff" which was half updated with puppet a few days back. Since the issue with authentication was intermittent,
    maybe it happened only when requests went to this host
  • An example of HTTP 401 was seen when user-login failed:
2015-09-15 07:40:01.569 145632 ERROR keystone.auth.plugins.password [-] Could not find user: ospadmin
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password Traceback (most recent call last):
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password   File "/usr/lib/python2.7/site-packages/keystone/auth/plugins/password.py", line 101, in _validate_and_normalize_auth_data
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password     user_name, domain_ref['id'])
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password   File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 202, in wrapper
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password     return f(self, *args, **kwargs)
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password   File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 213, in wrapper
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password     return f(self, *args, **kwargs)
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password   File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 600, in get_user_by_name
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password     ref = driver.get_user_by_name(user_name, domain_id)
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password   File "/usr/lib/python2.7/site-packages/keystone/identity/backends/sql.py", line 153, in get_user_by_name
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password     raise exception.UserNotFound(user_id=user_name)
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password UserNotFound: Could not find user: ospadmin
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password 
2015-09-15 07:40:01.615 145632 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 192.168.10.21
2015-09-15 07:40:01.616 145632 INFO eventlet.wsgi.server [-] 192.168.10.21 - - [15/Sep/2015 07:40:01] "POST /v3/auth/tokens HTTP/1.1" 401 357 0.067372

  • In their nova.conf:
auth_host=ldap_server_fqdn
  • ldap_server_fqdn is the hostname for the ip pointing to the keystone service currently on controller node.
auth_version=  (blank)

Environment

  • Red Hat OpenStack 6.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In