openstack Unauthorized HTTP 401 on running any commands from controller nodes.
Issue
- Running client commands like nova, they get a :
Wed Sep 23 15:37:31 CEST 2015 ERROR: openstack Unauthorized (HTTP 401) (Request-ID: req-46c2dccb-806f-468e-89b0-6f7129d6a986)
- Doing a sequence of 'penStack-service list' approximately 1 out of 6 fail with authorization failed.
- Restarting all controller nodes possibly solved the issue for now; that is they have not had the issue since they rebooted the controller nodes.
- one controller node "macb083fee7c1ff" which was half updated with puppet a few days back. Since the issue with authentication was intermittent,
maybe it happened only when requests went to this host - An example of HTTP 401 was seen when user-login failed:
2015-09-15 07:40:01.569 145632 ERROR keystone.auth.plugins.password [-] Could not find user: ospadmin
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password Traceback (most recent call last):
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password File "/usr/lib/python2.7/site-packages/keystone/auth/plugins/password.py", line 101, in _validate_and_normalize_auth_data
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password user_name, domain_ref['id'])
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 202, in wrapper
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password return f(self, *args, **kwargs)
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 213, in wrapper
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password return f(self, *args, **kwargs)
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 600, in get_user_by_name
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password ref = driver.get_user_by_name(user_name, domain_id)
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password File "/usr/lib/python2.7/site-packages/keystone/identity/backends/sql.py", line 153, in get_user_by_name
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password raise exception.UserNotFound(user_id=user_name)
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password UserNotFound: Could not find user: ospadmin
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password
2015-09-15 07:40:01.615 145632 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 192.168.10.21
2015-09-15 07:40:01.616 145632 INFO eventlet.wsgi.server [-] 192.168.10.21 - - [15/Sep/2015 07:40:01] "POST /v3/auth/tokens HTTP/1.1" 401 357 0.067372
- In their nova.conf:
auth_host=ldap_server_fqdn
ldap_server_fqdn
is the hostname for the ip pointing to the keystone service currently on controller node.
auth_version= (blank)
Environment
- Red Hat OpenStack 6.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.