openstack Unauthorized HTTP 401 on running any commands from controller nodes.

Issue

• Running client commands like nova, they get a :

Wed Sep 23 15:37:31 CEST 2015 ERROR: openstack Unauthorized (HTTP 401) (Request-ID: req-46c2dccb-806f-468e-89b0-6f7129d6a986)

• Doing a sequence of 'penStack-service list' approximately 1 out of 6 fail with authorization failed.
• Restarting all controller nodes possibly solved the issue for now; that is they have not had the issue since they rebooted the controller nodes.
• one controller node "macb083fee7c1ff" which was half updated with puppet a few days back. Since the issue with authentication was intermittent,
  maybe it happened only when requests went to this host
• An example of HTTP 401 was seen when user-login failed:

2015-09-15 07:40:01.569 145632 ERROR keystone.auth.plugins.password [-] Could not find user: ospadmin
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password Traceback (most recent call last):
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password   File "/usr/lib/python2.7/site-packages/keystone/auth/plugins/password.py", line 101, in _validate_and_normalize_auth_data
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password     user_name, domain_ref['id'])
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password   File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 202, in wrapper
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password     return f(self, *args, **kwargs)
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password   File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 213, in wrapper
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password     return f(self, *args, **kwargs)
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password   File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 600, in get_user_by_name
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password     ref = driver.get_user_by_name(user_name, domain_id)
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password   File "/usr/lib/python2.7/site-packages/keystone/identity/backends/sql.py", line 153, in get_user_by_name
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password     raise exception.UserNotFound(user_id=user_name)
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password UserNotFound: Could not find user: ospadmin
2015-09-15 07:40:01.569 145632 TRACE keystone.auth.plugins.password 
2015-09-15 07:40:01.615 145632 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 192.168.10.21
2015-09-15 07:40:01.616 145632 INFO eventlet.wsgi.server [-] 192.168.10.21 - - [15/Sep/2015 07:40:01] "POST /v3/auth/tokens HTTP/1.1" 401 357 0.067372

• In their nova.conf:

auth_host=ldap_server_fqdn

• ldap_server_fqdn is the hostname for the ip pointing to the keystone service currently on controller node.

auth_version=  (blank)