Weird issues with apparent credential timeout or memcached behavior

Issue

• We suspect what's at the heart of this issue is something is having trouble talking to memcached or memcached is having issues. Just a guess though.

• We've noticed that sessions in horizon become useless inside of about 5-10 minutes, despite the timeout for horizon set to about an hour and credential timeout in keystone set to two hours. When this happens you can't list anything in horizon an get constant error messages in the top left of the window until you log out. In the logs we'll see messages about not having permissions to a project when this occurs along with stack traces containing:

openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: https://public:13000/v3/projects, a bytes-like object is required, not 'str'

• This gets returned to the python sdk at times implying the issue isn't limited to horizon.

• It's like the session gets corrupt in someway, or that it's not actually being distributed by memcache to the other controller nodes

• Using LDAP authenticated users will break after some time of usage but using locally created users in the default domain are not hitting this issue.

• The CLI is also affected by this issue:

openstack --os-auth-type token --os-auth-url https://public.localdomain:13000/v3  --os-user-id e50461c7e4cfb42184b04bf391650a580d6a33345c2ee9ea760d993f31d6411c --os-token gAAAAABeTWbGINSV3uuIzTeqEsLZftX4D_Gagzip0uAsIoIurRXHW5A1wu6gdgd1gwgY2xTh_DICcGLxhKdXYxwknov6gn-U-jZVpm7CMFs2YZnDDWMJHgLxwWvb5vkHadZElPIIWVJgz7Dkn6LxT7c11bHfFtCgTY5NeHRBUQsjL0styNKSZMt0Bo4bCxLVZI6BGdkSHIHOA0bQPb9-PJFD1qlePHv3Rw --os-project-domain-name domain --os-project-name project server list
a bytes-like object is required, not 'str' (HTTP 400) (Request-ID: req-22f11bab-d189-4710-890d-7caa3a6f16b8)