Cannot retrieve an oauth token with Load Balanced osconsole

Solution In Progress - Updated -

Issue

  • Requesting the token using /oauth/token/request stops working in load balanced osconsole.

We are running OpenShift Enterprise in AWS with 3 master nodes that host osconsole (atomic-openshift-master-api service).
There are two Elastic Load Balancers setup - one for internal traffic and one internet facing. All the internal communication towards osconsole goes over internal load balancer.

Reproduction steps:
1. Point both load balancers to master01
2. Login
3. Retrieve token by going to /oauth/token/display and clicking request a new token
4. The token should be correctly displayed
5. Point both load balancers to master02
6. Retrieve token by going to /oauth/token/display and clicking request a new token
7. Error 400 is displayed even after requesting another token

Expected:
Token is successfully retrieved and displayed

Fix/Workaround:
restart of the api service

This is the easiest way to reproduce the issue. I also noticed that even without modifying the load balancers and pointing them to all 3 instances causes the problem. Could not determine yet how long it takes, but it makes the product unstable. Currently the developers can use oc client using only the token, so when the service does not work, it is not possible to work.
Even though the token is visible in the URL, it could not be used with oc client (but this needs to be verified next time the issue occurs).

At present, as a workaround, we point a LB to single instance and after restart of the api it seems to work. We have to observe the instance if the issue does not appear in such case.

Switching Load Balancers causes it every time.
In other cases it's hard to say how long it takes before it stops working.

I expected that pointing LB towards 3 instances and not making any further changes won't trigger the issue.
In current situation we have to have a workaround or some plan for a fix.

Environment

  • OpenShift 3.1.0.4 and OpenShift 3.1.1.6 deployed in AWS

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.