Authentication operator fails to upgrade

Solution Unverified - Updated -

Environment

  • Red Hat OpenShift Container Platform
    • 4.x

Issue

  • Upgrade to 4.0.0-0.11 fails to upgrade cluster operator authentication.
  • The error is:
[sgaikwad@sgaikwad beta3]$ oc get clusterversion
NAME      VERSION      AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.11   True        False         130m    Error while reconciling: the cluster operator authentication has not yet successfully rolled out

Events:
  Type     Reason       Age               From                                   Message
  ----     ------       ----              ----                                   -------
  Normal   Scheduled    38s               default-scheduler                      Successfully assigned openshift-authentication/openshift-authentication-5d99d469bb-t62rm to ip-192.168.0.1.ec2.internal
  Warning  FailedMount  6s (x7 over 38s)  kubelet, ip-192.168.0.1.ec2.internal  MountVolume.SetUp failed for volume "v4-0-config-user-idp-0-file-data" : references non-existent secret key

Resolution

  • Create the secret with the correct key name.
  • In the below example, the reference to the key is users.htpasswd:
[user@host beta3]$ oc get secret v4-0-config-user-idp-0-file-data -o yaml
apiVersion: v1
data:
  users.htpasswd: dXNlcjE6JDJ5JDA1JGQ1d2kzRldrVlhUWmJIOXlUNFBzMmVweWhiZzB6R2hrcWxqVHBaTWNpeWJNMFYxcTZxT3dHCg==
kind: Secret
metadata:
  creationTimestamp: 2019-04-12T15:37:58Z
  name: v4-0-config-user-idp-0-file-data
  namespace: openshift-authentication
  • This secret is pulling the information from another secret created in openshift-config project
[user@host beta3]$ oc get secret  htpass-secret -o yaml -n openshift-config
apiVersion: v1
data:
  users.htpasswd: dXNlcjE6JDJ5JDA1JGQ1d2kzRldrVlhUWmJIOXlUNFBzMmVweWhiZzB6R2hrcWxqVHBaTWNpeWJNMFYxcTZxT3dHCg==
kind: Secret
metadata:
  creationTimestamp: 2019-04-12T15:37:58Z
  name: htpass-secret
  namespace: openshift-config
  resourceVersion: "61924"
  selfLink: /api/v1/namespaces/openshift-config/secrets/htpass-secret
  uid: f28a9d73-5d38-11e9-af8f-0e5563093fdc
type: Opaque

resourceVersion: "61925"
selfLink: /api/v1/namespaces/openshift-authentication/secrets/v4-0-config-user-idp-0-file-data
uid: f28c155d-5d38-11e9-af8f-0e5563093fdc
type: Opaque

* Delete and recreate the secret by specifying the correct key name:

[user@host beta3]$ oc delete secret htpass-secret -n openshift-config
secret "htpass-secret" deleted
[user@host beta3]$ oc create secret generic htpass-secret --from-file=htpasswd=users.htpasswd -n openshift-config
secret/htpass-secret created
~~~

Root Cause

  • The secret was generated with the wrong key name.

Diagnostic Steps

  • Check the output of below commands:
$ oc get secret -n openshift-config
$ oc get secret -n openshift-authentication
$ oc get secret <secretname> -o yaml

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.