Pod is in ImagePullBackOff or PodInitializing status in OSD/ROSA cluster

Solution Unverified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA)
    • 4
  • Red Hat OpenShift Dedicated (OSD)
    • 4
  • Red Hat OpenShift Cluster Manager (OCM)

Issue

  • A pod is failing with ImagePullBackOff or PodInitializing status, and the following event in OSD/ROSA cluster:

    Failed to pull image "registry.redhat.io/[registry_namespace]/[image_name]@sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxx": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531

Resolution

It is possible to check if there is an issue with the cluster pull-secret trying to pull the image manually from within a node following the manual podman pull fails within OCP 4 nodes article instructions.

If the manual pull fails, it could be caused by the cluster owner being banned (for example, if the account has been deleted). In this case, it is needed to request an Ownership transfer of OSD/ROSA clusters.

Root Cause

If the user who created the cluster and installed the add-ons has been removed, the user will be marked as "banned user" by the OpenShift Cluster Manager (think of it as deleted instead of banned user).

Diagnostic Steps

When issuing a rollout, the events log shows the message "unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials" when fetching the POD image:

Failed to pull image "registry.redhat.io/3scale-mas/system-rhel7@sha256:5f1b280f5eadc16e74e81fc4441a8db8b163a2f7bc1c3c86a768accd40059322": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments