GitOps operator 1.4.4 pods not deploying due to ImagePullBackOff

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS
  • Red Hat OpenShift Container Platform (OCP) 4
  • Red Hat OpenShift GitOps 1.4.4
  • After GitOps operator install or update to version 1.4.4

Issue

  • Multiple pods are stuck in states as ErrImagePullBackOff or ImagePullBackOff messages.

    Back-off pulling image "registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:ff4ad30752cf0d321cd6c2c6fd4490b716607ea2960558347440f2f370a586a8"

Or

Failed to pull image "registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:ff4ad30752cf0d321cd6c2c6fd4490b716607ea2960558347440f2f370a586a8": rpc error: code = Unknown desc = reading manifest sha256:ff4ad30752cf0d321cd6c2c6fd4490b716607ea2960558347440f2f370a586a8 in registry.redhat.io/openshift-gitops-1/argocd-rhel8: StatusCode: 404, <HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY>

Resolution

The issue has been fixed at v1.4.5 which released as RHBA-2022:1063 in 03/24, 2022. Please update your GitOps operator appropriately. If you cannot update immediately, you can consider to apply the following workaround, the pods can be redeployed by replacing the wrong image digest value with correct one through the following procedures.

  1. Move to the namespace you intalled the GitOps operators

    $ oc project openshift-operators

  2. Check the CSV of GitOps operator

    $ oc get csv
NAME                                DISPLAY                      VERSION    REPLACES                            PHASE
:
openshift-gitops-operator.v1.4.4    Red Hat OpenShift GitOps     1.4.4      openshift-gitops-operator.v1.4.3    Succeeded

  3. Replace the existing image digest with new correct one as follows.

    $ oc edit csv openshift-gitops-operator.v1.4.4
...Open an editor(in this time, it's vim)...
:%s/ff4ad30752cf0d321cd6c2c6fd4490b716607ea2960558347440f2f370a586a8/28dfb790f234e8819c7641971956a08e8c7167d6fe8d61594bb952eb5ca84ab1/g

  4. Wait to re-deploy the pods, or restart manually as removing the target pods.

    // Before
$ oc get pod
NAME                                                         READY   STATUS                  RESTARTS   AGE
:
openshift-gitops-application-controller-0                    0/1     ImagePullBackOff        0          8h
openshift-gitops-applicationset-controller-7ddb47944-82xm2   1/1     Running                 0          8h
openshift-gitops-dex-server-7f47bbf4b5-cjwhk                 1/1     Running                 0          6d3h
openshift-gitops-dex-server-864cb88696-8v9pd                 0/1     Init:ImagePullBackOff   0          8h
openshift-gitops-redis-7867d74fb4-vvmdp                      1/1     Running                 0          6d3h
openshift-gitops-repo-server-547c78cffb-rfv9q                0/1     ImagePullBackOff        0          8h
openshift-gitops-repo-server-5b66697f8f-6mr67                1/1     Running                 0          6d3h
openshift-gitops-server-589b4d5cf6-gffwb                     0/1     ImagePullBackOff        0          8h
openshift-gitops-server-7bf7f946d7-p69z6                     1/1     Running                 0          6d3h

// After
$ oc get pod
NAME                                                         READY   STATUS    RESTARTS   AGE
:
openshift-gitops-application-controller-0                    1/1     Running   0          7m28s
openshift-gitops-applicationset-controller-7ddb47944-82xm2   1/1     Running   0          8h
openshift-gitops-dex-server-5cf56fd458-gdvz4                 1/1     Running   0          8m19s
openshift-gitops-redis-7867d74fb4-vvmdp                      1/1     Running   0          6d3h
openshift-gitops-repo-server-7577d4c59d-mwnbf                1/1     Running   0          8m19s
openshift-gitops-server-6449c96f58-86xd2                     1/1     Running   0          8m21s

Root Cause

There is an incorrect image digest value. Operator keeps trying to deploy the pods using non-existant image digest values.

Specifically, registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:ff4ad30752cf0d321cd6c2c6fd4490b716607ea2960558347440f2f370a586a8 is incorrect and does not exist.

Refer Red Hat OpenShift GitOps Argo CD to find the correct image digest value.

// I've identified the digest value when tagging v1.4.4-6 as 1.4.4.
// As you can see, the correct one is 28dfb790f234e8819c7641971956a08e8c7167d6fe8d61594bb952eb5ca84ab1.
Tag: v1.4.4-6 (v1.4.4)
:
Digest: registry.access.redhat.com/openshift-gitops-1/argocd-rhel8@sha256:28dfb790f234e8819c7641971956a08e8c7167d6fe8d61594bb952eb5ca84ab1

Diagnostic Steps

  • View the event logs to verify the problem is the failure to pull the image with this specific digest value: "ff4ad30752cf0d321cd6c2c6fd4490b716607ea2960558347440f2f370a586a8".

    $ oc describe pod <a pod name failed to pull image>
:
Events:
Type    Reason   Age  From     Message
----    ------   ---- ----     -------
Normal  Pulling  xxx  kubelet  Pulling image "registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:ff4ad30752cf0d321cd6c2c6fd4490b716607ea2960558347440f2f370a586a8"
Normal  BackOff  xxx  kubelet  Back-off pulling image "registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:ff4ad30752cf0d321cd6c2c6fd4490b716607ea2960558347440f2f370a586a8"

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments