The image-registry co is Degraded with Unable to apply resources: unable to sync storage configuration error in GCP.

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform 4.12
  • GCP

Issue

  • The image-registry and cloud credential co are stuck in Degraded state with below error:
NAME                                      VERSION  AVAILABLE  PROGRESSING  DEGRADED  SINCE
cloud-credential                          4.12.21  True       True         True      3d
image-registry                            4.12.21  False      True         True      1h1m

message: 'Progressing: Unable to apply resources: unable to sync storage configuration:
      Get "https://storage.googleapis.com/storage/xxxx?alt=json&prettyPrint=false&projection=full":
      oauth2: cannot fetch token: 400 Bad Request

Resolution

  • Provide the necessary permission to the iam user to access the cluster resources and make sure GCP secret has all valid arguments as per the GCP documentation.

Root Cause

  • The lack of permissions at GCP end caused this issue.

Diagnostic Steps

  • Check for the image-registry and cloud credentials co status why it is stuck in degraded state.
$ oc get co image-registry -o yaml
Message        : Progressing: Unable to apply resources: unable to sync storage configuration:
  • If image-registry is stuck in degraded due to above error, check for cloud credentials operator pod logs and look for below error:
$ oc logs cloud-credential-operator-xxxxx -c cloud-credential-operator
level=error msg="error syncing credentials: error syncing creds in mint-mode: error creating key: rpc error: code = FailedPrecondition desc = Key creation is not allowed on this service account.\nerror details: name = PreconditionFailure type = constraints/iam.disableServiceAccountKeyCreation subj = projects/xxxx/serviceAccounts/xxxx?configvalue=xxxx desc = Key creation is not allowed on this service account." controller=credreq cr=openshift-cloud-credential-operator/openshift-gcp-xxx secret=openshift-cloud-controller-manager/gcp-ccm-cloud-credentials
level=error msg="errored with condition: CredentialsProvisionFailure" controller=credreq cr=openshift-cloud-credential-operator/openshift-gcp-xxx secret=openshift-cloud-controller-manager/gcp-ccm-cloud-credentials

 level=error msg="error syncing creds in mint-mode" actuator=gcp cr=openshift-cloud-credential-operator/openshift-gcp-xxx error="error creating key: rpc error: code = FailedPrecondition desc = Key creation is not allowed on this service account.\nerror details: name = PreconditionFailure type = constraints/iam.disableServiceAccountKeyCreation subj = projects/xxxx/serviceAccounts/xxxx?configvalue=xxxx desc = Key creation is not allowed on this service account."

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments