Quay operator reporting that it cannot inflate Kubernetes objects

Solution Verified - Updated -

Environment

  • Red Hat Quay Operator
    • 3.x
  • Red Hat OpenShift Container Platform
    • >=4.12.x

Issue

  • Quay operator cannot reconcile the Quay deployment after changes are made in the config tool.

Resolution

  1. Find the custom config bundle secret name that the operator is using to deploy Quay:

    $ oc get quayregistry name-of-registry -n quay-enterprise-namespace -o yaml | grep -i configbundle
configBundleSecret: quay-config-bundle-xxxxxxxxxxx

  2. Open OpenShift console and locate the namespace where Quay is deployed. Click on Workloads -> Secrets on the left side
    and find the custom config bundle secret. Open the secret and set it to editing mode by clicking Actions -> Edit on the left side.

  3. Find the key named extra_ca_cert_ca-bundle.crt and delete it by clicking Remove.

  4. Save the new config bundle and let the operator reconcile the deployment. New pods should come online soon. If the operator does not reconcile properly, delete the operator pod and let it restart:

    $ oc get pods -n openshift-operators
$ oc delete pod quay-operator-pod -n openshift-operators

    If the operator is not installed in the default location, replace the -n openshift-operators with the correct value.

Root Cause

The error is most likely caused by the accentuated letters found in one of the CA authorities that are stored in the CA bundle.

Diagnostic Steps

  • The following log is observed in the operator logs:
$ oc logs quay-operator-pod
...
3271.692005884362491e+09 DEBUG events Warning {"object": {"kind":"QuayRegistry","namespace":"quay-namespace","name":"quay-registry","uid":"2a5a97de-8147-4ca5-948e-5f66f65ed422","apiVersion":"quay.redhat.com/v1","resourceVersion":"1555539382"}, "reason": "ComponentCreationFailed", "message": "could not inflate kubernetes objects: error converting JSON to YAML: yaml: control characters are not allowed"}
...
  • When examining the config bundle secret in more detail, the extra_ca_certs_ca-bundle.crt is added to it. Decoding the value of the CA bundle yields:
:~/cases/classroom-sa# cat quay-registry-quay-config-bundle-h7vmn.yaml | yq -r '.data."extra_ca_cert_ca-bundle.crt"' | base64 -d -w0 | grep -Po "^\#.*$"
...
# Classroom Global Root Certification Authority
# Classroom example (Class example) F��������tan�������ºs�������­tv�������¡ny
# Classroom Example Certificate Authority
...
  • The Classroom Example certificate contains unprintable UTF-8 characters, most likely caused by the fact that the original certificate has accentuated letters in its name:
bash-4.4$ cat tls-ca-bundle.pem | grep -i classroom
# Classroom Example (Class Example) Főtanúsítvány

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments