Chapter 7. Advanced Concepts

7.1. Customizing the Quay Deployment

The Quay Operator takes an opinionated strategy towards deploying Quay and its dependencies, however there are places where the Quay deployment can be customized.

7.1.1. Quay Application Configuration

Once deployed, the Quay application itself can be configured as normal using the config editor UI or by modifying the Secret containing the Quay configuration bundle. The Operator uses the Secret named in the spec.configBundleSecret field but does not watch this resource for changes. It is recommended that configuration changes be made to a new Secret resource and the spec.configBundleSecret field be updated to reflect the change. In the event there are issues with the new configuration, it is simple to revert the value of spec.configBundleSecret to the older Secret.

7.1.2. Customizing External Access to the Registry

When running on OpenShift, the Routes API is available and will automatically be used as a managed component. After creating the QuayRegistry, the external access point can be found in the status block of the QuayRegistry: 

status:
  registryEndpoint: some-quay.my-namespace.apps.mycluster.com

When running on native Kubernetes, the Operator creates a Service of type: ClusterIP for your registry. You are then responsible for external access (like Ingress). 

$ kubectl get services -n <namespace>
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP          PORT(S)             AGE
some-quay               ClusterIP   172.30.143.199   <none>               443/TCP,9091/TCP    23h

7.1.2.1. Using a Custom Hostname and TLS

By default, a Route will be created with the default generated hostname and a certificate/key pair will be generated for TLS. If you want to access Red Hat Quay using a custom hostname and bring your own TLS certificate/key pair, follow these steps.

If FEATURE_BUILD_SUPPORT: true, then make sure the certificate/key pair is also valid for the BUILDMAN_HOSTNAME.

If the given cert/key pair is invalid for the above hostnames, then the Quay Operator will reject your provided certificate/key pair and generate one to be used by Red Hat Quay.

Next, create a Secret with the following content: 

apiVersion: v1
kind: Secret
metadata:
  name: my-config-bundle
data:
  config.yaml: <must include SERVER_HOSTNAME field with your custom hostname>
  ssl.cert: <your TLS certificate>
  ssl.key: <your TLS key>

Then, create a QuayRegistry which references the created Secret: 

apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: some-quay
spec:
  configBundleSecret: my-config-bundle

7.1.2.2. Using OpenShift Provided TLS Certificate

It is preferred to have TLS terminated in the Quay app container. Therefore, to use the OpenShift provided TLS, you must create a Route with type "reencrypt", which will use the OpenShift provided TLS at the edge, and Quay Operator-generated TLS within the cluster. This is achieved by marking the route component as unmanaged, and creating your own Route which reencrypts TLS using the Operator-generated CA certificate.

Create a Secret with a config.yaml key containing the SERVER_HOSTNAME field of value <route-name>-<namespace>.apps.<cluster-domain> (the Route with this hostname will be created in a later step). 

apiVersion: v1
kind: Secret
metadata:
  name: my-config-bundle
data:
  config.yaml: <must include SERVER_HOSTNAME field with your custom hostname>

Create a QuayRegistry referencing the above Secret and with the route component unmanaged: 

apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: some-quay
spec:
  configBundleSecret: my-config-bundle
  components:
  - kind: route
    managed: false

Wait for the QuayRegistry to be fully reconciled by the Quay Operator. Then, acquire the generated TLS certificate by finding the Secret being mounted into the Quay app pods and copying the tls.cert value.

Create a Route with TLS reencryption and the destination CA certificate you copied above: 

apiVersion: v1
kind: Route
metadata:
  name: registry
  namespace: <namespace>
spec:
  to:
    kind: Service
    name: <quay-service-name>
  tls:
    termination: reencrypt
    destinationCACertificate:
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----

You can now access your Quay registry using the created Route.

7.1.3. Disabling Route Component

To prevent the Operator from creating a Route, mark the component as unmanaged in the QuayRegistry: 

apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: some-quay
spec:
  components:
    - kind: route
      managed: false
Note

Disabling the default Route means you are now responsible for creating a Route, Service, or Ingress in order to access the Quay instance and that whatever DNS you use must match the SERVER_HOSTNAME in the Quay config.

7.1.4. Resizing Managed Storage

The Quay Operator creates default object storage using the defaults provided by RHOCS when creating a NooBaa object (50 Gib). There are two ways to extend this storage; you can resize an existing PVC or add more PVCs to a new storage pool.

7.1.4.1. Resize Noobaa PVC

  1. Log into the OpenShift console and select Storage → Persistent Volume Claims.
  2. Select the PersistentVolumeClaim named like noobaa-default-backing-store-noobaa-pvc-*.
  3. From the Action menu, select Expand PVC.
  4. Enter the new size of the Persistent Volume Claim and select Expand.

After a few minutes (depending on the size of the PVC), the expanded size should reflect in the PVC’s Capacity field.

Note

Expanding CSI volumes is a Technology Preview feature only. For more information, see https://access.redhat.com/documentation/en-us/openshift_container_platform/4.6/html/storage/expanding-persistent-volumes.

7.1.4.2. Add Another Storage Pool

  1. Log into the OpenShift console and select Networking → Routes. Make sure the openshift-storage project is selected.
  2. Click on the Location field for the noobaa-mgmt Route.
  3. Log into the Noobaa Management Console.
  4. On the main dashboard, under Storage Resources, select Add Storage Resources.
  5. Select Deploy Kubernetes Pool
  6. Enter a new pool name. Click Next.
  7. Choose the number of Pods to manage the pool and set the size per node. Click Next.
  8. Click Deploy.

After a few minutes, the additional storage pool will be added to the Noobaa resources and available for use by Red Hat Quay.

7.1.5. Customizing Default Operator Images

Note

Using this mechanism is not supported for production Quay environments and is strongly encouraged only for development/testing purposes. There is no guarantee your deployment will work correctly when using non-default images with the Quay Operator.

In certain circumstances, it may be useful to override the default images used by the Operator. This can be done by setting one or more environment variables in the Quay Operator ClusterServiceVersion.

7.1.5.1. Environment Variables

The following environment variables are used in the Operator to override component images:

Environment Variable

Component

RELATED_IMAGE_COMPONENT_QUAY

base

RELATED_IMAGE_COMPONENT_CLAIR

clair

RELATED_IMAGE_COMPONENT_POSTGRES

postgres and clair databases

RELATED_IMAGE_COMPONENT_REDIS

redis

Note

Override images must be referenced by manifest (@sha256:), not by tag (:latest).

7.1.5.2. Applying Overrides to a Running Operator

When the Quay Operator is installed in a cluster via the Operator Lifecycle Manager (OLM), the managed component container images can be easily overridden by modifying the ClusterServiceVersion object, which is OLM’s representation of a running Operator in the cluster. Find the Quay Operator’s ClusterServiceVersion either by using a Kubernetes UI or kubectl/oc: 

$ oc get clusterserviceversions -n <your-namespace>

Using the UI, oc edit, or any other method, modify the Quay ClusterServiceVersion to include the environment variables outlined above to point to the override images:

JSONPath: spec.install.spec.deployments[0].spec.template.spec.containers[0].env 

- name: RELATED_IMAGE_COMPONENT_QUAY
  value: quay.io/projectquay/quay@sha256:c35f5af964431673f4ff5c9e90bdf45f19e38b8742b5903d41c10cc7f6339a6d
- name: RELATED_IMAGE_COMPONENT_CLAIR
  value: quay.io/projectquay/clair@sha256:70c99feceb4c0973540d22e740659cd8d616775d3ad1c1698ddf71d0221f3ce6
- name: RELATED_IMAGE_COMPONENT_POSTGRES
  value: centos/postgresql-10-centos7@sha256:de1560cb35e5ec643e7b3a772ebaac8e3a7a2a8e8271d9e91ff023539b4dfb33
- name: RELATED_IMAGE_COMPONENT_REDIS
  value: centos/redis-32-centos7@sha256:06dbb609484330ec6be6090109f1fa16e936afcf975d1cbc5fff3e6c7cae7542

Note that this is done at the Operator level, so every QuayRegistry will be deployed using these same overrides.

7.1.6. AWS S3 CloudFront

If you use AWS S3 CloudFront for backend registry storage, specify the private key as shown in the following example: 

$ oc create secret generic --from-file config.yaml=./config_awss3cloudfront.yaml --from-file default-cloudfront-signing-key.pem=./default-cloudfront-signing-key.pem test-config-bundle