Menu Close

Chapter 3. Configuring Quay before deployment

The Operator can manage all the Red Hat Quay components when deploying on OpenShift, and this is the default configuration. Alternatively, you can manage one or more components externally yourself, where you want more control over the set up, and then allow the Operator to manage the remaining components.

The standard pattern for configuring unmanaged components is:

  1. Create a config.yaml configuration file with the appropriate settings
  2. Create a Secret using the configuration file

    $ oc create secret generic --from-file config.yaml=./config.yaml config-bundle-secret
  3. Create a QuayRegistry YAML file quayregistry.yaml, identifying the unmanaged components and also referencing the created Secret, for example:

    quayregistry.yaml

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: example-registry
      namespace: quay-enterprise
    spec:
      configBundleSecret: config-bundle-secret
      components:
        - kind: objectstorage
          managed: false

  4. Deploy the registry using the YAML file:

    $ oc create -n quay-enterprise -f quayregistry.yaml

3.1. Pre-configuring Quay for automation

Quay has a number of configuration options that support automation. These options can be set before deployment, to minimize the need to interact with the user interface.

3.1.1. Allowing the API to create the first user

Set the config option FEATURE_USER_INITIALIZE to true, so that you can use the API /api/v1/user/initialize to create the first user. This API endpoint does not require authentication, unlike all other registry API calls which require an OAuth token which is generated by an OAuth application in an existing organization.

Once you have deployed Quay, you can use the API to create a user, for example, quayadmin, provided no other users have already been created. For more information, see the section on Creating the first user using the API.

3.1.2. Enabling general API access

Set the config option BROWSER_API_CALLS_XHR_ONLY to false, to allow general access to the Quay registry API.

3.1.3. Adding a super user

While you cannot create a user until after deployment, it is convenient to ensure that first user is an administrator with full permissions. It is easier to configure this in advance, using the SUPER_USER configuration object.

3.1.4. Restricting user creation

Once you have configured a super user, you can restrict the ability to create new users to the super user group. Set the FEATURE_USER_CREATION to false to restrict user creation.

3.1.5. Enabling new functionality

If you want to take advantage of the new functionality in Red Hat Quay 3.7, enable some or all of the following features:

...
FEATURE_QUOTA_MANAGEMENT: true
FEATURE_BUILD_SUPPORT: true
FEATURE_PROXY_CACHE: true
FEATURE_STORAGE_REPLICATION: true
DEFAULT_SYSTEM_REJECT_QUOTA_BYTES: 102400000
...

3.1.6. Suggested configuration for automation

Create a config.yaml configuration file that includes the appropriate settings:

config.yaml

...
FEATURE_USER_INITIALIZE: true
BROWSER_API_CALLS_XHR_ONLY: false
SUPER_USERS:
- quayadmin
FEATURE_USER_CREATION: false
...

3.1.7. Deploying the Operator using the initial configuration

  1. Create a Secret using the configuration file

    $ oc create secret generic -n quay-enterprise --from-file config.yaml=./config.yaml init-config-bundle-secret
  2. Create a QuayRegistry YAML file quayregistry.yaml, identifying the unmanaged components and also referencing the created Secret, for example:

    quayregistry.yaml

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: example-registry
      namespace: quay-enterprise
    spec:
      configBundleSecret: init-config-bundle-secret

  3. Deploy the registry:

    $ oc create -n quay-enterprise -f quayregistry.yaml
  4. Create the first user, quayadmin, using the API

3.2. Configuring object storage

You need to configure object storage before installing Red Hat Quay, irrespective of whether you are allowing the Operator to manage the storage or managing it yourself.

If you want the Operator to be responsible for managing storage, see the section on Managed storage for information on installing and configuring the NooBaa / RHOCS Operator.

If you are using a separate storage solution, set objectstorage as unmanaged when configuring the Operator. See the following section. Unmanaged storage, for details of configuring existing storage.

3.2.1. Unmanaged storage

Some configuration examples for unmanaged storage are provided in this section for convenience. See the Red Hat Quay configuration guide for full details for setting up object storage.

3.2.1.1. AWS S3 storage

DISTRIBUTED_STORAGE_CONFIG:
  s3Storage:
    - S3Storage
    - host: s3.us-east-2.amazonaws.com
      s3_access_key: ABCDEFGHIJKLMN
      s3_secret_key: OL3ABCDEFGHIJKLMN
      s3_bucket: quay_bucket
      storage_path: /datastorage/registry
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: []
DISTRIBUTED_STORAGE_PREFERENCE:
    - s3Storage

3.2.1.2. Google Cloud storage

DISTRIBUTED_STORAGE_CONFIG:
    googleCloudStorage:
        - GoogleCloudStorage
        - access_key: GOOGQIMFB3ABCDEFGHIJKLMN
          bucket_name: quay-bucket
          secret_key: FhDAYe2HeuAKfvZCAGyOioNaaRABCDEFGHIJKLMN
          storage_path: /datastorage/registry
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: []
DISTRIBUTED_STORAGE_PREFERENCE:
    - googleCloudStorage

3.2.1.3. Azure storage

DISTRIBUTED_STORAGE_CONFIG:
  azureStorage:
    - AzureStorage
      azure_container: azure_container_here
      storage_path: /datastorage/registry
    - azure_account_name: azure_account_name_here
      azure_account_key: azure_account_key_here
      sas_token: some/path/
      endpoint_url: https://[account-name].blob.core.usgovcloudapi.net 1
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: []
DISTRIBUTED_STORAGE_PREFERENCE:
    - azureStorage
1
The endpoint_url parameter for Azure storage is optional and can be used with Microsoft Azure Government (MAG) endpoints. If left blank, the endpoint_url will connect to the normal Azure region.

As of Red Hat Quay 3.7, you must use the Primary endpoint of your MAG Blob service. Using the Secondary endpoint of your MAG Blob service will result in the following error: AuthenticationErrorDetail:Cannot find the claimed account when trying to GetProperties for the account whusc8-secondary.

3.2.1.4. Ceph / RadosGW Storage / Hitachi HCP storage

DISTRIBUTED_STORAGE_CONFIG:
  radosGWStorage:
    - RadosGWStorage
    - access_key: access_key_here
      secret_key: secret_key_here
      bucket_name: bucket_name_here
      hostname: hostname_here
      is_secure: 'true'
      port: '443'
      storage_path: /datastorage/registry
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: []
DISTRIBUTED_STORAGE_PREFERENCE:
    - default

3.2.1.5. Swift storage

DISTRIBUTED_STORAGE_CONFIG:
  swiftStorage:
    - SwiftStorage
    - swift_user: swift_user_here
      swift_password: swift_password_here
      swift_container: swift_container_here
      auth_url: https://example.org/swift/v1/quay
      auth_version: 1
      ca_cert_path: /conf/stack/swift.cert"
      storage_path: /datastorage/registry
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: []
DISTRIBUTED_STORAGE_PREFERENCE:
    - swiftStorage

3.2.1.6. NooBaa unmanaged storage

Use the following procedure to deploy NooBaa as your unmanaged storage configuration.

Procedure

  1. Create a NooBaa Object Bucket Claim in the {product-title} console by navigating to StorageObject Bucket Claims.
  2. Retrieve the Object Bucket Claim Data details, including the Access Key, Bucket Name, Endpoint (hostname), and Secret Key.
  3. Create a config.yaml configuration file using the information for the Object Bucket Claim:

    DISTRIBUTED_STORAGE_CONFIG:
      default:
        - RHOCSStorage
        - access_key: WmrXtSGk8B3nABCDEFGH
          bucket_name: my-noobaa-bucket-claim-8b844191-dc6c-444e-9ea4-87ece0abcdef
          hostname: s3.openshift-storage.svc.cluster.local
          is_secure: true
          port: "443"
          secret_key: X9P5SDGJtmSuHFCMSLMbdNCMfUABCDEFGH+C5QD
          storage_path: /datastorage/registry
    DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: []
    DISTRIBUTED_STORAGE_PREFERENCE:
      - default

For more information about configuring an Object Bucket Claim, see Object Bucket Claim.

3.2.2. Managed storage

If you want the Operator to manage object storage for Quay, your cluster needs to be capable of providing object storage via the ObjectBucketClaim API. Using the Red Hat OpenShift Data Foundation (ODF) Operator, there are two supported options available:

  • A standalone instance of the Multi-Cloud Object Gateway backed by a local Kubernetes PersistentVolume storage

    • Not highly available
    • Included in the Quay subscription
    • Does not require a separate subscription for ODF
  • A production deployment of ODF with scale-out Object Service and Ceph

    • Highly available
    • Requires a separate subscription for ODF

To use the standalone instance option, continue reading below. For production deployment of ODF, please refer to the official documentation.

Note

Object storage disk space is allocated automatically by the Operator with 50 GiB. This number represents a usable amount of storage for most small to medium Red Hat Quay installations but may not be sufficient for your use cases. Resizing the RHOCS volume is currently not handled by the Operator. See the section below on resizing managed storage for more details.

3.2.2.1. About The Standalone Object Gateway

As part of a Red Hat Quay subscription, users are entitled to use the Multi-Cloud Object Gateway (MCG) component of the Red Hat OpenShift Data Foundation Operator (formerly known as OpenShift Container Storage Operator). This gateway component allows you to provide an S3-compatible object storage interface to Quay backed by Kubernetes PersistentVolume-based block storage. The usage is limited to a Quay deployment managed by the Operator and to the exact specifications of the MCG instance as documented below.

Since Red Hat Quay does not support local filesystem storage, users can leverage the gateway in combination with Kubernetes PersistentVolume storage instead, to provide a supported deployment. A PersistentVolume is directly mounted on the gateway instance as a backing store for object storage and any block-based StorageClass is supported.

By the nature of PersistentVolume, this is not a scale-out, highly available solution and does not replace a scale-out storage system like Red Hat OpenShift Data Foundation (ODF). Only a single instance of the gateway is running. If the pod running the gateway becomes unavailable due to rescheduling, updates or unplanned downtime, this will cause temporary degradation of the connected Quay instances.

3.2.2.1.1. Create A Standalone Object Gateway

To install the ODF (formerly known as OpenShift Container Storage) Operator and configure a single instance Multi-Cloud Gateway service, follow these steps:

  1. Open the OpenShift console and select Operators → OperatorHub, then select the OpenShift Data Foundation Operator.
  2. Select Install. Accept all default options and select Install again.
  3. Within a minute, the Operator will install and create a namespace openshift-storage. You can confirm it has completed when the Status column is marked Succeeded.

    When the installation of the ODF Operator is complete, you are prompted to create a storage system. Do not follow this instruction. Instead, create NooBaa object storage as outlined the following steps.
  4. Create NooBaa object storage. Save the following YAML to a file called noobaa.yaml.

    apiVersion: noobaa.io/v1alpha1
    kind: NooBaa
    metadata:
      name: noobaa
      namespace: openshift-storage
    spec:
     dbResources:
       requests:
         cpu: '0.1'
         memory: 1Gi
     dbType: postgres
     coreResources:
       requests:
         cpu: '0.1'
         memory: 1Gi

    This will create a single instance deployment of the Multi-cloud Object Gateway.

  5. Apply the configuration with the following command:

    $ oc create -n openshift-storage -f noobaa.yaml
    noobaa.noobaa.io/noobaa created
  6. After a couple of minutes, you should see that the MCG instance has finished provisioning (PHASE column will be set to Ready):

    $ oc get -n openshift-storage noobaas noobaa -w
    NAME     MGMT-ENDPOINTS              S3-ENDPOINTS                IMAGE                                                                                                            PHASE   AGE
    noobaa   [https://10.0.32.3:30318]   [https://10.0.32.3:31958]   registry.redhat.io/ocs4/mcg-core-rhel8@sha256:56624aa7dd4ca178c1887343c7445a9425a841600b1309f6deace37ce6b8678d   Ready   3d18h
  7. Next, configure a backing store for the gateway. Save the following YAML to a file called noobaa-pv-backing-store.yaml.

    noobaa-pv-backing-store.yaml

    apiVersion: noobaa.io/v1alpha1
    kind: BackingStore
    metadata:
      finalizers:
      - noobaa.io/finalizer
      labels:
        app: noobaa
      name: noobaa-pv-backing-store
      namespace: openshift-storage
    spec:
      pvPool:
        numVolumes: 1
        resources:
          requests:
            storage: 50Gi 1
        storageClass: STORAGE-CLASS-NAME 2
      type: pv-pool

    1
    The overall capacity of the object storage service, adjust as needed
    2
    The StorageClass to use for the PersistentVolumes requested, delete this property to use the cluster default
  8. Apply the configuration with the following command:

    $ oc create -f noobaa-pv-backing-store.yaml
    backingstore.noobaa.io/noobaa-pv-backing-store created

    This creates the backing store configuration for the gateway. All images in Quay will be stored as objects through the gateway in a PersistentVolume created by the above configuration.

  9. Finally, run the following command to make the PersistentVolume backing store the default for all ObjectBucketClaims issued by the Operator.

    $ oc patch bucketclass noobaa-default-bucket-class --patch '{"spec":{"placementPolicy":{"tiers":[{"backingStores":["noobaa-pv-backing-store"]}]}}}' --type merge -n openshift-storage

This concludes the setup of the Multi-Cloud Object Gateway instance for Red Hat Quay. Note that this configuration cannot be run in parallel on a cluster with Red Hat OpenShift Data Foundation installed.

3.3. Configuring the database

3.3.1. Using an existing Postgres database

  1. Create a configuration file config.yaml with the necessary database fields:

    config.yaml:

    DB_URI: postgresql://test-quay-database:postgres@test-quay-database:5432/test-quay-database

  2. Create a Secret using the configuration file:

    $ kubectl create secret generic --from-file config.yaml=./config.yaml config-bundle-secret
  3. Create a QuayRegistry YAML file quayregistry.yaml which marks the postgres component as unmanaged and references the created Secret:

    quayregistry.yaml

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: example-registry
      namespace: quay-enterprise
    spec:
      configBundleSecret: config-bundle-secret
      components:
        - kind: postgres
          managed: false

  4. Deploy the registry as detailed in the following sections.

3.3.2. Database configuration

You configure the connection to the database using the required DB_URI field and optional connection arguments in the DB_CONNECTION_ARGS structure. Some key-value pairs defined under DB_CONNECTION_ARGS are generic while others are database-specific. In particular, SSL configuration depends on the database you are deploying, and examples for PostgreSQL and MySQL are given below.

3.3.2.1. Database URI

Table 3.1. Database URI

FieldTypeDescription

DB_URI
(Required)

String

The URI for accessing the database, including any credentials

Example:

postgresql://quayuser:quaypass@quay-server.example.com:5432/quay

3.3.2.2. Database connection arguments

Table 3.2. Database connection arguments

FieldTypeDescription

DB_CONNECTION_ARGS

Object

Optional connection arguments for the database, such as timeouts and SSL

   .autorollback

Boolean

Whether to use thread-local connections
 
Should ALWAYS be true

   .threadlocals

Boolean

Whether to use auto-rollback connections
 
Should ALWAYS be true

3.3.2.2.1. PostgreSQL SSL connection arguments

A sample PostgreSQL SSL configuration is given below:

DB_CONNECTION_ARGS:
  sslmode: verify-ca
  sslrootcert: /path/to/cacert

The sslmode option determines whether or with what priority a secure SSL TCP/IP connection will be negotiated with the server. There are six modes:

  • disable: only try a non-SSL connection
  • allow: first try a non-SSL connection; if that fails, try an SSL connection
  • prefer: (default) first try an SSL connection; if that fails, try a non-SSL connection
  • require: only try an SSL connection. If a root CA file is present, verify the certificate in the same way as if verify-ca was specified
  • verify-ca: only try an SSL connection, and verify that the server certificate is issued by a trusted certificate authority (CA)
  • verify-full: only try an SSL connection, verify that the server certificate is issued by a trusted CA and that the requested server host name matches that in the certificate

More information on the valid arguments for PostgreSQL is available at https://www.postgresql.org/docs/current/libpq-connect.html.

3.3.2.2.2. MySQL SSL connection arguments

A sample MySQL SSL configuration follows:

DB_CONNECTION_ARGS:
  ssl:
    ca: /path/to/cacert

Information on the valid connection arguments for MySQL is available at https://dev.mysql.com/doc/refman/8.0/en/connecting-using-uri-or-key-value-pairs.html.

3.3.3. Using the managed PostgreSQL

Recommendations:

  • Database backups should be performed regularly using either the supplied tools on the Postgres image or your own backup infrastructure. The Operator does not currently ensure the Postgres database is backed up.
  • Restoring the Postgres database from a backup must be done using Postgres tools and procedures. Be aware that your Quay Pods should not be running while the database restore is in progress.
  • Database disk space is allocated automatically by the Operator with 50 GiB. This number represents a usable amount of storage for most small to medium Red Hat Quay installations but may not be sufficient for your use cases. Resizing the database volume is currently not handled by the Operator.

3.4. Configuring TLS and routes

Support for OpenShift Container Platform Edge-Termination Routes has been added by way of a new managed component, tls. This separates the route component from TLS and allows users to configure both separately. EXTERNAL_TLS_TERMINATION: true is the opinionated setting. Managed tls means that the default cluster wildcard cert is used. Unmanaged tls means that the user provided cert/key pair will be injected into the Route.

ssl.cert and ssl.key are now moved to a separate, persistent Secret, which ensures that the cert/key pair is not re-generated upon every reconcile. These are now formatted as edge routes and mounted to the same directory in the Quay container.

Multiple permutations are possible when configuring TLS and Routes, but the following rules apply:

  • If TLS is managed, then route must also be managed
  • If TLS is unmanaged then you must supply certs, either with the config tool or directly in the config bundle

The following table outlines the valid options:

Table 3.3. Valid configuration options for TLS and routes

OptionRouteTLSCerts providedResult

My own load balancer handles TLS

Managed

Managed

No

Edge Route with default wildcard cert

Red Hat Quay handles TLS

Managed

Unmanaged

Yes

Passthrough route with certs mounted inside the pod

Red Hat Quay handles TLS

Unmanaged

Unmanaged

Yes

Certificates are set inside the quay pod but route must be created manually

Note

Red Hat Quay 3.7 does not support builders when TLS is managed by the Operator.

3.4.1. Creating the config bundle secret with TLS cert, key pair:

To add your own TLS cert and key, include them in the config bundle secret as follows:

$ oc create secret generic --from-file config.yaml=./config.yaml --from-file ssl.cert=./ssl.cert --from-file ssl.key=./ssl.key config-bundle-secret

3.5. Configuring other components

3.5.1. Using external Redis

If you wish to use an external Redis database, set the component as unmanaged in the QuayRegistry instance:

  1. Create a configuration file config.yaml with the necessary redis fields:

    BUILDLOGS_REDIS:
        host: quay-server.example.com
        password: strongpassword
        port: 6379
    
    USER_EVENTS_REDIS:
        host: quay-server.example.com
        password: strongpassword
        port: 6379
  2. Create a Secret using the configuration file

    $ oc create secret generic --from-file config.yaml=./config.yaml config-bundle-secret
  3. Create a QuayRegistry YAML file quayregistry.yaml which marks redis component as unmanaged and references the created Secret:

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: example-registry
      namespace: quay-enterprise
    spec:
      configBundleSecret: config-bundle-secret
      components:
        - kind: redis
          managed: false
  4. Deploy the registry

3.5.1.1. Redis configuration fields

3.5.1.1.1. Build logs

Table 3.4. Build logs configuration

FieldTypeDescription

BUILDLOGS_REDIS
(Required)

Object

Redis connection details for build logs caching

   .host
   (Required)

String

The hostname at which Redis is accessible
 
Example:
quay-server.example.com

   .port
   (Required)

Number

The port at which Redis is accessible
 
Example:
6379

   .password

String

The port at which Redis is accessible
 
Example:
strongpassword

3.5.1.1.2. User events

Table 3.5. User events config

FieldTypeDescription

USER_EVENTS_REDIS
(Required)

Object

Redis connection details for user event handling

   .host
   (Required)

String

The hostname at which Redis is accessible
 
Example:
quay-server.example.com

   .port
   (Required)

Number

The port at which Redis is accessible
 
Example:
6379

   .password

String

The port at which Redis is accessible
 
Example:
strongpassword

3.5.1.1.3. Example redis configuration
BUILDLOGS_REDIS:
    host: quay-server.example.com
    password: strongpassword
    port: 6379

USER_EVENTS_REDIS:
    host: quay-server.example.com
    password: strongpassword
    port: 6379

3.5.2. Disabling the Horizontal Pod Autoscaler

HorizontalPodAutoscalers have been added to the Clair, Quay, and Mirror pods, so that they now automatically scale during load spikes.

As HPA is configured by default to be managed, the number of pods for Quay, Clair and repository mirroring is set to two. This facilitates the avoidance of downtime when updating / reconfiguring Quay via the Operator or during rescheduling events.

If you wish to disable autoscaling or create your own HorizontalPodAutoscaler, simply specify the component as unmanaged in the QuayRegistry instance:

apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: example-registry
  namespace: quay-enterprise
spec:
  components:
    - kind: horizontalpodautoscaler
      managed: false

3.5.3. Disabling Route Component

To prevent the Operator from creating a Route:

  1. Mark the component as unmanaged in the QuayRegistry:

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      name: example-registry
      namespace: quay-enterprise
    spec:
      components:
        - kind: route
          managed: false
  2. Specify that you want Quay to handle TLS in the configuration, by editing the config.yaml file:

    config.yaml

    ...
    EXTERNAL_TLS_TERMINATION: false
    ...
    SERVER_HOSTNAME: example-registry-quay-quay-enterprise.apps.user1.example.com
    ...
    PREFERRED_URL_SCHEME: https
    ...

    If you do not configure the unmanaged Route correctly, you will see an error similar to the following:

    {
      {
        "kind":"QuayRegistry",
        "namespace":"quay-enterprise",
        "name":"example-registry",
        "uid":"d5879ba5-cc92-406c-ba62-8b19cf56d4aa",
        "apiVersion":"quay.redhat.com/v1",
        "resourceVersion":"2418527"
      },
      "reason":"ConfigInvalid",
      "message":"required component `route` marked as unmanaged, but `configBundleSecret` is missing necessary fields"
    }
Note

Disabling the default Route means you are now responsible for creating a Route, Service, or Ingress in order to access the Quay instance and that whatever DNS you use must match the SERVER_HOSTNAME in the Quay config.

3.5.4. Unmanaged monitoring

If you install the Quay Operator in a single namespace, the monitoring component is automatically set to 'unmanaged'. To enable monitoring in this scenario, see the section Section 8.2, “Enabling monitoring when Operator is installed in a single namespace”.

To disable monitoring explicitly:

apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: example-registry
  namespace: quay-enterprise
spec:
  components:
    - kind: monitoring
      managed: false

3.5.5. Unmanaged mirroring

To disable mirroring explicitly:

apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: example-registry
  namespace: quay-enterprise
spec:
  components:
    - kind: mirroring
      managed: false