Chapter 16. Red Hat Quay build enhancements

Prior to Red Hat Quay 3.7, Quay ran podman commands in virtual machines launched by pods. Running builds on virtual platforms requires enabling nested virtualization, which is not featured in Red Hat Enterprise Linux or OpenShift Container Platform. As a result, builds had to run on bare-metal clusters, which is an inefficient use of resources.

With Red Hat Quay 3.7., the bare-metal constraint required to run builds has been removed by adding an additional build option which does not contain the virtual machine layer. As a result, builds can be run on virtualized platforms. Backwards compatibility to run previous build configurations are also available.

16.1. Red Hat Quay enhanced build architecture

The preceding image shows the expected design flow and architecture of the enhanced build features:

Enhanced Quay builds architecture

With this enhancement, the build manager first creates the Job Object. Then, the Job Object then creates a pod using the quay-builder-image. The quay-builder-image will contain the quay-builder binary and the Podman service. The created pod runs as unprivileged. The quay-builder binary then builds the image while communicating status and retrieving build information from the Build Manager.

16.2. Red Hat Quay build limitations

Running builds in Red Hat Quay in an unprivileged context might cause some commands that were working under the previous build strategy to fail. Attempts to change the build strategy could potentially cause performance issues and reliability with the build.

Running builds directly in a container will not have the same isolation as using virtual machines. Changing the build environment might also caused builds that were previously working to fail.

16.3. Creating a Red Hat Quay builders environment with OpenShift Container Platform

The procedures in this section explain how to create a Red Hat Quay virtual builders environment with OpenShift Container Platform.

16.3.1. OpenShift Container Platform TLS component

The tls component allows you to control TLS configuration.

Note

Red Hat Quay 3 does not support builders when the TLS component is managed by the Operator.

If you set tls to unmanaged, you supply your own ssl.cert and ssl.key files. In this instance, if you want your cluster to support builders, you must add both the Quay route and the builder route name to the SAN list in the cert, or use a wildcard.

To add the builder route, use the following format:

[quayregistry-cr-name]-quay-builder-[ocp-namespace].[ocp-domain-name]:443

16.3.2. Using OpenShift Container Platform for Red Hat Quay builders

Builders require SSL/TLS certificates. For more information about SSL/TLS certificates, see Adding TLS certificates to the Red Hat Quay container.

If you are using Amazon Web Service (AWS) S3 storage, you must modify your storage bucket in the AWS console, prior to running builders. See "Modifying your AWS S3 storage bucket" in the following section for the required parameters.

16.3.2.1. Preparing OpenShift Container Platform for virtual builders

Use the following procedure to prepare OpenShift Container Platform for Red Hat Quay virtual builders.

Note
  • This procedure assumes you already have a cluster provisioned and a Quay Operator running.
  • This procedure is for setting up a virtual namespace on OpenShift Container Platform.

Procedure

  1. Log in to your Red Hat Quay cluster using a cluster administrator account.
  2. Create a new project where your virtual builders will be run, for example, virtual-builders, by running the following command:

    $ oc new-project virtual-builders
  3. Create a ServiceAccount in the project that will be used to run builds by entering the following command:

    $ oc create sa -n virtual-builders quay-builder
  4. Provide the created service account with editing permissions so that it can run the build:

    $ oc adm policy -n virtual-builders add-role-to-user edit system:serviceaccount:virtual-builders:quay-builder
  5. Grant the Quay builder anyuid scc permissions by entering the following command:

    $ oc adm policy -n virtual-builders add-scc-to-user anyuid -z quay-builder
    Note

    This action requires cluster admin privileges. This is required because builders must run as the Podman user for unprivileged or rootless builds to work.

  6. Obtain the token for the Quay builder service account.

    1. If using OpenShift Container Platform 4.10 or an earlier version, enter the following command:

      oc sa get-token -n virtual-builders quay-builder
    2. If using OpenShift Container Platform 4.11 or later, enter the following command:

      $ oc create token quay-builder -n virtual-builders

      Example output

      eyJhbGciOiJSUzI1NiIsImtpZCI6IldfQUJkaDVmb3ltTHZ0dGZMYjhIWnYxZTQzN2dJVEJxcDJscldSdEUtYWsifQ...

  7. Determine the builder route by entering the following command:

    $ oc get route -n quay-enterprise

    Example output

    NAME                                  HOST/PORT                                                                    PATH   SERVICES                              PORT   TERMINATION     WILDCARD
    ...
    example-registry-quay-builder         example-registry-quay-builder-quay-enterprise.apps.docs.quayteam.org                example-registry-quay-app             grpc   edge/Redirect   None
    ...

  8. Generate a self-signed SSL/TlS certificate with the .crt extension by entering the following command:

    $ oc extract cm/kube-root-ca.crt -n openshift-apiserver

    Example output

    ca.crt

  9. Rename the ca.crt file to extra_ca_cert_build_cluster.crt by entering the following command:

    $ mv ca.crt extra_ca_cert_build_cluster.crt
  10. Locate the secret for you configuration bundle in the Console, and select ActionsEdit Secret and add the appropriate builder configuration:

    FEATURE_USER_INITIALIZE: true
    BROWSER_API_CALLS_XHR_ONLY: false
    SUPER_USERS:
    - <superusername>
    FEATURE_USER_CREATION: false
    FEATURE_QUOTA_MANAGEMENT: true
    FEATURE_BUILD_SUPPORT: True
    BUILDMAN_HOSTNAME: <sample_build_route> 1
    BUILD_MANAGER:
      - ephemeral
      - ALLOWED_WORKER_COUNT: 1
        ORCHESTRATOR_PREFIX: buildman/production/
        JOB_REGISTRATION_TIMEOUT: 3600 2
        ORCHESTRATOR:
          REDIS_HOST: <sample_redis_hostname> 3
          REDIS_PASSWORD: ""
          REDIS_SSL: false
          REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
        EXECUTORS:
          - EXECUTOR: kubernetesPodman
            NAME: openshift
            BUILDER_NAMESPACE: <sample_builder_namespace> 4
            SETUP_TIME: 180
            MINIMUM_RETRY_THRESHOLD:
            BUILDER_CONTAINER_IMAGE: <sample_builder_container_image> 5
            # Kubernetes resource options
            K8S_API_SERVER: <sample_k8s_api_server> 6
            K8S_API_TLS_CA: <sample_crt_file> 7
            VOLUME_SIZE: 8G
            KUBERNETES_DISTRIBUTION: openshift
            CONTAINER_MEMORY_LIMITS: 300m 8
            CONTAINER_CPU_LIMITS: 1G 9
            CONTAINER_MEMORY_REQUEST: 300m 10
            CONTAINER_CPU_REQUEST: 1G 11
            NODE_SELECTOR_LABEL_KEY: ""
            NODE_SELECTOR_LABEL_VALUE: ""
            SERVICE_ACCOUNT_NAME: <sample_service_account_name>
            SERVICE_ACCOUNT_TOKEN: <sample_account_token> 12
    1
    The build route is obtained by running oc get route -n with the name of your OpenShift Operator’s namespace. A port must be provided at the end of the route, and it should use the following format: [quayregistry-cr-name]-quay-builder-[ocp-namespace].[ocp-domain-name]:443.
    2
    If the JOB_REGISTRATION_TIMEOUT parameter is set too low, you might receive the following error: failed to register job to build manager: rpc error: code = Unauthenticated desc = Invalid build token: Signature has expired. It is suggested that this parameter be set to at least 240.
    3
    If your Redis host has a password or SSL/TLS certificates, you must update accordingly.
    4
    Set to match the name of your virtual builders namespace, for example, virtual-builders.
    5
    For early access, the BUILDER_CONTAINER_IMAGE is currently quay.io/projectquay/quay-builder:3.7.0-rc.2. Note that this might change during the early access window. If this happens, customers are alerted.
    6
    The K8S_API_SERVER is obtained by running oc cluster-info.
    7
    You must manually create and add your custom CA cert, for example, K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build_cluster.crt.
    8
    Defaults to 5120Mi if left unspecified.
    9
    For virtual builds, you must ensure that there are enough resources in your cluster. Defaults to 1000m if left unspecified.
    10
    Defaults to 3968Mi if left unspecified.
    11
    Defaults to 500m if left unspecified.
    12
    Obtained when running oc create sa.

    Sample configuration

    FEATURE_USER_INITIALIZE: true
    BROWSER_API_CALLS_XHR_ONLY: false
    SUPER_USERS:
    - quayadmin
    FEATURE_USER_CREATION: false
    FEATURE_QUOTA_MANAGEMENT: true
    FEATURE_BUILD_SUPPORT: True
    BUILDMAN_HOSTNAME: example-registry-quay-builder-quay-enterprise.apps.docs.quayteam.org:443
    BUILD_MANAGER:
      - ephemeral
      - ALLOWED_WORKER_COUNT: 1
        ORCHESTRATOR_PREFIX: buildman/production/
        JOB_REGISTRATION_TIMEOUT: 3600
        ORCHESTRATOR:
          REDIS_HOST: example-registry-quay-redis
          REDIS_PASSWORD: ""
          REDIS_SSL: false
          REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
        EXECUTORS:
          - EXECUTOR: kubernetesPodman
            NAME: openshift
            BUILDER_NAMESPACE: virtual-builders
            SETUP_TIME: 180
            MINIMUM_RETRY_THRESHOLD:
            BUILDER_CONTAINER_IMAGE: quay.io/projectquay/quay-builder:3.7.0-rc.2
            # Kubernetes resource options
            K8S_API_SERVER: api.docs.quayteam.org:6443
            K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build_cluster.crt
            VOLUME_SIZE: 8G
            KUBERNETES_DISTRIBUTION: openshift
            CONTAINER_MEMORY_LIMITS: 1G
            CONTAINER_CPU_LIMITS: 1080m
            CONTAINER_MEMORY_REQUEST: 1G
            CONTAINER_CPU_REQUEST: 580m
            NODE_SELECTOR_LABEL_KEY: ""
            NODE_SELECTOR_LABEL_VALUE: ""
            SERVICE_ACCOUNT_NAME: quay-builder
            SERVICE_ACCOUNT_TOKEN: "eyJhbGciOiJSUzI1NiIsImtpZCI6IldfQUJkaDVmb3ltTHZ0dGZMYjhIWnYxZTQzN2dJVEJxcDJscldSdEUtYWsifQ"

16.3.2.2. Manually adding SSL/TLS certificates

Due to a known issue with the configuration tool, you must manually add your custom SSL/TLS certificates to properly run builders. Use the following procedure to manually add custom SSL/TLS certificates.

For more information creating SSL/TLS certificates, see Adding TLS certificates to the Red Hat Quay container.

16.3.2.2.1. Creating and signing certificates

Use the following procedure to create and sign an SSL/TLS certificate.

Procedure

  • Create a certificate authority and sign a certificate. For more information, see Create a Certificate Authority and sign a certificate.

    openssl.cnf

    [req]
    req_extensions = v3_req
    distinguished_name = req_distinguished_name
    [req_distinguished_name]
    [ v3_req ]
    basicConstraints = CA:FALSE
    keyUsage = nonRepudiation, digitalSignature, keyEncipherment
    subjectAltName = @alt_names
    [alt_names]
    DNS.1 = example-registry-quay-quay-enterprise.apps.docs.quayteam.org 1
    DNS.2 = example-registry-quay-builder-quay-enterprise.apps.docs.quayteam.org 2

    1
    An alt_name for the URL of your Red Hat Quay registry must be included.
    2
    An alt_name for the BUILDMAN_HOSTNAME

    Sample commands

    $ openssl genrsa -out rootCA.key 2048
    $ openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.pem
    $ openssl genrsa -out ssl.key 2048
    $ openssl req -new -key ssl.key -out ssl.csr
    $ openssl x509 -req -in ssl.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out ssl.cert -days 356 -extensions v3_req -extfile openssl.cnf

16.3.2.2.2. Setting TLS to unmanaged

Use the following procedure to set king:tls to unmanaged.

Procedure

  1. In your Red Hat Quay Registry YAML, set kind: tls to managed: false:

      - kind: tls
        managed: false
  2. On the Events page, the change is blocked until you set up the appropriate config.yaml file. For example:

        - lastTransitionTime: '2022-03-28T12:56:49Z'
          lastUpdateTime: '2022-03-28T12:56:49Z'
          message: >-
            required component `tls` marked as unmanaged, but `configBundleSecret`
            is missing necessary fields
          reason: ConfigInvalid
          status: 'True'
16.3.2.2.3. Creating temporary secrets

Use the following procedure to create temporary secrets for the CA certificate.

Procedure

  1. Create a secret in your default namespace for the CA certificate:

    $ oc create secret generic -n quay-enterprise temp-crt --from-file extra_ca_cert_build_cluster.crt
  2. Create a secret in your default namespace for the ssl.key and ssl.cert files:

    $ oc create secret generic -n quay-enterprise quay-config-ssl --from-file ssl.cert --from-file ssl.key
16.3.2.2.4. Copying secret data to the configuration YAML

Use the following procedure to copy secret data to your config.yaml file.

Procedure

  1. Locate the new secrets in the console UI at WorkloadsSecrets.
  2. For each secret, locate the YAML view:

    kind: Secret
    apiVersion: v1
    metadata:
      name: temp-crt
      namespace: quay-enterprise
      uid: a4818adb-8e21-443a-a8db-f334ace9f6d0
      resourceVersion: '9087855'
      creationTimestamp: '2022-03-28T13:05:30Z'
    ...
    data:
      extra_ca_cert_build_cluster.crt: >-
        LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURNakNDQWhxZ0F3SUJBZ0l....
    type: Opaque
    kind: Secret
    apiVersion: v1
    metadata:
      name: quay-config-ssl
      namespace: quay-enterprise
      uid: 4f5ae352-17d8-4e2d-89a2-143a3280783c
      resourceVersion: '9090567'
      creationTimestamp: '2022-03-28T13:10:34Z'
    ...
    data:
      ssl.cert: >-
        LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVaakNDQTA2Z0F3SUJBZ0lVT...
      ssl.key: >-
        LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBc...
    type: Opaque
  3. Locate the secret for your Red Hat Quay registry configuration bundle in the UI, or through the command line by running a command like the following:

    $ oc get quayregistries.quay.redhat.com -o jsonpath="{.items[0].spec.configBundleSecret}{'\n'}"  -n quay-enterprise
  4. In the OpenShift Container Platform console, select the YAML tab for your configuration bundle secret, and add the data from the two secrets you created:

    kind: Secret
    apiVersion: v1
    metadata:
      name: init-config-bundle-secret
      namespace: quay-enterprise
      uid: 4724aca5-bff0-406a-9162-ccb1972a27c1
      resourceVersion: '4383160'
      creationTimestamp: '2022-03-22T12:35:59Z'
    ...
    data:
      config.yaml: >-
        RkVBVFVSRV9VU0VSX0lOSVRJQUxJWkU6IHRydWUKQlJ...
      extra_ca_cert_build_cluster.crt: >-
        LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURNakNDQWhxZ0F3SUJBZ0ldw....
      ssl.cert: >-
        LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVaakNDQTA2Z0F3SUJBZ0lVT...
      ssl.key: >-
        LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBc...
    type: Opaque
  5. Click Save.
  6. Enter the following command to see if your pods are restarting:

    $ oc get pods -n quay-enterprise

    Example output

    NAME                                                   READY   STATUS              RESTARTS   AGE
    ...
    example-registry-quay-app-6786987b99-vgg2v             0/1     ContainerCreating   0          2s
    example-registry-quay-app-7975d4889f-q7tvl             1/1     Running             0          5d21h
    example-registry-quay-app-7975d4889f-zn8bb             1/1     Running             0          5d21h
    example-registry-quay-app-upgrade-lswsn                0/1     Completed           0          6d1h
    example-registry-quay-config-editor-77847fc4f5-nsbbv   0/1     ContainerCreating   0          2s
    example-registry-quay-config-editor-c6c4d9ccd-2mwg2    1/1     Running             0          5d21h
    example-registry-quay-database-66969cd859-n2ssm        1/1     Running             0          6d1h
    example-registry-quay-mirror-764d7b68d9-jmlkk          1/1     Terminating         0          5d21h
    example-registry-quay-mirror-764d7b68d9-jqzwg          1/1     Terminating         0          5d21h
    example-registry-quay-redis-7cc5f6c977-956g8           1/1     Running             0          5d21h

  7. After your Red Hat Quay registry has reconfigured, enter the following command to check if the Red Hat Quay app pods are running:

    $ oc get pods -n quay-enterprise

    Example output

    example-registry-quay-app-6786987b99-sz6kb             1/1     Running            0          7m45s
    example-registry-quay-app-6786987b99-vgg2v             1/1     Running            0          9m1s
    example-registry-quay-app-upgrade-lswsn                0/1     Completed          0          6d1h
    example-registry-quay-config-editor-77847fc4f5-nsbbv   1/1     Running            0          9m1s
    example-registry-quay-database-66969cd859-n2ssm        1/1     Running            0          6d1h
    example-registry-quay-mirror-758fc68ff7-5wxlp          1/1     Running            0          8m29s
    example-registry-quay-mirror-758fc68ff7-lbl82          1/1     Running            0          8m29s
    example-registry-quay-redis-7cc5f6c977-956g8           1/1     Running            0          5d21h

  8. In your browser, access the registry endpoint and validate that the certificate has been updated appropriately. For example:

    Common Name (CN)	example-registry-quay-quay-enterprise.apps.docs.quayteam.org
    Organisation (O)	DOCS
    Organisational Unit (OU)	QUAY

16.3.2.3. Using the UI to create a build trigger

Use the following procedure to use the UI to create a build trigger.

Procedure

  1. Log in to your Red Hat Quay repository.
  2. Click Create New Repository and create a new registry, for example, testrepo.
  3. On the Repositories page, click the Builds tab on the navigation pane. Alternatively, use the corresponding URL directly:

    https://example-registry-quay-quay-enterprise.apps.docs.quayteam.org/repository/quayadmin/testrepo?tab=builds
    Important

    In some cases, the builder might have issues resolving hostnames. This issue might be related to the dnsPolicy being set to default on the job object. Currently, there is no workaround for this issue. It will be resolved in a future version of Red Hat Quay.

  4. Click Create Build TriggerCustom Git Repository Push.
  5. Enter the HTTPS or SSH style URL used to clone your Git repository, then click Continue. For example:

    https://github.com/gabriel-rh/actions_test.git
  6. Check Tag manifest with the branch or tag name and then click Continue.
  7. Enter the location of the Dockerfile to build when the trigger is invoked, for example, /Dockerfile and click Continue.
  8. Enter the location of the context for the Docker build, for example, /, and click Continue.
  9. If warranted, create a Robot Account. Otherwise, click Continue.
  10. Click Continue to verify the parameters.
  11. On the Builds page, click Options icon of your Trigger Name, and then click Run Trigger Now.
  12. Enter a commit SHA from the Git repository and click Start Build.
  13. You can check the status of your build by clicking the commit in the Build History page, or by running oc get pods -n virtual-builders. For example:

    $ oc get pods -n virtual-builders

    Example output

    NAME                                               READY   STATUS    RESTARTS   AGE
    f192fe4a-c802-4275-bcce-d2031e635126-9l2b5-25lg2   1/1     Running   0          7s

    $ oc get pods -n virtual-builders

    Example output

    NAME                                               READY   STATUS        RESTARTS   AGE
    f192fe4a-c802-4275-bcce-d2031e635126-9l2b5-25lg2   1/1     Terminating   0          9s

    $ oc get pods -n virtual-builders

    Example output

    No resources found in virtual-builders namespace.

  14. When the build is finished, you can check the status of the tag under Tags on the navigation pane.

    Note

    With early access, full build logs and timestamps of builds are currently unavailable.

16.3.2.4. Modifying your AWS S3 storage bucket

If you are using AWS S3 storage, you must change your storage bucket in the AWS console, prior to running builders.

Procedure

  1. Log in to your AWS console at s3.console.aws.com.
  2. In the search bar, search for S3 and then click S3.
  3. Click the name of your bucket, for example, myawsbucket.
  4. Click the Permissions tab.
  5. Under Cross-origin resource sharing (CORS), include the following parameters:

      [
          {
              "AllowedHeaders": [
                  "Authorization"
              ],
              "AllowedMethods": [
                  "GET"
              ],
              "AllowedOrigins": [
                  "*"
              ],
              "ExposeHeaders": [],
              "MaxAgeSeconds": 3000
          },
          {
              "AllowedHeaders": [
                  "Content-Type",
                  "x-amz-acl",
                  "origin"
              ],
              "AllowedMethods": [
                  "PUT"
              ],
              "AllowedOrigins": [
                  "*"
              ],
              "ExposeHeaders": [],
              "MaxAgeSeconds": 3000
          }
      ]