Deploy Red Hat Quay on OpenShift
Deploy Red Hat Quay on OpenShift
Red Hat Quay is an enterprise-quality container registry. Use Quay to build and store containers, then make them available to deploy across your enterprise.
This procedure describes how to deploy a Red Hat Quay setup to run on an OpenShift cluster.
Chapter 1. Overview
Features of Quay include:
- High availability
- Docker v2, schema 2 (multiarch) support
- Continuous integration
- Security scanning
- Custom log rotation
- 24/7 support
Quay provides support for multiple:
- Authentication and access methods
- Storage backends
- SSL certificates
- Application registries
- Container image types
Chapter 2. Architecture
Quay is made up of three core components for a basic setup. In highly available setups, an additional object storage component is needed. The three core components are:
- Database: Used by Red Hat Quay as its primary metadata storage (not for image storage).
- Redis (key, value store): Used for providing real time events.
- Quay (container registry): Runs the quay container as a service, consisting of several components in the pod.
For the high availability installation, you need to use one of the following types of storage:
- Public cloud storage: In public cloud environments, you should use the cloud provider’s object storage, such as Amazon S3 (for AWS) or Google Cloud Storage (for Google Cloud).
- Private cloud storage: In private clouds, an S3 or Swift compliant Object Store is needed, such as Ceph RADOS, or OpenStack Swift.
Local storage is supported for the Red Hat Quay test-only installation, but not for high-availability.
Chapter 3. Installing Red Hat Quay on OpenShift
This procedure presents guidance on how to set up a deployment of Red Hat Quay on an OpenShift cluster.
Here are a few things you need to know before you begin the Red Hat Quay on OpenShift deployment:
- OpenShift cluster: You need a privileged account to an OpenShift cluster on which to deploy the Red Hat Quay. That account must have the ability to create namespaces at the cluster scope.
Storage: AWS cloud storage is used as an example in the following procedure. As an alternative, you can create Ceph cloud storage using steps from the Set up Ceph section of the high availability Red Hat Quay deployment guide. The following is a list of other supported cloud storage:
- Amazon S3 (see S3 IAM Bucket Policy for details on configuring an S3 bucket policy for Quay)
- Azure Blob Storage
- Google Cloud Storage
- Ceph Object Gateway
- OpenStack Swift
- CloudFront + S3
Services: Because you are deploying to an OpenShift cluster, instead of to bare metal or virtual machines, you simply need to make sure that your cluster has enough capacity to run the following containerized services:
Database: We recommend you use an enterprise-quality database for production use of Red Hat Quay. Postgres is used as an example in this document.Note
Because many organizations already have high-availability (HA) databases, instead of building one into the cost of the Red Hat Quay product, we require that you provide your own HA database to use with Red Hat Quay. The example Postgresql database shown here is provided for illustration purposes and should be substituted with a HA configured database which would typically be available from Red Hat partners.
- Key-value database: Redis is used to serve non-critical data to your Red Hat Quay configuration.
- Red Hat Quay: The quay container provides the features to manage the Quay registry.
Chapter 4. Set up Red Hat Quay services
Deploying Red Hat Quay on OpenShift requires you to create a set of yaml files that you use with the
oc command to set up name spaces, add secrets, configure networking, and start the required containerized services. Although the
oc command is used to configure the Red Hat Quay registry here, you could use the OpenShift web UI instead, if you prefer.
Refer to Appendix A for the contents of these yaml files.
Here are a few things to keep in mind:
- Your OpenShift account must have permission to create namespaces at the cluster scope.
Quay runs under its own namespace inside a Kubernetes cluster, so that needs to be created first. You can create it through the
New projectin the OpenShift web console or using quay-enterprise-namespace.yaml (as described here).
- You need a working enterprise-quality database. In our example, we illustrate PostgreSQL (version 9.4 or above is required, although we recommend 9.6).
You can use an existing Redis service (needed for build logs) or start one as described in this procedure.
Log in with oc cli: Login as a user with cluster scope permissions to the OpenShift cluster. For example:
$ oc login -u system:admin
Create namespace. Create a
quay-enterprise-namespace.yamlfile with the content shown here and run
oc createon it to create the
quay-enterprisenamespace. All objects will be deployed to this namespace:
$ oc create -f quay-enterprise-namespace.yaml namespace "quay-enterprise" created
Create the database. If you are not using your own enterprise-quality database (recommended), this procedure illustrates how to set up a Postgresql database on an OpenShift cluster. This entails creating AWS storage, a postgres deployment, and postgres service, then adding an extension to the database:
$ oc create -f quay-storageclass.yaml storageclass.storage.k8s.io/quay-storageclass created $ oc create -f db-pvc.yaml persistentvolumeclaim/postgres-storage created $ oc create -f postgres-deployment.yaml deployment.extensions/postgres-new created $ oc create -f postgres-service.yaml service/postgres created
$ oc get pods -n quay-enterprise NAME READY STATUS RESTARTS AGE postgres-6d9cc4f5c8-l4qxk 1/1 Running 0 3m26s $ oc exec -it postgres-6d9cc4f5c8-l4qxk -n quay-enterprise -- /bin/bash -c 'echo "CREATE EXTENSION IF NOT EXISTS pg_trgm" | /opt/rh/rh-postgresql10/root/usr/bin/psql -d quay'Note
-d database_namemust not be omitted. If it is, the extension will be created on the default PostgreSQL database.
Create the config secret. Refer to Accessing Red Hat Quay to get the credentials you need to add to the quay-enterprise-redhat-quay-pull-secret.yaml file, then run oc create:
apiVersion: v1 kind: Secret metadata: namespace: quay-enterprise name: redhat-quay-pull-secret data: .dockerconfigjson: <Add credentials from https://access.redhat.com/solutions/3533201> type: kubernetes.io/dockerconfigjson
$ oc create -f quay-enterprise-redhat-quay-pull-secret.yaml secret/redhat-quay-pull-secret created
Create the role and the role binding: Red Hat Quay has native Kubernetes integrations. These integrations require Service Account to have access to the Kubernetes API. When Kubernetes RBAC is enabled, Role Based Access Control policy manifests also have to be deployed. This role will be used to run Quay and also to write the config.yaml file that Quay creates at the end of the web interface setup:
$ oc create -f quay-servicetoken-role-k8s1-6.yaml $ oc create -f quay-servicetoken-role-binding-k8s1-6.yaml
Add privilege: Make sure that the service account has root privileges, because Quay runs strictly under root (this will be changed in the future versions). Replace
quay-enterpriseif you are using a different namespace name:
$ oc adm policy add-scc-to-user anyuid \ system:serviceaccount:quay-enterprise:default
Create Redis deployment: If you haven’t already deployed Redis, create a
quay-enterprise-redis.yamlfile and deploy it:
$ oc create -f quay-enterprise-redis.yaml
Set up to configure Red Hat Quay: Red Hat Quay V3 added a tool for configuring the Red Hat Quay service before deploying it. Although the config tools is in the same container as the full Red Hat Quay service, it is deployed in a different way, as follows:
$ oc create -f quay-enterprise-config.yaml $ oc create -f quay-enterprise-config-service-clusterip.yaml $ oc create -f quay-enterprise-config-route.yaml
The quay configuration container is now set up to be accessed from port 443 from your Web browser. Before creating the configuration, however, you need to create a route to the permanent Red Hat Quay service. This is because we need the Red Hat Quay service’s publicly available FQDN when setting up the application.
Start the Red Hat Quay service and route: Identify it the Red Hat Quay Kubernetes service and create a route for it as follows:
$ oc create -f quay-enterprise-service-clusterip.yaml $ oc create -f quay-enterprise-app-route.yaml
You will need to know the location of the service when you do the Red Hat Quay configuration and before you start the quay container itself later.
Begin to configure Red Hat Quay: Open the public route to the Red Hat Quay configuration container in a Web browser. To see the route to the quay configuration service, type the following:
$ oc get route -n quay-enterprise quay-enterprise-config NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD quay-enterprise-config quay-enterprise-config-quay-enterprise.apps.test.example.com quay-enterprise-config <all> passthrough None
For this example, you would open this URL in your web browser: https://quay-enterprise-config-quay-enterprise.apps.test.example.com
Log in as quayconfig: When prompted, enter
- User Name: quayconfig
- Password: secret
Choose configuration mode: You are prompted to choose to either create a new Red Hat Quay configuration file or edit an existing one in these two modes:
Start New Registry Setup: The result of this selection is the creation of a new configuration file (
config.yaml) and optional
ssl.keyfiles. Those files are bundled into a tarball file you can use to actually deploy all your Red Hat Quay nodes.
Modify an existing configuration: With this selection, you are prompted to provide an existing tarball and modify it before you use it to start your Red Hat Quay nodes.
The following figure shows an example of the resulting
Choose an optionpage:
For an initial setup, you are asked to identify the database type. For a subsequent configuration, you are prompted for a tarball containing the
config.yamland credential files (optional). Then you can continue on with the configuration.
- Start New Registry Setup: The result of this selection is the creation of a new configuration file (
Identify the database: For the initial setup, add the following information about the type and location of the database to be used by Red Hat Quay:
- Database Type: Choose MySQL or PostgreSQL. MySQL will be used in the basic example; PostgreSQL is used with the high availability Red Hat Quay on OpenShift examples and is shown here.
- Database Server: Identify the IP address or hostname of the database, along with the port number if it is different from 3306.
- Username: Identify a user with full access to the database.
- Password: Enter the password you assigned to the selected user.
- Database Name: Enter the database name you assigned when you started the database server.
SSL Certificate: For production environments, you should provide an SSL certificate to connect to the database.
To find the address of the postgres service, type the following:
$ oc get services -n quay-enterprise postgres NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE postgres NodePort 172.30.127.41 <none> 5432:32212/TCP 19h
The following figure shows an example of the screen for identifying the database used by Red Hat Quay. The example yaml file set the user name to
username, the password to
password, and the database to
Validate database: Select
Validate Database Settingsand proceed to the next screen.
Create Quay superuser: You need to set up an account with superuser privileges to Quay, to use for editing Quay configuration settings. That information includes a Username, Email address, and Password (entered twice).
The following figure shows an example of the Red Hat Quay Setup screen for setting up a Quay superuser account:
Create Super User, and proceed to the next screen.
Identify settings: Go through each of the following settings. The minimum you must enter includes:
- Server hostname: The URL to the Red Hat Quay service.
Redis hostname: The URL or IP address to the Redis service.
Here are the settings you need to consider:
Custom SSL Certificates: Upload custom or self-signed SSL certificates for use by Quay. See Using SSL to protect connections to Red Hat Quay for details. Recommended for high availability.Important
Using SSL certificates is recommended for both basic and high availability deployments. If you decide to not use SSL, you must configure your container clients to use your new Quay setup as an insecure registry as described in Test an Insecure Registry.
- Basic Configuration: Upload a company logo to rebrand your Quay registry.
Server Configuration: Hostname or IP address to reach the Quay service, along with TLS indication (recommended for production installations). To get the route to the permanent Red Hat Quay service, type the following:
$ oc get route -n quay-enterprise quay-enterprise NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD quay-enterprise quay-enterprise-quay-enterprise.apps.cnegus-ocp.devcluster.openshift.com quay-enterprise-clusterip <all> None
TLS termination can be done in two different ways:
- On the instance itself, with all TLS traffic governed by the nginx server in the quay container (recommended).
- On the load balancer. This is not recommended. Access to Quay could be lost if the TLS setup is not done correctly on the load balancer.
- Data Consistency Settings: Select to relax logging consistency guarantees to improve performance and availability.
- Time Machine: Allow older image tags to remain in the repository for set periods of time and allow users to select their own tag expiration times.
redis: Identify the hostname or IP address (and optional password) to connect to the redis service used by Quay. To find the address of the redis service, type the following:
$ oc get services -n quay-enterprise quay-enterprise-redis NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE quay-enterprise-redis ClusterIP 172.30.207.35 <none> 6379/TCP 40m
- Registry Storage: Identify the location of storage. A variety of cloud and local storage options are available. Remote storage is required for high availability. Identify the Ceph storage location if you are following the example for Red Hat Quay high availability storage. On OpenShift, the example uses Amazon S3 storage.
- Action Log Rotation and Archiving: Select to enable log rotation, which moves logs older than 30 days into storage, then indicate storage area.
- Security Scanner: Enable security scanning by selecting a security scanner endpoint and authentication key. To setup Clair to do image scanning, refer to Clair Setup and Configuring Clair. Recommended for high availability.
- Application Registry: Enable an additional application registry that includes things like Kubernetes manifests or Helm charts (see the App Registry specification).
BitTorrent-based download: Allow all registry images to be downloaded using BitTorrent protocol (using the
rkt Conversion: Allow
rkt fetchto be used to fetch images from Quay registry. Public and private GPG2 keys are needed (see Generating signing keys for ACI conversion for details. This field is deprecated.
- E-mail: Enable e-mail to use for notifications and user password resets.
- Internal Authentication: Change default authentication for the registry from Local Database to LDAP, Keystone (OpenStack), JWT Custom Authentication, or External Application Token.
- External Authorization (OAuth): Enable to allow GitHub or GitHub Enterprise to authenticate to the registry.
- Google Authentication: Enable to allow Google to authenticate to the registry.
- Access settings: Basic username/password authentication is enabled by default. Other authentication types that can be enabled include: external application tokens (user-generated tokens used with docker or rkt commands), anonymous access (enable for public access to anyone who can get to the registry), user creation (let users create their own accounts), encrypted client password (require command-line user access to include encrypted passwords), and prefix username autocompletion (disable to require exact username matches on autocompletion).
- Dockerfile Build Support: Enable to allow users to submit Dockerfiles to be built and pushed to Quay. This is not recommended for multitenant environments.
Save the changes: Select
Save Configuration Changes. You are presented with the following Download Configuration screen:
Download configuration: Select the
Download Configurationbutton and save the tarball (
quay-config.tar.gz) to a local directory.
Untar config files: Untar the configuration files in your local directory. This includes the config.yaml and possibly SSL keys:
$ tar xvf quay-config.tar.gz config.yaml
Apply config to Red Hat Quay: Apply the
config.yamlfile as a secret to your namespace:
$ oc create secret generic quay-enterprise-secret \ -n quay-enterprise --from-file=config.yaml
Start Red Hat Quay: Create the actual Red Hat Quay deployment:
$ oc create -f quay-enterprise-app-rc.yaml deployment.extensions/quay-enterprise-app created
Check pods: In a couple of minutes (depending on your connection speed), Red Hat Quay should be up and running and the following pods should be visible in the quay-enterprise namespace You might get a mount error at first, but that should resolve itself:
$ oc get pods -n quay-enterprise NAME READY STATUS RESTARTS AGE postgres-7b87b9c978-j4ktp 1/1 Running 0 21h quay-enterprise-app-c9b7cf7dc-5pmqj 1/1 Running 0 26s quay-enterprise-config-app-5dcd4bcd76-82fx5 1/1 Running 0 19h quay-enterprise-redis-56977989bd-pqxzp 1/1 Running 0 21h
Get the URL for Red Hat Quay: Type the following to get the hostname of the new Red Hat Quay installation:
$ oc get routes -n quay-enterprise quay-enterprise NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD quay-enterprise quay-enterprise-quay-enterprise.apps.test.example.com quay-enterprise-clusterip <all> None
- Start using Red Hat Quay: Open the hostname in a web browser to start using Red Hat Quay.
Chapter 5. Starting to use Red Hat Quay
With Red Hat Quay now running, you can:
- Select Tutorial from the Quay home page to try the 15-minute tutorial. In the tutorial, you learn to log into Quay, start a container, create images, push repositories, view repositories, and change repository permissions with Quay.
- Refer to the Use Red Hat Quay for information on working with Red Hat Quay repositories.
Chapter 6. Appendix A: Red Hat Quay on OpenShift configuration files
The following yaml files were created to deploy Red Hat Quay on OpenShift. They are used throughout the deployment procedure in this document. We recommend you copy the files from this document into a directory, review the contents, and make any changes necessary for your deployment.
apiVersion: v1 kind: Namespace 1 metadata: name: quay-enterprise 2
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: quay-storageclass parameters: type: gp2 provisioner: kubernetes.io/aws-ebs reclaimPolicy: Delete
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: postgres-storage namespace: quay-enterprise spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 5Gi storageClassName: quay-storageclass
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: postgres namespace: quay-enterprise spec: replicas: 1 template: metadata: labels: app: postgres spec: containers: - name: postgres image: registry.redhat.io/rhscl/postgresql-10-rhel7:1-35 imagePullPolicy: "IfNotPresent" ports: - containerPort: 5432 env: - name: POSTGRESQL_USER value: "username" - name: POSTGRESQL_DATABASE value: quay - name: POSTGRESQL_PASSWORD value: "password" volumeMounts: - mountPath: /var/lib/pgsql/data name: postgredb volumes: - name: postgredb persistentVolumeClaim: claimName: postgres-storage
apiVersion: v1 kind: Service metadata: name: postgres namespace: quay-enterprise labels: app: postgres spec: type: NodePort ports: - port: 5432 selector: app: postgres
apiVersion: v1 kind: Secret metadata: namespace: quay-enterprise name: quay-enterprise-config-secret
apiVersion: v1 kind: Secret metadata: namespace: quay-enterprise name: redhat-quay-pull-secret data: .dockerconfigjson: <Add credentials> type: kubernetes.io/dockerconfigjson
apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: quay-enterprise-serviceaccount namespace: quay-enterprise rules: - apiGroups: - "" resources: - secrets verbs: - get - put - patch - update - apiGroups: - "" resources: - namespaces verbs: - get
apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: quay-enterprise-secret-writer namespace: quay-enterprise roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: quay-enterprise-serviceaccount subjects: - kind: ServiceAccount name: default
apiVersion: extensions/v1beta1 kind: Deployment metadata: namespace: quay-enterprise name: quay-enterprise-redis labels: quay-enterprise-component: redis spec: replicas: 1 selector: matchLabels: quay-enterprise-component: redis template: metadata: namespace: quay-enterprise labels: quay-enterprise-component: redis spec: containers: - name: redis-master image: registry.redhat.io/rhscl/redis-32-rhel7 ports: - containerPort: 6379 --- apiVersion: v1 kind: Service metadata: namespace: quay-enterprise name: quay-enterprise-redis labels: quay-enterprise-component: redis spec: ports: - port: 6379 selector: quay-enterprise-component: redis
apiVersion: extensions/v1beta1 kind: Deployment metadata: namespace: quay-enterprise name: quay-enterprise-config-app labels: quay-enterprise-component: config-app spec: replicas: 1 selector: matchLabels: quay-enterprise-component: config-app template: metadata: namespace: quay-enterprise labels: quay-enterprise-component: config-app spec: containers: - name: quay-enterprise-config-app image: quay.io/redhat/quay:v3.0.4 ports: - containerPort: 8443 command: ["/quay-registry/quay-entrypoint.sh"] args: ["config", "your_secret"] imagePullSecrets: - name: redhat-quay-pull-secret
apiVersion: v1 kind: Service metadata: namespace: quay-enterprise name: quay-enterprise-config spec: type: ClusterIP ports: - protocol: TCP name: https port: 8443 targetPort: 8443 selector: quay-enterprise-component: config-app
apiVersion: v1 kind: Route metadata: name: quay-enterprise-config namespace: quay-enterprise spec: to: kind: Service name: quay-enterprise-config tls: termination: passthrough
apiVersion: v1 kind: Service metadata: namespace: quay-enterprise name: quay-enterprise-clusterip spec: type: ClusterIP ports: - protocol: TCP name: http port: 80 targetPort: 8080 - protocol: TCP name: https port: 8443 targetPort: 8443 selector: quay-enterprise-component: app
apiVersion: v1 kind: Route metadata: name: quay-enterprise namespace: quay-enterprise spec: to: kind: Service name: quay-enterprise-clusterip
apiVersion: extensions/v1beta1 kind: Deployment metadata: namespace: quay-enterprise name: quay-enterprise-app labels: quay-enterprise-component: app spec: replicas: 1 selector: matchLabels: quay-enterprise-component: app template: metadata: namespace: quay-enterprise labels: quay-enterprise-component: app spec: volumes: - name: configvolume secret: secretName: quay-enterprise-config-secret containers: - name: quay-enterprise-app image: quay.io/redhat/quay:v3.0.4 ports: - containerPort: 8080 name: http - containerPort: 8443 name: https volumeMounts: - name: configvolume readOnly: false mountPath: /conf/stack imagePullSecrets: - name: redhat-pull-secret