Chapter 1. Advanced Red Hat Quay configuration

You can configure your Red Hat Quay after initial deployment using several different interfaces:

  • The Red Hat Quay Config Tool: Running the Quay container in config mode presents a Web-based interface for configuring the Red Hat Quay cluster. This is the recommended method for most configuration of the Red Hat Quay service itself.
  • Editing the config.yaml: The config.yaml file holds most of the configuration information for the Red Hat Quay cluster. Editing that file directly is possible, but it is only recommended for advanced tuning and performance features that are not available through the Config Tool.
  • Red Hat Quay API: Some Red Hat Quay configuration can be done through the API.

While configuration for specific features is covered in separate sections, this section describes how to use each of those interfaces and perform some more advanced configuration.

1.1. Using Red Hat Quay Config Tool to modify Red Hat Quay

The Red Hat Quay Config Tool is made available by running a Quay container in config mode alongside the regular Red Hat Quay service. Running the Config Tool is different for Red Hat Quay clusters running on OpenShift than it is for those running directly on host systems.

1.1.1. Running the Config Tool from the Red Hat Quay Operator

If you are running the Red Hat Quay Operator from OpenShift, the Config Tool is probably already available for you to use. To access the Config Tool, do the following:

  1. From the OpenShift console, select the project in which Red Hat Quay is running. For example, quay-enterprise.

  2. From the left column, select Networking → Routes. You should see routes to both the Red Hat Quay application and Config Tool, as shown in the following image:

    View the route to the Red Hat Quay Config Tool

  3. Select the route to the Config Tool (for example, example-quayecosystem-quay-config) and select it. The Config tool Web UI should open in your browser.

  4. Select Modify configuration for this cluster. You should see the Config Tool, ready for you to change features of your Red Hat Quay cluster, as shown in the following image:

    Modify Red Hat Quay cluster settings from the Config Tool

  5. When you have made the changes you want, select Save Configuration Changes. The Config Tool will validate your changes.
  6. Make any corrections as needed by selecting Continue Editing or select Next to continue on.
  7. When prompted, it is recommended that you select Download Configuration. That will download a tarball of your new config.yaml, as well as any certificates and keys used with your Red Hat Quay setup.
  8. Select Go to deployment rollout, then Populate the configuration to deployments. The Red Hat Quay pods will be restarted and the changes will take effect.

The config.yaml file you saved can be used to make advanced changes to your configuration or just kept for future reference.

1.1.2. Running the Config Tool from the command line

If you are running Red Hat Quay directly from a host system, using tools such as the podman or docker commands, after the initial Red Hat Quay deployment, you can restart the Config Tool to modify your Red Hat Quay cluster. Here’s how:

  1. Start quay in config mode: On the first quay node run the following, replacing my-secret-password with your password. If you would like to modify an existing config bundle, you can simply mount your configuration directory into the Quay container as you would in registry mode. 

    # podman run --rm -it --name quay_config -p 8080:8080 \
    -v path/to/config-bundle:/conf/stack \
    registry.redhat.io/quay/quay-rhel8:v3.6.8 config my-secret-password
  2. Open browser: When the quay configuration tool starts up, open a browser to the URL and port 8080 of the system you are running the configuration tool on (for example https://myquay.example.com:8080). You are prompted for a username and password.

At this point, you can begin modifying your Red Hat Quay cluster as described earlier.

1.2. Using the API to modify Red Hat Quay

See the Red Hat Quay API Guide for information on how to access Red Hat Quay API.

1.3. Editing the config.yaml file to modify Red Hat Quay

Some advanced Red Hat Quay configuration that is not available through the Config Tool can be achieved by editing the config.yaml file directly. Available settings are described in the Schema for Red Hat Quay configuration The following are examples of settings you can change directly in the config.yaml file.

1.3.1. Add name and company to Red Hat Quay sign-in

Setting the following will cause users to be prompted for their name and company when they first sign in. Although this is optional, it can provide you with extra data about your Red Hat Quay users:

+ FEATURE_USER_METADATA: true

1.3.2. Disable TLS Protocols

You can change the SSL_PROTOCOLS setting to remove SSL protocols that you do not want to support in your Red Hat Quay instance. For example, to remove TLS v1 support from the default SSL_PROTOCOLS : ['TLSv1','TLSv1.1','TLSv1.2'], change it as follows:

+ SSL_PROTOCOLS : ['TLSv1.1','TLSv1.2']

1.3.3. Rate limit API calls

Adding the FEATURE_RATE_LIMITS parameter to the config.yaml causes nginx to limit certain API calls to 30 per second. If that feature is not set, API calls are limied to 300 per second (effectively unlimited). Rate limiting can be an important feature, if you need to make sure the resources available are not overwhelmed with traffic.

Some namespace may require unlimited access (perhaps they are important to CI/CD and take priority, for example). In this case, those namespace may be placed in a list in config.yaml for NON_RATE_LIMITED_NAMESPACES.

1.3.4. Adjust database connection pooling

Red Hat Quay is composed of many different processes which all run within the same container. Many of these processes interact with the database.

If enabled, each process that interacts with the database will contain a connection pool. These per-process connection pools are configured to maintain a maximum of 20 connections. Under heavy load, it is possible to fill the connection pool for every process within a Red Hat Quay container. Under certain deployments and loads, this may require analysis to ensure Red Hat Quay does not exceed the database’s configured maximum connection count.

Overtime, the connection pools will release idle connections. To release all connections immediately, Red Hat Quay requires a restart.

Database connection pooling may be toggled by setting the environment variable DB_CONNECTION_POOLING={true|false}

If database connection pooling is enabled, it is possible to change the maximum size of the connection pool. This can be done through the following config.yaml option: 

DB_CONNECTION_ARGS:
  max_connections: 10

1.3.4.1. Database connection arguments

You can customize Red Hat Quay database connection settings within the config.yaml file. These are entirely dependent upon the underlying database driver, such as psycopg2 for Postgres and pymysql for MySQL. It is also possible to pass in arguments used by Peewee’s Connection Pooling mechanism as seen below. 

DB_CONNECTION_ARGS:
  max_connections: n  # Max Connection Pool size. (Connection Pooling only)
  timeout: n  # Time to hold on to connections. (Connection Pooling only)
  stale_timeout: n  # Number of seconds to block when the pool is full. (Connection Pooling only)

1.3.4.2. Database SSL configuration

Some key-value pairs defined under DB_CONNECTION_ARGS are generic while others are database-specific. In particular, SSL configuration depends on the database you are deploying.

1.3.4.2.1. PostgreSQL SSL connection arguments

A sample PostgreSQL SSL configuration is given below: 

DB_CONNECTION_ARGS:
  sslmode: verify-ca
  sslrootcert: /path/to/cacert

The sslmode option determines whether or with what priority a secure SSL TCP/IP connection will be negotiated with the server. There are six modes:

  • disable: only try a non-SSL connection
  • allow: first try a non-SSL connection; if that fails, try an SSL connection
  • prefer: (default) first try an SSL connection; if that fails, try a non-SSL connection
  • require: only try an SSL connection. If a root CA file is present, verify the certificate in the same way as if verify-ca was specified
  • verify-ca: only try an SSL connection, and verify that the server certificate is issued by a trusted certificate authority (CA)
  • verify-full: only try an SSL connection, verify that the server certificate is issued by a trusted CA and that the requested server host name matches that in the certificate

More information on the valid arguments for PostgreSQL is available at https://www.postgresql.org/docs/current/libpq-connect.html.

1.3.4.2.2. MySQL SSL connection arguments

A sample MySQL SSL configuration follows: 

DB_CONNECTION_ARGS:
  ssl:
    ca: /path/to/cacert

Information on the valid connection arguments for MySQL is available at https://dev.mysql.com/doc/refman/8.0/en/connecting-using-uri-or-key-value-pairs.html.

1.3.4.3. HTTP connection counts

It is possible to specify the quantity of simultaneous HTTP connections using environment variables. These can be specified as a whole, or for a specific component. The default for each is 50 parallel connections per process.

Environment variables: 

WORKER_CONNECTION_COUNT_REGISTRY=n
WORKER_CONNECTION_COUNT_WEB=n
WORKER_CONNECTION_COUNT_SECSCAN=n
WORKER_CONNECTION_COUNT=n

Specifying a count for a specific component will override any value set in WORKER_CONNECTION_COUNT.

1.3.4.4. Dynamic process counts

To estimate the quantity of dynamically sized processes, the following calculation is used by default.

Note

Red Hat Quay queries the available CPU count from the entire machine. Any limits applied using kubernetes or other non-virtualized mechanisms will not affect this behavior; Red Hat Quay will makes its calculation based on the total number of processors on the Node. The default values listed are simply targets, but shall not exceed the maximum or be lower than the minimum.

Each of the following process quantities can be overridden using the environment variable specified below.

  • registry - Provides HTTP endpoints to handle registry action

    • minimum: 8
    • maximum: 64
    • default: $CPU_COUNT x 4
    • environment variable: WORKER_COUNT_REGISTRY

  • web - Provides HTTP endpoints for the web-based interface

    • minimum: 2
    • maximum: 32
    • default: $CPU_COUNT x 2
    • environment_variable: WORKER_COUNT_WEB

  • secscan - Interacts with Clair

    • minimum: 2
    • maximum: 4
    • default: $CPU_COUNT x 2
    • environment variable: WORKER_COUNT_SECSCAN

1.3.4.5. Environment variables

Red Hat Quay allows overriding default behavior using environment variables. This table lists and describes each variable and the values they can expect.

Table 1.1. Worker count environment variables

VariableDescriptionValues

WORKER_COUNT_REGISTRY

Specifies the number of processes to handle Registry requests within the Quay container.

Integer between 8 and 64

WORKER_COUNT_WEB

Specifies the number of processes to handle UI/Web requests within the container.

Integer between 2 and 32

WORKER_COUNT_SECSCAN

Specifies the number of processes to handle Security Scanning (e.g. Clair) integration within the container.

Integer between 2 and 4

DB_CONNECTION_POOLING

Toggle database connection pooling. In 3.4, it is disabled by default.

"true" or "false"

1.3.4.6. Turning off connection pooling

Red Hat Quay deployments with a large amount of user activity can regularly hit the 2k maximum database connection limit. In these cases, connection pooling, which is enabled by default for Red Hat Quay, can cause database connection count to rise exponentially and require you to turn off connection pooling.

If turning off connection pooling is not enough to prevent hitting that 2k database connection limit, you need to take additional steps to deal with the problem. In this case you might need to increase the maximum database connections to better suit your workload.