Chapter 10. Repository mirroring
10.1. Repository mirroring
Red Hat Quay repository mirroring lets you mirror images from external container registries (or another local registry) into your Red Hat Quay cluster. Using repository mirroring, you can synchronize images to Red Hat Quay based on repository names and tags.
From your Red Hat Quay cluster with repository mirroring enabled, you can:
- Choose a repository from an external registry to mirror
- Add credentials to access the external registry
- Identify specific container image repository names and tags to sync
- Set intervals at which a repository is synced
- Check the current state of synchronization
To use the mirroring functionality, you need to:
- Enable Repository Mirroring in the Red Hat Quay configuration
- Run a repository mirroring worker
- Create mirrored repositories
All repository mirroring configuration can be performed using the configuration tool UI or via the Quay API
10.2. Repository mirroring versus geo-replication
Quay geo-replication mirrors the entire image storage backend data between 2 or more different storage backends while the database is shared (one Quay registry with two different blob storage endpoints). The primary use cases for geo-replication are:
- Speeding up access to the binary blobs for geographically dispersed setups
- Guaranteeing that the image content is the same across regions
Repository mirroring synchronizes selected repositories (or subsets of repositories) from one registry to another. The registries are distinct, with registry is separate database and image storage. The primary use cases for mirroring are:
- Independent registry deployments in different datacenters or regions, where a certain subset of the overall content is supposed to be shared across the datacenters / regions
- Automatic synchronization or mirroring of selected (whitelisted) upstream repositories from external registries into a local Quay deployment
Repository mirroring and geo-replication can be used simultaneously.
Table 10.1. Red Hat Quay Repository mirroring versus geo-replication
|Feature / Capability||Geo-replication||Repository mirroring|
What is the feature designed to do?
A shared, global registry
Distinct, different registries
What happens if replication or mirroring hasn’t been completed yet?
The remote copy is used (slower)
No image is served
Is access to all storage backends in both regions required?
Yes (all Red Hat Quay nodes)
No (distinct storage)
Can users push images from both sites to the same repository?
Is all registry content and configuration identical across all regions (shared database)
Can users select individual namespaces or repositories to be mirrored?
Can users apply filters to synchronization rules?
10.3. Using repository mirroring
Here are some features and limitations of Red Hat Quay repository mirroring:
- With repository mirroring, you can mirror an entire repository or selectively limit which images are synced. Filters can be based on a comma-separated list of tags, a range of tags, or other means of identifying tags through regular expressions.
- Once a repository is set as mirrored, you cannot manually add other images to that repository.
- Because the mirrored repository is based on the repository and tags you set, it will hold only the content represented by the repo/tag pair. In other words, if you change the tag so that some images in the repository no longer match, those images will be deleted.
- Only the designated robot can push images to a mirrored repository, superseding any role-based access control permissions set on the repository.
- With a mirrored repository, a user can pull images (given read permission) from the repository but not push images to the repository.
- Changing settings on your mirrored repository is done from the Mirrors tab on the Repositories page for the mirrored repository you create.
- Images are synced at set intervals, but can also be synced on demand.
10.4. Mirroring configuration UI
Quaycontainer in configuration mode and select the Enable Repository Mirroring check box. If you want to require HTTPS communications and verify certificates during mirroring, select the HTTPS and cert verification check box.
Validate and download the
configurationfile, and then restart Quay in registry mode using the updated config file.
10.5. Mirroring configuration fields
Table 10.2. Mirroring configuration
Enable or disable repository mirroring
The number of seconds between checking for repository mirror candidates
Require HTTPS and verify certificates of Quay registry during mirror.
10.6. Mirroring worker
To run the repository mirroring worker, start by running a
Quaypod with the
$ sudo podman run -d --name mirroring-worker \ -v $QUAY/config:/conf/stack:Z \ registry.redhat.io/quay/quay-rhel8:v3.5.3 repomirror
If you have configured TLS communications using a certificate
/root/ca.crt, then the following example shows how to start the mirroring worker:
$ sudo podman run -d --name mirroring-worker \ -v $QUAY/config:/conf/stack:Z \ -v /root/ca.crt:/etc/pki/ca-trust/source/anchors/ca.crt \ registry.redhat.io/quay/quay-rhel8:v3.5.3 repomirror
10.7. Creating a mirrored repository
The steps shown in this section assume you already have enabled repository mirroring in the configuration for your Red Hat Quay cluster and that you have a deployed a mirroring worker.
When mirroring a repository from an external container registry, create a new private repository. Typically the same name is used as the target repository, for example,
10.7.1. Repository mirroring settings
In the Settings tab, set the Repository State to
In the Mirror tab, enter the details for connecting to the external registry, along with the tags, scheduling and access information:
Enter the details as required in the following fields:
Registry Location: The external repository you want to mirror, for example,
Tags: This field is required. You may enter a comma-separated list of individual tags or tag patterns. (See Tag Patterns section for details.)Note
In order for Quay to get the list of tags in the remote repository, one of the following requirements must be met:
- An image with the "latest" tag must exist in the remote repository OR
- At least one explicit tag, without pattern matching, must exist in the list of tags that you specify
- Start Date: The date on which mirroring begins. The current date and time is used by default.
- Sync Interval: Defaults to syncing every 24 hours. You can change that based on hours or days.
- Robot User: Create a new robot account or choose an existing robot account to do the mirroring.
- Username: The username for accessing the external registry holding the repository you are mirroring.
- Password: The password associated with the Username. Note that the password cannot include characters that require an escape character (\).
- Registry Location: The external repository you want to mirror, for example,
10.7.2. Advanced settings
- In the Advanced Settings section, configure TLS and proxy, if required:
- Verify TLS: Check this box if you want to require HTTPS and to verify certificates, when communicating with the target remote registry.
- HTTP Proxy: Identify the HTTP proxy server needed to access the remote site, if one is required.
- HTTPS Proxy: Identify the HTTPS proxy server needed to access the remote site, if one is required.
- No Proxy: List of locations that do not require proxy
10.7.3. Synchronize now
To perform an immediate mirroring operation, press the Sync Now button on the repository’s Mirroring tab. The logs are available on the Usage Logs tab:
When the mirroring is complete, the images will appear in the Tags tab:
Below is an example of a completed Repository Mirroring screen:
10.8. Event notifications for mirroring
There are three notification events for repository mirroring:
- Repository Mirror Started
- Repository Mirror Success
- Repository Mirror Unsuccessful
The events can be configured inside the Settings tab for each repository, and all existing notification methods such as email, slack, Quay UI and webhooks are supported.
10.9. Mirroring tag patterns
As noted above, at least one Tag must be explicitly entered (ie. not a tag pattern) or the tag "latest" must exist in the report repository. (The tag "latest" will not be synced unless specified in the tag list.). This is required for Quay to get the list of tags in the remote repository to compare to the specified list to mirror.
10.9.1. Pattern syntax
Matches all characters
Matches any single character
Matches any character in seq
Matches any character not in seq
10.9.2. Example tag patterns
v32, v3.1, v3.2, v3.2-4beta, v3.3
v3.1, v3.2, v3.2-4beta
v3.1, v3.2, v3.3
v3.1, v3.2, v3.2-4beta
v3.2, v3.2-4beta, v3.3
10.10. Working with mirrored repositories
Once you have created a mirrored repository, there are several ways you can work with that repository. Select your mirrored repository from the Repositories page and do any of the following:
- Enable/disable the repository: Select the Mirroring button in the left column, then toggle the Enabled check box to enable or disable the repository temporarily.
Check mirror logs: To make sure the mirrored repository is working properly, you can check the mirror logs. To do that, select the Usage Logs button in the left column. Here’s an example:
- Sync mirror now: To immediately sync the images in your repository, select the Sync Now button.
- Change credentials: To change the username and password, select DELETE from the Credentials line. Then select None and add the username and password needed to log into the external registry when prompted.
- Cancel mirroring: To stop mirroring, which keeps the current images available but stops new ones from being synced, select the CANCEL button.
Set robot permissions: Red Hat Quay robot accounts are named tokens that hold credentials for accessing external repositories. By assigning credentials to a robot, that robot can be used across multiple mirrored repositories that need to access the same external registry.
You can assign an existing robot to a repository by going to Account Settings, then selecting the Robot Accounts icon in the left column. For the robot account, choose the link under the REPOSITORIES column. From the pop-up window, you can:
- Check which repositories are assigned to that robot.
- Assign read, write or Admin privileges to that robot from the PERMISSION field shown in this figure:
Change robot credentials: Robots can hold credentials such as Kubernetes secrets, Docker login information, and Mesos bundles. To change robot credentials, select the Options gear on the robot’s account line on the Robot Accounts window and choose View Credentials. Add the appropriate credentials for the external repository the robot needs to access.
- Check and change general setting: Select the Settings button (gear icon) from the left column on the mirrored repository page. On the resulting page, you can change settings associated with the mirrored repository. In particular, you can change User and Robot Permissions, to specify exactly which users and robots can read from or write to the repo.
10.11. Repository mirroring recommendations
- Repository mirroring pods can run on any node including other nodes where Quay is already running
- Repository mirroring is scheduled in the database and run in batches. As a result, more workers could mean faster mirroring, since more batches will be processed.
The optimal number of mirroring pods depends on:
- The total number of repositories to be mirrored
- The number of images and tags in the repositories and the frequency of changes
- Parallel batches
- You should balance your mirroring schedule across all mirrored repositories, so that they do not all start up at the same time.
- For a mid-size deployment, with approximately 1000 users and 1000 repositories, and with roughly 100 mirrored repositories, it is expected that you would use 3-5 mirroring pods, scaling up to 10 if required.