Red Hat Quay Release Notes
Red Hat Quay
Abstract
Preface
Red Hat Quay container registry platform provides secure storage, distribution, and governance of containers and cloud-native artifacts on any infrastructure. It is available as a standalone component or as an Operator on OpenShift Container Platform. Red Hat Quay includes the following features and benefits:
- Granular security management
- Fast and robust at any scale
- High velocity CI/CD
- Automated installation and upates
- Enterprise authentication and team-based access control
- OpenShift Container Platform integration
Red Hat Quay is regularly released, containing new features, bug fixes, and software updates. To upgrade Red Hat Quay for both standalone and OpenShift Container Platform deployments, see Upgrade Red Hat Quay.
Red Hat Quay only supports rolling back, or downgrading, to previous z-stream versions, for example, 3.7.2 → 3.7.1. Rolling back to previous y-stream versions (3.7.0 → 3.6.0) is not supported. This is because Red Hat Quay updates might contain database schema upgrades that are applied when upgrading to a new version of Red Hat Quay. Database schema upgrades are not considered backwards compatible.
Downgrading to previous z-streams is neither recommended nor supported by either Operator based deployments or virtual machine based deployments. Downgrading should only be done in extreme circumstances. The decision to rollback your Red Hat Quay deployment must be made in conjunction with the Red Hat Quay support and development teams. For more information, contact Red Hat Quay support.
Documentation for Red Hat Quay is versioned with each release. The latest Red Hat Quay documentation is available from the Red Hat Quay Documentation page. Currently, version 3 is the latest major version.
Prior to version 2.9.2, Red Hat Quay was called Quay Enterprise. Documentation for 2.9.2 and prior versions are archived on the Product Documentation for Red Hat Quay 2.9 page.
Chapter 1. RHBA-2023:5345 - Red Hat Quay 3.9.2 release
Issued 2023-09-26
Red Hat Quay release 3.9.2 is now available.
As of September 25, 2023, the Code Ready Dependency Analytics (CRDA) service for Java vulnerability matching is no longer usable with Clair. The service’s API moved to a different endpoint and there are no plans to update Clair to support this new endpoint. Instead, users should upgrade to Red Hat Quay 3 in order to keep getting CVE reports on Java Maven packages indexed by Clair from container images stored in Red Hat Quay, with the additional benefit of offline support and without the need for separate API keys.
The bug fixes that are included in the update are listed in the RHBA-2023:5345 advisory.
1.1. Bug fixes
- PROJQUAY-5174. Quay Operator doesn’t trust internal service CA when it is rotated.
- PROJQUAY-5931. Duplicate Robot accounts
- PROJQUAY-5256. Storage replication not triggered on manifest list mirror
Chapter 2. RHBA-2023:4974 - Red Hat Quay 3.9.1 release
Issued 2023-09-05
Red Hat Quay release 3.9.1 is now available with Clair 4.7.1. The bug fixes that are included in the update are listed in the RHBA-2023:4974 advisory.
2.1. Bug fixes
- PROJQUAY-5581. Should show total quota consumption for user account namespace in UI.
- PROJQUAY-5691. CVE-2023-33733 python-reportlab: remote code execution via supplying a crafted PDF file [quay-3.9].
- PROJQUAY-5702. CVE-2023-36464 quay-registry-container: pypdf: Possible Infinite Loop when a comment isn’t followed by a character [quay-3].
- PROJQUAY-5874. CVE-2021-33194 Vulnerabilities in dependency usr/local/bin/pushgateway (gobinary).
- PROJQUAY-5925. A lot of quotatotalworker error in quayregistry-quay-config-editor pod log.
- PROJQUAY-5914. Bulk update Repo settings in Robot accounts tab.
- PROJQUAY-5967. Quay 3.9.1 High Image Vulnerability reported by Redhat ACS.
Chapter 3. RHBA-2023:3256 - Red Hat Quay 3.9.0 release
Issued 2023-08-14
Red Hat Quay release 3.9.0 is now available with Clair 4.7. The bug fixes that are included in the update are listed in the RHBA-2023:3256 advisory.
3.1. Red Hat Quay release cadence
With the next release of Red Hat Quay, version 3.10, the product will begin to align its release cadence and lifecycle with OpenShift Container Platform. As a result, Red Hat Quay 3.10 will be generally available within approximately four weeks of the OpenShift Container Platform 4.14 release, which is currently scheduled for release in early Q4, 2024.
With the current release model, the total support length of Red Hat Quay 3.8 and Red Hat Quay 3.9 would have been cut short due to the release of Red Hat Quay 3.10 being scheduled earlier than previous releases. In order to provide customers with proper time to prepare for updates, the full support and maintenance phases of Red Hat Quay 3.8 and Red Hat Quay 3.9 have been amended to go beyond the release of Red Hat Quay 3.10. This is a one time amendment. After the release of Red Hat Quay 3.10 and subsequent releases, customers can expect the support lifecycle phases of Red Hat Quay to align with OpenShift Container Platform releases.
For more information, see the Red Hat Quay Life Cycle Policy.
3.2. Red Hat Quay new features and enhancements
The following updates have been made to Red Hat Quay:
3.2.1. Clair 4.7
Clair 4.7 was released as part of Red Hat Quay 3.9.
As of September 25, 2023, the Code Ready Dependency Analytics (CRDA) service for Java vulnerability matching will no longer be usable with Clair. The service’s API moved to a different endpoint and there are no plans to update Clair to support this new endpoint. Instead, users should upgrade to Red Hat Quay 3 in order to keep getting CVE reports on Java Maven packages indexed by Clair from container images stored in Red Hat Quay, with the additional benefit of offline support and without the need for separate API keys.
Additional enhancements to Clair include the following:
- Native support for indexing Golang modules and RubeGems in container images.
Change to OSV.dev as the vulnerability database source for any programming language package managers.
- This includes popular sources like GitHub Security Advisories or PyPA.
- This allows offline capability.
- Use of pyup.io for Python and CRDA for Java is suspended.
- Clair now supports Java, Golang, Python, and Ruby dependencies.
3.2.2. Removal of a single site in a geo-replicated environment
Red Hat Quay administrators can now remove a specific site from their geo-replicated environment.
For more information, see Removing a geo-replicated site from your Red Hat Quay Operator deployment.
3.2.3. Quota management enhancements
Prior to Red Hat Quay 3.9, the quota management feature created totals by combining the manifest sizes at the repository and namespace level. This created an issue wherein a single blob could be counted multiple times within the total. For example, in previous versions of Red Hat Quay, if blobs were referenced multiple times within a repository and namespace, the blob was counted towards the allotted quota for every time it was referenced.
With this release, individual blob sizes are summed at the repository and namespace level. For example, if two tags in the same repository reference the same blob, the size of that blob is now only counted once towards the repository total. This enhancement to the quota management feature works by calculating the size of existing repositories and namespace with a backfill worker, and then adding or subtracting from the total for every image that is pushed or garbage collected afterwords. Additionally, the subtraction from the total happens when the manifest is garbage collected, whereas in the past it occurred when the tag was deleted.
NoteBecause subtraction occurs from the total when the manifest is garbage collected, there is a delay in the size calculation until it is able to be garbage collected. For more information about Red Hat Quay garbage collection, see Red Hat Quay garbage collection.
Additionally, manifest list totals are now counted toward the repository total, the total quota consumed when upgrading from a previous version of Red Hat Quay might be reportedly differently in Red Hat Quay 3.9. In some cases, the new total might go over a repository’s previously-set limit. Red Hat Quay administrators might have to adjust the allotted quota of a repository to account for these changes.
Collectively, the quota management feature in Red Hat Quay 3.9 provides a more accurate depiction of storage growth and registry consumption. As a result, users can place quota limits on the namespace and repository sizes based on the actual usage of storage by Red Hat Quay.
For more information, see Quota management for Red Hat Quay 3.9
3.2.4. Configuring action log storage for Splunk
With this release, Red Hat Quay administrators can forward logs to a Splunk deployment. This allows administrators to perform log analyses and offload the internal database.
For more information, see Configuring action log storage for Splunk.
3.2.5. Red Hat Quay UI v2 enhancements
In Red Hat Quay 3.8, a new UI was introduced as a technology preview. With Red Hat Quay 3.9, the following enhancements have been made to the UI v2:
- A tab for robot account creation.
- A tab for Organization settings.
- A tab for image tags.
- A tab for Repository settings.
- Overview, Security Reports, and Package vulnerability reports.
For more information about UI v2 enablement, see Using the Red Hat Quay v2 UI.
3.2.6. Nutanix Object Storage
With this release, Nutanix Object Storage is now supported. For more information, see Nutanix Object Storage.
3.3. New Red Hat Quay configuration fields
The following configuration fields have been added to Red Hat Quay 3.9:
The following configuration fields have been added to the quota management feature:
QUOTA_BACKFILL: Enables the quota backfill worker to calculate the size of pre-existing blobs. Because this parameter sums the de-duplicated totals in the database, it might increase database load.
Default:
TrueQUOTA_TOTAL_DELAY_SECONDS:The time delay for starting the quota backfill. Rolling deployments can cause incorrect totals. This field must be set to a time longer than it takes for the rolling deployment to complete.
Default:
1800PERMANENTLY_DELETE_TAGS: Enables functionality related to the removal of tags from the time machine window.
Default:
FalseRESET_CHILD_MANIFEST_EXPIRATION: Resets the expirations of temporary tags targeting the child manifests. With this feature set to
True, child manifests are immediately garbage collected.Default:
False
For more information, see Configuration updates for Red Hat Quay 3.9.
The following configuration field has been added to enhance the Red Hat Quay security scanner feature:
FEATURE_SECURITY_SCANNER_NOTIFY_ON_NEW_INDEX: Whether to allow sending notifications about vulnerabilities for new pushes.
Default:
TrueFor more information, see Security scanner configuration fields.
The following configuration field has been added to configure whether Red Hat Quay automatically removes old persistent volume claims (PVCs) when upgrading from version 3.8 → 3.9:
POSTGRES_UPGRADE_RETAIN_BACKUP: When set to
True, persistent volume claims from PostgreSQL 10 are backed up.Default:
False
The following configuration field has been added to track various events:
ACTION_LOG_AUDIT_LOGINS: When set to
True, tracks advanced events such as logging into, and out of, the UI, and logging in using Docker for regular users, robot accounts, and for application-specific token accounts.Default:
True
3.4. Red Hat Quay Operator
The following updates have been made to the Red Hat Quay Operator:
Currently, the Red Hat Quay Operator and Clair use PostgreSQL 10. PostgreSQL 10 had its final release on November 10, 2022 and is no longer supported.
With this release, if your database is managed by the Red Hat Quay Operator, updating from Red Hat Quay 3.8 → 3.9 automatically handles upgrading PostgreSQL 10 to PostgreSQL 13.
ImportantUsers with a managed database will be required to upgrade their PostgreSQL database from 10 → 13.
If you do not want the Red Hat Quay Operator to upgrade your PostgreSQL deployment from 10 → 13, you must set the PostgreSQL parameter to
managed: falsein yourquayregistry.yamlfile. For more information about setting your database to unmanaged, see Using an existing Postgres database.Important- It is highly recommended that you upgrade to PostgreSQL 13. PostgreSQL 10 had its final release on November 10, 2022 and is no longer supported. For more information, see the PostgreSQL Versioning Policy.
If you want your PostgreSQL database to match the same version as your Red Hat Enterprise Linux (RHEL) system, see Migrating to a RHEL 8 version of PostgreSQL for RHEL 8 or Migrating to a RHEL 9 version of PostgreSQL for RHEL 9.
For more information about the Red Hat Quay 3.8 → 3.9 procedure, see Upgrading the Red Hat Quay Operator overview.
3.5. Red Hat Quay 3.9 known issues and limitations
The following sections note known issues and limitations for Red Hat Quay 3.9.
3.5.1. Known issues:
3.5.1.1. Upgrading known issues
There are two known issues when upgrading your Red Hat Quay deployment:
-
If your Red Hat Quay deployment is upgrading from one y-stream to the next, for example, from 3.8.10 → 3.8.11, you must not switch the upgrade channel from
stable-3.8tostable-3.9. Changing the upgrade channel in the middle of a y-stream upgrade will disallow Red Hat Quay from upgrading to 3.9. This is a known issue and will be fixed in a future version of Red Hat Quay. -
When upgrading from Red Hat Quay 3.7 to 3.9, you might receive the following error:
pg_dumpall: error: query failed: ERROR: xlog flush request 1/B446CCD8 is not satisfied --- flushed only to 1/B0013858. As a workaround to this issue, you can delete thequayregistry-clair-postgres-upgradejob on your OpenShift Container Platform deployment, which should resolve the issue.
3.5.1.2. Other known issues
Using
conftest pullcommands to obtain policies might return the following error:Error: download policies: client get: stat /policy/quayregistry-quay-quay-enterprise-847.apps.quaytest-847.qe.devcluster.openshift.com/conftest/policy:latest: no such file or directory. As a workaround, you can add theoci://prefix on your registry host. For example:$ conftest pull oci://mkoktest.quaydev.org/admin/conftest:v1
This is a known issue and will be fixed in a future version of Red Hat Quay. (PROJQUAY-5573)
Red Hat Quay 3.9 introduced changes to the quota management feature. One of these changes is that tags in the time machine window now count towards the quota total of your organization.
There is a known issue when the proxy cache feature is enabled and configured in a new organization with a hard quota check and time machine settings set to longer than a few seconds under their organization settings. In sum, tags in a proxy organization are all given a tag expiration that defaults to 1 day. If your proxy organization has a time machine policy set to longer than a few seconds under your organization settings, and the tag expires, it is not immediately available for garbage collection; it must wait to be outside of the time machine window before it can be garbage collected. Because subtraction happens upon garbage collection, and pruned tags are kept within the time frame allotted by your organization’s settings, image tags are not immediately garbage collected. This results in the quota consumption metric not being updated, and runs the risk of your proxy organization going over the allotted quota.
When a hard quota check is configured for a proxy organization, Red Hat Quay administrators will want to reclaim the space taken by tags within the time machine window to prevent organizations from hitting their allotted quota. As a temporary workaround, you can set the time machine expiration for proxy organizations to a few seconds under Organizations → Settings on the Red Hat Quay UI. This immediately removes image tags and allows for more accurate quota consumption metrics.
This is a non-issue for proxy organizations employing a soft quota check and can be ignored.
-
When removing a site from your geo-replicated Red Hat Quay deployment, you might receive the following error when running
python -m util.removelocation:/app/lib/python3.9/site-packages/tzlocal/unix.py:141: SyntaxWarning: "is not" with a literal. Did you mean "!="? while start is not 0: /app/lib/python3.9/site-packages/netaddr/strategy/{}init{}.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="? if word_sep is not ''. You can confirm the deletion of your site by enteringy. The error is a known issue and will be removed in a future version of Red Hat Quay.
3.5.2. Red Hat Quay 3.9 limitations
-
You must use the Splunk UI to view Red Hat Quay action logs. At this time, viewing Splunk action logs on the Red Hat Quay Usage Logs page is unsupported, and returns the following message:
Method not implemented. Splunk does not support log lookups.
3.6. Red Hat Quay bug fixes
- Previously, on Red Hat Quay Lightweight Directory Access Protocol (LDAP) deployments, there was a bug that disallowed referrals from being used with team synchronization and in other circumstances. With this update, referrals can be turned off globally for Red Hat Quay to ensure proper behavior across all components.
Previously, only last access timestamps were recorded in Red Hat Quay. This issue has been fixed, and now the following timestamps are recorded:
- Login to the Red Hat Quay UI.
- Logout of the Red Hat Quay UI.
- Login via Docker CLI (registry API) for regular users.
- Login via Docker CLI (Registry API) for robot accounts.
Login via Docker CLI (Registry API) for app-specific tokens accounts.
You can disable this timestamp feature by setting
ACTION_LOG_AUDIT_LOGINStofalsein yourconfig.yamlfile. This field is set totrueby default.NoteLogout events from the client side (Docker or Podman) are not causing requests to the registry API and are therefore not trackable.
- PROJQUAY-4614. Add conftest mediatypes to default Quay configuration.
- PROJQUAY-4865. Remove unused dependencies.
- PROJQUAY-4957. Limit indexing of manifests that continuously fail.
- PROJQUAY-5009. secscan: add api client timeout.
- PROJQUAY-5018. Ignore unknown media types in manifests.
- PROJQUAY-5237. The number of repositories in organization is incorrect in new UI.
- PROJQUAY-4993. Support Action Log Forward to Splunk.
- PROJQUAY-4567. Robot Tokens.
- PROJQUAY-5289. Create a new username for accounts that login via SSO in the new UI.
- PROJQUAY-5362. API: Add filtering to Tags API.
- PROJQUAY-5207. Phase 3: Quay.io Summit Deliverables.
- PROJQUAY-4608. Quay Operator should install a fully supported version of Postgres for Quay and Clair.
- PROJQUAY-5050. Can’t provide a link to quay directly to an image that works in both old UI and new UI.
- PROJQUAY-5253. Don’t convert dashes to underscores during first login.
- PROJQUAY-4303. Multi-arch images are ignored in storage consumption calculation.
- PROJQUAY-4304. Empty repositories are reporting storage consumption.
- PROJQUAY-5634. oci: Allow optional components in the image config to be set to "null".
- PROJQUAY-5639. Quay 3.9.0 delete organization under normal user by superuser was failed with unauthorized error.
- PROJQUAY-5642. Quay 3.9.0 image High Vulnerability reported by Redhat ACS.
- PROJQUAY-5630. Quay 3.9.0 Quay image High vulnerability issue CVE-2022-28948.
3.7. Red Hat Quay feature tracker
New features have been added to Red Hat Quay, some of which are currently in Technology Preview. Technology Preview features are experimental features and are not intended for production use.
Some features available in previous releases have been deprecated or removed. Deprecated functionality is still included in Red Hat Quay, but is planned for removal in a future release and is not recommended for new deployments. For the most recent list of deprecated and removed functionality in Red Hat Quay, refer to Table 1.1. Additional details for more fine-grained functionality that has been deprecated and removed are listed after the table.
Table 3.1. Technology Preview tracker
| Feature | Quay 3.9 | Quay 3.8 | Quay 3.7 |
|---|---|---|---|
| General Availability | - | - | |
| General Availability | - | - | |
| General Availability | - | - | |
| Docker v1 support | Deprecated | Deprecated | General Availability |
| Technology Preview | Technology Preview | - | |
| General Availability | General Availability | - | |
| General Availability | General Availability | - | |
| General Availability | General Availability | - | |
| General Availability | General Availability | - | |
| General Availability | General Availability | - | |
| General Availability | General Availability | - | |
| General Availability | General Availability | - | |
| General Availability | General Availability | General Availability | |
| General Availability | General Availability | General Availability | |
| General Availability | General Availability | Technology Preview | |
| General Availability | General Availability | General Availability | |
| General Availability | General Availability | General Availability | |
| Support for Microsoft Azure Government (MAG) | General Availability | General Availability | General Availability |
| Technology Preview | Technology Preview | Technology Preview |