How to delete images with old tags from the registry in OpenShift 4?

Solution Verified - Updated -

Environment

  • Red Hat Openshift Container Platform (RHOCP)
    • 4
  • OpenShift internal Image Registry

Issue

  • The image registry's persistent volume (pv) gets filled by the images with old tags as the application deployment in the RHOCP cluster increases, especially in the environment where the application version release is frequent. Pruning the image either manually or using a cronjob does not help.
  • How to clean up images with older tags that are not being utilized in RHOCP 4 using image pruning?
  • Specifying the revision in image pruning does not delete images with old tags?

Resolution

Pruning in general does not take any specific tag version as an input, so as a workaround use the "time" as a factor to identify the old tagged images.

Manual Pruning

NOTE: Please make sure, oc binary version is same as cluster version with command oc version.

To delete old tagged images without retaining any image revisions of it in the registry, run the following command with appropriate time values:

$ oc adm prune images --keep-tag-revisions=2 --keep-younger-than=2400h  --ignore-invalid-refs=true

#### The "--keep-younger-than=2400h" flag identifies the images older than 2400 hours i.e 100 days.

#### The "--keep-tag-revisions" identifies all the revisions of each tagged image to be deleted, for example "--keep-tag-revisions=2"  will identify all the revisions of each tagged image to be deleted excluding the last two latest revisions of each tagged image. Using "--keep-tag-revisions=0" is **not** recommended.

Note: The above command is a dry run and it will not delete any image unless a --confirm flag is used with it.

Refer to manually pruning images for additional information about how to prune images manually. In some cases, it could be required to hard prune the registry to free up disk space. Some times it could be also required to add the parameter --loglevel=10 to increase the log verbosity for the manual prune. Refer also to delete images not deleted by image pruner and hard prune in OpenShift 4 and image pruner unable to delete image if images are not pruned as expected.

Automatic pruning

Edit the imagepruners.imageregistry.operator.openshift.io object:

$ oc edit imagepruners.imageregistry.operator.openshift.io/cluster

  • The following example uses values to delete all the revisions of the image older than 100 days (i.e 2400 hours):

    spec:
  schedule: 0 0 * * * 
  suspend: false 
  keepTagRevisions: 2 
  keepYoungerThanDuration: 2400h
  resources: {} 
  affinity: {} 
  nodeSelector: {} 
  tolerations: [] 
  successfulJobsHistoryLimit: 3 
  failedJobsHistoryLimit: 3

Refer to automatically pruning images for additional information about how to configure the automatic pruning of images.

Root Cause

The image pruning does not remove the older tags. Image pruning only removes the older image revision for each tagged image.

Diagnostic Steps

  1. Verify the space available in the storage provided to the internal image registry:

    $ oc -n openshift-image-registry rsh image-registry-xxxx 
$ df -h /registry

  2. Pruning the images with the below commands will not clean the space or remove the images with old tags:

    $ oc adm prune images --keep-tag-revisions=3 --keep-younger-than=2400h  --ignore-invalid-refs=true
$ oc adm prune images --keep-tag-revisions=3 --keep-younger-than=2400h  --ignore-invalid-refs=true --confirm

  3. Space utilized by the image registry and old tags remains the same.

    $ oc project openshift-image-registry
$ oc rsh image-registry-xxxx -n openshift-image-registry
$ df -h /registry
$ oc get is

Note: Above example uses one revision of each tagged image to be saved from pruning.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments