Red Hat Advanced Cluster Management and ClusterImageSets
Red Hat Advanced Cluster Management (RHACM) is a solution that allows you to manage Kubernetes/Openshift clusters and infrastructures. It includes Openshift Hive, which is in charge of provisioning and managing clusters based on Openshift. Hive enables deployments with different cloud providers (such as, AWS, Azure or Google Cloud Platform) and also baremetal infrastructure.
Openshift Hive provides a custom resource called ClusterImageSet, which points to different available Openshift versions. You can easily check which available Openshift versions are available with:
> oc get clusterimagesets.hive.openshift.io
NAME RELEASE
img4.10.3-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.10.3-x86_64
img4.10.4-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.10.4-x86_64
img4.10.5-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.10.5-x86_64
img4.6.1-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.1-x86_64
img4.6.12-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.12-x86_64
img4.6.13-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.13-x86_64
img4.6.15-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.15-x86_64
img4.6.16-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.16-x86_64
img4.6.17-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.17-x86_64
img4.6.18-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.18-x86_64
img4.6.19-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.19-x86_64
.....
.....
.....
Using oc CLI provides the full list of available images. Although, when using ACM GUI, this list is automatically filtered to show only the latest versions. This encourages the creation of clusters with the latest release images , but does not avoid at all, to use previous versions.
> oc get clusterimagesets.hive.openshift.io img4.10.10-x86-64-appsub -o jsonpath={.metadata.labels.visible}
false
> oc get clusterimagesets.hive.openshift.io img4.10.15-x86-64-appsub -o jsonpath={.metadata.labels.visible}
true
The list of available images is automatically synced using a combination of Subscription and Channel resources. Following, some information about Subscription's manifest (simplified for better reading):
> oc get subscription hive-clusterimagesets-subscription-fast-0 -n open-cluster-management -oyaml
apiVersion: apps.open-cluster-management.io/v1
kind: Subscription
metadata:
annotations:
apps.open-cluster-management.io/git-branch: release-2.4
apps.open-cluster-management.io/git-path: clusterImageSets/fast
creationTimestamp: "2022-03-29T12:39:46Z"
labels:
subscription-pause: "false"
name: hive-clusterimagesets-subscription-fast-0
namespace: open-cluster-management
spec:
channel: open-cluster-management/acm-hive-openshift-releases-chn-0
placement:
local: true
status:
lastUpdateTime: "2022-05-24T22:20:38Z"
message: Active
phase: Subscribed
statuses:
/:
packages:
acm-hive-openshift-releases-chn-0-ClusterImageSet-img4.6.1-x86-64-appsub:
lastUpdateTime: "2022-04-17T14:43:26Z"
phase: Subscribed
acm-hive-openshift-releases-chn-0-ClusterImageSet-img4.6.3-x86-64-appsub:
lastUpdateTime: "2022-04-17T14:43:25Z"
phase: Subscribed
acm-hive-openshift-releases-chn-0-ClusterImageSet-img4.6.4-x86-64-appsub:
lastUpdateTime: "2022-04-17T14:43:27Z"
phase: Subscribed
acm-hive-openshift-releases-chn-0-ClusterImageSet-img4.6.6-x86-64-appsub:
Some important information we can check In the subscription:
- It is using a fast channel and the release 2.4 (which is the current version of RHACM we are using as example). You can learn more about Red Hat channels here.
- It is active (not paused)
- The date with the last update
- A list of all the ClusterImageSet received
The 'spec.channel' points to the corresponding Channel resource, which keeps the communication with the repository containing all the ClusterImageSets. In this case, the Subscription is using an ACM Channel called: 'open-cluster-management/acm-hive-openshift-releases-chn-0'
Dont confuse this ACM Channel with the Red Hat release channels showed in the previous link.
Lets get more info about this ACM Channel:
> oc get channel acm-hive-openshift-releases-chn-0 -n open-cluster-management -oyaml
apiVersion: apps.open-cluster-management.io/v1
kind: Channel
metadata:
annotations:
apps.open-cluster-management.io/reconcile-rate: low
creationTimestamp: "2022-03-29T12:39:46Z"
labels:
name: acm-hive-openshift-releases-chn-0
namespace: open-cluster-management
spec:
pathname: https://github.com/stolostron/acm-hive-openshift-releases.git
type: Git
The Channel points to a Git repository. This repository is maintained and updated by RedHat with the different available OCP versions, in the way of 'ClusterImageSets'.
The Subscription uses this Channel, selecting the git-repo and git-branch
apiVersion: apps.open-cluster-management.io/v1
kind: Subscription
metadata:
annotations:
apps.open-cluster-management.io/git-branch: release-2.4
apps.open-cluster-management.io/git-path: clusterImageSets/fast
By this mechanism, ACM keeps updated about all the available versions of OCP
Combining Git url configured by the Channel, the git-branch and the git-path from the Subscription, you can browse to the repo, where the different ClusterImageSets are downloaded.
By default, RHACM subscribes the list of Clusterimagesets to a fast channel. But the deployed clusters contains OCP's clusters subscribed to stable channel. This is a known issue/limitation.
In any case, you can play with the Subscription in order to point to another Red Hat channel, such as, the stable one.
Changing the channel to sync
RHACM comes with the fast channel installed by default. This can be changed by following next steps. If you are just using RHACM, by default, all the resources should be managed into the Namespace open-cluster-management:
- Pause the fast channel sub
> oc -n open-cluster-management patch appsub hive-clusterimagesets-subscription-fast-0 --type='json' -p='[{"op":"replace","path": "/metadata/labels/subscription-pause","value":"true"}]'
subscription.apps.open-cluster-management.io/hive-clusterimagesets-subscription-fast-0 patched
- Delete all the current available ClusterImageSets. These objects only contain references to download images. So you are not really deleting anything critical. The resources we are deleting belong to the fast channel. . Ensure, you did not manually created ClusterImageSets, these will be also deleted and maybe you want to keep them.
> oc delete clusterimageset --all
clusterimageset.hive.openshift.io "img4.10.10-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.11-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.12-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.13-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.14-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.15-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.3-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.4-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.5-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.6-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.8-x86-64-appsub" deleted
- Add a new subscription for stable channel. Suggestion: get the Manifest from your current subscription. Just clean the usual extra data created by Openshift/Kubernetes. Finally change the git-patch, ensure it is not paused, and change the name for stable-0.
Here an example, based on the pre-installed fast Subscription:
apiVersion: apps.open-cluster-management.io/v1
kind: Subscription
metadata:
annotations:
apps.open-cluster-management.io/git-branch: release-2.4
apps.open-cluster-management.io/git-path: clusterImageSets/stable
open-cluster-management.io/user-group: c3lzdGVtOnNlcnZpY2VhY2NvdW50cyxzeXN0ZW06c2VydmljZWFjY291bnRzOm9wZW4tY2x1c3Rlci1tYW5hZ2VtZW50LHN5c3RlbTphdXRoZW50aWNhdGVk
open-cluster-management.io/user-identity: c3lzdGVtOnNlcnZpY2VhY2NvdW50Om9wZW4tY2x1c3Rlci1tYW5hZ2VtZW50Om11bHRpY2x1c3Rlci1vcGVyYXRvcnM=
labels:
app: hive-clusterimagesets
subscription-pause: "false"
name: hive-clusterimagesets-subscription-stable-0
namespace: open-cluster-management
spec:
channel: open-cluster-management/acm-hive-openshift-releases-chn-0
placement:
local: true
- Create the object and wait for a while.
- You can list the new set of ClusterImageSets, and how the new ones belongs to stable channel:
> oc get clusterimagesets img4.9.9-x86-64-appsub -o jsonpath={.metadata.labels.channel}
stable
After that, you can change from one channel to other, just pausing and un-pausing the subscriptions.
ClusterImageSets refresh
The refresh time is managed by the Subscription resource:
> oc get channel acm-hive-openshift-releases-chn-0 -n open-cluster-management -o json | grep apps.open-cluster-management.io/reconcile-rate
"apps.open-cluster-management.io/reconcile-rate": "low",
In this case, it is configured as 'low'. Which means hourly. Other different values can be found here
Comments