Preparing to upgrade to OpenShift Container Platform 4.9

Updated -

OpenShift Container Platform 4.9 uses Kubernetes 1.22, which removed a significant number of deprecated v1beta1 APIs.

OpenShift Container Platform 4.8.14 introduced a requirement that an administrator must provide a manual acknowledgment before the cluster can be upgraded from OpenShift Container Platform 4.8 to 4.9. This is to help prevent issues after upgrading to OpenShift Container Platform 4.9, where APIs that have been removed are still in use by workloads, tools, or other components running on or interacting with the cluster. Administrators must evaluate their cluster for any APIs in use that will be removed and migrate the affected components to use the appropriate new API version. After this is done, the administrator can provide the administrator acknowledgment.

All clusters require this administrator acknowledgment before they can be upgraded to OpenShift Container Platform 4.9.

Removed Kubernetes APIs

Kubernetes 1.22 removed the following deprecated v1beta1 APIs. If your cluster, or any idle workloads or tools use any of these APIs, you must migrate them to the appropriate new version before upgrading to OpenShift Container Platform 4.9.

Resource API Notable changes
APIService apiregistration.k8s.io/v1beta1 No
CertificateSigningRequest certificates.k8s.io/v1beta1 Yes
ClusterRole rbac.authorization.k8s.io/v1beta1 No
ClusterRoleBinding rbac.authorization.k8s.io/v1beta1 No
CSIDriver storage.k8s.io/v1beta1 No
CSINode storage.k8s.io/v1beta1 No
CustomResourceDefinition apiextensions.k8s.io/v1beta1 Yes
Ingress extensions/v1beta1 Yes
Ingress networking.k8s.io/v1beta1 Yes
IngressClass networking.k8s.io/v1beta1 No
Lease coordination.k8s.io/v1beta1 No
LocalSubjectAccessReview authorization.k8s.io/v1beta1 Yes
MutatingWebhookConfiguration admissionregistration.k8s.io/v1beta1 Yes
PriorityClass scheduling.k8s.io/v1beta1 No
Role rbac.authorization.k8s.io/v1beta1 No
RoleBinding rbac.authorization.k8s.io/v1beta1 No
SelfSubjectAccessReview authorization.k8s.io/v1beta1 Yes
StorageClass storage.k8s.io/v1beta1 No
SubjectAccessReview authorization.k8s.io/v1beta1 Yes
TokenReview authentication.k8s.io/v1beta1 No
ValidatingWebhookConfiguration admissionregistration.k8s.io/v1beta1 Yes
VolumeAttachment storage.k8s.io/v1beta1 No

Evaluating your cluster for removed APIs

There are several methods to help administrators identify where APIs that will be removed are in use. However, OpenShift Container Platform cannot identify all instances, especially workloads that are idle or external tools that are used. It is the responsibility of the administrator to properly evaluate all workloads and other integrations for instances of removed APIs.

Reviewing alerts to identify uses of APIs that will be removed

OpenShift Container Platform 4.8 introduced two new alerts that fire when an API is in use that will be removed in the next release:

  • APIRemovedInNextReleaseInUse - for APIs that will be removed in the next OpenShift Container Platform release.
  • APIRemovedInNextEUSReleaseInUse - for APIs that will be removed in the next OpenShift Container Platform Extended Update Support (EUS) release.

If either of these alerts are firing in your cluster, review the alerts and take action to clear the alerts by migrating manifests and API clients to use the new API version. You can use the APIRequestCount API to get more information about which APIs are in use and which workloads are using APIs that will be removed. Additionally, some APIs might not trigger the above alerts, yet may still be captured by APIRequestCount.

Using APIRequestCount to identify uses of APIs that will be removed

You can use the APIRequestCount API to track API requests and review whether any of them are using one of the removed APIs.

NOTE: You must have access to the cluster as a user with the cluster-admin role in order to use use the APIRequestCount API.

Run the following command and examine the REMOVEDINRELEASE column of the output to identify APIs that will be removed in a future release but are currently in use:

$ oc get apirequestcounts

Example output:

NAME                                        REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
cloudcredentials.v1.operator.openshift.io                      32                      111
ingresses.v1.networking.k8s.io                                 28                      110
ingresses.v1beta1.extensions                1.22               16                      66
ingresses.v1beta1.networking.k8s.io         1.22               0                       1
installplans.v1alpha1.operators.coreos.com                     93                      167
...

You can also use -o jsonpath to filter the results:

$ oc get apirequestcounts -o jsonpath='{range .items[?(@.status.removedInRelease!="")]}{.status.removedInRelease}{"\t"}{.metadata.name}{"\n"}{end}'

Example output:

1.22    certificatesigningrequests.v1beta1.certificates.k8s.io
1.22    ingresses.v1beta1.extensions
1.22    ingresses.v1beta1.networking.k8s.io

Using APIRequestCount to identify which workloads are using the APIs that will be removed

You can examine the APIRequestCount resource for a given API version to help identify which workloads are using the API.

NOTE: You must have access to the cluster as a user with the cluster-admin role in order to use use the APIRequestCount API.

Run the following command and examine the username and userAgent fields to help identify the workloads that are using the API:

$ oc get apirequestcounts <resource>.<version>.<group> -o yaml

For example:

$ oc get apirequestcounts ingresses.v1beta1.networking.k8s.io -o yaml

You can also use -o jsonpath to extract the username values from an APIRequestCount resource:

$ oc get apirequestcounts ingresses.v1beta1.networking.k8s.io -o jsonpath='{range ..username}{$}{"\n"}{end}' | sort | uniq

Example output:

user1
user2
app:serviceaccount:delta

Migrating instances of removed APIs

For more information on migrating removed Kubernetes APIs, see the Deprecated API Migration Guide in the Kubernetes documentation.

Providing the administrator acknowledgment

After you have evaluated your cluster for any removed APIs and have migrated any removed APIs, you can acknowledge that your cluster is ready to upgrade to OpenShift Container Platform 4.9.

WARNING: Be aware that all responsibility falls on the administrator to ensure that all uses of removed APIs have been resolved and migrated as necessary before providing this administrator acknowledgment. OpenShift Container Platform can assist with the evaluation, but cannot identify all possible uses of removed APIs, especially idle workloads or external tools.

To acknowledge that you have completed the evaluation and your cluster is ready to upgrade to OpenShift Container Platform 4.9, run the following command:

$ oc -n openshift-config patch cm admin-acks --patch '{"data":{"ack-4.8-kube-1.22-api-removals-in-4.9":"true"}}' --type=merge

NOTE: You must have access to the cluster as a user with the cluster-admin role in order to run this command.

Your cluster is now ready to be upgraded to OpenShift Container Platform 4.9, once it is available.

3 Comments

Had a look at this after upgrading to 4.8 and in all but a few cases they are used by openshift-<something> and in one case kube-controller-manager. So regarding this "WARNING", why am I expected to take responsibility to ensure that the usage of these removed APIs have been resolved for cluster specific functionality? I would be nice if Red Hat can state that they will take care of their stuff and not have me to acknowledge that and subsequently assume responsibility.

Thanks for the feedback.

The counters currently look back over the past 24hrs which would include data before the upgrade had completed. We'll look at improving this such that the counters are reset when the 4.8 upgrade completes. If after 24hrs the counters are still showing activity can you please file a case and attach a must-gather so that we can look into the specifics of why your cluster is triggering these counters?

We'll look at improving this such that the counters are reset when the 4.8 upgrade completes.

is this already implemented/active in 4.8.z?