Navigating Kubernetes API deprecations and removals

Updated -

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Kubernetes follows a fairly strict API versioning policy; resulting in a number of API deprecation for v1beta1 and v2beta1 APIs happening over several releases. Beta API versions must be supported for 9 months or 3 releases (whichever is longer) after deprecation, according to the policy, at which point they can be removed.

Places where APIs that have been removed are still in use by workloads, tools, or other components running on or interacting with the cluster will begin to fail. Therefore, it is important for users/administrators must evaluate their cluster for any APIs in use that will be removed and migrate the affected components to use the appropriate new API version.

After this is done, the administrator can provide the administrator acknowledgment and proceed with updates where API removals are happening.

Migrating applications using removed APIs

For more information on migrating from removed Kubernetes APIs, see the following:

Evaluating your cluster for removed APIs

There are several methods to help administrators identify where APIs that will be removed are in use. However, OpenShift Container Platform cannot identify all instances, especially workloads that are idle or external tools that are used. It is the responsibility of the administrator to properly evaluate all workloads and other integrations for instances of removed APIs.

Reviewing alerts to identify uses of APIs that will be removed

OpenShift Container Platform provides two alerts that fire when an API is in use that will be removed in the next release:

  • APIRemovedInNextReleaseInUse - for APIs that will be removed in the next OpenShift Container Platform release.
  • APIRemovedInNextEUSReleaseInUse - for APIs that will be removed in the next OpenShift Container Platform Extended Update Support (EUS) release.

If either of these alerts are firing in your cluster, review the alerts and take action to clear the alerts by migrating manifests and API clients to use the new API version.

These alerts, meant for the overall monitoring of the cluster, do not provide information that helps determine which workloads are using the APIs that will be removed. Additionally, these alerts are tuned to not be overly sensitive so as to avoid causing alerting fatigue on a production system. You can use the APIRequestCount API to get more information about which APIs are in use and which workloads are using APIs that will be removed. Additionally, some APIs might not trigger the above alerts, yet may still be captured by APIRequestCount.

Using APIRequestCount to identify uses of APIs that will be removed

You can use the APIRequestCount API to track API requests and review whether any of them are using one of the removed APIs.

IMPORTANT: There was a bug in previous OCP 4.8 releases affecting the APIRequestCount (BZ 2021629), already fixed in OCP 4.8.25 by errata RHBA-2021:5209. The recommendation is to upgrade to a newer OCP 4.8.z release before checking the APIRequestCount.

Note: You must have access to the cluster as a user with the cluster-admin role in order to use use the APIRequestCount API.

Run the following command and examine the REMOVEDINRELEASE column of the output to identify APIs that will be removed in a future release but are currently in use:

$ oc get apirequestcounts

Example output:

NAME                                        REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
cloudcredentials.v1.operator.openshift.io                      32                      111
ingresses.v1.networking.k8s.io                                 28                      110
ingresses.v1beta1.extensions                1.22               16                      66
ingresses.v1beta1.networking.k8s.io         1.22               0                       1
installplans.v1alpha1.operators.coreos.com                     93                      167
...

You can also use -o jsonpath to filter the results:

$ oc get apirequestcounts -o jsonpath='{range .items[?(@.status.removedInRelease!="")]}{.status.removedInRelease}{"\t"}{.status.requestCount}{"\t"}{.metadata.name}{"\n"}{end}'

Example output:

1.22    11      certificatesigningrequests.v1beta1.certificates.k8s.io
1.22    356     ingresses.v1beta1.extensions
1.22    0       ingresses.v1beta1.networking.k8s.io

Using APIRequestCount to identify which workloads are using the APIs that will be removed

You can examine the APIRequestCount resource for a given API version to help identify which workloads are using the API.

IMPORTANT: There was a bug in previous OCP 4.8 releases affecting the APIRequestCount (BZ 2021629), already fixed in OCP 4.8.25 by errata RHBA-2021:5209. The recommendation is to upgrade to a newer OCP 4.8.z release before checking the APIRequestCount.

Note: You must have access to the cluster as a user with the cluster-admin role in order to use use the APIRequestCount API.

Run the following command and examine the username and userAgent fields to help identify the workloads that are using the API:

$ oc get apirequestcounts <resource>.<version>.<group> -o yaml

For example:

$ oc get apirequestcounts ingresses.v1beta1.networking.k8s.io -o yaml

You can also use -o jsonpath to extract the username and userAgent values from an APIRequestCount resource:

$ oc get apirequestcounts ingresses.v1beta1.networking.k8s.io \
  -o jsonpath='{range .status.last24h..byUser[*]}{..byVerb[*].verb}{","}{.username}{","}{.userAgent}{"\n"}{end}' \
  | sort -u | column -t -s ","

Note: change the ingresses.v1beta1.networking.k8s.io in the command with the name of each deprecated APIs still used.
Note: above command shows if the selected API was used in the last 24 hours. If changes/upgrades were done in some of the applications to use the newer APIs, it's possible to change last24h to currentHour to see if the API is still used in the last hour.

Example output:

VERBS  USERNAME                        USERAGENT
watch  bob                             oc/v4.8.11
watch  system:kube-controller-manager  cluster-policy-controller/v0.0.0

IMPORTANT NOTE: You can safely ignore the following entries that appear in the results:

  • The system:serviceaccount:kube-system:generic-garbage-collector and system:serviceaccount:kube-system:namespace-controller users might appear in the results because they walk through all registered APIs searching for resources to remove.
  • The system:kube-controller-manager and system:cluster-policy-controller users might appear in the results because they walk through all resources while enforcing various policies.
  • If OpenShift GitOps is installed in the cluster, the system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller user (refer to KCS 6635361 for additional information).
  • If OpenShift Pipelines is installed in the cluster, the openshift-pipelines-operator userAgent (refer to KCS 6821411 for additional information).
  • In OSD and ROSA clusters, or if Velero is installed in the cluster, the system:serviceaccount:openshift-velero:velero and system:serviceaccount:openshift-oadp:velero users (refer to KCS 6351332 for additional information).
  • The system:serviceaccount:openshift-ovn-kubernetes:ovn-kubernetes-controller may appear as using endpointslices.v1beta1.discovery.k8s.io in 4.10, while this API is removed in 4.12. This is not a problem because OVN-Kubernetes switches to v1 endpointslices API in 4.11 and an EUS upgrade from 4.10 to 4.12 must have its control plane traverse 4.11.
  • The system:serviceaccount:openshift-network-operator may appear as using poddisruptionbudgets.v1beta1.policy in 4.10, while this API is removed in 4.12. This is not a problem because cluster-network-operator switches to v1 poddisruptionbudgets API in 4.11 and an EUS upgrade from 4.10 to 4.12 must have its control plane traverse 4.11.
  • The system:serviceaccount:openshift-monitoring:kube-state-metrics may appear as using horizontalpodautoscalers.v2beta2.autoscaling. This can safely ignored because all 4.12 versions switch to v2 API as per resolution of OCPBUGS-4112.
  • The system:serviceaccount:prod-rhsso:rhsso-operator may appear using poddisruptionbudgets.v1beta1.policy when trying to upgrade to 4.12. If the version of RHSSO is up-to-date, this can be safely ignored as explained in Red Hat Single Sign-On (RH SSO) Operator is using deprecated 'poddisruptionbudgets.v1beta1.policy' API.
  • The system:serviceaccount:openshift-kube-storage-version-migrator may appear using flowschemas.v1beta1.flowcontrol.apiserver.k8s.io when trying to upgrade to 4.13. This is safe to ignore because it is a serviceaccount used to migrate old versions in etcd. There is a BUG raised to Remove openshift-kube-storage-version-migrator flowschemas APIs from OCP 4.12 to OCP 4.13 upgrade.

Comments