Labels and taints are not removed from nodes when removed from MachinePool in OSD/ROSA

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA)
    • 4
  • Red Hat OpenShift Dedicated (OSD)
    *
  • OpenShift Cluster Manager (OCM) web console
  • rosa CLI
  • MachinePool

Issue

  • After removing labels/taints from a MachinePool via OCM web console or rosa CLI, the labels/taints are not removed from the nodes of that MachinePool.

Resolution

Red Hat is aware of this issue, and there is an internal task to allow the deletion of labels/taints.

Root Cause

Previous change in the code to manage how the labels and taints are synchronized from the MachinePools to the Nodes is preventing the labels/taints to be removed.

Diagnostic Steps

The following example is using labels, but the same behavior is shown for taints.

  1. Create labels/taints from the OCM web console or via rosa CLI:

    $ rosa edit machinepool --cluster [cluster_name] --labels="custom-lable=true" [machinepool_name]

  2. Wait for some time after the label is reconciled to the Nodes:

    $ oc get nodes --show-labels
[...]
worker-0.example.com   Ready    worker                 1d20h   v1.25.11+1485cc9   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,host-files=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker-0.example.com,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhcos,custom-lable=true
[...]

  3. Delete the labels from the OCM web console or with rosa CLI:

    $ rosa edit machinepool --cluster [cluster_name] --labels="" [machinepool_name]

  4. After waiting for a long time, the deleted labels are still present:

    $ oc get nodes --show-labels
[...]
worker-0.example.com   Ready    worker                 1d20h   v1.25.11+1485cc9   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,host-files=true,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker-0.example.com,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhcos,custom-lable=true
[...]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments