How to remove a specific worker node on ROSA ?

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS

Issue

  • How to resize a ROSA cluster as removing a specific worker node?
  • How to drain the running pods on a specific worker node to remove, before scaling-in on a ROSA cluster?

Resolution

  • Unfortunately, a specific worker node cannot be removed. When scaling the cluster, decreasing the worker replicas count, a random worker node would be removed.
  • Instead of it, the worker node would be removed after all running pods are drained safely.
  • ROSA worker node manages worker nodes using MachinePool which depends on MachineSet and Machine resources on the backend. Refer What happens when I delete a Machine? for more details.
What happens a Machine is deleted?
  This section assumes that a Machine is deleted from the API, rather than deleting a VM or Instance in the cloud provider.

  When a Machine is marked deleted, the associated Node is immediately cordoned and drained. 
  Draining utilizes the eviction API, so PodDisruptionBudgets will be respected. 
  If a Node cannot be successfully drained due to a PDB, the operation will retry indefinitely.

  After the associated Node is drained, the associated instance or VM is deleted from the cloud provider. 
  After the instance is deleted in the cloud provider, the associated Node object is deleted from the API.

  Finally, the finalizer is removed from the Machine object and the Machine object is removed from the API.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments