Chapter 13. Node maintenance

13.1. About node maintenance

13.1.1. About node maintenance mode

Nodes can be placed into maintenance mode using the oc adm utility, or using NodeMaintenance custom resources (CRs).

Note

The node-maintenance-operator (NMO) is no longer shipped with OpenShift Virtualization. It is now available to deploy as a standalone Operator from the OperatorHub in the OpenShift Container Platform web console, or by using the OpenShift CLI (oc).

Placing a node into maintenance marks the node as unschedulable and drains all the virtual machines and pods from it. Virtual machine instances that have a LiveMigrate eviction strategy are live migrated to another node without loss of service. This eviction strategy is configured by default in virtual machine created from common templates but must be configured manually for custom virtual machines.

Virtual machine instances without an eviction strategy are shut down. Virtual machines with a RunStrategy of Running or RerunOnFailure are recreated on another node. Virtual machines with a RunStrategy of Manual are not automatically restarted.

Important

Virtual machines must have a persistent volume claim (PVC) with a shared ReadWriteMany (RWX) access mode to be live migrated.

The Node Maintenance Operator watches for new or deleted NodeMaintenance CRs. When a new NodeMaintenance CR is detected, no new workloads are scheduled and the node is cordoned off from the rest of the cluster. All pods that can be evicted are evicted from the node. When a NodeMaintenance CR is deleted, the node that is referenced in the CR is made available for new workloads.

Note

Using a NodeMaintenance CR for node maintenance tasks achieves the same results as the oc adm cordon and oc adm drain commands using standard OpenShift Container Platform custom resource processing.

13.1.2. Maintaining bare metal nodes

When you deploy OpenShift Container Platform on bare metal infrastructure, there are additional considerations that must be taken into account compared to deploying on cloud infrastructure. Unlike in cloud environments where the cluster nodes are considered ephemeral, re-provisioning a bare metal node requires significantly more time and effort for maintenance tasks.

When a bare metal node fails, for example, if a fatal kernel error happens or a NIC card hardware failure occurs, workloads on the failed node need to be restarted elsewhere else on the cluster while the problem node is repaired or replaced. Node maintenance mode allows cluster administrators to gracefully power down nodes, moving workloads to other parts of the cluster and ensuring workloads do not get interrupted. Detailed progress and node status details are provided during maintenance.

13.1.3. Additional resources

13.2. Automatic renewal of TLS certificates

All TLS certificates for OpenShift Virtualization components are renewed and rotated automatically. You are not required to refresh them manually.

13.2.1. TLS certificates automatic renewal schedules

TLS certificates are automatically deleted and replaced according to the following schedule:

  • KubeVirt certificates are renewed daily.
  • Containerized Data Importer controller (CDI) certificates are renewed every 15 days.
  • MAC pool certificates are renewed every year.

Automatic TLS certificate rotation does not disrupt any operations. For example, the following operations continue to function without any disruption:

  • Migrations
  • Image uploads
  • VNC and console connections

13.3. Managing node labeling for obsolete CPU models

You can schedule a virtual machine (VM) on a node as long as the VM CPU model and policy are supported by the node.

13.3.1. About node labeling for obsolete CPU models

The OpenShift Virtualization Operator uses a predefined list of obsolete CPU models to ensure that a node supports only valid CPU models for scheduled VMs.

By default, the following CPU models are eliminated from the list of labels generated for the node:

Example 13.1. Obsolete CPU models

"486"
Conroe
athlon
core2duo
coreduo
kvm32
kvm64
n270
pentium
pentium2
pentium3
pentiumpro
phenom
qemu32
qemu64

This predefined list is not visible in the HyperConverged CR. You cannot remove CPU models from this list, but you can add to the list by editing the spec.obsoleteCPUs.cpuModels field of the HyperConverged CR.

13.3.2. About node labeling for CPU features

Through the process of iteration, the base CPU features in the minimum CPU model are eliminated from the list of labels generated for the node.

For example:

  • An environment might have two supported CPU models: Penryn and Haswell.

  • If Penryn is specified as the CPU model for minCPU, each base CPU feature for Penryn is compared to the list of CPU features supported by Haswell.

    Example 13.2. CPU features supported by Penryn

    apic
clflush
cmov
cx16
cx8
de
fpu
fxsr
lahf_lm
lm
mca
mce
mmx
msr
mtrr
nx
pae
pat
pge
pni
pse
pse36
sep
sse
sse2
sse4.1
ssse3
syscall
tsc

    Example 13.3. CPU features supported by Haswell

    aes
apic
avx
avx2
bmi1
bmi2
clflush
cmov
cx16
cx8
de
erms
fma
fpu
fsgsbase
fxsr
hle
invpcid
lahf_lm
lm
mca
mce
mmx
movbe
msr
mtrr
nx
pae
pat
pcid
pclmuldq
pge
pni
popcnt
pse
pse36
rdtscp
rtm
sep
smep
sse
sse2
sse4.1
sse4.2
ssse3
syscall
tsc
tsc-deadline
x2apic
xsave

  • If both Penryn and Haswell support a specific CPU feature, a label is not created for that feature. Labels are generated for CPU features that are supported only by Haswell and not by Penryn.

    Example 13.4. Node labels created for CPU features after iteration

    aes
avx
avx2
bmi1
bmi2
erms
fma
fsgsbase
hle
invpcid
movbe
pcid
pclmuldq
popcnt
rdtscp
rtm
sse4.2
tsc-deadline
x2apic
xsave

13.3.3. Configuring obsolete CPU models

You can configure a list of obsolete CPU models by editing the HyperConverged custom resource (CR).

Procedure

  • Edit the HyperConverged custom resource, specifying the obsolete CPU models in the obsoleteCPUs array. For example: 

    apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  obsoleteCPUs:
    cpuModels: 1
      - "<obsolete_cpu_1>"
      - "<obsolete_cpu_2>"
    minCPUModel: "<minimum_cpu_model>" 2
    1
    Replace the example values in the cpuModels array with obsolete CPU models. Any value that you specify is added to a predefined list of obsolete CPU models. The predefined list is not visible in the CR.
    2
    Replace this value with the minimum CPU model that you want to use for basic CPU features. If you do not specify a value, Penryn is used by default.

13.4. Preventing node reconciliation

Use skip-node annotation to prevent the node-labeller from reconciling a node.

13.4.1. Using skip-node annotation

If you want the node-labeller to skip a node, annotate that node by using the oc CLI.

Prerequisites

  • You have installed the OpenShift CLI (oc).

Procedure

  • Annotate the node that you want to skip by running the following command: 

    $ oc annotate node <node_name> node-labeller.kubevirt.io/skip-node=true 1
    1
    Replace <node_name> with the name of the relevant node to skip.

    Reconciliation resumes on the next cycle after the node annotation is removed or set to false.

13.4.2. Additional resources