CSI-Plugin pods are not running on all OCP nodes

Solution Verified - Updated -

Environment

  • Openshift Container Storage 4.x

Issue

  • The CSI-Plugin pods are not running on all nodes in the OCP cluster.

Resolution

  • Verify that Configmap rook-ceph-operator-config is having toleration of the nodes
   # oc describe cm rook-ceph-operator-config -n openshift-storage

  • If the toleration is not present add the taints by following doc.

  • If it's not present describe the openshift-storage namespace for 'node-selector'.

 # oc describe ns openshift-storage

Name:         openshift-storage
Labels:       kubernetes.io/metadata.name=openshift-storage
              openshift.io/cluster-monitoring=true
Annotations:  openshift.io/node-selector: cluster.ocs.openshift.io/openshift-storage=
  • Override the cluster-wide default node selector from openshift-storage namespace:
   # oc annotate namespace openshift-storage openshift.io/node-selector=  --overwrite

Root Cause

  • The CSI-plugin pods are not able to tolerate the taints on other nodes.

  • The node-selector on the namespace blocks the plugin pods to be scheduled on the non-OCS nodes.

Diagnostic Steps

  • Verify if CSI-plugin plugin pods are running on all expected nodes.
 # oc get pods -n openshift-storage -o wide |grep csi
  • Verify the Configmap for the tolerations:
  # oc describe cm rook-ceph-operator-config -n openshift-storage
  • Verify the node-selector on openshift-storage namespace:
  # oc describe project openshift-storage

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments