Pods stuck in Pending state with error ' X node(s) didn't match the node selector'

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.x

Issue

  • Fluentd pods stuck in Pending state.
fluentd-2nbcv                                 0/1    Pending  0         10m
fluentd-68mdv                                 1/1    Running  0         9m50s
fluentd-7ll2m                                 0/1    Pending  0         10m
fluentd-7pbsb                                 0/1    Pending  0         10m
fluentd-867nf                                 1/1    Running  0         9m48s
fluentd-g8769                                 0/1    Pending  0         10m
fluentd-j6lps                                 0/1    Pending  0         10m
fluentd-js5cb                                 0/1    Pending  0         10m
fluentd-l22w8                                 1/1    Running  0         9m57s
fluentd-srgxn                                 0/1    Pending  0         10m
  • Event logs in the openshift-logging project showing below error messages.
 0/X nodes are available: X node(s) didn't match the node selector.

Resolution

  • Check the default scheduler configuration.
$ oc get scheduler cluster -o yaml
  • Check if the defaultNodeSelector is configured.
  • Remove the defaultNodeSelector from default scheduler.
$ oc edit scheduler cluster

  • kubeapiserver will restart to apply the changes. Once it is applied, all the pods stuck in Pending will start Running.

  • Once the kubeapiserver pods are restarted. Delete the pods that are in thePending` state.

$ oc delete pod <podinpendingstate>

Root Cause

  • If defaultNodeSelector is configured then the scheduler stops pods from scheduling on any other node. Only the node which matches defaultNodeSelector is available for scheduling new pods.

Diagnostic Steps

  • Check the default scheduler configuration.
$ oc get scheduler cluster -o yaml

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments