Pod is in Pending state with Warning "No preemption victims found for incoming pod" in RHOCP 4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform 4

Issue

  • The Pod fails with below warning in events:
3h41m       Warning   FailedScheduling                 pod/abc-deploy              0/11 nodes are available: 2 node(s) had untolerated taint {node-role.kubernetes.io/infra-router: }, 3 Too many pods, 3 node(s) had untolerated taint {node-role.kubernetes.io/infra: }, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/11 nodes are available: 3 No preemption victims found for incoming pod, 8 Preemption is not helpful for scheduling.

Resolution

  • Get the label associated with the static MachineConfigPool CRD for the type of node you want to configure by entering the following command:
$ oc edit machineconfigpool <name>

OR create a custom label for target MachineConfigPool.

$ oc label machineconfigpool <name> custom-kubelet=small-pods
  • Increase the maxPods per node on the worker nodes by creating a custom resource file that contains the below kubelet configuration.
$ vi custom-kubelet-config.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: set-max-pods
spec:
  machineConfigPoolSelector:
    matchLabels:
      custom-kubelet: small-pods
  kubeletConfig:
    maxPods: 500   

$ oc create -f custom-kubelet-config.yaml

Root Cause

  • The default value for maxPods is 250.
  • This issue happened as the number of pods present on each worker node have been exceeded more than 250 pod's per node.

Diagnostic Steps

  • Check if any pods are in pending state.
$ oc get pods | grep 'Pending'
abc-deploy                  0/1     Pending     0          2h
pqr-deploy                  0/1     Pending     0          20m
  • Notice the events gives the below warning:
$ oc get events
3h41m       Warning   FailedScheduling                 pod/abc-deploy              0/11 nodes are available: 2 node(s) had untolerated taint {node-role.kubernetes.io/infra-router: }, 3 Too many pods, 3 node(s) had untolerated taint {node-role.kubernetes.io/infra: }, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/11 nodes are available: 3 No preemption victims found for incoming pod, 8 Preemption is not helpful for scheduling.
  • Check currently how many pods are scheduled on each worker node.
$ oc get pod -A -o wide | grep -i worker-abc | wc
250    2616   68750

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments