Pods getting stuck in the pending state

Solution Verified - Updated -

Environment

  • Azure Red Hat OpenShift (ARO)
    • 4.10

Issue

  • Despite having enough of the resources on the nodes, pods are not able to schedule on to the node.
  • Namespace events showing below errors:
1h34m      Normal   NotTriggerScaleUp       pod/cp-kafka-dev-2                pod didn't trigger scale-up: 9 max cluster cpu, memory limit reached


Unknown    Warning  FailedScheduling        pod/cp-kafka-dev-2                0/16 nodes are available: 1 Insufficient cpu, 1 Insufficient memory, 1 node(s) had taint {sonarqube: true}, that the pod didn't tolerate, 1 node(s) were unschedulable, 3 node(s) had taint {kafka: true}, that the pod didn't tolerate, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 7 node(s) had volume node affinity conflict.
  • The clusterautoscaler configuration is provided with the below values:
resourceLimits:
    cores:
      max: 256   <===
      min: 4
    maxNodesTotal: 48
    memory:
      max: 1024  <===
      min: 4
  • The memory used across all the nodes is approximately 1232Gb.
oc get nodes -o json | jq '.items[].status.capacity.memory'
"32923624Ki"
"32923624Ki"
"32923624Ki"
"132008628Ki"
"132008624Ki"
"32923624Ki"
"132008624Ki"
"32923624Ki"
"132008628Ki"
"132008280Ki"
"32923624Ki"
"32923624Ki"
"132008628Ki"
"32923624Ki"
"132008280Ki"
"16409424Ki"
  • The CPU used across all the nodes is approximately 292.
oc get nodes -o json | jq '.items[].status.capacity.cpu'
"8"
"8"
"8"
"32"
"32"
"8"
"32"
"8"
"32"
"32"
"8"
"8"
"32"
"8"
"32"
"4"

Resolution

  • Check the CPU and Memory Utilization on all the nodes across the cluster.
  • Adjust the appropriate max and min values for the CPU and Memory in the clusterautoscaler configuration to avoid resource deficiency.
  • Additionally, make sure that the pods will be using appropriate requests and limits to avoid max resource utilization.

Root Cause

  • The cluster autoscaler adjusts the size of an OpenShift Container Platform cluster to meet its current deployment needs.
  • cores min/max means the minimum and maximum number of cores in the cluster. The Cluster autoscaler will not scale the cluster beyond these numbers. (default "0:320000")
  • memory min/max means the minimum and maximum number of gigabytes of memory in the cluster, in the format <min>:<max>. The cluster autoscaler will not scale the cluster beyond these numbers. (default "0:6400000")
  • Even though the maxNodesTotal: 48 value is set and the cluster has only 16 Nodes in total, it will not scale further since the memory/cpu allocated in the clusterautoscaler definition has crossed its limit.

Diagnostic Steps

  • Check the total memory utilization on the nodes:
    oc get nodes -o json | jq '.items[].status.capacity.memory'
  • Check the total cpu utilization on the nodes:
    oc get nodes -o json | jq '.items[].status.capacity.cpu'
  • Check the clusterautoscaler min and max values:
resourceLimits:
    cores:
      max: 256
      min: 4
    maxNodesTotal: 48
    memory:
      max: 1024
      min: 4

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments