OpenShift pods stay in Pending, then roll over to Evicted

Solution In Progress - Updated -

Environment

  • OpenShift Container Platform

Issue

  • Pod deployment is failing with FailedScheduling Insufficient memory and/or Insufficient cpu.
  • Pods are shown as Evicted.

Resolution

First, check the pod limits:

# oc describe pod <evicted-pod-id>

    Limits:
      cpu:      2
      memory:   3Gi
    Requests:
      cpu:      1
      memory:   1Gi

Second, check to see if any limit/request configuration block exists in the project:

# oc edit project <projectname>

apiVersion: v1
kind: Project
metadata:
  annotations:
    openshift.io/description: Management Infrastructure
    openshift.io/display-name: ""
    openshift.io/sa.scc.mcs: s0:c8,c2
    openshift.io/sa.scc.supplemental-groups: 1000060000/10000
    openshift.io/sa.scc.uid-range: 1000060000/10000
  creationTimestamp: 2019-07-21T03:06:17Z
  name: management-infra
  resourceVersion: "1115"
  selfLink: /oapi/v1/projects/management-infra
  uid: 81a7271f-ab64-11e9-92e4-fa163e6256c5
spec:
  finalizers:
  - openshift.io/origin
  - kubernetes
status:
  phase: Active

Lastly, check the kubeletArguments in the node-config.yaml file:

kubeletArguments:
  kube-reserved:
  - cpu=200m,memory=256Mi
  system-reserved:
  - cpu=200m,memory=256Mi

If it isn't present, add it, as this causes the following error:

kernel: java invoked oom-killer: gfp_mask=0x50, order=0, oom_score_adj=999

Keep in mind that changes to the node-config.yaml file will not take effect until the node service is restarted.

Root Cause

  • Evicted pods are usually due to a node not ready, or a lack of resources available.
  • Tight pod limits can cause pods to be Evicted.
  • Limits in the project deployment configuration can cause pods to be Evicted.
  • Missing kubeletArguments configuration can cause pods to be Evicted.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments