Pods in status "OutOfcpu" and Kubelet showing "OutOfpods" errors when many pods are created in a short time in the same node in OpenShift

Issue

• The issues below happen when many pods are scheduled at the same time into the same node although its hardware resources are more than enough to host them.

  • Status of some pods OutOfcpu:

    NAME                    READY   STATUS      RESTARTS   AGE
    out-of-cpu-test-424z2   0/1     Completed   0          14s
    out-of-cpu-test-6qpdv   0/1     Completed   0          9s
    out-of-cpu-test-6w92k   0/1     Completed   0          12s
    out-of-cpu-test-b2q9l   0/1     Completed   0          17s
    out-of-cpu-test-c2rgc   0/1     Completed   0          10s
    out-of-cpu-test-c5qzz   0/1     Completed   0          16s
    out-of-cpu-test-cbxln   0/1     Completed   0          11s
    out-of-cpu-test-chkcp   0/1     Completed   0          13s
    out-of-cpu-test-dbqct   0/1     Completed   0          8s
    out-of-cpu-test-dtpxv   0/1     Completed   0          11s
    out-of-cpu-test-ghjgx   0/1     Completed   0          16s
    out-of-cpu-test-mvttr   0/1     Completed   0          15s
    out-of-cpu-test-mwvrp   0/1     Completed   0          8s
    out-of-cpu-test-qb5rr   0/1     OutOfcpu    0          10s
    out-of-cpu-test-qsgxr   0/1     OutOfcpu    0          12s
    out-of-cpu-test-rvqxr   0/1     Completed   0          14s
    out-of-cpu-test-vgvnx   0/1     OutOfcpu    0          9s
    out-of-cpu-test-xfmwn   0/1     Completed   0          13s
    out-of-cpu-test-xxnct   0/1     Completed   0          15s
    out-of-cpu-test-zwzbd   0/1     Completed   0          7s
    

  • Kubelet returns OutOfpods errors:

    Warning  OutOfpods  71m   kubelet            Node didn't have enough resource: pods, requested: 1, used: 251, capacity: 250
    

  • Following message on Kubelet logs:

    Feb 08 15:26:48.999380 ip-10-0-146-92 hyperkube[808496]: I0208 15:26:48.999252  808496 predicate.go:144] "Predicate failed on Pod" pod="default/test667--1-jzvhm" err="Node didn't have enough resource: pods, requested: 1, used: 50, capacity: 50"