Pod creation stuck in INIT status randomly

Issue

• Pod creation stuck in INIT status randomly

$ oc get pods
NAME                                 READY   STATUS     RESTARTS   AGE
hello-openshift-0-7d59599c8b-q7b52   2/2     Running    0          5h2m
hello-openshift-1-7d59599c8b-6zdv7   2/2     Running    0          5h2m
hello-openshift-2-7d59599c8b-r8z2c   2/2     Running    0          5h2m
hello-openshift-3-7d59599c8b-5rccg   2/2     Running    0          5h2m
hello-openshift-4-7d59599c8b-d5xvn   2/2     Running    0          5h2m
hello-openshift-5-7d59599c8b-8hrmc   2/2     Running    0          5h2m
hello-openshift-6-7d59599c8b-6mv5h   2/2     Running    0          5h2m
hello-openshift-7-7d59599c8b-gghdz   2/2     Running    0          5h2m
hello-openshift-8-7d59599c8b-zjrwg   2/2     Running    0          5h2m
hello-openshift-9-7d59599c8b-w4z7w   0/2     Init:1/4   0          5h2m

• The affected Pod's has initContainer and its initContainer is in Unknown status

$ crictl ps -a | grep Unknown
83937abf1f953       8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a                                                             5 hours ago         Unknown             config-init-2                                 0                   a7a536a2f76c3

• Journal log shows CreatedAt is not set error.

$ journalctl -f | grep 83937abf1f953
Jul 28 23:10:18 dell-per730-08.gsslab.pek2.redhat.com hyperkube[658981]: E0728 23:10:18.040501  658981 remote_runtime.go:633] "UpdateContainerResources from runtime service failed" err="rpc error: code = Unknown desc = container 83937abf1f953467ceff9b48e62ed92f85e260ef3345b20b614bc951860a53bc is not running or created state: paused" containerID="83937abf1f953467ceff9b48e62ed92f85e260ef3345b20b614bc951860a53bc"
Jul 28 23:10:18 dell-per730-08.gsslab.pek2.redhat.com hyperkube[658981]: E0728 23:10:18.040578  658981 cpu_manager.go:470] "ReconcileState: failed to update container" err="rpc error: code = Unknown desc = container 83937abf1f953467ceff9b48e62ed92f85e260ef3345b20b614bc951860a53bc is not running or created state: paused" pod="test-vdu/hello-openshift-9-7d59599c8b-w4z7w" containerName="config-init-2" containerID="83937abf1f953467ceff9b48e62ed92f85e260ef3345b20b614bc951860a53bc" cpuSet="0-1,3-31"
Jul 28 23:10:18 dell-per730-08.gsslab.pek2.redhat.com hyperkube[658981]: E0728 23:10:18.756449  658981 remote_runtime.go:607] "verify ContainerStatus failed" err="status.CreatedAt is not set" containerID="83937abf1f953467ceff9b48e62ed92f85e260ef3345b20b614bc951860a53bc"
Jul 28 23:10:18 dell-per730-08.gsslab.pek2.redhat.com hyperkube[658981]: E0728 23:10:18.758109  658981 remote_runtime.go:607] "verify ContainerStatus failed" err="status.CreatedAt is not set" containerID="83937abf1f953467ceff9b48e62ed92f85e260ef3345b20b614bc951860a53bc"
Jul 28 23:10:19 dell-per730-08.gsslab.pek2.redhat.com hyperkube[658981]: E0728 23:10:19.765505  658981 remote_runtime.go:607] "verify ContainerStatus failed" err="status.CreatedAt is not set" containerID="83937abf1f953467ceff9b48e62ed92f85e260ef3345b20b614bc951860a53bc"
Jul 28 23:10:19 dell-per730-08.gsslab.pek2.redhat.com hyperkube[658981]: E0728 23:10:19.767410  658981 remote_runtime.go:607] "verify ContainerStatus failed" err="status.CreatedAt is not set" containerID="83937abf1f953467ceff9b48e62ed92f85e260ef3345b20b614bc951860a53bc"
Jul 28 23:10:20 dell-per730-08.gsslab.pek2.redhat.com hyperkube[658981]: E0728 23:10:20.775925  658981 remote_runtime.go:607] "verify ContainerStatus failed" err="status.CreatedAt is not set" containerID="83937abf1f953467ceff9b48e62ed92f85e260ef3345b20b614bc951860a53bc"
Jul 28 23:10:20 dell-per730-08.gsslab.pek2.redhat.com hyperkube[658981]: E0728 23:10:20.777378  658981 remote_runtime.go:607] "verify ContainerStatus failed" err="status.CreatedAt is not set" containerID="83937abf1f953467ceff9b48e62ed92f85e260ef3345b20b614bc951860a53bc"

• This issue could cause Released status PVs and stale pod sandbox remain on the node forever until crio is restarted.

$ oc get pv | grep Released
$ crictl pods