Pods are unable to start due to missing backingFsBlockDev file in ROSA/ARO
Environment
- Red Hat OpenShift Services on AWS (ROSA)
- Azure Red Hat OpenShift (ARO)
- OpenShift Dedicated (OSD)
Issue
- Pods on nodes stuck in pending state and below event is streamed:
Warning FailedCreatePodSandBox pod/podname (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_podname_namespace_32148648-d572-4622-b3f0-2111417e7a18_0": error creating read-write layer with ID "67473e77e49fe090bf14ed9ded93def90591b24e73448b9436a45df7f756e34b": Failed to set quota limit for projid 31409 on /var/lib/containers/storage/overlay/backingFsBlockDev: no such file or directory
Resolution
Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.
This issue has been fixed and the patch is included by default in RHOCP 4.12 and above and was backported in the following RHOCP versions:
| Version | Related BZ/Jira | Errata | Crio Version |
|---|---|---|---|
| 4.11.19 | Jira OCPBUGS-4470 | Errata RHBA-2022:8892 | 1.24.3-7 |
| 4.10.46 | Jira OCPBUGS-4471 | Errata RHSA-2022:9098 | 1.23.4-3 |
| 4.9.54 | Jira OCPBUGS-4472 | Errata RHSA-2022:9110 | 1.22.5-16 |
| 4.8.57 | Jira OCPBUGS-1230 | Errata RHBA-2023:0236 | 1.21.8-11 |
IMPORTANT: As the fix is already included in newer OCP releases, the recommendation is to upgrade the cluster or open a case with Red Hat for a workaround to be applied to the cluster. Please Note The workaround is only a temporary solution to enable cluster to upgrade and the issue might re-surface if the cluster is not on the bug-fix version.
Root Cause
The underline root cause of this issue is currently under investigation.
The Packageserver Operator has been identified as a common operator running on all impacted environment.
There is an internal Jira ticket OCPBUGS-13403 to review it.
Diagnostic Steps
Check events generated for the pod which got stuck in pending state:
$ oc get events -n [project_name] | grep [pod_name]
Check for backingFsBlockDev in the journalctl from the node logs:
$ journalctl -u kubelet | grep backingFsBlockDev | head
or in a SOSREPORT:
$ grep "backingFsBlockDev" ${SOSREPORT_PATH}/sos_commands/openshift/journalctl_--no-pager_--unit_kubelet | head
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments