OpenShift pods failing to start with input/output error for overlay mount.
Issue
- OpenShift Pods fail to start returning input/output errors for overlay mounts:
kubenswrapper[6201]: E1202 05:05:36.086157 6201 remote_runtime.go:176] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to mount container k8s_POD_redhat-marketplace-tb68b_openshift-marketplace_5f499bfa-c979-40fc-9c99-090953949a8b_0 in pod sandbox k8s_redhat-marketplace-tb68b_openshift-marketplace_5f499bfa-979-40fc-9c99-090953949a8b_0(0bfc8ae01878ed1670c56f0e19f7a1051424444f0d0f1300d319619bcc6c5693): creating overlay mount to /var/lib/containers/storage/overlay/5c8b87c00d507e4fa1ff66e8f20201bc3677280d43b6e1d5387e4004f6a835e/merged, mount_data=\"lowerdir=/var/lib/containers/storage/overlay/l/I7PBCDCIAVKI52AKRX237EV7NL:/var/lib/containers/storage/overlay/l/LTK3TXU7X4GKRYJZQQ25HKQQH3:/var/lib/containers/storage/overlay/l/CKYQRATSEFXANYVP4AV262KIY5:/var/lib/containers/storage/overlay/l/XHKNB5OCJ45VIABZQLQYEKYNAT:/var/lib/containers/storage/overlay/l/JW36VSNS6LERG7EQLE5A7A2AER,upperdir=/var/lib/containers/storage/overlay/d5c8b87c00d507e4fa1ff66e8f20201bc3677280d43b6e1d5387e4004f6a835e/diff,workdir=/var/lib/containers/storage/overlay/d5c8b87c00d507e4fa1ff66e8f20201bc3677280d43b6e1d5387e4004f6a835e/work,,volatile,context=\\\"system_u:object_r:container_file_t:s0:c14,c9\\\"\": input/output error
- The
dmesgoutput shows overlayfs errors:
kernel: overlayfs: Cannot mount volatile when upperdir has an unseen error. Sync upperdir fs to clear state.
Environment
- Red Hat OpenShift Container Platform (RHOCP) 4
- Red Hat Enterprise Linux CoreOS (RHCOS) 4
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.