OpenShift pods failing to start with input/output error for overlay mount.

Solution Unverified - Updated -

Issue

  • OpenShift Pods fail to start returning input/output errors for overlay mounts:
kubenswrapper[6201]: E1202 05:05:36.086157 6201 remote_runtime.go:176] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to mount container k8s_POD_redhat-marketplace-tb68b_openshift-marketplace_5f499bfa-c979-40fc-9c99-090953949a8b_0 in pod sandbox k8s_redhat-marketplace-tb68b_openshift-marketplace_5f499bfa-979-40fc-9c99-090953949a8b_0(0bfc8ae01878ed1670c56f0e19f7a1051424444f0d0f1300d319619bcc6c5693): creating overlay mount to /var/lib/containers/storage/overlay/5c8b87c00d507e4fa1ff66e8f20201bc3677280d43b6e1d5387e4004f6a835e/merged, mount_data=\"lowerdir=/var/lib/containers/storage/overlay/l/I7PBCDCIAVKI52AKRX237EV7NL:/var/lib/containers/storage/overlay/l/LTK3TXU7X4GKRYJZQQ25HKQQH3:/var/lib/containers/storage/overlay/l/CKYQRATSEFXANYVP4AV262KIY5:/var/lib/containers/storage/overlay/l/XHKNB5OCJ45VIABZQLQYEKYNAT:/var/lib/containers/storage/overlay/l/JW36VSNS6LERG7EQLE5A7A2AER,upperdir=/var/lib/containers/storage/overlay/d5c8b87c00d507e4fa1ff66e8f20201bc3677280d43b6e1d5387e4004f6a835e/diff,workdir=/var/lib/containers/storage/overlay/d5c8b87c00d507e4fa1ff66e8f20201bc3677280d43b6e1d5387e4004f6a835e/work,,volatile,context=\\\"system_u:object_r:container_file_t:s0:c14,c9\\\"\": input/output error
  • The dmesg output shows overlayfs errors:
kernel: overlayfs: Cannot mount volatile when upperdir has an unseen error. Sync upperdir fs to clear state.

Environment

  • Red Hat OpenShift Container Platform (RHOCP) 4
  • Red Hat Enterprise Linux CoreOS (RHCOS) 4

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content