container failing with "Error: readlink : no such file or directory" in OpenShift 4 disconnected
Issue
On a disconnected OpenShift environment with no external registry, CRI-O will fail coming up because it wipes and pulls all the container images. That is a feature in the case of a non-gracefully shutdown of the cluster when the storage could have gotten corrupted. With this configuration, the wipe can be disabled and the images won't be wiped but in the case of an image corruption, manual intervention would be required.
- Crictl messages:
# crictl ps -a
time="2021-08-01T01:50:58Z" level=fatal msg="connect: connect endpoint 'unix:///var/run/crio/crio.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded"
# crictl pods
time="2021-08-01T01:51:00Z" level=fatal msg="connect: connect endpoint 'unix:///var/run/crio/crio.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded"
- Podman also fails wit errors like:
Error: readlink /var/lib/containers/storage/overlay/l/DM... : no such file or directory
Environment
- Red Hat OpenShift Container Platform 4.9
- Disconnected, without external registry
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.