container failing with "Error: readlink : no such file or directory" in OpenShift 4 disconnected

Solution Verified - Updated -

Issue

On a disconnected OpenShift environment with no external registry, CRI-O will fail coming up because it wipes and pulls all the container images. That is a feature in the case of a non-gracefully shutdown of the cluster when the storage could have gotten corrupted. With this configuration, the wipe can be disabled and the images won't be wiped but in the case of an image corruption, manual intervention would be required.

  • Crictl messages:
# crictl ps -a
time="2021-08-01T01:50:58Z" level=fatal msg="connect: connect endpoint 'unix:///var/run/crio/crio.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded"

# crictl pods
time="2021-08-01T01:51:00Z" level=fatal msg="connect: connect endpoint 'unix:///var/run/crio/crio.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded"
  • Podman also fails wit errors like:
Error: readlink /var/lib/containers/storage/overlay/l/DM... : no such file or directory

Environment

  • Red Hat OpenShift Container Platform 4.9
    • Disconnected, without external registry

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content