How to clean CRI-O storage in Red Hat OpenShift 4

Issue

• Note: This should only be used if CRI-O is not able to create new workloads. It should not be a catch-all remediation for delays in pod startup.
  • A case this should especially not be used is when encountering errors of the form Error reserving ctr name %s for id %s: name is reserved. These errors happen because of a bottleneck on the node. Removing and recreating all of the pods is only likely to exacerbate the issue.
• How to wipe CRI-O ephemeral storage?
• The kubelet service is restarting continuously on a node.

• A node can't run any pods with the container or CRI-O errors, including but not limited to the following ones:

  • Failing pod sandboxes due to container-related errors:

    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container XXX: error recreating the missing symlinks: error reading name of symlink for XXX: open /var/lib/containers/storage/overlay/XXX/link: no such file or directory
    

  • Container creation is failing with the can't stat lower layer error

    can't stat lower layer ...  because it does not exist.  Going through storage to recreate the missing symlinks.
    

  • CRI-O failures due to container storage issues:

    Failed to remove storage directory: unlinkat /var/lib/containers/storage/overlay-containers/586f92c81c4576e97be1091af010449e04feedaed726d28b8eb840ff87ec10c8/userdata/shm: device or resource busy
    

  • CRI-O is continuously killed by a SIG ABRT, generating a stack trace that may look similar to the example in "diagnostic steps".

  • Failing to pull images due to error committing the images:

    Failed to pull image "registry.redhat.io/ocs4/cephcsi-rhel8@sha256:bf274db28eb745135995c598c7c938509b1a22e4d526956c66eef4416af919c6": rpc error: code = Unknown desc = Error committing the finished image: error adding layer with blob "sha256:2c2be27b4878555a6af9e39d5494444d6152e2a94aafff5d7132c515c373dcf5": error creating layer with ID "a5a1f55d627a5893af066519c6113c122becb8627cb9aca8e299138e406a46e1": Stat /var/lib/containers/storage/overlay/0c51fae87c95e09f4dfccee417713913004c97821d72473d07faeba9c4132761: no such file or directory