Filesystem corruption on an Elasticsearch pod on OpenShift 4.x
Issue
-
An Elasticsearch instance is stuck on
CreateContainerError
with the following error in the OpenShift events in theopenshift-logging
namespace:Error: relabel failed /var/lib/kubelet/pods/374138d9-a02e-4470-91a8-b63653c7ccc4/volumes/kubernetes.io~csi/pvc-76158b52-88f2-43ed-ae9b-8bf4175a1bfd/mount: lstat /var/lib/kubelet/pods/374138d9-a02e-4470-91a8-b63653c7ccc4/volumes/kubernetes.io~csi/pvc-76158b52-88f2-43ed-ae9b-8bf4175a1bfd/mount/elasticsearch/data/nodes/0/indices/OcBsYffpRBepn2EDbTrh_A/_state/state-545.st: bad message
-
This error comes from the OpenShift node kubelet logs:
Jun 17 12:15:02 node01.example.com hyperkube[1541]: E0617 12:15:02.200829 1541 remote_runtime.go:200] CreateContainer in sandbox "50f97a52935c10c919033166e3e3459fa6f2ccb6998839b084f122af89db7c47" from runtime service failed: rpc error: code = Unknown desc = relabel failed /var/lib/kubelet/pods/c8f949ed-0bcd-432c-9ebf-7df762b5ea7e/volumes/kubernetes.io~csi/pvc-76158b52-88f2-43ed-ae9b-8bf4175a1bfd/mount: lstat /var/lib/kubelet/pods/c8f949ed-0bcd-432c-9ebf-7df762b5ea7e/volumes/kubernetes.io~csi/pvc-76158b52-88f2-43ed-ae9b-8bf4175a1bfd/mount/elasticsearch/data/nodes/0/indices/OcBsYffpRBepn2EDbTrh_A/_state/state-545.st: bad message
Environment
- Red Hat Openshift Container Platform (RHOCP) 4.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.