How to manually repair a gluster-block filesystem used by OpenShift
Issue
-
Need to manually repair the filesystem inside a gluster-block volume;
-
Elasticsearch pod has
CrashLoopBackOff
status. The pod log shows the error:main ERROR Unable to create file /elasticsearch/persistent/logging-es/logs/logging-es.log java.io.IOException: Could not create directory /elasticsearch/persistent/logging-es/logs
-
Pod has a Persistent Volume Claim using a gluster-block volume, but the pod recognizes the mount point as
tmpfs
(emptyDir
) instead of the proper gluster-block volume (/dev/mapper/mpathN
); -
The logs for service
atomic-openshift-node
shows:nestedpendingoperations.go:267] Operation for "\"kubernetes.io/iscsi/10.0.0.10:iqn.2016-12.org.gluster-block:b54c4cb0-074a-4c38-8ed3-76cb27a8bdbb:0\"" failed. No retries permitted until 2020-01-01 00:00:00.242007101 -0000 -00 m=+28779.167712731 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-4119b152-a57c-11e9-8a9c-0050569a7d6a\" (UniqueName: \"kubernetes.io/iscsi/10.0.0.10:iqn.2016-12.org.gluster-block:b54c4cb0-074a-4c38-8ed3-76cb27a8bdbb:0\") pod \"logging-es-data-master-q2t5c59s-9-sfz8f\" (UID: \"3d04e48c-3d0c-11ea-b6bd-0050569afe07\") "
Environment
- Red Hat OpenShift Container Platform (OCP) 3.x
- Red Hat OpenShift Container Storage 3.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.