How to manually repair a gluster-block filesystem used by OpenShift

Solution Verified - Updated -

Issue

  • Need to manually repair the filesystem inside a gluster-block volume;

  • Elasticsearch pod has CrashLoopBackOff status. The pod log shows the error:

    main ERROR Unable to create file /elasticsearch/persistent/logging-es/logs/logging-es.log java.io.IOException: Could not create directory /elasticsearch/persistent/logging-es/logs
    
  • Pod has a Persistent Volume Claim using a gluster-block volume, but the pod recognizes the mount point as tmpfs (emptyDir) instead of the proper gluster-block volume (/dev/mapper/mpathN);

  • The logs for service atomic-openshift-node shows:

    nestedpendingoperations.go:267] Operation for "\"kubernetes.io/iscsi/10.0.0.10:iqn.2016-12.org.gluster-block:b54c4cb0-074a-4c38-8ed3-76cb27a8bdbb:0\"" failed. No retries permitted until 2020-01-01 00:00:00.242007101 -0000 -00 m=+28779.167712731 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-4119b152-a57c-11e9-8a9c-0050569a7d6a\" (UniqueName: \"kubernetes.io/iscsi/10.0.0.10:iqn.2016-12.org.gluster-block:b54c4cb0-074a-4c38-8ed3-76cb27a8bdbb:0\") pod \"logging-es-data-master-q2t5c59s-9-sfz8f\" (UID: \"3d04e48c-3d0c-11ea-b6bd-0050569afe07\") "
    

Environment

  • Red Hat OpenShift Container Platform (OCP) 3.x
  • Red Hat OpenShift Container Storage 3.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content