Why is a Gluster mount point returning 'Transport endpoint is not connected' after a rolling reboot of the Gluster nodes?

Solution Verified - Updated -

Issue

  • After correctly restarting Gluster nodes / pods one by one, a Gluster file system mount point is returning Transport endpoint is not connected:

    Filesystem                             Size  Used Avail Use% Mounted on
    192.168.1.1:glusterfs-registry-volume  5.0G  1.1G  4.0G  22% /registry
    
    df -h /registry
    df: '/registry': Transport endpoint is not connected
    
  • However, the matching Gluster volume is online, with all its bricks up and there's no pending healing:

    Status of volume: glusterfs-registry-volume
    Gluster process                             TCP Port  RDMA Port  Online  Pid
     ------------------------------------------------------------------------------
    Brick 192.168.1.1:/var/lib/heketi/mounts/vg
    _192cd678d5d6bb3ebc5111f906f576c4/brick_0cc
    a0b2f00e4fa82dca7859550c5fc66/brick         49154     0          Y             359
    Brick 192.168.1.2:/var/lib/heketi/mounts/vg
    _248d829a7538a61c5e7d43c8325652e8/brick_13f
    f85e4bf0362bc2218600c860dc229/brick         49152     0          Y       358
    Brick 192.168.1.3:/var/lib/heketi/mounts/v
    g_41c604c20fe4c4de54b9b7b120448756/brick_0e
    3d063b30e02fc5dc03eafb3e65f06b/brick        49152     0          Y       358
    Self-heal Daemon on localhost               N/A       N/A        Y       318
    Self-heal Daemon on 192.168.1.2             N/A       N/A        Y       315
    Self-heal Daemon on 192.168.1.3             N/A       N/A        Y       329
    
  • Why is this issue occurring? How to fix it?

Environment

  • Red Hat Gluster Storage version 3.x
  • Red Hat Openshift Container Storage version 3.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content