Gluster commands fails with "Request timed out" when SSL is enabled

Issue

• Gluster commands fail randomly when SSL is enabled. Sometimes they may generate the correct output and other times they may timeout with the below error.

sh-4.2# gluster v status XYZ_volume
Error : Request timed out

• The " Transport endpoint is not connected" error is observed also observed as a side-effect when running heal info for multiple volumes at once. This error is not observed when heal info is run for the same volume again individually.

# for i in `gluster v list`; echo $i; gluster v heal $i info; done

vol_7fdbf0c580325912a5ee261def8538da
Brick 10.70.46.77:/var/lib/heketi/mounts/vg_1d18c75f8eccda61ef34d9a95bdf6672/brick_2d574c6220fbc796110a68e23a2e7184/brick
Status: Transport endpoint is not connected
Number of entries: -

Brick 10.70.47.87:/var/lib/heketi/mounts/vg_7728a790942b699e0f416aa32ca3cb28/brick_fa6f11d3378ee375800b37859570e4f7/brick
Status: Connected
Number of entries: 0

Brick 10.70.46.95:/var/lib/heketi/mounts/vg_b0402be94b303df4c5ea3e6c794da00b/brick_505fb126a3528e61465aaf48bb73d04c/brick
Status: Connected
Number of entries: 0


When running individually 

# gluster v heal vol_7fdbf0c580325912a5ee261def8538da info
Brick 10.70.46.77:/var/lib/heketi/mounts/vg_1d18c75f8eccda61ef34d9a95bdf6672/brick_2d574c6220fbc796110a68e23a2e7184/brick
Status: Connected
Number of entries: 0

Brick 10.70.47.87:/var/lib/heketi/mounts/vg_7728a790942b699e0f416aa32ca3cb28/brick_fa6f11d3378ee375800b37859570e4f7/brick
Status: Connected
Number of entries: 0

Brick 10.70.46.95:/var/lib/heketi/mounts/vg_b0402be94b303df4c5ea3e6c794da00b/brick_505fb126a3528e61465aaf48bb73d04c/brick
Status: Connected
Number of entries: 0

• Volume creation/deletion does not complete in OCS 3.x due to gluster command timeout errors.

Heketi logs

[kubeexec] ERROR 2021/01/12 13:11:38 heketi/pkg/remoteexec/log/commandlog.go:56:log.(*CommandLogger).Error: Failed to run command [gluster --mode=script --timeout=600 volume create vol_5f8581e1037d75ca8281e0e4cf32bbf7 replica 3 10.70.47.87:/var/lib/heketi/mounts/vg_7728a790942b699e0f416aa32ca3cb28/brick_4fbeb8d3dde689ed18f014f76592ba0f/brick 10.70.46.95:/var/lib/heketi/mounts/vg_9d1167dd9339385e8f5ec93ab2a2f299/brick_00afd673db78cd90354998dbd7552c9a/brick 10.70.46.77:/var/lib/heketi/mounts/vg_1d18c75f8eccda61ef34d9a95bdf6672/brick_f5fb5b23fd980834261d1122127f145f/brick ] on [pod:glusterfs-storage-p6hz8 c:glusterfs ns:glusterfs (from host:dhcp47-87.lab.eng.blr.redhat.com selector:glusterfs-node)]: Err[command terminated with exit code 1]: Stdout [Error : Request timed out
[kubeexec] ERROR 2021/01/12 13:26:01 heketi/pkg/remoteexec/log/commandlog.go:56:log.(*CommandLogger).Error: Failed to run command [gluster --mode=script --timeout=600 volume create vol_06757f2c6711b5dafbeff5a9a8d59869 replica 3 10.70.47.87:/var/lib/heketi/mounts/vg_db41061739a58ef2682d7cfe3b4a2021/brick_83258b27709e1c38bb7ebed677096cf3/brick 10.70.47.68:/var/lib/heketi/mounts/vg_69f4ca3e062b07b41daa4d3efff29082/brick_e2712952e1042199ea0ed8855c02baef/brick 10.70.46.95:/var/lib/heketi/mounts/vg_9d1167dd9339385e8f5ec93ab2a2f299/brick_ff51e9314b49e0f3d074d3722475746e/brick ] on [pod:glusterfs-storage-p6hz8 c:glusterfs ns:glusterfs (from host:dhcp47-87.lab.eng.blr.redhat.com selector:glusterfs-node)]: Err[command terminated with exit code 1]: Stdout [Error : Request timed out