Why slave node goes faulty with operation not permitted error for gluster geo-replica session in RHGS 3.1.3 ?

Issue

• Why non-root user geo-replication session status shows "Faulty" state in Red Hat Gluster Storage?

# gluster volume geo-replication geovol geouser@slave-node1::geovol status

MASTER NODE                MASTER VOL    MASTER BRICK               SLAVE USER    SLAVE                                      SLAVE NODE    STATUS    CRAWL STATUS    LAST_SYNCED          
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
master-node1    ap_config        /brick/brick_georepl_01    geoaccount       geouser@slave-node1::ap_config    N/A           Faulty    N/A             N/A                  
master-node4    ap_config        /brick/brick_georepl_04    geoaccount       geouser@slave-node1::ap_config    N/A           Faulty    N/A             N/A                  
master-node3    ap_config        /brick/brick_georepl_03    geoaccount       geouser@slave-node1::ap_config    N/A           Faulty    N/A             N/A                  
master-node2    ap_config        /brick/brick_georepl_02    geoaccount       geouser@slave-node1::ap_config    N/A           Faulty    N/A             N/A 

• The master node glusterd failed to connect slave node glusterd during geo-replica session establishment which can be seen in master geo-replication logs:

[2016-12-14 01:03:43.534657] I [monitor(monitor):267:monitor] Monitor: starting gsyncd worker
[2016-12-14 01:03:43.812525] I [gsyncd(/brick/ap_config):710:main_i] <top>: syncing: gluster://localhost:ap_config -> ssh://geoaccount@gluster-node5:gluster://localhost:ap_config
[2016-12-14 01:03:43.815692] I [changelogagent(agent):73:__init__] ChangelogAgent: Agent listining...
[2016-12-14 01:03:44.888118] E [syncdutils(/brick/ap_config):252:log_raise_exception] <top>: connection to peer is broken
[2016-12-14 01:03:44.888609] E [resource(/brick/ap_config):226:errlog] Popen: command "ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-Kgx3p1/83d98d35abb59f50517f40249b52fb49.sock geoaccount@glusterc2-node5 /nonexistent/gsyncd --session-owner ee116313-eb4b-4523-8bfc-c5a6b4058570 -N --listen --timeout 120 gluster://localhost:ap_config" returned with 1, saying:
[2016-12-14 01:03:44.888797] E [resource(/brick/ap_config):230:logerr] Popen: ssh> [2016-12-14 01:03:44.471701] I [cli.c:721:main] 0-cli: Started running /usr/sbin/gluster with version 3.7.9
[2016-12-14 01:03:44.888935] E [resource(/brick/ap_config):230:logerr] Popen: ssh> [2016-12-14 01:03:44.471751] I [cli.c:608:cli_rpc_init] 0-cli: Connecting to remote glusterd at localhost
[2016-12-14 01:03:44.889067] E [resource(/brick/ap_config):230:logerr] Popen: ssh> [2016-12-14 01:03:44.606874] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-12-14 01:03:44.889213] E [resource(/brick/ap_config):230:logerr] Popen: ssh> [2016-12-14 01:03:44.606962] I [socket.c:2472:socket_event_handler] 0-transport: disconnecting now
[2016-12-14 01:03:44.889374] E [resource(/brick/ap_config):230:logerr] Popen: ssh> [2016-12-14 01:03:44.609049] I [cli-rpc-ops.c:6514:gf_cli_getwd_cbk] 0-cli: Received resp to getwd
[2016-12-14 01:03:44.889497] E [resource(/brick/ap_config):230:logerr] Popen: ssh> [2016-12-14 01:03:44.609165] I [input.c:36:cli_batch] 0-: Exiting with: 0
[2016-12-14 01:03:44.889853] I [syncdutils(/brick/ap_config):220:finalize] <top>: exiting.
[2016-12-14 01:03:44.891433] I [repce(agent):92:service_loop] RepceServer: terminating on reaching EOF.
[2016-12-14 01:03:44.891706] I [syncdutils(agent):220:finalize] <top>: exiting.
[2016-12-14 01:03:44.892102] I [monitor(monitor):333:monitor] Monitor: worker(/brick/ap_config) died before establishing connection