Why slave node goes faulty with operation not permitted error for gluster geo-replica session in RHGS 3.1.3 ?

Solution Verified - Updated -

Issue

  • Why non-root user geo-replication session status shows "Faulty" state in Red Hat Gluster Storage?
# gluster volume geo-replication geovol geouser@slave-node1::geovol status

MASTER NODE                MASTER VOL    MASTER BRICK               SLAVE USER    SLAVE                                      SLAVE NODE    STATUS    CRAWL STATUS    LAST_SYNCED          
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
master-node1    ap_config        /brick/brick_georepl_01    geoaccount       geouser@slave-node1::ap_config    N/A           Faulty    N/A             N/A                  
master-node4    ap_config        /brick/brick_georepl_04    geoaccount       geouser@slave-node1::ap_config    N/A           Faulty    N/A             N/A                  
master-node3    ap_config        /brick/brick_georepl_03    geoaccount       geouser@slave-node1::ap_config    N/A           Faulty    N/A             N/A                  
master-node2    ap_config        /brick/brick_georepl_02    geoaccount       geouser@slave-node1::ap_config    N/A           Faulty    N/A             N/A 
  • The master node glusterd failed to connect slave node glusterd during geo-replica session establishment which can be seen in master geo-replication logs:
[2016-12-14 01:03:43.534657] I [monitor(monitor):267:monitor] Monitor: starting gsyncd worker
[2016-12-14 01:03:43.812525] I [gsyncd(/brick/ap_config):710:main_i] <top>: syncing: gluster://localhost:ap_config -> ssh://geoaccount@gluster-node5:gluster://localhost:ap_config
[2016-12-14 01:03:43.815692] I [changelogagent(agent):73:__init__] ChangelogAgent: Agent listining...
[2016-12-14 01:03:44.888118] E [syncdutils(/brick/ap_config):252:log_raise_exception] <top>: connection to peer is broken
[2016-12-14 01:03:44.888609] E [resource(/brick/ap_config):226:errlog] Popen: command "ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-Kgx3p1/83d98d35abb59f50517f40249b52fb49.sock geoaccount@glusterc2-node5 /nonexistent/gsyncd --session-owner ee116313-eb4b-4523-8bfc-c5a6b4058570 -N --listen --timeout 120 gluster://localhost:ap_config" returned with 1, saying:
[2016-12-14 01:03:44.888797] E [resource(/brick/ap_config):230:logerr] Popen: ssh> [2016-12-14 01:03:44.471701] I [cli.c:721:main] 0-cli: Started running /usr/sbin/gluster with version 3.7.9
[2016-12-14 01:03:44.888935] E [resource(/brick/ap_config):230:logerr] Popen: ssh> [2016-12-14 01:03:44.471751] I [cli.c:608:cli_rpc_init] 0-cli: Connecting to remote glusterd at localhost
[2016-12-14 01:03:44.889067] E [resource(/brick/ap_config):230:logerr] Popen: ssh> [2016-12-14 01:03:44.606874] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-12-14 01:03:44.889213] E [resource(/brick/ap_config):230:logerr] Popen: ssh> [2016-12-14 01:03:44.606962] I [socket.c:2472:socket_event_handler] 0-transport: disconnecting now
[2016-12-14 01:03:44.889374] E [resource(/brick/ap_config):230:logerr] Popen: ssh> [2016-12-14 01:03:44.609049] I [cli-rpc-ops.c:6514:gf_cli_getwd_cbk] 0-cli: Received resp to getwd
[2016-12-14 01:03:44.889497] E [resource(/brick/ap_config):230:logerr] Popen: ssh> [2016-12-14 01:03:44.609165] I [input.c:36:cli_batch] 0-: Exiting with: 0
[2016-12-14 01:03:44.889853] I [syncdutils(/brick/ap_config):220:finalize] <top>: exiting.
[2016-12-14 01:03:44.891433] I [repce(agent):92:service_loop] RepceServer: terminating on reaching EOF.
[2016-12-14 01:03:44.891706] I [syncdutils(agent):220:finalize] <top>: exiting.
[2016-12-14 01:03:44.892102] I [monitor(monitor):333:monitor] Monitor: worker(/brick/ap_config) died before establishing connection

Environment

  • Red Hat Gluster Storage 3.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content