geo replication paused due to OSError: [Errno 39] Directory not empty: '.gfid/<GFID>

Solution Verified - Updated -

Issue

geo-rep session status turns 'Faulty' soon after initialization.

We see something similar to the following on the logs:

I [master(/gluster/vg00/VolName/brick):1521:crawl] _GMaster: starting history crawl    turns=1    stime=(1574842874, 0)    entry_stime=(1574842874, 0)    etime=1578493885
I [master(/gluster/vg00/VolName/brick):1550:crawl] _GMaster: slave's time    stime=(1574842874, 0)
E [repce(/gluster/vg00/VolName/brick):209:__call__] RepceClient: call failed    call=87244:140001104545600:1578493886.7    method=entry_ops    error=OSError
E [syncdutils(/gluster/vg00/VolName/brick):355:log_raise_exception] <top>: FAIL: Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 210, in main    main_i()
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 805, in main_i    local.service_loop(*[r for r in [remote] if r])
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1592, in service_loop    g3.crawlwrap(oneshot=True)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 597, in crawlwrap    self.crawl()
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1559, in crawl    self.changelogs_batch_process(changes)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1459, in changelogs_batch_process    self.process(batch)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1293, in process    self.process_change(change, done, retry)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1189, in process_change    failures = self.slave.server.entry_ops(entries)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 228, in __call__    return self.ins(self.meth, *a)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 210, in __call__    raise res
OSError: [Errno 39] Directory not empty: '.gfid/fe150ee3-f3f4-4227-840d-f7d81ba31815/ABCDEFG'
I [syncdutils(/gluster/vg00/VolName/brick):295:finalize] <top>: exiting.
I [repce(/gluster/vg00/VolName/brick):92:service_loop] RepceServer: terminating on reaching EOF.
I [syncdutils(/gluster/vg00/VolName/brick):295:finalize] <top>: exiting.
I [monitor(monitor):299:monitor] Monitor: worker died in startup phase    brick=/gluster/vg00/VolName/brick
I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change    status=Faulty

Environment

glusterfs 3.*

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content