geo replication paused due to OSError: [Errno 39] Directory not empty: '.gfid/<GFID>
Issue
geo-rep session status turns 'Faulty' soon after initialization.
We see something similar to the following on the logs:
I [master(/gluster/vg00/VolName/brick):1521:crawl] _GMaster: starting history crawl turns=1 stime=(1574842874, 0) entry_stime=(1574842874, 0) etime=1578493885
I [master(/gluster/vg00/VolName/brick):1550:crawl] _GMaster: slave's time stime=(1574842874, 0)
E [repce(/gluster/vg00/VolName/brick):209:__call__] RepceClient: call failed call=87244:140001104545600:1578493886.7 method=entry_ops error=OSError
E [syncdutils(/gluster/vg00/VolName/brick):355:log_raise_exception] <top>: FAIL: Traceback (most recent call last):
File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 210, in main main_i()
File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 805, in main_i local.service_loop(*[r for r in [remote] if r])
File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1592, in service_loop g3.crawlwrap(oneshot=True)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 597, in crawlwrap self.crawl()
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1559, in crawl self.changelogs_batch_process(changes)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1459, in changelogs_batch_process self.process(batch)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1293, in process self.process_change(change, done, retry)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1189, in process_change failures = self.slave.server.entry_ops(entries)
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 228, in __call__ return self.ins(self.meth, *a)
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 210, in __call__ raise res
OSError: [Errno 39] Directory not empty: '.gfid/fe150ee3-f3f4-4227-840d-f7d81ba31815/ABCDEFG'
I [syncdutils(/gluster/vg00/VolName/brick):295:finalize] <top>: exiting.
I [repce(/gluster/vg00/VolName/brick):92:service_loop] RepceServer: terminating on reaching EOF.
I [syncdutils(/gluster/vg00/VolName/brick):295:finalize] <top>: exiting.
I [monitor(monitor):299:monitor] Monitor: worker died in startup phase brick=/gluster/vg00/VolName/brick
I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change status=Faulty
Environment
glusterfs 3.*
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.