geo-replication failed on peer with OSError: [Errno 5] Input/output error

Solution In Progress - Updated -

Issue

Traceback

[2016-12-07 16:44:54.377336] E [repce(/bricks/brick5):207:call] RepceClient: call 123914:139678469535552:1481129094.04 (entry_ops) failed on peer with OSError
[2016-12-07 16:44:54.377635] E [syncdutils(/bricks/brick5):276:log_raise_exception] : FAIL:
Traceback (most recent call last):
File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 201, in main
main_i()
File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 720, in main_i
local.service_loop(*[r for r in [remote] if r])
File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1504, in service_loop
g1.crawlwrap(oneshot=True, register_time=register_time)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 571, in crawlwrap
self.crawl()
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1282, in crawl
self.process([item[1]], 0)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 992, in process
self.process_change(change, done, retry)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 934, in process_change
failures = self.slave.server.entry_ops(entries)
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 226, in call
return self.ins(self.meth, *a)
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 208, in call
raise res
OSError: [Errno 5] Input/output error

=-=-=-=-=-=
It looks like the parent directory, .trashcan/internal_op//, is missing from the bricks:

$ grep -r ".trashcan/internal_op" *
bricks-brick15.log:[2016-12-05 18:27:26.249500] E [MSGID: 113018] [posix.c:2334:posix_rename] 0-XXXX-posix: pre-operation lstat on parent of /bricks/brick15/.trashcan/internal_op//C43.1 failed [No such file or directory]
bricks-brick15.log:[2016-12-05 18:27:26.263829] W [posix.c:1422:posix_mkdir] (-->/usr/lib64/glusterfs/3.7.9/xlator/storage/posix.so(posix_rename+0x820) [0x7f8f627d8660] -->/usr/lib64/glusterfs/3.7.9/xlator/features/trash.so(trash_unlink_rename_cbk+0x73e) [0x7f8f61f99bde] -->/usr/lib64/glusterfs/3.7.9/xlator/storage/posix.so(posix_mkdir+0x9ca) [0x7f8f627cbeda] ) 0-mrgglus2-posix: mkdir (/.trashcan/internal_op//C43.1): is issued without gfid-req 0x7f8f702fedd0 [Operation not permitted]

Environment

glusterfs 3.*

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content