Frequently getting : Remote operation failed. - [No such file or directory]

Posted on

Server : glusterfs 4.0.2, OS: RHEL 7.6
Client : glusterfs 3.12.2, OS: RHEL 7.7 (2 node replica with Arbiter)
Opensource Version.

Client side:
We are receiving the below message very frequently. We are running an app which creates temporary file and deletes it immediately after process..

[2020-04-12 04:24:21.768738] W [MSGID: 114031] [client-rpc-fops.c:2865:client3_3_lookup_cbk] 0-gvol_qa-client-0: remote operation failed. Path: (918115fc-f1c9-4a65-8b07-31ff41798b1a) [No such file or directory]
[2020-04-12 04:24:21.769106] W [MSGID: 114031] [client-rpc-fops.c:2865:client3_3_lookup_cbk] 0-gvol_qa-client-2: remote operation failed. Path: (918115fc-f1c9-4a65-8b07-31ff41798b1a) [No such file or directory]
[2020-04-12 04:24:21.769417] W [fuse-bridge.c:1486:fuse_fd_cbk] 0-glusterfs-fuse: 562804974: OPEN() => -1 (Stale file handle)

Also we get the below message very frequently:

[2020-04-08 13:47:18.934790] W [fuse-bridge.c:1486:fuse_fd_cbk] 0-glusterfs-fuse: 98112911: OPEN() <FILEPATH###> => -1 (Stale file handle)
[2020-04-08 13:47:18.934640] E [MSGID: 109040] [dht-helper.c:1387:dht_migration_complete_check_task] 0-gvol_prod-dht: <FILEPATH###>: failed to lookup the file on gvol_prod-dht [Stale file handle]

What could be the reason and how to avoid these as it eats up lots of space on /var/log location.?