Why is the tcmu-runner log consuming a lot of space?
Issue
-
The tcmu-runner log is spanning several GBs:
ls -lha /var/log/gluster-block total 120G 120G -rw-------. 1 root root 120G May 28 21:53 tcmu-runner-glfs.log
This is because the following entries are constantly being added to this file, hundreds of times per second:
[2020-05-28 20:37:37.290333] W [inode.c:1638:inode_table_prune] (-->/lib64/libglusterfs.so.0(fd_unref+0x1d3) [0x7f6567ac0d03] -->/lib64/libglusterfs.so.0(inode_unref+0x36) [0x7f6567aa8676] -->/lib64/libglusterfs.so.0(+0x38607) [0x7f6567aa8607] ) 0-vol_373db344457515f36db17714447245d7-client-2: Empty inode lru list found but with (-1) lru_size [2020-05-28 20:37:37.292657] W [inode.c:1638:inode_table_prune] (-->/usr/lib64/glusterfs/6.0/xlator/cluster/replicate.so(+0x5f0c7) [0x7f65170c80c7] -->/lib64/libglusterfs.so.0(inode_unref+0x36) [0x7f6567aa8676] -->/lib64/libglusterfs.so.0(+0x38607) [0x7f6567aa8607] ) 0-vol_373db344457515f36db17714447245d7-replicate-0: Empty inode lru list found but with (-1) lru_size [2020-05-28 20:37:37.292766] W [inode.c:1638:inode_table_prune] (-->/usr/lib64/glusterfs/6.0/xlator/features/shard.so(+0x4aca) [0x7f6516b84aca] -->/lib64/libglusterfs.so.0(inode_unref+0x36) [0x7f6567aa8676] -->/lib64/libglusterfs.so.0(+0x38607) [0x7f6567aa8607] ) 0-vol_373db344457515f36db17714447245d7-shard: Empty inode lru list found but with (-1) lru_size
-
Also, the CPU usage of the process is very high:
$ top top - 09:24:20 up 125 days, 22:50, 1 user, load average: 2.59, 2.49, 2.54 Tasks: 24264 total, 2 running, 24262 sleeping, 0 stopped, 0 zombie %Cpu(s): 15.8 us, 18.6 sy, 0.0 ni, 60.8 id, 2.5 wa, 0.0 hi, 2.2 si, 0.0 st KiB Mem : 16266620 total, 200820 free, 11995632 used, 4070168 buff/cache KiB Swap: 16777212 total, 250372 free, 16526840 used. 3073320 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21188 root 20 0 11.7g 125032 92728 S 95.0 0.8 23462:40 tcmu- runner
Why is this so? How to stop these messages to show up?
Environment
Red Hat Openshift Container Storage version 3.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.