Why is the tcmu-runner log consuming a lot of space?

Solution Verified - Updated -

Issue

  • The tcmu-runner log is spanning several GBs:

    ls -lha /var/log/gluster-block
    total 120G
    120G -rw-------.  1 root root 120G May 28 21:53 tcmu-runner-glfs.log
    

    This is because the following entries are constantly being added to this file, hundreds of times per second:

    [2020-05-28 20:37:37.290333] W [inode.c:1638:inode_table_prune] (-->/lib64/libglusterfs.so.0(fd_unref+0x1d3) [0x7f6567ac0d03] -->/lib64/libglusterfs.so.0(inode_unref+0x36) [0x7f6567aa8676] -->/lib64/libglusterfs.so.0(+0x38607) [0x7f6567aa8607] ) 0-vol_373db344457515f36db17714447245d7-client-2: Empty inode lru list found but with (-1) lru_size
    [2020-05-28 20:37:37.292657] W [inode.c:1638:inode_table_prune] (-->/usr/lib64/glusterfs/6.0/xlator/cluster/replicate.so(+0x5f0c7) [0x7f65170c80c7] -->/lib64/libglusterfs.so.0(inode_unref+0x36) [0x7f6567aa8676] -->/lib64/libglusterfs.so.0(+0x38607) [0x7f6567aa8607] ) 0-vol_373db344457515f36db17714447245d7-replicate-0: Empty inode lru list found but with (-1) lru_size
    [2020-05-28 20:37:37.292766] W [inode.c:1638:inode_table_prune] (-->/usr/lib64/glusterfs/6.0/xlator/features/shard.so(+0x4aca)  [0x7f6516b84aca] -->/lib64/libglusterfs.so.0(inode_unref+0x36) [0x7f6567aa8676] -->/lib64/libglusterfs.so.0(+0x38607)     [0x7f6567aa8607] ) 0-vol_373db344457515f36db17714447245d7-shard: Empty inode lru list found but with (-1) lru_size
    
  • Also, the CPU usage of the process is very high:

    $ top
    top - 09:24:20 up 125 days, 22:50,  1 user,  load average: 2.59, 2.49, 2.54                                                                                                         
    Tasks: 24264 total,   2 running, 24262 sleeping,   0 stopped,   0 zombie
    %Cpu(s): 15.8 us, 18.6 sy,  0.0 ni, 60.8 id,  2.5 wa,  0.0 hi,  2.2 si,  0.0 st
    KiB Mem : 16266620 total,   200820 free, 11995632 used,  4070168 buff/cache
    KiB Swap: 16777212 total,   250372 free, 16526840 used.  3073320 avail Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    21188 root      20   0   11.7g 125032  92728 S  95.0  0.8  23462:40 tcmu-   runner
    

Why is this so? How to stop these messages to show up?

Environment

Red Hat Openshift Container Storage version 3.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content