GFS2 write performance degrades as filesystem becomes more full in a RHEL 6, 7 Resilient Storage cluster

Solution Verified - Updated -

Issue

  • We had some more GFS2 issues today with slowness and things like this appearing in the logs:
oracle        D ffffffff801566cc     0 20843      1         20845 20841 (NOTLB)
ffff8123d167fdb8 0000000000000082 ffff8111f833ad20 ffff8111ecbea650
ffff8123f4725920 0000000000000006 ffff8123e78d3860 ffff81120bf557a0
000000c9b525c862 0000000000002e3e ffff8123e78d3a48 0000000b8002dec8
Call Trace:
[<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b
[<ffffffff80063cad>] .text.lock.mutex+0xf/0x14
[<ffffffff8877755c>] :gfs2:gfs2_file_writev+0x37/0xa2
[<ffffffff887775c7>] :gfs2:gfs2_file_write+0x0/0xa7
[<ffffffff800e3f50>] do_readv_writev+0x172/0x291
[<ffffffff887775c7>] :gfs2:gfs2_file_write+0x0/0xa7
[<ffffffff800ba777>] audit_syscall_entry+0x1a8/0x1d3
[<ffffffff800e40f9>] sys_writev+0x45/0x93
[<ffffffff8005d28d>] tracesys+0xd5/0xe0
...
kernel:  ffff810fc33e7e58 0000000000000082 ffff8122723bd438 ffff810c064d5ed0
kernel:  ffff810f6e954550 0000000000000006 ffff810f5d429820 ffff81120be807e0
kernel:  0001c1bee99aa4ae 000000000000549c ffff810f5d429a08 00000008e9a8ce35
kernel: Call Trace:
kernel:  [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b
kernel:  [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14
kernel:  [<ffffffff887775fd>] :gfs2:gfs2_file_write+0x36/0xa7
kernel:  [<ffffffff80016b29>] vfs_write+0xce/0x174
kernel:  [<ffffffff800173f2>] sys_write+0x45/0x6e
kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
kernel:
kernel: INFO: task oracle:17413 blocked for more than 120 seconds.
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Environment

  • Red Hat Enterprise Linux (RHEL) 6,7, 8 with the Resilient Storage Add On
  • Global File System 2 (GFS2)
    • The filesystem is becoming full or is full

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In