GFS2 write performance degrades as filesystem becomes more full in a RHEL 6, 7 Resilient Storage cluster

Issue

• We had some more GFS2 issues today with slowness and things like this appearing in the logs:

oracle        D ffffffff801566cc     0 20843      1         20845 20841 (NOTLB)
ffff8123d167fdb8 0000000000000082 ffff8111f833ad20 ffff8111ecbea650
ffff8123f4725920 0000000000000006 ffff8123e78d3860 ffff81120bf557a0
000000c9b525c862 0000000000002e3e ffff8123e78d3a48 0000000b8002dec8
Call Trace:
[<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b
[<ffffffff80063cad>] .text.lock.mutex+0xf/0x14
[<ffffffff8877755c>] :gfs2:gfs2_file_writev+0x37/0xa2
[<ffffffff887775c7>] :gfs2:gfs2_file_write+0x0/0xa7
[<ffffffff800e3f50>] do_readv_writev+0x172/0x291
[<ffffffff887775c7>] :gfs2:gfs2_file_write+0x0/0xa7
[<ffffffff800ba777>] audit_syscall_entry+0x1a8/0x1d3
[<ffffffff800e40f9>] sys_writev+0x45/0x93
[<ffffffff8005d28d>] tracesys+0xd5/0xe0
...
kernel:  ffff810fc33e7e58 0000000000000082 ffff8122723bd438 ffff810c064d5ed0
kernel:  ffff810f6e954550 0000000000000006 ffff810f5d429820 ffff81120be807e0
kernel:  0001c1bee99aa4ae 000000000000549c ffff810f5d429a08 00000008e9a8ce35
kernel: Call Trace:
kernel:  [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b
kernel:  [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14
kernel:  [<ffffffff887775fd>] :gfs2:gfs2_file_write+0x36/0xa7
kernel:  [<ffffffff80016b29>] vfs_write+0xce/0x174
kernel:  [<ffffffff800173f2>] sys_write+0x45/0x6e
kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
kernel:
kernel: INFO: task oracle:17413 blocked for more than 120 seconds.
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.