GFS2 write performance degrades as filesystem becomes more full in a RHEL Resilient Storage pacemaker cluster

Issue

• We had some more GFS2 issues today with slowness and things like this appearing in the logs:

  oracle        D ffffffff801566cc     0 20843      1         20845 20841 (NOTLB)
  ffff8123d167fdb8 0000000000000082 ffff8111f833ad20 ffff8111ecbea650
  ffff8123f4725920 0000000000000006 ffff8123e78d3860 ffff81120bf557a0
  000000c9b525c862 0000000000002e3e ffff8123e78d3a48 0000000b8002dec8
  Call Trace:
  [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b
  [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14
  [<ffffffff800e3f50>] do_readv_writev+0x172/0x291
  [<ffffffff800ba777>] audit_syscall_entry+0x1a8/0x1d3
  [<ffffffff800e40f9>] sys_writev+0x45/0x93
  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
  ...
  kernel:  ffff810fc33e7e58 0000000000000082 ffff8122723bd438 ffff810c064d5ed0
  kernel:  ffff810f6e954550 0000000000000006 ffff810f5d429820 ffff81120be807e0
  kernel:  0001c1bee99aa4ae 000000000000549c ffff810f5d429a08 00000008e9a8ce35
  kernel: Call Trace:
  kernel:  [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b
  kernel:  [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14
  kernel:  [<ffffffff887775fd>] :gfs2:gfs2_file_write+0x36/0xa7
  kernel:  [<ffffffff80016b29>] vfs_write+0xce/0x174
  kernel:  [<ffffffff800173f2>] sys_write+0x45/0x6e
  kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
  kernel:
  kernel: INFO: task oracle:17413 blocked for more than 120 seconds.
  kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.