Applications using GFS2 filesystem are hanging intermittently while trying to create/delete large files

Issue

• The applications using GFS2 filesystem are getting hung intermittently while creating, deleting large files (of size ~150 GB) and following error messages are getting logged in /var/log/messages file on all cluster nodes:

  Sep 18 12:33:32 node1 kernel: INFO: task dgraph:21510 blocked for more than 120 seconds.
  Sep 18 12:33:32 node1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  Sep 18 12:33:32 node1 kernel: dgraph        D ffff815078233120     0 21510      1         21511 21509 (NOTLB)
  Sep 18 12:33:32 node1 kernel:  ffff811637f93ba8 0000000000000082 0000000000000018 ffffffff8868f4f8
  Sep 18 12:33:32 node1 kernel:  0000000000000292 0000000000000001 ffff81119bb7b0c0 ffff8130783b2820
  Sep 18 12:33:32 node1 kernel:  00045608465f13ca 0000000000012d97 ffff81119bb7b2a8 0000001a88690e5f
  Sep 18 12:33:32 node1 kernel: Call Trace:
  Sep 18 12:33:32 node1 kernel:  [<ffffffff8868f4f8>] :dlm:request_lock+0x93/0xa0
  Sep 18 12:33:32 node1 kernel:  [<ffffffff886baf54>] :gfs2:just_schedule+0x0/0xe
  Sep 18 12:33:32 node1 kernel:  [<ffffffff886baf5d>] :gfs2:just_schedule+0x9/0xe
  Sep 18 12:33:32 node1 kernel:  [<ffffffff800639f6>] __wait_on_bit+0x40/0x6e
  Sep 18 12:33:32 node1 kernel:  [<ffffffff886baf54>] :gfs2:just_schedule+0x0/0xe
  Sep 18 12:33:32 node1 kernel:  [<ffffffff80063a90>] out_of_line_wait_on_bit+0x6c/0x78
  [...]