A pacemaker gfs2 filesystem resource failed to stop and umount appear to fail

Issue

A pacemaker gfs2 filesystem resource failed to stop and umount appear to fail. The umount process appear to hang for a long period of time after the Filesystem resource timed out.

May  2 12:44:07 node42 Filesystem(data)[22988]: INFO: Running stop for /dev/clustervg/data on /data
May  2 12:44:07 node42 Filesystem(data)[22988]: INFO: Trying to unmount /data
May  2 12:45:07 node42 lrmd[5311]: warning: data_stop_0 process (PID 22988) timed out
May  2 12:45:07 node42 lrmd[5311]: warning: data_stop_0:22988 - timed out after 60000ms
May  2 12:45:07 node42 crmd[5314]:   error: Result of stop operation for data on node42: Timed Out
[....]
May  2 12:53:22 node42 kernel: INFO: task umount:23282 blocked for more than 120 seconds.
May  2 12:53:22 node42 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  2 12:53:22 node42 kernel: umount          D ffff8c00ffd1ac80     0 23282      1 0x10000086
May  2 12:53:22 node42 kernel: Call Trace:
May  2 12:53:22 node42 kernel: [<ffffffffacb80ddd>] ? wait_for_completion+0x11d/0x140
May  2 12:53:22 node42 kernel: [<ffffffffacb80a09>] schedule+0x29/0x70
May  2 12:53:22 node42 kernel: [<ffffffffacb7e458>] schedule_timeout+0x168/0x2d0
May  2 12:53:22 node42 kernel: [<ffffffffac4adce0>] ? __internal_add_timer+0x130/0x130
May  2 12:53:22 node42 kernel: [<ffffffffac4c6dd6>] ? prepare_to_wait+0x56/0x90
May  2 12:53:22 node42 kernel: [<ffffffffc0d0d408>] gfs2_gl_hash_clear+0xa8/0x120 [gfs2]
May  2 12:53:22 node42 kernel: [<ffffffffac4c72e0>] ? wake_up_atomic_t+0x30/0x30
May  2 12:53:22 node42 kernel: [<ffffffffc0d2b792>] gfs2_put_super+0x132/0x1c0 [gfs2]
May  2 12:53:22 node42 kernel: [<ffffffffac64d37d>] generic_shutdown_super+0x6d/0x100
May  2 12:53:22 node42 kernel: [<ffffffffac64d7f7>] kill_block_super+0x27/0x70
May  2 12:53:22 node42 kernel: [<ffffffffc0d193a2>] gfs2_kill_sb+0x72/0x80 [gfs2]
May  2 12:53:22 node42 kernel: [<ffffffffac64db5e>] deactivate_locked_super+0x4e/0x70
May  2 12:53:22 node42 kernel: [<ffffffffac64e2e6>] deactivate_super+0x46/0x60
May  2 12:53:22 node42 kernel: [<ffffffffac66ce5f>] cleanup_mnt+0x3f/0x80
May  2 12:53:22 node42 kernel: [<ffffffffac66cef2>] __cleanup_mnt+0x12/0x20
May  2 12:53:22 node42 kernel: [<ffffffffac4c2d2b>] task_work_run+0xbb/0xe0
May  2 12:53:22 node42 kernel: [<ffffffffac42cc65>] do_notify_resume+0xa5/0xc0
May  2 12:53:22 node42 kernel: [<ffffffffacb8e23b>] int_signal+0x12/0x17
May  2 12:54:11 node42 kernel: G:  s:UN n:2/ca0a05d f:Iqo t:UN d:EX/0 a:0 v:0 r:2 m:200
May  2 12:54:11 node42 kernel: G:  s:SH n:5/c8649b4 f:DIqob t:SH d:UN/600386000 a:0 v:0 r:3 m:200
May  2 12:54:11 node42 kernel:  H: s:SH f:EH e:0 p:1179 [(ended)] gfs2_inode_lookup+0x228/0x440 [gfs2]
May  2 12:54:11 node42 kernel: G:  s:UN n:2/c8649b4 f:Iqo t:UN d:EX/0 a:0 v:0 r:2 m:200
May  2 12:53:22 node42 kernel: INFO: task umount:23282 blocked for more than 120 seconds.
May  2 12:53:22 node42 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  2 12:53:22 node42 kernel: umount          D ffff8c00ffd1ac80     0 23282      1 0x10000086
May  2 12:53:22 node42 kernel: Call Trace:
May  2 12:53:22 node42 kernel: [<ffffffffacb80ddd>] ? wait_for_completion+0x11d/0x140
May  2 12:53:22 node42 kernel: [<ffffffffacb80a09>] schedule+0x29/0x70
May  2 12:53:22 node42 kernel: [<ffffffffacb7e458>] schedule_timeout+0x168/0x2d0
May  2 12:53:22 node42 kernel: [<ffffffffac4adce0>] ? __internal_add_timer+0x130/0x130
May  2 12:53:22 node42 kernel: [<ffffffffac4c6dd6>] ? prepare_to_wait+0x56/0x90
May  2 12:53:22 node42 kernel: [<ffffffffc0d0d408>] gfs2_gl_hash_clear+0xa8/0x120 [gfs2]
May  2 12:53:22 node42 kernel: [<ffffffffac4c72e0>] ? wake_up_atomic_t+0x30/0x30
May  2 12:53:22 node42 kernel: [<ffffffffc0d2b792>] gfs2_put_super+0x132/0x1c0 [gfs2]
May  2 12:53:22 node42 kernel: [<ffffffffac64d37d>] generic_shutdown_super+0x6d/0x100
May  2 12:53:22 node42 kernel: [<ffffffffac64d7f7>] kill_block_super+0x27/0x70
May  2 12:53:22 node42 kernel: [<ffffffffc0d193a2>] gfs2_kill_sb+0x72/0x80 [gfs2]
May  2 12:53:22 node42 kernel: [<ffffffffac64db5e>] deactivate_locked_super+0x4e/0x70
May  2 12:53:22 node42 kernel: [<ffffffffac64e2e6>] deactivate_super+0x46/0x60
May  2 12:53:22 node42 kernel: [<ffffffffac66ce5f>] cleanup_mnt+0x3f/0x80
May  2 12:53:22 node42 kernel: [<ffffffffac66cef2>] __cleanup_mnt+0x12/0x20
May  2 12:53:22 node42 kernel: [<ffffffffac4c2d2b>] task_work_run+0xbb/0xe0
May  2 12:53:22 node42 kernel: [<ffffffffac42cc65>] do_notify_resume+0xa5/0xc0
May  2 12:53:22 node42 kernel: [<ffffffffacb8e23b>] int_signal+0x12/0x17
May  2 12:54:11 node42 kernel: G:  s:UN n:2/ca0a05d f:Iqo t:UN d:EX/0 a:0 v:0 r:2 m:200
May  2 12:54:11 node42 kernel: G:  s:SH n:5/c8649b4 f:DIqob t:SH d:UN/600386000 a:0 v:0 r:3 m:200
May  2 12:54:11 node42 kernel:  H: s:SH f:EH e:0 p:1179 [(ended)] gfs2_inode_lookup+0x228/0x440 [gfs2]
May  2 12:54:11 node42 kernel: G:  s:UN n:2/c8649b4 f:Iqo t:UN d:EX/0 a:0 v:0 r:2 m:200
May  2 12:54:11 node42 kernel: G:  s:UN n:2/cffa835 f:Iqo t:UN d:EX/0 a:0 v:0 r:2 m:200
May  2 12:54:11 node42 kernel: G:  s:SH n:5/ca0a05d f:DIqob t:SH d:UN/600366000 a:0 v:0 r:3 m:200
May  2 12:54:11 node42 kernel:  H: s:SH f:EH e:0 p:30019 [(ended)] gfs2_inode_lookup+0x228/0x440 [gfs2]
May  2 12:54:11 node42 kernel: G:  s:SH n:5/cffa835 f:DIqob t:SH d:UN/600365000 a:0 v:0 r:3 m:200
May  2 12:54:11 node42 kernel:  H: s:SH f:EH e:0 p:2472 [(ended)] gfs2_inode_lookup+0x228/0x440 [gfs2]
May  2 12:54:11 node42 kernel: dlm: datalock: group leave failed -512 0
May  2 12:54:11 node42 kernel: VFS: Busy inodes after unmount of dm-32. Self-destruct in 5 seconds.  Have a nice day...