Applications using GFS2 filesystem are hanging intermittently while trying to create/delete large files
Issue
-
The applications using GFS2 filesystem are getting hung intermittently while creating, deleting large files (of size ~150 GB) and following error messages are getting logged in
/var/log/messagesfile on all cluster nodes:Sep 18 12:33:32 node1 kernel: INFO: task dgraph:21510 blocked for more than 120 seconds. Sep 18 12:33:32 node1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 18 12:33:32 node1 kernel: dgraph D ffff815078233120 0 21510 1 21511 21509 (NOTLB) Sep 18 12:33:32 node1 kernel: ffff811637f93ba8 0000000000000082 0000000000000018 ffffffff8868f4f8 Sep 18 12:33:32 node1 kernel: 0000000000000292 0000000000000001 ffff81119bb7b0c0 ffff8130783b2820 Sep 18 12:33:32 node1 kernel: 00045608465f13ca 0000000000012d97 ffff81119bb7b2a8 0000001a88690e5f Sep 18 12:33:32 node1 kernel: Call Trace: Sep 18 12:33:32 node1 kernel: [<ffffffff8868f4f8>] :dlm:request_lock+0x93/0xa0 Sep 18 12:33:32 node1 kernel: [<ffffffff886baf54>] :gfs2:just_schedule+0x0/0xe Sep 18 12:33:32 node1 kernel: [<ffffffff886baf5d>] :gfs2:just_schedule+0x9/0xe Sep 18 12:33:32 node1 kernel: [<ffffffff800639f6>] __wait_on_bit+0x40/0x6e Sep 18 12:33:32 node1 kernel: [<ffffffff886baf54>] :gfs2:just_schedule+0x0/0xe Sep 18 12:33:32 node1 kernel: [<ffffffff80063a90>] out_of_line_wait_on_bit+0x6c/0x78 [...]
Environment
- Red Hat Enterprise Linux Server 5 (with the High Availability and Resilient Storage Add Ons)
- Global Filesystem 2 (GFS2)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Close
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.