IOZone deadlocked observed during random I/O test in Red Hat Gluster storage
Issue
- IOZone deadlocked during a random I/O test
Mar 19 14:39:42 kernel: INFO: task iozone:13184 blocked for more than 120 seconds.
Mar 19 14:39:42 kernel: Not tainted 2.6.32-431.el6.x86_64 #1
Mar 19 14:39:42 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 19 14:39:42 kernel: iozone D 0000000000000000 0 13184 13183 0x00000084
Mar 19 14:39:42 kernel: ffff880be811fda8 0000000000000086 0000000000000000 0000000000000000
Mar 19 14:39:42 kernel: ffff880be6fa0a80 ffff880be839f980 ffff880be811fdf8 ffffffffa02fc78b
Mar 19 14:39:42 kernel: ffff880be80db058 ffff880be811ffd8 000000000000fbc8 ffff880be80db058
Mar 19 14:39:42 kernel: Call Trace:
Mar 19 14:39:42 kernel: [<ffffffffa02fc78b>] ? fuse_dentry_revalidate+0x2fb/0x340 [fuse]
Mar 19 14:39:42 kernel: [<ffffffff8152907e>] __mutex_lock_slowpath+0x13e/0x180
Mar 19 14:39:42 kernel: [<ffffffff8122752f>] ? security_inode_permission+0x1f/0x30
Mar 19 14:39:42 kernel: [<ffffffff81528f1b>] mutex_lock+0x2b/0x50
Mar 19 14:39:42 kernel: [<ffffffff8119796c>] vfs_unlink+0x5c/0xf0
Mar 19 14:39:42 kernel: [<ffffffff81196c1a>] ? lookup_hash+0x3a/0x50
Mar 19 14:39:42 kernel: [<ffffffff8119a723>] do_unlinkat+0x163/0x260
Mar 19 14:39:42 kernel: [<ffffffff810e2067>] ? audit_syscall_entry+0x1d7/0x200
Mar 19 14:39:42 kernel: [<ffffffff8119a836>] sys_unlink+0x16/0x20
Mar 19 14:39:42 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
-
IOZone tests hang, disconnects and hung tasks seen in logs for NFS
- Running multi node sequential writes from iozone gnfs lead to disconnects and hung tasks
Mar 19 18:03:43 kernel: INFO: task iozone:16077 blocked for more than 120 seconds.
Mar 19 18:03:43 kernel: Not tainted 2.6.32-431.el6.x86_64 #1
Mar 19 18:03:43 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 19 18:03:43 kernel: iozone D 0000000000000008 0 16077 1 0x00000080
Mar 19 18:03:43 kernel: ffff8806231bdcc8 0000000000000086 0000000000000000 ffffffffa0498940
Mar 19 18:03:43 kernel: ffff8806231bdc38 ffffffff810149b9 ffff8806231bdc78 ffffffff810a70a1
Mar 19 18:03:43 kernel: ffff880624317ab8 ffff8806231bdfd8 000000000000fbc8 ffff880624317ab8
Mar 19 18:03:43 kernel: Call Trace:
Mar 19 18:03:43 kernel: [<ffffffffa0498940>] ? rpc_execute+0x50/0xa0 [sunrpc]
Mar 19 18:03:43 kernel: [<ffffffff810149b9>] ? read_tsc+0x9/0x20
Mar 19 18:03:43 kernel: [<ffffffff810a70a1>] ? ktime_get_ts+0xb1/0xf0
Mar 19 18:03:43 kernel: [<ffffffff810a70a1>] ? ktime_get_ts+0xb1/0xf0
Mar 19 18:03:43 kernel: [<ffffffff8111f930>] ? sync_page+0x0/0x50
Mar 19 18:03:43 kernel: [<ffffffff815280a3>] io_schedule+0x73/0xc0
Mar 19 18:03:43 kernel: [<ffffffff8111f96d>] sync_page+0x3d/0x50
Mar 19 18:03:43 kernel: [<ffffffff81528b6f>] __wait_on_bit+0x5f/0x90
Mar 19 18:03:43 kernel: [<ffffffff8111fba3>] wait_on_page_bit+0x73/0x80
Mar 19 18:03:43 kernel: [<ffffffff8109b320>] ? wake_bit_function+0x0/0x50
Mar 19 18:03:43 kernel: [<ffffffff81135bf5>] ? pagevec_lookup_tag+0x25/0x40
Mar 19 18:03:43 kernel: [<ffffffff8111ffcb>] wait_on_page_writeback_range+0xfb/0x190
Mar 19 18:03:43 kernel: [<ffffffff81120198>] filemap_write_and_wait_range+0x78/0x90
Mar 19 18:03:43 kernel: [<ffffffff811baa3e>] vfs_fsync_range+0x7e/0x100
Mar 19 18:03:43 kernel: [<ffffffff811bab2d>] vfs_fsync+0x1d/0x20
Mar 19 18:03:43 kernel: [<ffffffff811bab6e>] do_fsync+0x3e/0x60
Mar 19 18:03:43 kernel: [<ffffffff811babc0>] sys_fsync+0x10/0x20
Mar 19 18:03:43 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Environment
- Red Hat Gluster Storage 3.0.4
- glusterfs-3.6.0.53
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
