IOZone deadlocked observed during random I/O test in Red Hat Gluster storage

Issue

• IOZone deadlocked during a random I/O test

Mar 19 14:39:42  kernel: INFO: task iozone:13184 blocked for more than 120 seconds.
Mar 19 14:39:42  kernel:      Not tainted 2.6.32-431.el6.x86_64 #1
Mar 19 14:39:42 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 19 14:39:42  kernel: iozone        D 0000000000000000     0 13184  13183 0x00000084
Mar 19 14:39:42  kernel: ffff880be811fda8 0000000000000086 0000000000000000 0000000000000000
Mar 19 14:39:42  kernel: ffff880be6fa0a80 ffff880be839f980 ffff880be811fdf8 ffffffffa02fc78b
Mar 19 14:39:42  kernel: ffff880be80db058 ffff880be811ffd8 000000000000fbc8 ffff880be80db058
Mar 19 14:39:42  kernel: Call Trace:
Mar 19 14:39:42  kernel: [<ffffffffa02fc78b>] ? fuse_dentry_revalidate+0x2fb/0x340 [fuse]
Mar 19 14:39:42  kernel: [<ffffffff8152907e>] __mutex_lock_slowpath+0x13e/0x180
Mar 19 14:39:42  kernel: [<ffffffff8122752f>] ? security_inode_permission+0x1f/0x30
Mar 19 14:39:42  kernel: [<ffffffff81528f1b>] mutex_lock+0x2b/0x50
Mar 19 14:39:42  kernel: [<ffffffff8119796c>] vfs_unlink+0x5c/0xf0
Mar 19 14:39:42  kernel: [<ffffffff81196c1a>] ? lookup_hash+0x3a/0x50
Mar 19 14:39:42  kernel: [<ffffffff8119a723>] do_unlinkat+0x163/0x260
Mar 19 14:39:42  kernel: [<ffffffff810e2067>] ? audit_syscall_entry+0x1d7/0x200
Mar 19 14:39:42  kernel: [<ffffffff8119a836>] sys_unlink+0x16/0x20
Mar 19 14:39:42  kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

• IOZone tests hang, disconnects and hung tasks seen in logs for NFS

  • Running multi node sequential writes from iozone gnfs lead to disconnects and hung tasks

Mar 19 18:03:43  kernel: INFO: task iozone:16077 blocked for more than 120 seconds.
Mar 19 18:03:43  kernel:      Not tainted 2.6.32-431.el6.x86_64 #1
Mar 19 18:03:43  kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 19 18:03:43  kernel: iozone        D 0000000000000008     0 16077      1 0x00000080
Mar 19 18:03:43  kernel: ffff8806231bdcc8 0000000000000086 0000000000000000 ffffffffa0498940
Mar 19 18:03:43  kernel: ffff8806231bdc38 ffffffff810149b9 ffff8806231bdc78 ffffffff810a70a1
Mar 19 18:03:43  kernel: ffff880624317ab8 ffff8806231bdfd8 000000000000fbc8 ffff880624317ab8
Mar 19 18:03:43  kernel: Call Trace:
Mar 19 18:03:43  kernel: [<ffffffffa0498940>] ? rpc_execute+0x50/0xa0 [sunrpc]
Mar 19 18:03:43  kernel: [<ffffffff810149b9>] ? read_tsc+0x9/0x20
Mar 19 18:03:43  kernel: [<ffffffff810a70a1>] ? ktime_get_ts+0xb1/0xf0
Mar 19 18:03:43  kernel: [<ffffffff810a70a1>] ? ktime_get_ts+0xb1/0xf0
Mar 19 18:03:43  kernel: [<ffffffff8111f930>] ? sync_page+0x0/0x50
Mar 19 18:03:43  kernel: [<ffffffff815280a3>] io_schedule+0x73/0xc0
Mar 19 18:03:43  kernel: [<ffffffff8111f96d>] sync_page+0x3d/0x50
Mar 19 18:03:43  kernel: [<ffffffff81528b6f>] __wait_on_bit+0x5f/0x90
Mar 19 18:03:43  kernel: [<ffffffff8111fba3>] wait_on_page_bit+0x73/0x80
Mar 19 18:03:43  kernel: [<ffffffff8109b320>] ? wake_bit_function+0x0/0x50
Mar 19 18:03:43  kernel: [<ffffffff81135bf5>] ? pagevec_lookup_tag+0x25/0x40
Mar 19 18:03:43  kernel: [<ffffffff8111ffcb>] wait_on_page_writeback_range+0xfb/0x190
Mar 19 18:03:43  kernel: [<ffffffff81120198>] filemap_write_and_wait_range+0x78/0x90
Mar 19 18:03:43  kernel: [<ffffffff811baa3e>] vfs_fsync_range+0x7e/0x100
Mar 19 18:03:43  kernel: [<ffffffff811bab2d>] vfs_fsync+0x1d/0x20
Mar 19 18:03:43  kernel: [<ffffffff811bab6e>] do_fsync+0x3e/0x60
Mar 19 18:03:43  kernel: [<ffffffff811babc0>] sys_fsync+0x10/0x20
Mar 19 18:03:43  kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b