RHEL 6 nfs_migrate_page() deadlocks when truncate_inode_pages_range() tries to lock page
Issue
- System or application hung when using NFS.
-
Our application reads files off an NFS mount and also writes data to the nfs. In this particular case, it was reading the data from one NAS device and writing some data to another nas device. The application runs without any problems on CentOS 5.6 and RHEL 5.6 however when we run the application on RHEL 6.1, one of the java threads of our application goes into uninterruptible sleep mode and from that point onwards, the application cannot function properly. The process that shows up as the one in uninterruptible sleep mode is shown below with its stack trace:
PID: 7465 TASK: ffff880f3f4ba080 CPU: 12 COMMAND: "java" #0 [ffff880f3f6cb3b8] schedule at ffffffff814dabd9 #1 [ffff880f3f6cb480] io_schedule at ffffffff814db3c3 #2 [ffff880f3f6cb4a0] sync_page at ffffffff8110d35d #3 [ffff880f3f6cb4b0] __wait_on_bit_lock at ffffffff814dbada #4 [ffff880f3f6cb500] __lock_page at ffffffff8110d2f7 #5 [ffff880f3f6cb560] truncate_inode_pages_range at ffffffff81124a8c #6 [ffff880f3f6cb650] truncate_inode_pages at ffffffff81124ab5 #7 [ffff880f3f6cb660] generic_delete_inode at ffffffff8118d20e #8 [ffff880f3f6cb690] generic_drop_inode at ffffffff8118d2b5 #9 [ffff880f3f6cb6b0] iput at ffffffff8118c122 #10 [ffff880f3f6cb6d0] nfs_dentry_iput at ffffffffa04b309e [nfs] #11 [ffff880f3f6cb6f0] dentry_iput at ffffffff81188eac #12 [ffff880f3f6cb710] d_kill at ffffffff81189021 #13 [ffff880f3f6cb730] dput at ffffffff8118a90c #14 [ffff880f3f6cb750] path_put at ffffffff8117f5da #15 [ffff880f3f6cb770] __put_nfs_open_context at ffffffffa04b9f62 [nfs] #16 [ffff880f3f6cb7b0] put_nfs_open_context at ffffffffa04ba040 [nfs] #17 [ffff880f3f6cb7c0] nfs_clear_request at ffffffffa04c2fbc [nfs] #18 [ffff880f3f6cb7f0] nfs_free_request at ffffffffa04c2ffa [nfs] #19 [ffff880f3f6cb810] kref_put at ffffffff812662c7 #20 [ffff880f3f6cb830] nfs_release_request at ffffffffa04c2f59 [nfs] #21 [ffff880f3f6cb840] nfs_find_and_lock_request at ffffffffa04c73e3 [nfs] #22 [ffff880f3f6cb880] nfs_migrate_page at ffffffffa04c74b1 [nfs] #23 [ffff880f3f6cb8c0] move_to_new_page at ffffffff8115f9e8 #24 [ffff880f3f6cb900] migrate_pages at ffffffff8115fef4 #25 [ffff880f3f6cb9b0] compact_zone at ffffffff811559bb #26 [ffff880f3f6cba70] compact_zone_order at ffffffff81155ee1 #27 [ffff880f3f6cbb20] try_to_compact_pages at ffffffff8115603c #28 [ffff880f3f6cbb90] __alloc_pages_nodemask at ffffffff8111ff95 #29 [ffff880f3f6cbcb0] alloc_pages_vma at ffffffff8115475a #30 [ffff880f3f6cbd00] do_huge_pmd_wp_page at ffffffff8116cb18 #31 [ffff880f3f6cbdb0] handle_mm_fault at ffffffff81138394 #32 [ffff880f3f6cbe00] __do_page_fault at ffffffff810414e9 #33 [ffff880f3f6cbf20] do_page_fault at ffffffff814e054e #34 [ffff880f3f6cbf50] page_fault at ffffffff814dd8d5 RIP: 00007f06f93838b3 RSP: 00007f0294a56070 RFLAGS: 00010202 RAX: 0000000000000000 RBX: 00007f0326b782b0 RCX: 00007f0326b78300 RDX: 0000000000000000 RSI: 00007f0326b782c8 RDI: 00007f0397d65b60 RBP: 00007f06bfefee48 R8: 00007ec34cbdf000 R9: 0000000000000020 R10: 00007f04e3876d50 R11: 00007f0326b78268 R12: 00007f0326b78288 R13: 00007f0326b782c8 R14: 00007f04e3876d18 R15: 00007f0248015800 ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b
Environment
- Red Hat Enterprise Linux 6.1
- Kernel 2.6.32-131.0.15.el6.x86_64
- Kernel 2.6.32-131.6.1.el6.x86_64
- kernel 2.6.32-220.el6.x86_64
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.