Kernel panic with blocked tasks due to mmfslinux module.
Issue
- The system crashed via hung_task panic with the following call traces from the stalled task.
INFO: task nmon:38387 blocked for more than 120 seconds.
Tainted: GW -------------- 2.6.32-573.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nmon D 0000000000000000 0 38387 1 0x00000080
ffff8805bab03738 0000000000000086 0000000000000000 ffff88085c4559c0
0000000000000000 0000000000000008 0006bf321f1ecc9f ffffffff8106faa4
ffff8805bab036c8 000000017110c719 ffff880694dab068 ffff8805bab03fd8
Call Trace:
[<ffffffff8106faa4>] ? enqueue_task_fair+0x64/0x100
[<ffffffffa0873264>] cxiWaitEventWait+0x144/0x230 [mmfslinux]
[<ffffffff810672b0>] ? default_wake_function+0x0/0x20
[<ffffffffa096c4e1>] _ZN6ThCond12internalWaitEP16KernelSynchStatejPv+0x61/0x200 [mmfs26]
[<ffffffffa096d69f>] ? _ZN6ThCond5kWaitEiPKc+0x15f/0x2d0 [mmfs26]
[<ffffffffa096cf45>] ? _ZN14BaseMutexClass11releaseSlowEv+0x195/0x1f0 [mmfs26]
[<ffffffffa08be99d>] ? _ZN13KernelMailbox21sendToDaemonWithReplyEv+0x21d/0x300 [mmfs26]
[<ffffffffa08e21cb>] ? _Z12kShHashFetchP15KernelOperationP8CacheObjtsiPvij+0x1cb/0x220 [mmfs26]
[<ffffffffa08c8552>] ? _Z11lockGetattrP15KernelOperationP13gpfsVfsData_tR7FileUIDiR13LookupDetailsR10WhatLockedRP8OpenFileP10cxiVattr_t+0xeb2/0x1550 [mmfs26]
[<ffffffffa08e120b>] ? _ZN13SharedHashTab13lookupAndHoldEPP8CacheObjRK6ObjKeyP11StripeGroupiP15KernelOperation+0x76b/0x9a0 [mmfs26]
[<ffffffff810a76be>] ? down+0x2e/0x50
[<ffffffff810a760f>] ? up+0x2f/0x50
[<ffffffff810a76be>] ? down+0x2e/0x50
[<ffffffffa08ef8d8>] ? _Z11kSFSGetattrP15KernelOperationP13gpfsVfsData_tP10gpfsNode_tiP10cxiVattr_t+0x198/0x4a0 [mmfs26]
[<ffffffff810a760f>] ? up+0x2f/0x50
[<ffffffffa08f932e>] ? _Z8getVattrP15KernelOperationP13gpfsVfsData_tP10gpfsNode_tP10cxiVattr_ti+0x4e/0x100 [mmfs26]
[<ffffffffa09178f1>] ? _Z11gpfsGetattrP13gpfsVfsData_tP9cxiNode_tP10cxiVattr_ti+0x201/0x2a0 [mmfs26]
[<ffffffffa087488b>] ? gpfs_i_getattr_internal+0x6b/0x110 [mmfslinux]
[<ffffffff811b43d0>] ? mntput_no_expire+0x30/0x110
[<ffffffff8119f776>] ? final_putname+0x26/0x50
[<ffffffff8119fd7b>] ? putname+0x2b/0x40
[<ffffffff811a4352>] ? user_path_at+0x62/0xa0
[<ffffffffa088a57a>] ? gpfs_i_getattr+0x4a/0xc0 [mmfslinux]
[<ffffffff811976d1>] ? vfs_getattr+0x51/0x80
[<ffffffff81197514>] ? cp_new_stat+0xe4/0x100
[<ffffffff81197764>] ? vfs_fstatat+0x64/0xa0
[<ffffffff811978cb>] ? vfs_stat+0x1b/0x20
[<ffffffff811978f4>] ? sys_newstat+0x24/0x50
[<ffffffff810e8ab7>] ? audit_syscall_entry+0x1d7/0x200
[<ffffffff810e88ae>] ? __audit_syscall_exit+0x25e/0x290
[<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
Kernel panic - not syncing: hung_task: blocked tasks
Pid: 175, comm: khungtaskd Tainted: GW -------------- 2.6.32-573.el6.x86_64 #1
Call Trace:
[<ffffffff81537a84>] ? panic+0xa7/0x16f
[<ffffffff810ecd94>] ? watchdog+0x254/0x290
[<ffffffff810ecb40>] ? watchdog+0x0/0x290
[<ffffffff810a101e>] ? kthread+0x9e/0xc0
[<ffffffff8100c28a>] ? child_rip+0xa/0x20
[<ffffffff810a0f80>] ? kthread+0x0/0xc0
[<ffffffff8100c280>] ? child_rip+0x0/0x20
- Task blocked in the uninterruptible (UN) state with the following call traces.
PID: 53354 TASK: ffff94493c166300 CPU: 5 COMMAND: "MessageListener"
#0 [ffff944bb44f2e20] __schedule at ffffffffa4d8c028
#1 [ffff944bb44f2e88] schedule at ffffffffa4d8c3f9
#2 [ffff944bb44f2e98] cxiWaitEventWait at ffffffffc07209c1 [mmfslinux]
#3 [ffff944bb44f2f60] _ZN6ThCond12internalWaitEP16KernelSynchStatejPv at ffffffffc08a2950 [mmfs26]
#4 [ffff944bb44f2fb0] _ZN6ThCond5kWaitEiPKc at ffffffffc08a401b [mmfs26]
#5 [ffff944bb44f3060] _ZN13KernelMailbox21sendToDaemonWithReplyEv at ffffffffc07c3a15 [mmfs26]
#6 [ffff944bb44f30a0] _Z14kSFSAcquireBRTP8OpenFileRK9ByteRangeS3_N6BRLock10BRLockModeEi at ffffffffc07f634c [mmfs26]
#7 [ffff944bb44f3100] _ZN3BRL7brLockMERK9ByteRangeN6BRLock10BRLockModeEPPvixPS0_PS4_ at ffffffffc08a0dc1 [mmfs26]
#8 [ffff944bb44f3280] _ZN3BRL6brLockERK9ByteRangeN6BRLock10BRLockModeEPPvixPS0_PS4_ at ffffffffc08a181a [mmfs26]
#9 [ffff944bb44f32e0] _ZN8Dirblock12daccessEntryEP8OpenFileR13DirHashedName at ffffffffc07c630d [mmfs26]
#10 [ffff944bb44f3350] _Z9DirSearchP15KernelOperationP8OpenFilePN5LkObj12LockModeEnumEP10gpfsNode_tPvP8DirblockPKcjiPPcP7FileUIDPjP13LookupDetailsSH_P at ffffffffc07cada5 [mmfs26]
#11 [ffff944bb44f34c0] _Z10lookupFileP15KernelOperationP13gpfsVfsData_tP10gpfsNode_tPvPcjjP10ext_cred_tP7FileUIDPjP13LookupDetailsSB_SB_PP8OpenFilePiS at ffffffffc07cda40 [mmfs26]
#12 [ffff944bb44f35b0] _ZN10gpfsNode_t6lookupEP15KernelOperationP13gpfsVfsData_tPS_PvPcjjP10ext_cred_tPS5_PS4_PjS9_PiS6_SC_Px at ffffffffc07d27e0 [mmfs26]
#13 [ffff944bb44f36e0] _Z10gpfsLookupP13gpfsVfsData_tPvP9cxiNode_tS1_S1_PcjPS1_PS3_PyP10cxiVattr_tPjP10ext_cred_tjS5_PiS4_SD_ at ffffffffc0848c2b [mmfs26]
#14 [ffff944bb44f3820] gpfs_i_lookup at ffffffffc0717fa6 [mmfslinux]
#15 [ffff944bb44f39b0] lookup_real at ffffffffa4859023
#16 [ffff944bb44f39d0] __lookup_hash at ffffffffa4859a42
#17 [ffff944bb44f3a00] lookup_slow at ffffffffa4d83175
#18 [ffff944bb44f3a38] path_lookupat at ffffffffa485d62e
#19 [ffff944bb44f3ad0] filename_lookup at ffffffffa485d68b
#20 [ffff944bb44f3b08] user_path_at_empty at ffffffffa4861397
#21 [ffff944bb44f3bd8] user_path_at at ffffffffa4861401
#22 [ffff944bb44f3be8] bmhook_get_file_info_from_upath at ffffffffc0ce12aa [bmhook]
#23 [ffff944bb44f3c18] tmbpf_get_file_info_from_upath at ffffffffc0cdbb3e [bmhook]
#24 [ffff944bb44f3cd0] ttwu_do_activate at ffffffffa46d761f
#25 [ffff944bb44f3cf8] try_to_wake_up at ffffffffa46db022
#26 [ffff944bb44f3d48] wake_up_q at ffffffffa46db29b
#27 [ffff944bb44f3da0] bmhook_prehook_handler at ffffffffc0cec73c [bmhook]
#28 [ffff944bb44f3e38] tmhook_invoke_prehooks at ffffffffc0c1bc4c [tmhook]
#29 [ffff944bb44f3e78] tmhook_handler at ffffffffc0c1bec9 [tmhook]
#30 [ffff944bb44f3f50] tracesys at ffffffffa4d9a226 (via system_call)
RIP: 00007f54447e2efd RSP: 00007f530b2fe700 RFLAGS: 00200293
RAX: ffffffffffffffda RBX: 00007f530b2fe830 RCX: ffffffffffffffff
RDX: 00000000000001b6 RSI: 0000000000000241 RDI: 00007f530b2fe830
RBP: 00000000000001b6 R8: 0000000000000000 R9: 00007f530b2fe400
R10: 0000000063456a4c R11: 0000000000200293 R12: 0000000000000241
R13: 00007f530b2fe830 R14: 00007f530b2fe830 R15: 00007f54430cc1c0
ORIG_RAX: 0000000000000002 CS: 0033 SS: 002b
Environment
- Red Hat Enterprise Linux 6
- Red Hat Enterprise Linux 7
- Red Hat Enterprise Linux 8
- IBM GPFS
- Out-of-tree (O) kernel module:
[mmfslinux]
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.