Kernel panic with blocked tasks due to mmfslinux module.

Solution Verified - Updated -

Issue

  • The system crashed via hung_task panic with the following call traces from the stalled task.
INFO: task nmon:38387 blocked for more than 120 seconds.
      Tainted: GW  --------------   2.6.32-573.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nmon          D 0000000000000000     0 38387      1 0x00000080
 ffff8805bab03738 0000000000000086 0000000000000000 ffff88085c4559c0
 0000000000000000 0000000000000008 0006bf321f1ecc9f ffffffff8106faa4
 ffff8805bab036c8 000000017110c719 ffff880694dab068 ffff8805bab03fd8
Call Trace:
 [<ffffffff8106faa4>] ? enqueue_task_fair+0x64/0x100
 [<ffffffffa0873264>] cxiWaitEventWait+0x144/0x230 [mmfslinux]
 [<ffffffff810672b0>] ? default_wake_function+0x0/0x20
 [<ffffffffa096c4e1>] _ZN6ThCond12internalWaitEP16KernelSynchStatejPv+0x61/0x200 [mmfs26]
 [<ffffffffa096d69f>] ? _ZN6ThCond5kWaitEiPKc+0x15f/0x2d0 [mmfs26]
 [<ffffffffa096cf45>] ? _ZN14BaseMutexClass11releaseSlowEv+0x195/0x1f0 [mmfs26]
 [<ffffffffa08be99d>] ? _ZN13KernelMailbox21sendToDaemonWithReplyEv+0x21d/0x300 [mmfs26]
 [<ffffffffa08e21cb>] ? _Z12kShHashFetchP15KernelOperationP8CacheObjtsiPvij+0x1cb/0x220 [mmfs26]
 [<ffffffffa08c8552>] ? _Z11lockGetattrP15KernelOperationP13gpfsVfsData_tR7FileUIDiR13LookupDetailsR10WhatLockedRP8OpenFileP10cxiVattr_t+0xeb2/0x1550 [mmfs26]
 [<ffffffffa08e120b>] ? _ZN13SharedHashTab13lookupAndHoldEPP8CacheObjRK6ObjKeyP11StripeGroupiP15KernelOperation+0x76b/0x9a0 [mmfs26]
 [<ffffffff810a76be>] ? down+0x2e/0x50
 [<ffffffff810a760f>] ? up+0x2f/0x50
 [<ffffffff810a76be>] ? down+0x2e/0x50
 [<ffffffffa08ef8d8>] ? _Z11kSFSGetattrP15KernelOperationP13gpfsVfsData_tP10gpfsNode_tiP10cxiVattr_t+0x198/0x4a0 [mmfs26]
 [<ffffffff810a760f>] ? up+0x2f/0x50
 [<ffffffffa08f932e>] ? _Z8getVattrP15KernelOperationP13gpfsVfsData_tP10gpfsNode_tP10cxiVattr_ti+0x4e/0x100 [mmfs26]
 [<ffffffffa09178f1>] ? _Z11gpfsGetattrP13gpfsVfsData_tP9cxiNode_tP10cxiVattr_ti+0x201/0x2a0 [mmfs26]
 [<ffffffffa087488b>] ? gpfs_i_getattr_internal+0x6b/0x110 [mmfslinux]
 [<ffffffff811b43d0>] ? mntput_no_expire+0x30/0x110
 [<ffffffff8119f776>] ? final_putname+0x26/0x50
 [<ffffffff8119fd7b>] ? putname+0x2b/0x40
 [<ffffffff811a4352>] ? user_path_at+0x62/0xa0
 [<ffffffffa088a57a>] ? gpfs_i_getattr+0x4a/0xc0 [mmfslinux]
 [<ffffffff811976d1>] ? vfs_getattr+0x51/0x80
 [<ffffffff81197514>] ? cp_new_stat+0xe4/0x100
 [<ffffffff81197764>] ? vfs_fstatat+0x64/0xa0
 [<ffffffff811978cb>] ? vfs_stat+0x1b/0x20
 [<ffffffff811978f4>] ? sys_newstat+0x24/0x50
 [<ffffffff810e8ab7>] ? audit_syscall_entry+0x1d7/0x200
 [<ffffffff810e88ae>] ? __audit_syscall_exit+0x25e/0x290
 [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
Kernel panic - not syncing: hung_task: blocked tasks
Pid: 175, comm: khungtaskd Tainted: GW  -------------- 2.6.32-573.el6.x86_64 #1
Call Trace:
 [<ffffffff81537a84>] ? panic+0xa7/0x16f
 [<ffffffff810ecd94>] ? watchdog+0x254/0x290
 [<ffffffff810ecb40>] ? watchdog+0x0/0x290
 [<ffffffff810a101e>] ? kthread+0x9e/0xc0
 [<ffffffff8100c28a>] ? child_rip+0xa/0x20
 [<ffffffff810a0f80>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
  • Task blocked in the uninterruptible (UN) state with the following call traces.
PID: 53354  TASK: ffff94493c166300  CPU: 5   COMMAND: "MessageListener"
 #0 [ffff944bb44f2e20] __schedule at ffffffffa4d8c028
 #1 [ffff944bb44f2e88] schedule at ffffffffa4d8c3f9
 #2 [ffff944bb44f2e98] cxiWaitEventWait at ffffffffc07209c1 [mmfslinux]
 #3 [ffff944bb44f2f60] _ZN6ThCond12internalWaitEP16KernelSynchStatejPv at ffffffffc08a2950 [mmfs26]
 #4 [ffff944bb44f2fb0] _ZN6ThCond5kWaitEiPKc at ffffffffc08a401b [mmfs26]
 #5 [ffff944bb44f3060] _ZN13KernelMailbox21sendToDaemonWithReplyEv at ffffffffc07c3a15 [mmfs26]
 #6 [ffff944bb44f30a0] _Z14kSFSAcquireBRTP8OpenFileRK9ByteRangeS3_N6BRLock10BRLockModeEi at ffffffffc07f634c [mmfs26]
 #7 [ffff944bb44f3100] _ZN3BRL7brLockMERK9ByteRangeN6BRLock10BRLockModeEPPvixPS0_PS4_ at ffffffffc08a0dc1 [mmfs26]
 #8 [ffff944bb44f3280] _ZN3BRL6brLockERK9ByteRangeN6BRLock10BRLockModeEPPvixPS0_PS4_ at ffffffffc08a181a [mmfs26]
 #9 [ffff944bb44f32e0] _ZN8Dirblock12daccessEntryEP8OpenFileR13DirHashedName at ffffffffc07c630d [mmfs26]
#10 [ffff944bb44f3350] _Z9DirSearchP15KernelOperationP8OpenFilePN5LkObj12LockModeEnumEP10gpfsNode_tPvP8DirblockPKcjiPPcP7FileUIDPjP13LookupDetailsSH_P at ffffffffc07cada5 [mmfs26]
#11 [ffff944bb44f34c0] _Z10lookupFileP15KernelOperationP13gpfsVfsData_tP10gpfsNode_tPvPcjjP10ext_cred_tP7FileUIDPjP13LookupDetailsSB_SB_PP8OpenFilePiS at ffffffffc07cda40 [mmfs26]
#12 [ffff944bb44f35b0] _ZN10gpfsNode_t6lookupEP15KernelOperationP13gpfsVfsData_tPS_PvPcjjP10ext_cred_tPS5_PS4_PjS9_PiS6_SC_Px at ffffffffc07d27e0 [mmfs26]
#13 [ffff944bb44f36e0] _Z10gpfsLookupP13gpfsVfsData_tPvP9cxiNode_tS1_S1_PcjPS1_PS3_PyP10cxiVattr_tPjP10ext_cred_tjS5_PiS4_SD_ at ffffffffc0848c2b [mmfs26]
#14 [ffff944bb44f3820] gpfs_i_lookup at ffffffffc0717fa6 [mmfslinux]
#15 [ffff944bb44f39b0] lookup_real at ffffffffa4859023
#16 [ffff944bb44f39d0] __lookup_hash at ffffffffa4859a42
#17 [ffff944bb44f3a00] lookup_slow at ffffffffa4d83175
#18 [ffff944bb44f3a38] path_lookupat at ffffffffa485d62e
#19 [ffff944bb44f3ad0] filename_lookup at ffffffffa485d68b
#20 [ffff944bb44f3b08] user_path_at_empty at ffffffffa4861397
#21 [ffff944bb44f3bd8] user_path_at at ffffffffa4861401
#22 [ffff944bb44f3be8] bmhook_get_file_info_from_upath at ffffffffc0ce12aa [bmhook]
#23 [ffff944bb44f3c18] tmbpf_get_file_info_from_upath at ffffffffc0cdbb3e [bmhook]
#24 [ffff944bb44f3cd0] ttwu_do_activate at ffffffffa46d761f
#25 [ffff944bb44f3cf8] try_to_wake_up at ffffffffa46db022
#26 [ffff944bb44f3d48] wake_up_q at ffffffffa46db29b
#27 [ffff944bb44f3da0] bmhook_prehook_handler at ffffffffc0cec73c [bmhook]
#28 [ffff944bb44f3e38] tmhook_invoke_prehooks at ffffffffc0c1bc4c [tmhook]
#29 [ffff944bb44f3e78] tmhook_handler at ffffffffc0c1bec9 [tmhook]
#30 [ffff944bb44f3f50] tracesys at ffffffffa4d9a226 (via system_call)
    RIP: 00007f54447e2efd  RSP: 00007f530b2fe700  RFLAGS: 00200293
    RAX: ffffffffffffffda  RBX: 00007f530b2fe830  RCX: ffffffffffffffff
    RDX: 00000000000001b6  RSI: 0000000000000241  RDI: 00007f530b2fe830
    RBP: 00000000000001b6   R8: 0000000000000000   R9: 00007f530b2fe400
    R10: 0000000063456a4c  R11: 0000000000200293  R12: 0000000000000241
    R13: 00007f530b2fe830  R14: 00007f530b2fe830  R15: 00007f54430cc1c0
    ORIG_RAX: 0000000000000002  CS: 0033  SS: 002b

Environment

  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • IBM GPFS
  • Out-of-tree (O) kernel module: [mmfslinux]

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content