Kernel panic occurred in third party [fp_gpl] module's code path due to soft lockup

Solution In Progress - Updated -

Issue

Kernel panic occurred in third party [fp_gpl] module's code path due to soft lockup.

        CPUS: 8
        DATE: Tue Jun 25 21:55:27 2019
      UPTIME: 20 days, 15:13:36
LOAD AVERAGE: 4.15, 2.62, 2.36
       TASKS: 1094
    NODENAME: host-name
     RELEASE: 3.10.0-693.11.1.el7.x86_64
     VERSION: #1 SMP Fri Oct 27 05:39:05 EDT 2017
     MACHINE: x86_64  (2094 Mhz)
      MEMORY: 64 GB
       PANIC: "Kernel panic - not syncing: softlockup: hung tasks"
         PID: 2092
     COMMAND: "in:imjournal"
        TASK: ffff880ff26faf70  [THREAD_INFO: ffff880ff1b98000]
         CPU: 3
       STATE: TASK_RUNNING (PANIC)
crash> log
[..]
[1782915.503215] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [in:imjournal:2092]
[1782915.504215] Modules linked in: iscsi_target_mod target_core_mod ip6_tables iptable_filter drbg ansi_cprng dm_crypt ext4 mbcache jbd2 fp_rcfs(POE) gfs2 dlm fp_fe(POE) fp_audit(POE) fp_delegation(POE) fp_ac(POE) fp_proc(POE) fp_core(POE) fp_syscall(POE) fp_gpl(OE) hipsec(OE) schook(OE) vmw_vsock_vmci_transport vsock sb_edac edac_core iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper vmw_balloon ablk_helper cryptd joydev pcspkr sg shpchp vmw_vmci nfit i2c_piix4 libnvdimm parport_pc parport nfsd nfs_acl lockd grace binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix ahci libahci drm crct10dif_pclmul crct10dif_common
[1782915.504376]  libata crc32c_intel serio_raw vmxnet3 i2c_core vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod
[1782915.504402] CPU: 3 PID: 2092 Comm: in:imjournal Tainted: P        W  OE  ------------   3.10.0-693.11.1.el7.x86_64 #1
[1782915.504404] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
[1782915.504405] task: ffff880ff26faf70 ti: ffff880ff1b98000 task.ti: ffff880ff1b98000
[1782915.504407] RIP: 0010:[<ffffffffc0508464>]  [<ffffffffc0508464>] __PK_rawVnodeGetVfsmount+0x18/0x104 [fp_gpl]
[1782915.504420] RSP: 0018:ffff880ff1b9b9d0  EFLAGS: 00000287
[1782915.504421] RAX: ffff880ff89af800 RBX: ffff88065df7a038 RCX: ffff880765d10d80
[1782915.504422] RDX: ffff880ff1b9ba48 RSI: ffff880ffcae6540 RDI: ffff880765d10dd0
[1782915.504423] RBP: ffff880ff1b9b9e0 R08: ffff880ff9344300 R09: ffff880ff9344320
[1782915.504424] R10: ffff880ffa6b0b40 R11: 0001f972cd5f2d11 R12: ffff880ff1b9b960
[1782915.504425] R13: 00000000b158c6b7 R14: 0000000000000000 R15: ffffffff81328cf2
[1782915.504426] FS:  00007f5491991700(0000) GS:ffff88103fcc0000(0000) knlGS:0000000000000000
[1782915.504428] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1782915.504428] CR2: 00007f54960fd000 CR3: 0000000ff2f04000 CR4: 00000000000407e0
[1782915.504469] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1782915.504501] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1782915.504503] Stack:
[1782915.504503]  ffffffff8120b1e7 ffff880ffb568944 ffff880ff1b9ba28 ffffffffc0508618
[1782915.504505]  ffff880ff9344320 ffff880ffa6b0b40 00000000b158c6b7 ffff880ff1b9bab0
[1782915.504507]  ffff880fe2eaece8 ffff88003b7789c0 ffff880ffcae6540 ffff880ff1b9ba80
[1782915.504509] Call Trace:
[1782915.504520]  [<ffffffff8120b1e7>] ? path_get+0x27/0x30
[1782915.504526]  [<ffffffffc0508618>] PK_rawVnodeGetVfsmount+0xc8/0xd2 [fp_gpl]
[1782915.504545]  [<ffffffffc06f794d>] PK_vfscallHookGetParams+0x4e5/0x1590 [fp_core]
[1782915.504551]  [<ffffffffc06a4145>] PK_feVfscallIoPrevRename+0x287/0x510 [fp_fe]
[1782915.504554]  [<ffffffffc06933bf>] PK_feVfscallPrevOperations+0x2f/0x46 [fp_fe]
[1782915.504563]  [<ffffffffc06fb698>] PK_vfscallHookStackPrevOperation+0x9c/0xfc [fp_core]
[1782915.504572]  [<ffffffffc06fd31d>] PK_vfscalliRename2+0x71/0x10c [fp_core]
[1782915.504686]  [<ffffffffc02a6d60>] ? xfs_vn_follow_link+0xa0/0xa0 [xfs]
[1782915.504701]  [<ffffffffc02a6d60>] ? xfs_vn_follow_link+0xa0/0xa0 [xfs]
[1782915.504787]  [<ffffffffc05513f7>] PK_vfscallStub_iRename2+0x69/0xc0 [fp_syscall]
[1782915.504829]  [<ffffffffc0551472>] PK_wrapper0x00PK_vfscallStub_iRename2+0x24/0x26 [fp_syscall]
[1782915.504831]  [<ffffffff812115c0>] vfs_rename+0x5a0/0x930
[1782915.504872]  [<ffffffffc055144e>] ? PK_vfscallStub_iRename2+0xc0/0xc0 [fp_syscall]
[1782915.504874]  [<ffffffff812128c3>] SYSC_renameat2+0x503/0x5a0
[1782915.504882]  [<ffffffffc06f021a>] ? PK_readUnlock+0xe/0x10 [fp_core]
[1782915.504891]  [<ffffffffc04f0937>] ? hip_sys_rename+0x37/0xe0 [hipsec]
[1782915.504898]  [<ffffffffc06f01fa>] ? PK_writeUnlock+0xe/0x10 [fp_core]
[1782915.504904]  [<ffffffffc0665bbe>] ? PK_pextRele+0xa2/0xa8 [fp_proc]
[1782915.504911]  [<ffffffffc06f0f2e>] ? PK_moduleMsgSend.part.13+0x20/0x24 [fp_core]
[1782915.504918]  [<ffffffffc06f0f6d>] ? PK_moduleMsgSend+0x3b/0x40 [fp_core]
[1782915.504921]  [<ffffffffc06a38cd>] ? PK_feSyscallIoPrevRename+0x113/0x54c [fp_fe]
[1782915.504924]  [<ffffffff812137d0>] ? SyS_renameat+0x20/0x20
[1782915.504926]  [<ffffffff812137ae>] SyS_renameat2+0xe/0x10
[1782915.504927]  [<ffffffff812137ee>] SyS_rename+0x1e/0x20
[1782915.504936]  [<ffffffffc0496a31>] schook_stub_rename+0x91/0xa0 [schook]
[1782915.504938]  [<ffffffffc04969a0>] ? schook_stub_chmod+0xa0/0xa0 [schook]
[1782915.504946]  [<ffffffffc0702e2b>] PK_sysRename+0x177/0x1ae [fp_core]
[1782915.504968]  [<ffffffffc0540062>] PK_stubRename+0x40/0x6e [fp_syscall]
[1782915.504973]  [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
[1782915.504974] Code: fe 48 8b 3d 1f 6f 00 00 48 89 e5 e8 eb fe ff ff 5d c3 90 66 66 66 66 90 55 48 89 e5 53 48 83 ec 08 48 89 fb 4c 8d 47 e0 eb 20 90 <48> 8b 4b 30 48 8d 43 30 48 8d 7b e0 48 39 c1 0f 84 89 00 00 00

Environment

  • Red Hat Enterprise Linux 7
  • softlockup_panic is enabled

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content