Kernel panic at vx_ioship_resend_pkts+0xc0

Issue

• Kernel panic with below logs:

[1094312.443222] Could not insert probe at compat_sys_setfsgid+0: -2
[1094312.457444] Could not insert probe at compat_sys_setfsgid+0: -2
[1517168.448748] BUG: unable to handle kernel NULL pointer dereference at           (null)
[1517168.461252] IP: [<ffffffffc080e0c0>] vx_ioship_resend_pkts+0xc0/0x250 [llt]
[1517168.469368] PGD 0
[1517168.474733] Oops: 0002 [#1] SMP
[1517168.480723] Modules linked in: vxodm(POE) vxglm(POE) vxfen(POE) vxgms(POE) dmpaa(POE) gab(POE) vxspec(POE) vxio(POE) vxdmp(POE) vxcafs(POE) vxportal(POE) fdd(POE) vxfs(POE) llt(POE) rdma_cm iw_cm ib_cm ib_core amf(POE) nf_conntrack_netlink nf_conntrack tcp_diag inet_diag veki(POE) nfit libnvdimm iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw parport_pc pcspkr gf128mul glue_helper ablk_helper cryptd parport i2c_piix4 binfmt_misc ip_tables xfs libcrc32c nvme crct10dif_pclmul crct10dif_common crc32c_intel ena serio_raw nvme_core sunrpc dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vxodm]
[1517168.540679] CPU: 0 PID: 11639 Comm: vx_ioship_resen Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
[1517168.554040] Hardware name: Amazon EC2 m5.large/, BIOS 1.0 10/16/2017
[1517168.560842] task: ffff918928b06300 ti: ffff91892589c000 task.ti: ffff91892589c000
[1517168.571783] RIP: 0010:[<ffffffffc080e0c0>]  [<ffffffffc080e0c0>] vx_ioship_resend_pkts+0xc0/0x250 [llt]
[1517168.583767] RSP: 0018:ffff91892589fdc8  EFLAGS: 00010282
[1517168.590094] RAX: 0000000000000000 RBX: ffff9187c85eec00 RCX: ffff91880a546d80
[1517168.600903] RDX: 0000000000017c4f RSI: 0000000000000000 RDI: ffff9187c85eed00
[1517168.611794] RBP: ffff91892589fe18 R08: ffff91892589c000 R09: 0000000000000001
[1517168.622611] R10: 0000000000000001 R11: 0000000000000008 R12: ffff918936d07b40
[1517168.633432] R13: ffff9189a45eae00 R14: 0000000000000001 R15: ffff9189a45eae00
[1517168.644277] FS:  0000000000000000(0000) GS:ffff9189ad000000(0000) knlGS:0000000000000000
[1517168.655590] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1517168.662171] CR2: 0000000000000000 CR3: 0000000152810000 CR4: 00000000007606f0
[1517168.672994] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1517168.683855] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[1517168.694672] PKRU: 00000000
[1517168.699703] Call Trace:
[1517168.704614]  [<ffffffffb538a010>] ? schedule_timeout+0x170/0x2d0
[1517168.711298]  [<ffffffffc0810449>] vx_ioship_resend_internal+0x69/0xa0 [llt]
[1517168.718431]  [<ffffffffc08104bb>] vx_ioship_node_resend+0x3b/0x50 [llt]
[1517168.725438]  [<ffffffffc081118f>] vx_ioship_process_resend+0x1bf/0x210 [llt]
[1517168.736239]  [<ffffffffc0810fd0>] ? vx_ioship_cleanup_tgtnode+0x180/0x180 [llt]
[1517168.747152]  [<ffffffffb4cc5f91>] kthread+0xd1/0xe0
[1517168.753248]  [<ffffffffb4cc5ec0>] ? insert_kthread_work+0x40/0x40
[1517168.759920]  [<ffffffffb5399df7>] ret_from_fork_nospec_begin+0x21/0x21
[1517168.766808]  [<ffffffffb4cc5ec0>] ? insert_kthread_work+0x40/0x40
[1517168.773487] Code: 44 24 08 00 00 00 00 49 c7 04 24 00 00 00 00 49 8b 44 24 50 48 8b 53 08 48 89 50 08 48 83 7b 38 00 0f 84 54 01 00 00 48 8b 43 40 <4c> 89 20 48 8b 43 40 49 89 44 24 08 4c 89 63 40 48 83 43 48 01
[1517168.797915] RIP  [<ffffffffc080e0c0>] vx_ioship_resend_pkts+0xc0/0x250 [llt]
[1517168.808856]  RSP <ffff91892589fdc8>
[1517168.814329] CR2: 0000000000000000

• Another pattern of logs:

28921369.328264] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[28921369.337410] IP: [<ffffffffc0867e9e>] vx_ioship_change_gen+0xee/0x190 [llt]
[28921369.343000] PGD 71db35067 PUD 721622067 PMD 0
[28921369.347938] Oops: 0000 [#1] SMP
[28921369.352327] Modules linked in: seos(POE) vxfen(POE) vxodm(POE) vxglm(POE) vxgms(POE) gab(POE) dmpaa(POE) vxspec(POE) vxio(POE) vxcafs(POE) vxportal(POE) fdd(POE) vxdmp(POE) llt(POE) amf(POE) rdma_cm vxfs(POE) iw_cm ib_cm ib_core veki(POE) ppdev sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel parport_pc parport i2c_piix4 lrw gf128mul glue_helper ablk_helper cryptd pcspkr nls_utf8 isofs loop binfmt_misc ip_tables xfs libcrc32c nvme crct10dif_pclmul crct10dif_common crc32c_intel ena nvme_core serio_raw
[28921369.391891] CPU: 3 PID: 12507 Comm: llt_ioship_rsnd Kdump: loaded Tainted: P           OE  ------------   3.10.0-1062.1.2.el7.x86_64 #1
[28921369.402585] Hardware name: Amazon EC2 m6i.2xlarge/, BIOS 1.0 10/16/2017
[28921369.408291] task: ffff96b0f9223150 ti: ffff96b101338000 task.ti: ffff96b101338000
[28921369.417600] RIP: 0010:[<ffffffffc0867e9e>]  [<ffffffffc0867e9e>] vx_ioship_change_gen+0xee/0x190 [llt]
[28921369.427182] RSP: 0018:ffff96b10133be08  EFLAGS: 00010286
[28921369.432126] RAX: ffff96adcd58e500 RBX: ffff96b0a929ac00 RCX: 0000000000000000
[28921369.440608] RDX: 0000000000000000 RSI: 000000000006d998 RDI: 000000000006de40
[28921369.449114] RBP: ffff96b10133be10 R08: ffff96b101338000 R09: 0000000000000001
[28921369.457605] R10: 0000000000000000 R11: 7fffffffffffffff R12: ffff96b0a929ad08
[28921369.466078] R13: ffff96b137bfeb00 R14: ffff96b137bfeb00 R15: 0000000000000001
[28921369.474551] FS:  0000000000000000(0000) GS:ffff96b15ecc0000(0000) knlGS:0000000000000000
[28921369.483373] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[28921369.488485] CR2: 0000000000000008 CR3: 000000057cfd2000 CR4: 00000000007606e0
[28921369.496979] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[28921369.505486] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[28921369.513991] PKRU: 00000000
[28921369.517969] Call Trace:
[28921369.521848]  [<ffffffffc086ac6e>] vx_ioship_resend_internal+0x8e/0xd0 [llt]
[28921369.530259]  [<ffffffffc086aceb>] vx_ioship_node_resend+0x3b/0x50 [llt]
[28921369.535657]  [<ffffffffc086bd7f>] vx_ioship_process_resend+0x20f/0x2a0 [llt]
[28921369.544067]  [<ffffffffc086bb70>] ? vx_ioship_cleanup_tgtnode+0x180/0x180 [llt]
[28921369.552593]  [<ffffffffbbec50d1>] kthread+0xd1/0xe0
[28921369.557364]  [<ffffffffbbec5000>] ? insert_kthread_work+0x40/0x40
[28921369.562603]  [<ffffffffbc58cd1d>] ret_from_fork_nospec_begin+0x7/0x21
[28921369.567953]  [<ffffffffbbec5000>] ? insert_kthread_work+0x40/0x40
[28921369.573157] Code: 48 8b 93 90 00 00 00 48 89 50 08 48 89 83 90 00 00 00 48 8b 43 38 48 83 83 98 00 00 00 01 48 85 c0 74 3a 48 8b 50 50 48 8b 7b 10 <48> 39 7a 08 73 2c 48 8b 10 48 85 d2 75 84 48 8b 50 08 48 89 53
[28921369.591818] RIP  [<ffffffffc0867e9e>] vx_ioship_change_gen+0xee/0x190 [llt]
[28921369.600265]  RSP <ffff96b10133be08>
[28921369.604530] CR2: 0000000000000008