RHEL7: kernel panic at vx_ioship_resend_pkts+0xc0

Issue

• Kernel panic with logs:

[8864293.896298] BUG: unable to handle kernel NULL pointer dereference at           (null)
[8864293.905859] IP: [<ffffffffc0a6f040>] vx_ioship_resend_pkts+0xc0/0x250 [llt]
[8864293.911915] PGD 80000006fc849067 PUD 9a2019067 PMD 0 
[8864293.917244] Oops: 0002 [#1] SMP 
[8864293.921805] Modules linked in: iptable_filter tcp_diag inet_diag vxfen(POE) vxgms(POE) vxglm(POE) vxodm(POE) gab(POE) dmpaa(POE) nf_conntrack_netlink nf_conntrack vxspec(POE) vxio(POE) llt(POE) rdma_cm amf(POE) iw_cm ib_cm ib_core vxdmp(POE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache vxcafs(POE) vxportal(POE) fdd(POE) vxfs(POE) veki(POE) nfit libnvdimm iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper parport_pc ablk_helper cryptd parport i2c_piix4 pcspkr binfmt_misc ip_tables xfs libcrc32c nvme crct10dif_pclmul crct10dif_common ena nvme_core crc32c_intel serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod
[8864293.977039] CPU: 1 PID: 13833 Comm: vx_ioship_resen Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
[8864293.988222] Hardware name: Amazon EC2 m5.4xlarge/, BIOS 1.0 10/16/2017
[8864293.994007] task: ffff8ca10a7dd280 ti: ffff8ca06ab20000 task.ti: ffff8ca06ab20000
[8864294.003192] RIP: 0010:[<ffffffffc0a6f040>]  [<ffffffffc0a6f040>] vx_ioship_resend_pkts+0xc0/0x250 [llt]
[8864294.013226] RSP: 0018:ffff8ca06ab23dc8  EFLAGS: 00010286
[8864294.018555] RAX: 0000000000000000 RBX: ffff8ca08f486400 RCX: ffff8ca108c075c0
[8864294.027641] RDX: 0000000001ca4724 RSI: 0000000000000000 RDI: ffff8ca08f486500
[8864294.036737] RBP: ffff8ca06ab23e18 R08: ffff8ca06ab20000 R09: 0000000000000001
[8864294.045812] R10: 0000000000000001 R11: 001f7e5fd689fc80 R12: ffff8c92c2e13680
[8864294.054905] R13: ffff8ca103d72f00 R14: 0000000000000001 R15: ffff8ca103d72f00
[8864294.063989] FS:  0000000000000000(0000) GS:ffff8ca10d240000(0000) knlGS:0000000000000000
[8864294.073381] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[8864294.078813] CR2: 0000000000000000 CR3: 0000000f3eb62000 CR4: 00000000007606e0
[8864294.087824] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[8864294.096815] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[8864294.105804] PKRU: 00000000
[8864294.109997] Call Trace:
[8864294.114081]  [<ffffffffbd98a010>] ? schedule_timeout+0x170/0x2d0
[8864294.119622]  [<ffffffffc0a713c9>] vx_ioship_resend_internal+0x69/0xa0 [llt]
[8864294.125549]  [<ffffffffc0a7143b>] vx_ioship_node_resend+0x3b/0x50 [llt]
[8864294.131340]  [<ffffffffc0a7210f>] vx_ioship_process_resend+0x1bf/0x210 [llt]
[8864294.140339]  [<ffffffffc0a71f50>] ? vx_ioship_cleanup_tgtnode+0x180/0x180 [llt]
[8864294.149456]  [<ffffffffbd2c5f91>] kthread+0xd1/0xe0
[8864294.154567]  [<ffffffffbd2c5ec0>] ? insert_kthread_work+0x40/0x40
[8864294.160178]  [<ffffffffbd999df7>] ret_from_fork_nospec_begin+0x21/0x21
[8864294.165979]  [<ffffffffbd2c5ec0>] ? insert_kthread_work+0x40/0x40
[8864294.171586] Code: 44 24 08 00 00 00 00 49 c7 04 24 00 00 00 00 49 8b 44 24 50 48 8b 53 08 48 89 50 08 48 83 7b 38 00 0f 84 54 01 00 00 48 8b 43 40 <4c> 89 20 48 8b 43 40 49 89 44 24 08 4c 89 63 40 48 83 43 48 01 
[8864294.191754] RIP  [<ffffffffc0a6f040>] vx_ioship_resend_pkts+0xc0/0x250 [llt]
[8864294.200823]  RSP <ffff8ca06ab23dc8>

• Another log:

[23019850.278797] general protection fault: 0000 [#1] SMP
[23019850.284401] Modules linked in: nfsv4 dns_resolver nfs lockd grace fscache iptable_filter vxfen(POE) vxodm(POE) vxglm(POE) vxgms(POE) dmpaa(POE) gab(POE) vxspec(POE) vxio(POE) vxcafs(POE) vxportal(POE) fdd(POE) vxfs(POE) vxdmp(POE) rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm llt(POE) ib_uverbs ib_umad rdma_cm iw_cm ib_cm amf(POE) ib_core nf_conntrack_netlink nf_conntrack tcp_diag inet_diag veki(POE) nfit libnvdimm iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper parport_pc cryptd i2c_piix4 parport pcspkr binfmt_misc ip_tables xfs libcrc32c nvme crct10dif_pclmul crct10dif_common ena nvme_core crc32c_intel serio_raw sunrpc
[23019850.349993]  dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vxfs]
[23019850.355753] CPU: 0 PID: 11944 Comm: vx_ioship_recv Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
[23019850.367380] Hardware name: Amazon EC2 m5.4xlarge/, BIOS 1.0 10/16/2017
[23019850.373452] task: ffff96491f3d1080 ti: ffff964893e24000 task.ti: ffff964893e24000
[23019850.383136] RIP: 0010:[<ffffffff85597da7>]  [<ffffffff85597da7>] __x86_indirect_thunk_rcx+0x17/0x20
[23019850.393435] RSP: 0018:ffff964893e27de8  EFLAGS: 00010206
[23019850.398980] RAX: ffff9640177e0fe0 RBX: 0000000000000fff RCX: 4b1d1d1d33383430
[23019850.408493] RDX: 2b408ad444531c45 RSI: 000000001f363531 RDI: 30454b4c45531f1d
[23019850.418014] RBP: ffff964893e27e10 R08: ffff9649487b7180 R09: 0000000180190012
[23019850.427563] R10: 00000000487b7101 R11: ffff9649487b7180 R12: ffff9640177c1000
[23019850.437114] R13: ffff9649487b72c0 R14: 0000000000000000 R15: ffff96464eeeef00
[23019850.446664] FS:  0000000000000000(0000) GS:ffff96494d200000(0000) knlGS:0000000000000000
[23019850.456630] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[23019850.462390] CR2: 00007f4e4aaa9000 CR3: 00000005ea90a000 CR4: 00000000007606f0
[23019850.472008] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[23019850.481545] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[23019850.491022] PKRU: 00000000
[23019850.495505] Call Trace:
[23019850.499879]  [<ffffffffc0fd1d7a>] ? vx_ioship_free_packet+0x4a/0xf0 [llt]
[23019850.506054]  [<ffffffffc0fd2490>] vx_free_dummy_pkt+0x50/0xa0 [llt]
[23019850.512011]  [<ffffffffc0fd308e>] vx_ioship_process_ack+0xde/0x390 [llt]
[23019850.518160]  [<ffffffffc0fd42db>] vx_ioship_recv_data+0x5bb/0x640 [llt]
[23019850.524254]  [<ffffffffc0fd3d20>] ? vx_ioship_process_data+0x390/0x390 [llt]
[23019850.533727]  [<ffffffff852c5f91>] kthread+0xd1/0xe0
[23019850.539104]  [<ffffffff852c5ec0>] ? insert_kthread_work+0x40/0x40
[23019850.545014]  [<ffffffff85999df7>] ret_from_fork_nospec_begin+0x21/0x21
[23019850.551070]  [<ffffffff852c5ec0>] ? insert_kthread_work+0x40/0x40
[23019850.556955] Code: f3 90 0f ae e8 eb f9 48 89 1c 24 c3 0f 1f 84 00 00 00 00 00 e9 02 00 00 00 ff e1 e8 07 00 00 00 f3 90 0f ae e8 eb f9 48 89 0c 24 <c3> 0f 1f 84 00 00 00 00 00 e9 02 00 00 00 ff e2 e8 07 00 00 00
[23019850.577827] RIP  [<ffffffff85597da7>] __x86_indirect_thunk_rcx+0x17/0x20
[23019850.583995]  RSP <ffff964893e27de8>