Server crashed with the error "general protection fault: 0000 [#1] SMP "

Solution Verified - Updated -

Issue

  • The kernel crashed in code from third-party module vxio provided by Veritas.
  • Kernel panics with following messages:
[22944113.979277] general protection fault: 0000 [#1] SMP 
[22944113.979418] Modules linked in: traps(OE) dsa_filter(POE) dsa_filter_hook(OE) ip6table_filter ip6_tables fuse btrfs raid6
_pq xor vfat msdos fat xfs unix_diag af_packet_diag netlink_diag tcp_diag udp_diag inet_diag nfs_layout_nfsv41_files rpcsec_gs
s_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache vxfen(POE) vxglm(POE) vxgms(POE) dmpaa(POE) gab(POE) vxspec(POE)
 vxio(POE) vxdmp(POE) vxcafs(POE) vxportal(POE) fdd(POE) vxfs(POE) llt(POE) amf(POE) rdma_cm iw_cm ib_cm nf_conntrack_ipv4 nf_
defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ib_core veki(POE) team_mode_activebackup team ipt_REJECT nf_reject_ipv4 iptabl
e_filter sunrpc skx_edac nfit libnvdimm intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni
_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd
[22944113.980978]  ses pcspkr enclosure i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt ipmi_ssif fb_sys_fop
s drm drm_panel_orientation_quirks ipmi_si sg joydev ipmi_devintf hpwdt ipmi_msghandler mei_me hpilo mei lpc_ich wmi acpi_powe
r_meter binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic qla2xxx hpsa crct10dif_pclmul crct10dif_co
mmon crc32c_intel serio_raw nvme_fc smartpqi nvme_fabrics tg3 nvme_core scsi_transport_sas scsi_transport_fc ptp scsi_tgt pps_
core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: traps]
[22944113.982160] CPU: 10 PID: 31073 Comm: vxiod Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-1160.118.1.el7.x
86_64 #1
[22944113.982445] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 10/19/2023
[22944113.982716] task: ffff9baa65999080 ti: ffff9baa56cb0000 task.ti: ffff9baa56cb0000
[22944113.982968] RIP: 0010:[<ffffffffc34bf475>]  [<ffffffffc34bf475>] vxvm_freerbuf+0x5/0x30 [vxio]
[22944113.983271] RSP: 0018:ffff9baaff9038c8  EFLAGS: 00010286
[22944113.983400] RAX: 0000000000000000 RBX: ffff9ba61fc6f5c0 RCX: 0000000000000000
[22944113.983645] RDX: ffff9baaff903b48 RSI: ffff9b8af4e2e000 RDI: 6d616e6976657220
[22944113.983892] RBP: ffff9baaff9038d8 R08: 0000000000014b80 R09: 000000010020001d
[22944113.984151] R10: 00000000d5df5001 R11: ffff9b8ad5df5400 R12: ffff9b9463dd8000
[22944113.984401] R13: ffff9b8af4e2e000 R14: ffff9b8b78415a40 R15: ffff9b95ba8c7000
[22944113.984651] FS:  0000000000000000(0000) GS:ffff9baaff900000(0000) knlGS:0000000000000000
[22944113.984904] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[22944113.985037] CR2: 00007f45e2339384 CR3: 0000000c29210000 CR4: 00000000007607e0
[22944113.985290] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[22944113.985535] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[22944113.985775] PKRU: 00000000
[22944113.985895] Call Trace:
[22944113.986017]  <IRQ> 
[22944113.986072]  [<ffffffffc35459e5>] ? volkiofree+0x15/0x20 [vxio]
[22944113.986360]  [<ffffffffc370bcc0>] volrv_seclog_wsio_childdone+0x50/0x5b0 [vxio]
[22944113.986613]  [<ffffffff812e0e92>] ? try_to_wake_up+0x192/0x3a0
[22944113.986752]  [<ffffffff812e1182>] ? default_wake_function+0x12/0x20
[22944113.986936]  [<ffffffffc3587f4c>] volsiodone+0x23c/0x7e0 [vxio]
[22944113.987069]  [<ffffffff814332fd>] ? __slab_free+0x9d/0x2b0
[22944113.987245]  [<ffffffffc358c81f>] vol_subdisksio_done+0xff/0x300 [vxio]
[22944113.987421]  [<ffffffffc35456b2>] volkcontext_process+0x92/0x240 [vxio]
[22944113.987598]  [<ffffffffc3518a45>] voldiskiodone+0x2a5/0x4f0 [vxio]
[22944113.987734]  [<ffffffff815b255a>] ? sg_init_table+0x1a/0x40
[22944113.987865]  [<ffffffff8157158b>] ? blk_rq_map_sg+0x9b/0x220
[22944113.987999]  [<ffffffffc045770d>] ? sd_setup_read_write_cmnd+0x3d/0x950 [sd_mod]
[22944113.988252]  [<ffffffff81701cca>] ? __scsi_get_command+0x1a/0x100
[22944113.988425]  [<ffffffffc351c0fe>] voldiskiodone_intr+0x15e/0x1b0 [vxio]
[22944113.988596]  [<ffffffffc34be480>] voldmp_iodone+0x60/0x70 [vxio]
[22944113.988734]  [<ffffffff8149d25c>] bio_endio+0x8c/0x130
[22944113.988873]  [<ffffffffc0ad5955>] gendmpiodone+0x2b5/0x3b0 [vxdmp]
[22944113.989014]  [<ffffffffc0ad6e50>] ? gendmpioctl+0x950/0x950 [vxdmp]
[22944113.989153]  [<ffffffffc0ad6e8e>] dmpiodone+0x3e/0xc0 [vxdmp]
[22944113.989288]  [<ffffffff8149d25c>] bio_endio+0x8c/0x130
[22944113.989422]  [<ffffffff8156ac90>] blk_update_request+0x90/0x380
[22944113.989561]  [<ffffffff8170ce44>] scsi_end_request+0x34/0x200
[22944113.989693]  [<ffffffff8170d1e0>] scsi_io_completion+0x170/0x720
[22944113.989825]  [<ffffffff81701fa0>] scsi_finish_command+0xe0/0x140
[22944113.989958]  [<ffffffff8170c6f8>] scsi_softirq_done+0x138/0x170
[22944113.990089]  [<ffffffff81572a06>] blk_done_softirq+0x96/0xd0
[22944113.990222]  [<ffffffff812a9595>] __do_softirq+0xf5/0x290
[22944113.990354]  [<ffffffff819c8aac>] call_softirq+0x1c/0x30
[22944113.990487]  [<ffffffff81230825>] do_softirq+0x65/0xa0
[22944113.990617]  [<ffffffff812a9945>] irq_exit+0x115/0x120
[22944113.990750]  [<ffffffff8125c829>] smp_call_function_single_interrupt+0x39/0x40
[22944113.991007]  [<ffffffff819c7772>] call_function_single_interrupt+0x172/0x180
[22944113.991302]  <EOI> 
[22944113.991351]  [<ffffffffc370d7b0>] ? volrv_seclog_handle_dvsio+0x530/0x530 [vxio]
[22944113.991766]  [<ffffffffc370d7b0>] ? volrv_seclog_handle_dvsio+0x530/0x530 [vxio]
[22944113.992064]  [<ffffffffc3544a06>] ? voliod_iohandle+0x46/0x380 [vxio]
[22944113.992248]  [<ffffffffc3544e1c>] voliod_loop+0xdc/0x3a0 [vxio]
[22944113.992437]  [<ffffffffc3544d40>] ? voliod_iohandle+0x380/0x380 [vxio]
[22944113.992590]  [<ffffffff812cb621>] kthread+0xd1/0xe0
[22944113.992720]  [<ffffffff812cb550>] ? insert_kthread_work+0x40/0x40
[22944113.992854]  [<ffffffff819c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[22944113.992987]  [<ffffffff812cb550>] ? insert_kthread_work+0x40/0x40
[22944113.993122] Code: 7f 70 48 c1 e6 04 e8 1b 52 13 00 48 c7 43 70 00 00 00 00 c7 43 68 00 00 00 00 eb c4 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 <f0> ff 4f 6c 0f 94 c0 84 c0 75 08 c3 0f 1f 80 00 00 00 00 55 48 
[22944113.993671] RIP  [<ffffffffc34bf475>] vxvm_freerbuf+0x5/0x30 [vxio]
[22944113.993843]  RSP <ffff9baaff9038c8>

Environment

  • Red Hat Enterprise Linux 7.9
  • A third party kernel [vxio] module

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content