RHEL9 crash in __submit_bio() with third-party module [snapapi26] loaded.

Solution Unverified - Updated -

Issue

  • Kernel crash with a null pointer dereference:
PID: 123      TASK: ff243e9a2a9ca2c0  CPU: 1    COMMAND: "md126_raid1"
 #0 [ff6b83f8c0a5fa28] machine_kexec at ffffffffb48695b1
 #1 [ff6b83f8c0a5fa38] __ipmi_set_host_system_status at ffffffffc01d660b [ossn]
 #2 [ff6b83f8c0a5fa58] ipmi_set_host_system_status_panic at ffffffffc01d6691 [ossn]
 #3 [ff6b83f8c0a5fa68] ossn_machine_kexec_handler at ffffffffc01d7a95 [ossn]
 #4 [ff6b83f8c0a5fa80] kprobe_ftrace_handler at ffffffffb486d2bb
 #5 [ff6b83f8c0a5fb28] machine_kexec at ffffffffb486973d
 #6 [ff6b83f8c0a5fb78] __crash_kexec at ffffffffb49be29d
 #7 [ff6b83f8c0a5fc40] crash_kexec at ffffffffb49bf528
 #8 [ff6b83f8c0a5fc48] oops_end at ffffffffb48282ab
 #9 [ff6b83f8c0a5fc68] exc_page_fault at ffffffffb52da502
#10 [ff6b83f8c0a5fc90] asm_exc_page_fault at ffffffffb5400b62
#11 [ff6b83f8c0a5fd48] __submit_bio at ffffffffb4ce81a6
#12 [ff6b83f8c0a5fd68] __submit_bio_noacct at ffffffffb4ce82d1
#13 [ff6b83f8c0a5fdb0] flush_bio_list at ffffffffc01809f8 [raid1]
#14 [ff6b83f8c0a5fdd0] flush_pending_writes at ffffffffc0180b01 [raid1]
#15 [ff6b83f8c0a5fe20] raid1d at ffffffffc01857ec [raid1]
#16 [ff6b83f8c0a5feb0] md_thread at ffffffffb4ffedb8
#17 [ff6b83f8c0a5ff10] kthread at ffffffffb491c456
#18 [ff6b83f8c0a5ff50] ret_from_fork at ffffffffb48036ef
  • Another call stack:
[20630222.867995] BUG: kernel NULL pointer dereference, address: 0000000000000000
[20630222.868919] #PF: supervisor instruction fetch in kernel mode
[20630222.868921] #PF: error_code(0x0010) - not-present page
[20630222.868922] PGD 10a86c067 P4D 0 
[20630222.868924] Oops: 0010 [#1] PREEMPT SMP NOPTI
[20630222.869881] CPU: 8 PID: 1406 Comm: systemd-journal Kdump: loaded Tainted: G           OE     -------  ---  5.14.0-362.8.1.el9_3.x86_64 #1
[20630222.870924] session_init(service_process,2712559): OK. kdev=fd:1, bs=4096.
[20630222.871197] Hardware name: Dell Inc. PowerEdge R7615/0G9DHV, BIOS 1.7.2 12/19/2023
[20630222.873789] RIP: 0010:0x0
[20630222.874236] Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
[20630222.874663] RSP: 0000:ff595207c13ef9c8 EFLAGS: 00010246
[20630222.875086] RAX: 0000000000000000 RBX: ff44386e24d94c00 RCX: ff44387e9cfe2738
[20630222.875510] RDX: ff4438a3216e4680 RSI: 0000000000000000 RDI: ff44387e9cfe2738
[20630222.875926] RBP: ff44387e9cfe2738 R08: 0000000000000000 R09: ff44386e0de63ed0
[20630222.876338] R10: 0000000000000000 R11: 0000000000000000 R12: ff44386e24d60c00
[20630222.876744] R13: ffffffffc0b2a170 R14: 0000000000003000 R15: ff595207c13efc20
[20630222.877148] FS:  00007fd91656c580(0000) GS:ff4438ac7cc00000(0000) knlGS:0000000000000000
[20630222.877553] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[20630222.877952] CR2: ffffffffffffffd6 CR3: 000000362294c005 CR4: 0000000000771ee0
[20630222.878353] PKRU: 55555554
[20630222.878745] Call Trace:
[20630222.879132]  <TASK>
[20630222.879512]  ? show_trace_log_lvl+0x1c4/0x2df
[20630222.879894]  ? show_trace_log_lvl+0x1c4/0x2df
[20630222.880263]  ? __submit_bio+0x86/0x130
[20630222.880628]  ? __die_body.cold+0x8/0xd
[20630222.880984]  ? page_fault_oops+0x134/0x170
[20630222.881338]  ? ktime_get+0x35/0xa0
[20630222.881688]  ? exc_page_fault+0x62/0x150
[20630222.882031]  ? asm_exc_page_fault+0x22/0x30
[20630222.882370]  __submit_bio+0x86/0x130
[20630222.882701]  __submit_bio_noacct+0x81/0x1f0
[20630222.883025]  iomap_readahead+0x1ef/0x350
[20630222.883348]  read_pages+0x58/0x2f0
[20630222.883662]  page_cache_ra_unbounded+0x10c/0x180
[20630222.883971]  do_sync_mmap_readahead+0x138/0x250
[20630222.884277]  filemap_fault+0x3e3/0x740
[20630222.884577]  __do_fault+0x33/0x140
[20630222.884870]  do_read_fault+0xf0/0x160
[20630222.885158]  do_fault+0xa9/0x390
[20630222.885440]  __handle_mm_fault+0x585/0x650
[20630222.885718]  ? rseq_ip_fixup+0x6e/0x1a0
[20630222.885990]  handle_mm_fault+0xc5/0x2a0
[20630222.885718]  ? rseq_ip_fixup+0x6e/0x1a0
[20630222.885990]  handle_mm_fault+0xc5/0x2a0
[20630222.886254]  do_user_addr_fault+0x1b4/0x6a0
[20630222.886513]  ? syscall_trace_enter.constprop.0+0x9a/0x1a0
[20630222.886771]  exc_page_fault+0x62/0x150
[20630222.887028]  asm_exc_page_fault+0x22/0x30
[20630222.887281] RIP: 0033:0x7fd916438698
[20630222.887536] Code: 35 68 03 0a 00 48 8d 3d 93 ea 07 00 e8 f1 c7 e3 ff 90 55 53 48 83 ec 08 48 85 ff 0f 84 41 01 00 00 48 85 f6 0f 84 58 01 00 00 <48> 8b 5e 08 48 89 cd 48 85 db 74 44 48 83 fb 0f 76 66 0f b6 06 84
[20630222.888073] RSP: 002b:00007ffd35a266a0 EFLAGS: 00010202
[20630222.888342] RAX: 0000000000000001 RBX: 00007ffd35a26740 RCX: 0000000000f56580
[20630222.888607] RDX: 0000000000000001 RSI: 00007fd911756580 RDI: 000056546c88c3c0
[20630222.888862] RBP: 000056546c88c3c0 R08: 0000000000000010 R09: 000056546c88c3f8
[20630222.889108] R10: 000056546c89dc40 R11: 000000004c000000 R12: 0000000000f56580
[20630222.889353] R13: 0000000000000001 R14: 00007ffd35a266c0 R15: 00007ffd35a26740
[20630222.889597]  </TASK>
[20630222.889832] Modules linked in: binfmt_misc snumbd26(OE) nls_utf8 cifs cifs_arc4 rdma_cm iw_cm ib_cm ib_core cifs_md4 rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs snapapi26(OE) bridge 8021q garp mrp stp llc bonding tls rfkill nvidia_drm(OE) nvidia_modeset(OE) sunrpc nvidia_uvm(OE) ipmi_ssif intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd nvidia(OE) kvm_amd dcdbas kvm irqbypass dell_smbios rapl dell_wmi_descriptor wmi_bmof pcspkr mgag200 i2c_algo_bit vfat drm_shmem_helper video fat ext4 acpi_ipmi drm_kms_helper ipmi_si syscopyarea ipmi_devintf mbcache sysfillrect sysimgblt k10temp i2c_piix4 acpi_power_meter ipmi_msghandler jbd2 joydev drm fuse xfs libcrc32c sd_mod t10_pi sg ahci libahci crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel libata mpi3mr bnxt_en tg3 ccp scsi_transport_sas wmi sp5100_tco dm_mirror dm_region_hash dm_log dm_mod
[20630222.892103] CR2: 0000000000000000

Environment

  • Red Hat Enterprise Linux 9
    • kernel-5.14.0-162.6.1.el9_1
    • kernel-5.14.0-362.8.1.el9_3
  • Third-party module snapapi26

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content