Kernel panic : BUG: unable to handle kernel NULL pointer dereference in bit9 module

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux
  • 3rd party module [b9k_87125], [b9k_87244]

Issue

  • Server crashed in a code path of bit9 while handling convert_raw_inode.
  • Kernel panic in the bit9 module.

Resolution

  • Engage the module vendor of the third-party Unsigned (E) Out-of-tree (O) no license Proprietary module (P) [b9k_87125] [b9k_87244] module to investigate the issue.

  • Check with the module vendor [b9k_87125] [b9k_87244] for a compatible module version with the currently installed kernel.

Workaround

  • Blacklist the module [b9k_87125] [b9k_87244] and check if this issue is reproducible.

Root Cause

  • The system has crashed in the code section of an Unsigned (E) Out-of-tree (O) no license Proprietary module (P) third-party module [b9k_87125] [b9k_87244].

Diagnostic Steps

  • The kernel ring buffer shows the panic string as BUG: unable to handle kernel NULL pointer dereference at 0000000000000040.
[81185.669028] BUG: unable to handle kernel NULL pointer dereference at 0000000000000040              <<-----
[81185.669061] PGD 0 P4D 0 
[81185.669070] Oops: 0000 [#1] SMP PTI
[81185.669079] CPU: 6 PID: 2620 Comm: pool-ds_am Kdump: loaded Tainted: P           OE    --------- -  - 4.18.0-348.el8.x86_64 #1
[81185.669108] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[81185.669172] RIP: 0010:convert_raw_inode+0x43/0x27c [b9k_87125]
[81185.669185] Code: 00 48 89 45 f8 31 c0 bf 48 30 00 00 e8 07 43 ff ff 48 89 45 c8 48 83 7d c8 00 75 0a b8 00 00 00 00 e9 27 02 00 00 48 8b 45 b8 <48> 8b 50 40 48 8b 45 c8 48 89 10 48 8b 45 b8 8b 50 04 48 8b 45 c8
[81185.669230] RSP: 0018:ffffb81411cf7d08 EFLAGS: 00010286
[81185.669242] RAX: 0000000000000000 RBX: ffffffffc0dbf7e0 RCX: 00000000c001c0de
[81185.669254] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 0000000000000286
[81185.669268] RBP: ffffb81411cf7d50 R08: 0000000000030318 R09: 0000000000000006
[81185.669282] R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000000
[81185.669300] R13: ffff91aa4fb68130 R14: ffff91a489438620 R15: ffff91aabee05c00
[81185.669322] FS:  00007f0864ff9700(0000) GS:ffff91af8df80000(0000) knlGS:0000000000000000
[81185.669338] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[81185.669349] CR2: 0000000000000040 CR3: 00000001a082c005 CR4: 00000000007706e0
[81185.669396] PKRU: 55555554
[81185.669404] Call Trace:
[81185.669419]  ? ktime_get_real_ts64+0x40/0xe0
[81185.669468]  ? bit9_printf+0x46/0xc4 [b9k_87125]
[81185.669502]  convert_inode_dentry+0x39/0x35d [b9k_87125]
[81185.669530]  ? _ZN22ReferenceCountedObject9ReferenceEPKc+0x28/0x66 [b9k_87125]
[81185.669567]  convert_inode+0x90/0xac [b9k_87125]
[81185.669601]  cbp_lsm_mmap_file+0x527/0x68d [b9k_87125]
[81185.669614]  cbstub_lsm_file_free_security+0x38/0x50 [cbproxy_cbp_8712_20221128]
[81185.669632]  security_file_free+0x22/0x60
[81185.669642]  __fput+0x12c/0x250
[81185.669651]  task_work_run+0x8a/0xb0
[81185.669660]  exit_to_usermode_loop+0xeb/0xf0
[81185.669670]  do_syscall_64+0x198/0x1a0
[81185.669681]  entry_SYSCALL_64_after_hwframe+0x65/0xca
[81185.669692] RIP: 0033:0x7f088452ea77
[81185.669702] Code: 12 b8 03 00 00 00 0f 05 48 3d 00 f0 ff ff 77 3b c3 66 90 53 89 fb 48 83 ec 10 e8 e4 fb ff ff 89 df 89 c2 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2b 89 d7 89 44 24 0c e8 26 fc ff ff 8b 44 24
[81185.669736] RSP: 002b:00007f0864ff8c60 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
[81185.669756] RAX: 0000000000000000 RBX: 000000000000000c RCX: 00007f088452ea77
[81185.669777] RDX: 0000000000000000 RSI: 00007f0768040c70 RDI: 000000000000000c
[81185.669797] RBP: 00007f0768294cf0 R08: 00007f07681447b0 R09: 0000000000000007
[81185.669820] R10: 00000000043b916a R11: 0000000000000293 R12: 00007f08847cc820
[81185.669837] R13: 00007ffd609597ff R14: 0000000000000001 R15: 0000000000000001
[81185.670423] Modules linked in: bmhook(OE) tmhook(OE) binfmt_misc b9k_87125(POE) cbproxy_cbp_8712_20221128(POE) dsa_filter(POE) dsa_filter_hook(OE) vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock intel_rapl_msr intel_rapl_common isst_if_mbox_msr isst_if_common nfit libnvdimm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl vmw_balloon pcspkr joydev i2c_piix4 vmw_vmci xfs libcrc32c sr_mod cdrom sd_mod t10_pi sg ata_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci ata_piix drm crc32c_intel libata serio_raw vmw_pvscsi vmxnet3 dm_mirror dm_region_hash dm_log dm_mod fuse
[81185.673449] Features: eBPF/event
[81185.674080] CR2: 0000000000000040
  • The backtrace of the panic task shows it panics when doing convert_raw_inode.
crash> bt
PID: 2620     TASK: ffff91a520b88000  CPU: 6    COMMAND: "pool-ds_am"
 #0 [ffffb81411cf7a28] machine_kexec at ffffffff926641ce
 #1 [ffffb81411cf7a80] __crash_kexec at ffffffff9279df1d
 #2 [ffffb81411cf7b48] crash_kexec at ffffffff9279ee0d
 #3 [ffffb81411cf7b60] oops_end at ffffffff9262613d
 #4 [ffffb81411cf7b80] no_context at ffffffff9267562f
 #5 [ffffb81411cf7bd8] __bad_area_nosemaphore at ffffffff9267598c
 #6 [ffffb81411cf7c20] do_page_fault at ffffffff92676267
 #7 [ffffb81411cf7c50] page_fault at ffffffff9300111e
    [exception RIP: convert_raw_inode+67]                  <<---------
    RIP: ffffffffc0caf567  RSP: ffffb81411cf7d08  RFLAGS: 00010286
    RAX: 0000000000000000  RBX: ffffffffc0dbf7e0  RCX: 00000000c001c0de
    RDX: 0000000000000001  RSI: 0000000000000001  RDI: 0000000000000286
    RBP: ffffb81411cf7d50   R8: 0000000000030318   R9: 0000000000000006
    R10: 0000000000000002  R11: 0000000000000000  R12: 0000000000000000
    R13: ffff91aa4fb68130  R14: ffff91a489438620  R15: ffff91aabee05c00
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #8 [ffffb81411cf7d58] convert_inode_dentry at ffffffffc0caf8b8 [b9k_87125]
 #9 [ffffb81411cf7da0] convert_inode at ffffffffc0caf830 [b9k_87125]
#10 [ffffb81411cf7dd8] cbp_lsm_mmap_file at ffffffffc0ca2fad [b9k_87125]
#11 [ffffb81411cf7ee8] task_work_run at ffffffff9270defa
#12 [ffffb81411cf7f20] exit_to_usermode_loop at ffffffff92603c6b
#13 [ffffb81411cf7f38] do_syscall_64 at ffffffff926043f8
#14 [ffffb81411cf7f50] entry_SYSCALL_64_after_hwframe at ffffffff930000ad
    RIP: 00007f088452ea77  RSP: 00007f0864ff8c60  RFLAGS: 00000293
    RAX: 0000000000000000  RBX: 000000000000000c  RCX: 00007f088452ea77
    RDX: 0000000000000000  RSI: 00007f0768040c70  RDI: 000000000000000c
    RBP: 00007f0768294cf0   R8: 00007f07681447b0   R9: 0000000000000007
    R10: 00000000043b916a  R11: 0000000000000293  R12: 00007f08847cc820
    R13: 00007ffd609597ff  R14: 0000000000000001  R15: 0000000000000001
    ORIG_RAX: 0000000000000003  CS: 0033  SS: 002b
  • Function convert_raw_inode is from the [b9k_87125] module.
crash> sym convert_raw_inode
ffffffffc0caf524 (t) convert_raw_inode [b9k_87125]
                      ^                     ^
                      |                     |
                    [Function Name]     [Module Name]
  • And [b9k_87125] is a tainted third-party module.
crash> mod -t
NAME                       TAINTS
dsa_filter_hook            OE
cbproxy_cbp_8712_20221128  POE
tmhook                     OE
bmhook                     OE
dsa_filter                 POE
b9k_87125                  POE   <<-----

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments