System panicked with BUG: unable to handle kernel paging request at ffffffffffffffaf in ec_mem_cache_get function of cbsensor_2_5_1230541 module.

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux.
  • Third-Party Module [cbsensor_2_5_1230541].

Issue

  • The server got crashed because it was unable to handle kernel paging requests.
[ 2080.356164] cbsensor_2_5_1230541: ec_mem_cache_get: Cache entry magic does not match.  Failed to free memory: ffff9046396eb730
[ 2080.356167] CPU: 9 PID: 1878 Comm: elastic-agent Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-1160.88.1.el7.x86_64 #1
[ 2080.356168] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[ 2080.356169] Call Trace:
[ 2080.356174]  [<ffffffffbcdb1bec>] dump_stack+0x19/0x1f
[ 2080.356180]  [<ffffffffc06d85fb>] ec_mem_cache_get+0x2b/0x50 [cbsensor_2_5_1230541]
[ 2080.356184]  [<ffffffffc06e4b5c>] ec_process_tracking_get_exec_identity_ref+0x1c/0x50 [cbsensor_2_5_1230541]
[ 2080.356187]  [<ffffffffc06e500f>] ec_process_exec_handle_set_exec_identity+0x2f/0x60 [cbsensor_2_5_1230541]
[ 2080.356190]  [<ffffffffc06e5113>] ec_process_handle_alloc+0x63/0x90 [cbsensor_2_5_1230541]
[ 2080.356193]  [<ffffffffc06e2a6e>] ec_hashtbl_handle_callback+0xe/0x10 [cbsensor_2_5_1230541]
[ 2080.356196]  [<ffffffffc06e7501>] ec_hashtbl_get+0x51/0xa0 [cbsensor_2_5_1230541]
[ 2080.356198]  [<ffffffffc06e7780>] ec_hashtbl_find+0x230/0x2c0 [cbsensor_2_5_1230541]
[ 2080.356201]  [<ffffffffc06e2bc2>] ec_process_tracking_get_handle+0x32/0x50 [cbsensor_2_5_1230541]
[ 2080.356204]  [<ffffffffc06da51c>] __ec_do_file_event+0xfc/0x410 [cbsensor_2_5_1230541]
[ 2080.356207]  [<ffffffffc06da9e6>] __ec_sys_open+0xb6/0x100 [cbsensor_2_5_1230541]
[ 2080.356210]  [<ffffffffc06db25d>] ec_sys_openat+0x8d/0xe0 [cbsensor_2_5_1230541]
[ 2080.356211]  [<ffffffffc05e1d95>] tmhook_invoke.isra.8+0x45/0xa0 [tmhook]
[ 2080.356213]  [<ffffffffc05e1f98>] tmhook_handler+0x1a8/0x2d0 [tmhook]
[ 2080.356215]  [<ffffffffbcdc539a>] system_call_fastpath+0x25/0x2a
[ 2080.356228] BUG: unable to handle kernel paging request at ffffffffffffffaf
[ 2080.356251] IP: [<ffffffffc06d85da>] ec_mem_cache_get+0xa/0x50 [cbsensor_2_5_1230541]
[ 2080.356273] PGD 64e214067 PUD 64e216067 PMD 0 
[ 2080.356285] Oops: 0000 [#1] SMP 
[ 2080.356294] Modules linked in: bmhook(OE) tmhook(OE) nfsv3 nfs_acl nfs lockd grace fscache dsa_filter(POE) dsa_filter_hook(OE) cbsensor_2_5_1230541(OE) vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci ata_piix crct10dif_pclmul nfit crct10dif_common libata crc32c_intel libnvdimm serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
[ 2080.356464] CPU: 9 PID: 1878 Comm: elastic-agent Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-1160.88.1.el7.x86_64 #1
[ 2080.356487] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[ 2080.356508] task: ffff90463b493180 ti: ffff904610114000 task.ti: ffff904610114000
[ 2080.356523] RIP: 0010:[<ffffffffc06d85da>]  [<ffffffffc06d85da>] ec_mem_cache_get+0xa/0x50 [cbsensor_2_5_1230541]
[ 2080.356546] RSP: 0018:ffff904610117b58  EFLAGS: 00010082
[ 2080.356556] RAX: ffff9046396eb730 RBX: ffffffffffffffff RCX: 0000000000000006
[ 2080.356571] RDX: ffff904610117da8 RSI: ffff904610117da8 RDI: ffffffffffffffdf
[ 2080.356585] RBP: ffff904610117b88 R08: 6170747361665f6c R09: 2f353278302b6874
[ 2080.356599] R10: 0000000000000835 R11: 613278302f353278 R12: ffffffffc06f6b80
[ 2080.356613] R13: ffff904610117da8 R14: ffffffffffffffdf R15: ffffad7506f0dfc0

[..]

[ 2080.356694] PKRU: 55555554
[ 2080.356700] Call Trace:
[ 2080.356710]  [<ffffffffc06e74e6>] ? ec_hashtbl_get+0x36/0xa0 [cbsensor_2_5_1230541]
[ 2080.356728]  [<ffffffffc06db7a3>] ec_path_cache_get+0x23/0x60 [cbsensor_2_5_1230541]
[ 2080.356745]  [<ffffffffbcdb1bec>] ? dump_stack+0x19/0x1f
[ 2080.356758]  [<ffffffffc06e4680>] __ec_process_tracking_get_path_data+0x30/0x80 [cbsensor_2_5_1230541]
[ 2080.356779]  [<ffffffffc06e4b5c>] ? ec_process_tracking_get_exec_identity_ref+0x1c/0x50 [cbsensor_2_5_1230541]
[ 2080.356800]  [<ffffffffc06e501d>] ec_process_exec_handle_set_exec_identity+0x3d/0x60 [cbsensor_2_5_1230541]
[ 2080.356827]  [<ffffffffc06e5113>] ec_process_handle_alloc+0x63/0x90 [cbsensor_2_5_1230541]
[ 2080.356846]  [<ffffffffc06e2a6e>] ec_hashtbl_handle_callback+0xe/0x10 [cbsensor_2_5_1230541]
[ 2080.356864]  [<ffffffffc06e7501>] ec_hashtbl_get+0x51/0xa0 [cbsensor_2_5_1230541]
[ 2080.356881]  [<ffffffffc06e7780>] ec_hashtbl_find+0x230/0x2c0 [cbsensor_2_5_1230541]
[ 2080.356898]  [<ffffffffc06e2bc2>] ec_process_tracking_get_handle+0x32/0x50 [cbsensor_2_5_1230541]
[ 2080.356917]  [<ffffffffc06da51c>] __ec_do_file_event+0xfc/0x410 [cbsensor_2_5_1230541]
[ 2080.356935]  [<ffffffffc06da9e6>] __ec_sys_open+0xb6/0x100 [cbsensor_2_5_1230541]
[ 2080.356951]  [<ffffffffc06db25d>] ec_sys_openat+0x8d/0xe0 [cbsensor_2_5_1230541]
[ 2080.356967]  [<ffffffffc05e1d95>] tmhook_invoke.isra.8+0x45/0xa0 [tmhook]
[ 2080.356982]  [<ffffffffc05e1f98>] tmhook_handler+0x1a8/0x2d0 [tmhook]
[ 2080.356996]  [<ffffffffbcdc539a>] system_call_fastpath+0x25/0x2a
[ 2080.357012] Code: 32 6d fc eb d6 48 8d 70 5c 48 89 da 48 c7 c7 e0 fe 6e c0 31 c0 e8 76 32 6d fc eb cf 0f 1f 44 00 00 0f 1f 44 00 00 48 85 ff 74 22 <81> 7f d0 ef be ad de 75 06 f0 48 ff 47 f0 c3 f6 05 40 e5 01 00 
[ 2080.357087] RIP  [<ffffffffc06d85da>] ec_mem_cache_get+0xa/0x50 [cbsensor_2_5_1230541]
[ 2080.357106]  RSP <ffff904610117b58>
[ 2080.357113] CR2: ffffffffffffffaf

Resolution

  • The [cbsensor_2_5_1230541] module is not shipped by Red Hat.

  • Engage the [cbsensor_2_5_1230541] module vendor to investigate further on the issue.

    Workaround:

Root Cause

  • The server was crashed in the function of the third-party kernel module [cbsensor_2_5_1230541].

Diagnostic Steps

  • The kernel ring buffer shows call traces of the panic task which indicates a panic when operating in the code section of the third-party module [cbsensor_2_5_1230541].

Kernel ring buffer:

crash> log
[ 2080.356164] cbsensor_2_5_1230541: ec_mem_cache_get: Cache entry magic does not match.  Failed to free memory: ffff9046396eb730
[ 2080.356167] CPU: 9 PID: 1878 Comm: elastic-agent Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-1160.88.1.el7.x86_64 #1
[ 2080.356168] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[ 2080.356169] Call Trace:
[ 2080.356174]  [<ffffffffbcdb1bec>] dump_stack+0x19/0x1f
[ 2080.356180]  [<ffffffffc06d85fb>] ec_mem_cache_get+0x2b/0x50 [cbsensor_2_5_1230541]
[ 2080.356184]  [<ffffffffc06e4b5c>] ec_process_tracking_get_exec_identity_ref+0x1c/0x50 [cbsensor_2_5_1230541]
[ 2080.356187]  [<ffffffffc06e500f>] ec_process_exec_handle_set_exec_identity+0x2f/0x60 [cbsensor_2_5_1230541]
[ 2080.356190]  [<ffffffffc06e5113>] ec_process_handle_alloc+0x63/0x90 [cbsensor_2_5_1230541]
[ 2080.356193]  [<ffffffffc06e2a6e>] ec_hashtbl_handle_callback+0xe/0x10 [cbsensor_2_5_1230541]
[ 2080.356196]  [<ffffffffc06e7501>] ec_hashtbl_get+0x51/0xa0 [cbsensor_2_5_1230541]
[ 2080.356198]  [<ffffffffc06e7780>] ec_hashtbl_find+0x230/0x2c0 [cbsensor_2_5_1230541]
[ 2080.356201]  [<ffffffffc06e2bc2>] ec_process_tracking_get_handle+0x32/0x50 [cbsensor_2_5_1230541]
[ 2080.356204]  [<ffffffffc06da51c>] __ec_do_file_event+0xfc/0x410 [cbsensor_2_5_1230541]
[ 2080.356207]  [<ffffffffc06da9e6>] __ec_sys_open+0xb6/0x100 [cbsensor_2_5_1230541]
[ 2080.356210]  [<ffffffffc06db25d>] ec_sys_openat+0x8d/0xe0 [cbsensor_2_5_1230541]
[ 2080.356211]  [<ffffffffc05e1d95>] tmhook_invoke.isra.8+0x45/0xa0 [tmhook]
[ 2080.356213]  [<ffffffffc05e1f98>] tmhook_handler+0x1a8/0x2d0 [tmhook]
[ 2080.356215]  [<ffffffffbcdc539a>] system_call_fastpath+0x25/0x2a
[ 2080.356228] BUG: unable to handle kernel paging request at ffffffffffffffaf
[ 2080.356251] IP: [<ffffffffc06d85da>] ec_mem_cache_get+0xa/0x50 [cbsensor_2_5_1230541]
[ 2080.356273] PGD 64e214067 PUD 64e216067 PMD 0 
[ 2080.356285] Oops: 0000 [#1] SMP 
[ 2080.356294] Modules linked in: bmhook(OE) tmhook(OE) nfsv3 nfs_acl nfs lockd grace fscache dsa_filter(POE) dsa_filter_hook(OE) cbsensor_2_5_1230541(OE) vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci ata_piix crct10dif_pclmul nfit crct10dif_common libata crc32c_intel libnvdimm serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
[ 2080.356464] CPU: 9 PID: 1878 Comm: elastic-agent Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-1160.88.1.el7.x86_64 #1
[ 2080.356487] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[ 2080.356508] task: ffff90463b493180 ti: ffff904610114000 task.ti: ffff904610114000
[ 2080.356523] RIP: 0010:[<ffffffffc06d85da>]  [<ffffffffc06d85da>] ec_mem_cache_get+0xa/0x50 [cbsensor_2_5_1230541]
[ 2080.356546] RSP: 0018:ffff904610117b58  EFLAGS: 00010082
[ 2080.356556] RAX: ffff9046396eb730 RBX: ffffffffffffffff RCX: 0000000000000006
[ 2080.356571] RDX: ffff904610117da8 RSI: ffff904610117da8 RDI: ffffffffffffffdf
[ 2080.356585] RBP: ffff904610117b88 R08: 6170747361665f6c R09: 2f353278302b6874
[ 2080.356599] R10: 0000000000000835 R11: 613278302f353278 R12: ffffffffc06f6b80
[ 2080.356613] R13: ffff904610117da8 R14: ffffffffffffffdf R15: ffffad7506f0dfc0

[..]

[ 2080.356694] PKRU: 55555554
[ 2080.356700] Call Trace:
[ 2080.356710]  [<ffffffffc06e74e6>] ? ec_hashtbl_get+0x36/0xa0 [cbsensor_2_5_1230541]
[ 2080.356728]  [<ffffffffc06db7a3>] ec_path_cache_get+0x23/0x60 [cbsensor_2_5_1230541]
[ 2080.356745]  [<ffffffffbcdb1bec>] ? dump_stack+0x19/0x1f
[ 2080.356758]  [<ffffffffc06e4680>] __ec_process_tracking_get_path_data+0x30/0x80 [cbsensor_2_5_1230541]
[ 2080.356779]  [<ffffffffc06e4b5c>] ? ec_process_tracking_get_exec_identity_ref+0x1c/0x50 [cbsensor_2_5_1230541]
[ 2080.356800]  [<ffffffffc06e501d>] ec_process_exec_handle_set_exec_identity+0x3d/0x60 [cbsensor_2_5_1230541]
[ 2080.356827]  [<ffffffffc06e5113>] ec_process_handle_alloc+0x63/0x90 [cbsensor_2_5_1230541]
[ 2080.356846]  [<ffffffffc06e2a6e>] ec_hashtbl_handle_callback+0xe/0x10 [cbsensor_2_5_1230541]
[ 2080.356864]  [<ffffffffc06e7501>] ec_hashtbl_get+0x51/0xa0 [cbsensor_2_5_1230541]
[ 2080.356881]  [<ffffffffc06e7780>] ec_hashtbl_find+0x230/0x2c0 [cbsensor_2_5_1230541]
[ 2080.356898]  [<ffffffffc06e2bc2>] ec_process_tracking_get_handle+0x32/0x50 [cbsensor_2_5_1230541]
[ 2080.356917]  [<ffffffffc06da51c>] __ec_do_file_event+0xfc/0x410 [cbsensor_2_5_1230541]
[ 2080.356935]  [<ffffffffc06da9e6>] __ec_sys_open+0xb6/0x100 [cbsensor_2_5_1230541]
[ 2080.356951]  [<ffffffffc06db25d>] ec_sys_openat+0x8d/0xe0 [cbsensor_2_5_1230541]
[ 2080.356967]  [<ffffffffc05e1d95>] tmhook_invoke.isra.8+0x45/0xa0 [tmhook]
[ 2080.356982]  [<ffffffffc05e1f98>] tmhook_handler+0x1a8/0x2d0 [tmhook]
[ 2080.356996]  [<ffffffffbcdc539a>] system_call_fastpath+0x25/0x2a
[ 2080.357012] Code: 32 6d fc eb d6 48 8d 70 5c 48 89 da 48 c7 c7 e0 fe 6e c0 31 c0 e8 76 32 6d fc eb cf 0f 1f 44 00 00 0f 1f 44 00 00 48 85 ff 74 22 <81> 7f d0 ef be ad de 75 06 f0 48 ff 47 f0 c3 f6 05 40 e5 01 00 
[ 2080.357087] RIP  [<ffffffffc06d85da>] ec_mem_cache_get+0xa/0x50 [cbsensor_2_5_1230541]
[ 2080.357106]  RSP <ffff904610117b58>
[ 2080.357113] CR2: ffffffffffffffaf

  • Backtrace of the panic task:

  • The server panicked in the function ec_mem_cache_get().

crash> bt
PID: 1878     TASK: ffff90463b493180  CPU: 9    COMMAND: "elastic-agent"
 #0 [ffff9046101177e0] machine_kexec at ffffffffbc669514
 #1 [ffff904610117840] __crash_kexec at ffffffffbc729e82
 #2 [ffff904610117910] crash_kexec at ffffffffbc729f78
 #3 [ffff904610117928] oops_end at ffffffffbcdbc818
 #4 [ffff904610117950] no_context at ffffffffbc67974c
 #5 [ffff9046101179a0] __bad_area_nosemaphore at ffffffffbc679a2a
 #6 [ffff9046101179f0] bad_area_nosemaphore at ffffffffbc679b54
 #7 [ffff904610117a00] __do_page_fault at ffffffffbcdbf8d0
 #8 [ffff904610117a70] do_page_fault at ffffffffbcdbfb05
 #9 [ffff904610117aa0] page_fault at ffffffffbcdbb7b8
    [exception RIP: ec_mem_cache_get+10]                      <<-----------
    RIP: ffffffffc06d85da  RSP: ffff904610117b58  RFLAGS: 00010082
    RAX: ffff9046396eb730  RBX: ffffffffffffffff  RCX: 0000000000000006
    RDX: ffff904610117da8  RSI: ffff904610117da8  RDI: ffffffffffffffdf
    RBP: ffff904610117b88   R8: 6170747361665f6c   R9: 2f353278302b6874
    R10: 0000000000000835  R11: 613278302f353278  R12: ffffffffc06f6b80
    R13: ffff904610117da8  R14: ffffffffffffffdf  R15: ffffad7506f0dfc0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#10 [ffff904610117b58] ec_hashtbl_get at ffffffffc06e74e6 [cbsensor_2_5_1230541]
#11 [ffff904610117b90] ec_path_cache_get at ffffffffc06db7a3 [cbsensor_2_5_1230541]
#12 [ffff904610117bb0] __ec_process_tracking_get_path_data at ffffffffc06e4680 [cbsensor_2_5_1230541]
#13 [ffff904610117bd8] ec_process_exec_handle_set_exec_identity at ffffffffc06e501d [cbsensor_2_5_1230541]
#14 [ffff904610117c00] ec_process_handle_alloc at ffffffffc06e5113 [cbsensor_2_5_1230541]
#15 [ffff904610117c30] ec_hashtbl_handle_callback at ffffffffc06e2a6e [cbsensor_2_5_1230541]
#16 [ffff904610117c40] ec_hashtbl_get at ffffffffc06e7501 [cbsensor_2_5_1230541]
#17 [ffff904610117c78] ec_hashtbl_find at ffffffffc06e7780 [cbsensor_2_5_1230541]
#18 [ffff904610117cb8] ec_process_tracking_get_handle at ffffffffc06e2bc2 [cbsensor_2_5_1230541]
#19 [ffff904610117cd8] __ec_do_file_event at ffffffffc06da51c [cbsensor_2_5_1230541]
#20 [ffff904610117d50] __ec_sys_open at ffffffffc06da9e6 [cbsensor_2_5_1230541]
#21 [ffff904610117d80] ec_sys_openat at ffffffffc06db25d [cbsensor_2_5_1230541]
#22 [ffff904610117e38] tmhook_invoke at ffffffffc05e1d95 [tmhook]
#23 [ffff904610117e78] tmhook_handler at ffffffffc05e1f98 [tmhook]
#24 [ffff904610117f50] system_call_fastpath at ffffffffbcdc539a
    RIP: 000056552b0f040e  RSP: 000000c000a3d418  RFLAGS: 00000202
    RAX: 0000000000000101  RBX: ffffffffffffff9c  RCX: ffffffffffffffff
    RDX: 0000000000080040  RSI: 000000c0000e66c0  RDI: ffffffffffffff9c
    RBP: 000000c000a3d420   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000180  R11: 0000000000000206  R12: 000000c0000e66c0
    R13: 0000000000000000  R14: 000000c00100a1a0  R15: ffffffffffffffff
    ORIG_RAX: 0000000000000101  CS: 0033  SS: 002b
crash>
  • Dis-assembly of address:
crash> sym ec_mem_cache_get
ffffffffc06d85d0 (t) ec_mem_cache_get [cbsensor_2_5_1230541]   <<-------
                              ^                    ^
                              |                    |
                       [Function Name]       [Module Name]

  • Third-party modules:

  • Details of the third-party module: [cbsensor_2_5_1230541].

crash> mod -t | grep -i 'cbsensor_2_5_1230541'
cbsensor_2_5_1230541  OE    <<--------
crash>

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments