Getting errors in /var/log/messages: "BUG: KFENCE: out-of-bounds"

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux
  • Third-party kernel module from McAfee (observed with mfe_aac_100716843, mfe_aac_100717487, mfe_aac_1007171140)

Issue

  • Getting errors in /var/log/messages: BUG: KFENCE: out-of-bounds when a task interacting with 3rd party module associated to McAfee

Resolution

Root Cause

  • Kernel Electric-Fence (KFENCE) is a low-overhead sampling-based memory safety error detector. KFENCE detects heap out-of-bounds access, use-after-free, and invalid-free errors mainly caused by third-party applications
  • This error will generally be reported if there is some memory corruption in the system.

Diagnostic Steps

  • Following call traces were observed in /var/log/messages when the container-shim process was interacting with McAfee module.
 kernel: BUG: KFENCE: out-of-bounds read in __memmove+0x128/0x1b0 <<----
 kernel: Out-of-bounds read at 0x0000000096c9271a (512B right of kfence-#202): <<----
 kernel: __memmove+0x128/0x1b0
 kernel: mfe_aac_get_initiator_cmdline_name+0xbc/0xe0 [mfe_aac_100716843] <<----
 kernel: mfe_aac_create_eventinfo_struct+0x302/0x630 [mfe_aac_100716843]
 kernel: mfe_aac_process_pre_events+0xa3/0x1b0 [mfe_aac_100716843]  <<----
 kernel: mfe_aac_sys_open_64_bit+0x262/0x2a0 [mfe_aac_100716843]
 kernel: mfe_fileaccess_sys_open_64_bit+0x30/0x1f0 [mfe_fileaccess_100716843] <<----
 kernel: do_syscall_64+0x59/0x90 
 kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
 kernel: 
 kernel: kfence-#202: 0x0000000006f67972-0x00000000beab78ee, size=512, cache=kmalloc-512
 kernel: allocated by task 11821 on cpu 2 at 1969990.110664s:
 kernel: mfe_aac_create_eventinfo_struct+0x2dd/0x630 [mfe_aac_100716843] <<----
 kernel: mfe_aac_process_pre_events+0xa3/0x1b0 [mfe_aac_100716843] <<----
 kernel: mfe_aac_sys_open_64_bit+0x262/0x2a0 [mfe_aac_100716843] <<----
 kernel: mfe_fileaccess_sys_open_64_bit+0x30/0x1f0 [mfe_fileaccess_100716843] <<----
 kernel: do_syscall_64+0x59/0x90
 kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
  • Log from RHEL8:
[1326857.907744] BUG: unable to handle kernel paging request at ffffa0360d664000
[1326857.907857] PGD 381601067 P4D 381601067 PUD 381606067 PMD 381609067 PTE 0
[1326857.907960] Oops: 0000 [#1] SMP PTI
[1326857.908017] CPU: 4 PID: 3308897 Comm: Ftp_FTPS10771_1 Kdump: loaded Tainted: G           OE    --------- -  - 4.18.0-477.27.1.el8_8.x86_64 #1
[1326857.908180] Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 Gen10, BIOS U30 02/22/2024
[1326857.908288] RIP: 0010:__memmove+0x1a0/0x1b0
[1326857.908360] Code: 66 44 8b 1e 66 44 8b 54 16 fe 66 44 89 1f 66 44 89 54 17 fe eb 0c 48 83 fa 01 72 06 44 8a 1e 44 88 1f e9 53 3f 41 00 48 89 d1 <f3> a4 e9 49 3f 41 00 90 90 90 90 90 90 90 90 90 eb 3e 0f 1f 00 49
[1326857.908588] RSP: 0018:ffffb231a4f07c40 EFLAGS: 00010206
[1326857.908662] RAX: ffffa0360d663e00 RBX: 00000000000001f4 RCX: 0000000000000001
[1326857.908755] RDX: 00000000000001f4 RSI: ffffa0360d664000 RDI: ffffa0360d663ff3
[1326857.908848] RBP: 0000000000000000 R08: 00000000000001ff R09: 0000000000000200
[1326857.908911] R10: 0000000000000000 R11: 0000000000000246 R12: ffffa0360d663e00
[1326857.908937] R13: ffffa0360d663fff R14: 00007ffff570817a R15: ffffa0371e0fa000
[1326857.908963] FS:  00007fa0545c9700(0000) GS:ffffa03d9fd00000(0000) knlGS:0000000000000000
[1326857.908993] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1326857.909015] CR2: ffffa0360d664000 CR3: 0000000fff608006 CR4: 00000000007706e0
[1326857.909041] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1326857.909066] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[1326857.909092] PKRU: 55555554
[1326857.909104] Call Trace:
[1326857.909118]  mfe_aac_get_initiator_cmdline_name+0xbe/0xe0 [mfe_aac_1007171140]
[1326857.909157]  mfe_aac_create_eventinfo_struct+0x2f7/0x640 [mfe_aac_1007171140]
[1326857.909191]  mfe_aac_process_pre_events+0xa7/0x1b0 [mfe_aac_1007171140]
[1326857.909223]  mfe_aac_sys_openat_64_bit+0x1c7/0x330 [mfe_aac_1007171140]
[1326857.909253]  ? __audit_getname+0x2d/0x50
[1326857.909273]  ? audit_filter_rules.constprop.18+0x680/0x1230
[1326857.909295]  ? __audit_syscall_entry+0xf2/0x140
[1326857.909316]  ? mfe_fileaccess_sys_openat_64_bit+0x34/0x210 [mfe_fileaccess_1007171140]
[1326857.909347]  mfe_fileaccess_sys_openat_64_bit+0x34/0x210 [mfe_fileaccess_1007171140]
[1326857.909378]  do_syscall_64+0x5b/0x1b0
[1326857.909396]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
[1326857.909418] RIP: 0033:0x7fa49d6932a6
[1326857.909433] Code: 89 54 24 08 e8 9b f4 ff ff 8b 74 24 0c 48 8b 3c 24 41 89 c0 44 8b 54 24 08 b8 01 01 00 00 89 f2 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 30 44 89 c7 89 44 24 08 e8 c6 f4 ff ff 8b 44
[1326857.909496] RSP: 002b:00007fa0545c7660 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[1326857.909524] RAX: ffffffffffffffda RBX: 00007fa0545c7860 RCX: 00007fa49d6932a6
[1326857.909550] RDX: 0000000000000042 RSI: 00007fa294121650 RDI: 00000000ffffff9c
[1326857.909576] RBP: 00007fa0545c77b0 R08: 0000000000000000 R09: 00000007a7be4728
[1326857.909602] R10: 00000000000001b6 R11: 0000000000000293 R12: 00007fa294121650
[1326857.909627] R13: 0000000000000042 R14: 00007fa0d0003348 R15: 00007fa0545c77c8
[1326857.909655] Modules linked in: mfe_fileaccess_1007171140(OE) mfe_aac_1007171140(OE) team_mode_loadbalance team ext4 mbcache jbd2 vfat fat dm_round_robin dm_multipath intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common isst_if_common nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl intel_cstate intel_uncore pcspkr ipmi_ssif ch acpi_ipmi ipmi_si ipmi_devintf mei_me ses enclosure hpilo mei hpwdt lpc_ich ipmi_msghandler st acpi_tad ioatdma wmi dca acpi_power_meter auth_rpcgss sunrpc xfs libcrc32c sd_mod sg qla2xxx mgag200 i2c_algo_bit drm_shmem_helper drm_kms_helper syscopyarea nvme_fc sysfillrect sysimgblt fb_sys_fops nvme_fabrics i40e drm crc32c_intel smartpqi nvme_core tg3 scsi_transport_sas t10_pi scsi_transport_fc dm_mirror dm_region_hash dm_log dm_mod
[1326857.911018] CR2: ffffa0360d664000

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments