The kernel got crashed due to the general protection fault occurred in a function of third party module [mfe_fileaccess_1007141206]

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux 7
  • Third party module [mfe_fileaccess_1007141206]

Issue

  • The kernel got crashed due to the general protection fault occurred from the 3rd party module mfe_fileaccess_1007141206.
  • Vmcore backtrace shows following RIP exception.
 #6 [ffff9ceee79cfcd0] general_protection at ffffffff8398c718
    [exception RIP: strncpy+24]
    :::::::
 #7 [ffff9ceee79cfd88] mfe_fileaccess_wake_up_all_events_and_reset_scan_and_scanning_queue at ffffffffc058d0bf [mfe_fileaccess_1007141206]

Resolution

  • The mfe_fileaccess_1007141206 module is not shipped by Red Hat. Contact the provider (McAffe) of the third-party kernel module mfe_fileaccess_1007141206 for further queries or investigation.

  • If the module provider does not provide any resolution, provide a new vmcore file captured at the time of the crash without the module mfe_fileaccess_1007141206 installed/loaded.

Possible Workaround:

  • Blacklist the third-party kernel module mfe_fileaccess_1007141206 and check whether issue reoccurs.

Root Cause

  • The system has panicked due to the general protection fault in function of third party module mfe_fileaccess_1007141206.

Diagnostic Steps

  • [1] From a vmcore file:
crash> set -p 
    PID: 18658
COMMAND: "kworker/1:3"
   TASK: ffff9cef17983180  [THREAD_INFO: ffff9ceee79cc000]
    CPU: 1
  STATE: TASK_RUNNING (PANIC)

crash> bt 
PID: 18658    TASK: ffff9cef17983180  CPU: 1    COMMAND: "kworker/1:3"
 #0 [ffff9ceee79cfaf8] machine_kexec at ffffffff832662f4
 #1 [ffff9ceee79cfb58] __crash_kexec at ffffffff83322a32
 #2 [ffff9ceee79cfc28] crash_kexec at ffffffff83322b20
 #3 [ffff9ceee79cfc40] oops_end at ffffffff8398d798
 #4 [ffff9ceee79cfc68] die at ffffffff83230a7b
 #5 [ffff9ceee79cfc98] do_general_protection at ffffffff8398d092
 #6 [ffff9ceee79cfcd0] general_protection at ffffffff8398c718
    [exception RIP: strncpy+24]
    RIP: ffffffff83590668  RSP: ffff9ceee79cfd80  RFLAGS: 00010206
    RAX: ffff9cf06c5a8000  RBX: ffff9cef5018d640  RCX: ffff9cf06c5a8000
    RDX: 0000000000001000  RSI: 3ce05b1176edae20  RDI: ffff9cf06c5a8000
    RBP: ffff9ceee79cfd80   R8: ffff9cf06c5a9000   R9: ffffffffc058cfc6
    R10: 8c425c8bf718d640  R11: ffff9cef5018de40  R12: ffff9cef37f91200
    R13: ffff9cf06c5aa080  R14: ffff9cf06c5a9000  R15: ffff9cf06c5a8000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffff9ceee79cfd88] mfe_fileaccess_wake_up_all_events_and_reset_scan_and_scanning_queue at ffffffffc058d0bf [mfe_fileaccess_1007141206]
 #8 [ffff9ceee79cfe20] process_one_work at ffffffff832bde8f
 #9 [ffff9ceee79cfe68] worker_thread at ffffffff832befa6
#10 [ffff9ceee79cfec8] kthread at ffffffff832c5e61
  • [2] Observed errors like following for third party module.
crash> log 
[...]
 general protection fault: 0000 [#1] SMP 
 Modules linked in: ip6table_mangle ip6table_nat nf_nat_ipv6 ip6table_raw iptable_mangle xt_CT iptable_raw xt_NFLOG ::::
 CPU: 1 PID: 18658 Comm: kworker/1:3 Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-1160.53.1.el7.x86_64 #1
 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
 Workqueue: nl_queue mfe_fileaccess_wake_up_all_events_and_reset_scan_and_scanning_queue [mfe_fileaccess_1007141206]
 task: ffff9cef17983180 ti: ffff9ceee79cc000 task.ti: ffff9ceee79cc000
 RIP: 0010:[<ffffffff83590668>]  [<ffffffff83590668>] strncpy+0x18/0x30
 RSP: 0018:ffff9ceee79cfd80  EFLAGS: 00010206
 RAX: ffff9cf06c5a8000 RBX: ffff9cef5018d640 RCX: ffff9cf06c5a8000
 RDX: 0000000000001000 RSI: 3ce05b1176edae20 RDI: ffff9cf06c5a8000
 RBP: ffff9ceee79cfd80 R08: ffff9cf06c5a9000 R09: ffffffffc058cfc6
 R10: 8c425c8bf718d640 R11: ffff9cef5018de40 R12: ffff9cef37f91200
 R13: ffff9cf06c5aa080 R14: ffff9cf06c5a9000 R15: ffff9cf06c5a8000
 FS:  0000000000000000(0000) GS:ffff9cf07fd00000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 00007f1250ea7190 CR3: 0000000069378000 CR4: 00000000001607e0
 Call Trace:
  [<ffffffffc058d0bf>] mfe_fileaccess_wake_up_all_events_and_reset_scan_and_scanning_queue+0x7af/0x1080 [mfe_fileaccess_1007141206]
  [<ffffffff832bb2fe>] ? move_linked_works+0x5e/0x90
  [<ffffffff832bba2f>] ? pwq_activate_delayed_work+0x3f/0xb0
  [<ffffffff832bde8f>] process_one_work+0x17f/0x440
  [<ffffffff832befa6>] worker_thread+0x126/0x3c0
  [<ffffffff832bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
  [<ffffffff832c5e61>] kthread+0xd1/0xe0
  [<ffffffff832c5d90>] ? insert_kthread_work+0x40/0x40
  [<ffffffff83995df7>] ret_from_fork_nospec_begin+0x21/0x21
  [<ffffffff832c5d90>] ? insert_kthread_work+0x40/0x40
 Code: c9 88 4a ff 75 ed 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 55 48 85 d2 48 89 f8 4c 8d 04 17 48 89 f9 48 89 e5 74 1a 0f 1f 44 00 00 <0f> b6 16 80 fa 01 88 11 48 83 de ff 48 83 c1 01 4c 39 c1 75 eb 
 RIP  [<ffffffff83590668>] strncpy+0x18/0x30
  • [3] From other vmcore file, it is observed following errors.
MFE_FILEACCESS_WARNING : Found scanning queue corruption while sending unicast failed. Recovering from queue corruption
MFE_FILEACCESS_WARNING : Scanning queue is corrupted . Recovering from queue corruption
BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<ffffffffab7a669b>] __list_add+0x1b/0xc0
PGD 8000000064fec067 PUD 0 
Oops: 0000 [#1] SMP 
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache nf_conntrack_netlink nf_conntrack ip_set unix_diag seos_1410_0_1707(POE) mfe_fileaccess_1007141206(OE) nfnetlink_queue nfnetlink_log vmw_vsock_vmci_transport vsock xfs libcrc32c sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev vmw_balloon ::::
CPU: 0 PID: 14164 Comm: kworker/0:0 Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
Workqueue: nl_queue mfe_fileaccess_wake_up_all_events_and_reset_scan_and_scanning_queue [mfe_fileaccess_1007141206]
task: ffff9da0e7caa100 ti: ffff9da04717c000 task.ti: ffff9da04717c000
RIP: 0010:[<ffffffffab7a669b>]  [<ffffffffab7a669b>] __list_add+0x1b/0xc0
RSP: 0018:ffff9da04717fcf8  EFLAGS: 00010246
RAX: 00000000ffffffff RBX: ffff9da04717fd20 RCX: 0000000000000000
RDX: ffff9da0d205d898 RSI: 0000000000000000 RDI: ffff9da04717fd20
RBP: ffff9da04717fd10 R08: 0000000000000000 R09: ffffffffc06f7fc6
R10: 89785a581ed26c80 R11: ffff9d9eaa357480 R12: ffff9da0d205d898
R13: 0000000000000000 R14: 00000000ffffffff R15: ffff9da0d205d898
FS:  0000000000000000(0000) GS:ffff9da27fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000028e93a000 CR4: 00000000001607f0
Call Trace:
 [<ffffffffabb88206>] __mutex_lock_slowpath+0xa6/0x1d0
 [<ffffffffab627022>] ? kmem_cache_free+0x1e2/0x200
 [<ffffffffabb875ff>] mutex_lock+0x1f/0x2f
 [<ffffffffc06f7fec>] mfe_fileaccess_wake_up_all_events_and_reset_scan_and_scanning_queue+0x6dc/0x1080 [mfe_fileaccess_1007141206]
 [<ffffffffc0c8e786>] ? nfs_pgio_release+0x16/0x20 [nfs]
 [<ffffffffab4bde8f>] process_one_work+0x17f/0x440
 [<ffffffffab4befa6>] worker_thread+0x126/0x3c0
 [<ffffffffab4bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
 [<ffffffffab4c5e61>] kthread+0xd1/0xe0
 [<ffffffffab4c5d90>] ? insert_kthread_work+0x40/0x40
 [<ffffffffabb95df7>] ret_from_fork_nospec_begin+0x21/0x21
 [<ffffffffab4c5d90>] ? insert_kthread_work+0x40/0x40
Code: ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 d4 53 4c 8b 42 08 48 89 fb 49 39 f0 75 2a <4d> 8b 45 00 4d 39 c4 75 68 4c 39 e3 74 3e 4c 39 eb 74 39 49 89 
RIP  [<ffffffffab7a669b>] __list_add+0x1b/0xc0
 RSP <ffff9da04717fcf8>
CR2: 0000000000000000

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments