[RHEL8.1/8.2] The kernel crashes due to slab corruption frequently with Symantec SEP modules

Solution Verified - Updated -

Issue

  • The kernel crashes frequently where Symantec Endpoint Protection (SEP) for Linux is running.
    • Panic message 1
[  106.623133] ------------[ cut here ]------------
[  106.623138] kernel BUG at mm/slub.c:3910!
[  106.625375] invalid opcode: 0000 [#1] SMP NOPTI
[  106.627294] CPU: 0 PID: 2031 Comm: rtvscand Kdump: loaded Tainted: P        W  OE    --------- -  - 4.18.0-193.6.3.el8_
2.x86_64 #1
[  106.631530] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.0.B64.1501130705 01/13/2015
[  106.635684] RIP: 0010:kfree+0x148/0x160
[  106.637354] Code: 89 d7 5d 41 5c e9 88 72 fd ff 48 89 d9 48 89 da 41 b8 01 00 00 00 5b 4c 89 d6 5d 41 5c e9 b0 f6 ff ff 49 8b 42 08 a8 01 75 c5 <0f> 0b 48 8b 3d 8f bf f7 00 e9 e7 fe ff ff 66 2e 0f 1f 84 00 00 00
[  106.640868] RSP: 0018:ffffb31782323e50 EFLAGS: 00210246
[  106.642487] RAX: fffff19184dc5008 RBX: ffff9545763d8778 RCX: 00000000000006d6
[  106.644132] RDX: 0000000000000000 RSI: ffff9545770a2f00 RDI: 00006abb40000000
[  106.645768] RBP: ffffffff8fe84700 R08: 00003dd20423b650 R09: ffffffff8f2ed851
[  106.647378] R10: fffff19184d8f600 R11: 0000000000000001 R12: ffffffff8f366a4d
[  106.649207] R13: ffff9545763d8778 R14: 0000000000000000 R15: ffff954571ac8a40
[  106.651152] FS:  0000000000000000(0000) GS:ffff95457ac00000(0063) knlGS:00000000e74d2b40
[  106.653004] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[  106.654616] CR2: 000056022b122320 CR3: 0000000118084000 CR4: 00000000000006f0
[  106.656336] Call Trace:
[  106.658045]  full_proxy_release+0x6d/0xb0
[  106.660119]  __fput+0xb7/0x230
[  106.661715]  task_work_run+0x8a/0xb0
[  106.663409]  exit_to_usermode_loop+0xeb/0xf0
[  106.664892]  do_fast_syscall_32+0x29a/0x2e1
[  106.666532]  entry_SYSCALL_compat_after_hwframe+0x45/0x4d
[  106.668285] Modules linked in: symap_custom_4_18_0_193_6_3_el8_2_x86_64(POE) symev_custom_4_18_0_193_6_3_el8_2_x86_64(OE) vmw_vsock_vmci_transport vsock vfat fat ext4 mbcache jbd2 ppdev vmw_balloon edac_mce_amd joydev pcspkr vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c vmwgfx drm_kms_helper sr_mod cdrom syscopyarea ata_generic sysfillrect sysimgblt fb_sys_fops ttm sd_mod sg drm ata_piix serio_raw libata vmxnet3 vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod
  • Panic message 2
[  194.633116] general protection fault: 0000 [#1] SMP NOPTI
[  194.635272] CPU: 1 PID: 2036 Comm: rtvscand Kdump: loaded Tainted: P           OE    --------- -  - 4.18.0-193.6.3.el8_2.x86_64 #1
[  194.638581] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.0.B64.1501130705 01/13/2015
[  194.642416] RIP: 0010:__x86_indirect_thunk_rax+0x10/0x20
[  194.644203] Code: 60 2f c8 ff b9 f2 ff ff ff 30 c0 e9 65 31 c8 ff b9 f2 ff ff ff e9 67 31 c8 ff e8 07 00 00 00 f3 90 0f ae e8 eb f9 48 89 04 24 <c3> 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 e8 07 00 00 00 f3
[  194.647370] RSP: 0018:ffffa66e40eeff00 EFLAGS: 00210282
[  194.648867] RAX: f6d7d4e1a5cd5229 RBX: ffff93b31e779e00 RCX: ffff93b31e710358
[  194.650717] RDX: ffff93b31e779e00 RSI: ffff93b31e7102c0 RDI: ffff93b31e779e00
[  194.652381] RBP: ffff93b31e7102c0 R08: 00000000f7ef6000 R09: 0000000000000001
[  194.654122] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
[  194.655732] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  194.657453] FS:  0000000000000000(0000) GS:ffff93b33ac40000(0063) knlGS:00000000e744fb40
[  194.659292] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[  194.661067] CR2: 00007f5ae4aa0000 CR3: 000000011e772000 CR4: 00000000000006e0
[  194.662796] Call Trace:
[  194.664320]  ? filp_close+0x31/0x60
[  194.665869]  ? __ia32_sys_close+0x1e/0x50
[  194.667556]  ? do_fast_syscall_32+0xa7/0x2e1
[  194.669144]  ? entry_SYSCALL_compat_after_hwframe+0x45/0x4d
[  194.670840] Modules linked in: symap_custom_4_18_0_193_6_3_el8_2_x86_64(POE) symev_custom_4_18_0_193_6_3_el8_2_x86_64(OE) vmw_vsock_vmci_transport vsock vfat fat ext4 mbcache jbd2 ppdev vmw_balloon edac_mce_amd joydev pcspkr vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c vmwgfx sr_mod drm_kms_helper cdrom ata_generic syscopyarea sysfillrect sysimgblt fb_sys_fops sd_mod ttm sg drm ata_piix libata serio_raw vmxnet3 vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod
  • Panic message 3
[80470.323111] stack segment: 0000 [#1] SMP NOPTI
[80470.323115] CPU: 4 PID: 457267 Comm: kworker/u12:2 Kdump: loaded Tainted: P           OE    --------- -  - 4.18.0-193.14.3.el8_2.x86_64 #1
[80470.323116] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[80470.323125] Workqueue: writeback wb_workfn (flush-253:0)
[80470.323132] RIP: 0010:__kmalloc+0xa0/0x200
[80470.323133] Code: 01 00 00 4d 8b 07 65 49 8b 50 08 65 4c 03 05 5f a6 77 5a 49 8b 28 48 85 ed 0f 84 21 01 00 00 41 8b 47 20 4d 8b 07 48 8d 4a 01 <48> 8b 5c 05 00 48 89 e8 65 49 0f c7 08 0f 94 c0 84 c0 74 c5 41 8b
[80470.323134] RSP: 0018:ffffad0d8352b6b0 EFLAGS: 00010206
[80470.323135] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000002028
[80470.323136] RDX: 0000000000002027 RSI: 0000000000600640 RDI: ffff931547c02400
[80470.323137] RBP: 0100000087c20d00 R08: 000000000002e1c0 R09: 0000000000000000
[80470.323138] R10: ffff9315f798e278 R11: 0000000000000001 R12: 0000000000600640
[80470.323139] R13: 0000000000001080 R14: ffff931547c02400 R15: ffff931547c02400
[80470.323140] FS:  0000000000000000(0000) GS:ffff9315f9b00000(0000) knlGS:0000000000000000
[80470.323141] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[80470.323142] CR2: 00007ff0f80ab000 CR3: 000000004860a004 CR4: 00000000001606e0
[80470.323192] Call Trace:
[80470.323282]  ? kmem_alloc+0x64/0x150 [xfs]
[80470.323313]  ? kmem_alloc_large+0x29/0x90 [xfs]
[80470.323341]  kmem_alloc+0x64/0x150 [xfs]
[80470.323379]  ? xfs_log_commit_cil+0x12d/0x630 [xfs]
[80470.323406]  kmem_alloc_large+0x29/0x90 [xfs]
[80470.323434]  xfs_log_commit_cil+0x12d/0x630 [xfs]
[80470.323466]  ? xfs_bmapi_convert_delalloc+0x333/0x460 [xfs]
[80470.323494]  __xfs_trans_commit+0xa4/0x350 [xfs]
[80470.323515]  xfs_bmapi_convert_delalloc+0x333/0x460 [xfs]
[80470.323551]  xfs_map_blocks+0x168/0x410 [xfs]
[80470.323580]  xfs_do_writepage+0x121/0x470 [xfs]
[80470.323587]  write_cache_pages+0x1aa/0x440
[80470.323612]  ? xfs_vm_releasepage+0x80/0x80 [xfs]
[80470.323617]  ? submit_bio+0x45/0x140
[80470.323641]  ? xfs_setfilesize_trans_alloc.isra.17+0x3d/0x90 [xfs]
[80470.323666]  xfs_vm_writepages+0x64/0xa0 [xfs]
[80470.323669]  do_writepages+0x41/0xd0
[80470.323672]  __writeback_single_inode+0x3d/0x360
[80470.323674]  writeback_sb_inodes+0x1e3/0x450
[80470.323676]  __writeback_inodes_wb+0x5d/0xb0
[80470.323678]  wb_writeback+0x25f/0x2f0
[80470.323682]  ? cpumask_next+0x17/0x20
[80470.323683]  wb_workfn+0x342/0x400
[80470.323686]  process_one_work+0x1a7/0x3b0
[80470.323688]  worker_thread+0x30/0x390
[80470.323689]  ? create_worker+0x1a0/0x1a0
[80470.323691]  kthread+0x112/0x130
[80470.323693]  ? kthread_flush_work_fn+0x10/0x10
[80470.323696]  ret_from_fork+0x1f/0x40
[80470.323698] Modules linked in: binfmt_misc nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace symap_custom_4_18_0_193_14_3_el8_2_x86_64(POE) fscache nf_nat_ftp nft_objref nf_conntrack_ftp nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_tables_set nft_chain_nat_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nft_chain_route_ipv6 nft_chain_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack nft_chain_route_ipv4 ip6_tables nft_compat ip_set nf_tables nfnetlink symev_custom_4_18_0_193_14_3_el8_2_x86_64(OE) vmw_vsock_vmci_transport vsock sunrpc intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel vmw_balloon joydev intel_rapl_perf pcspkr vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod sg ata_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci crc32c_intel serio_raw ata_piix vmxnet3 vmw_pvscsi
[80470.323736]  libata dm_mirror dm_region_hash dm_log dm_mod

Environment

  • Red Hat Enterprise Linux 8.1
    • kernel-4.18.0-147.el8
  • Symantec Endpoint Protection (SEP) for Linux

    • symap_custom_4_18_0_147_el8_x86_64
    • symev_custom_4_18_0_147_el8_x86_64
  • Red Hat Enterprise Linux 8.2

    • kernel-4.18.0-193.6.3.el8_2
  • Symantec Endpoint Protection (SEP) for Linux

    • symev_custom_4_18_0_193_6_3_el8_2_x86_64
    • symap_custom_4_18_0_193_6_3_el8_2_x86_64)
  • Red Hat Enterprise Linux 8.2

    • kernel-4.18.0-193.14.3.el8_2
  • Symantec Endpoint Protection (SEP) for Linux
    • symev_custom_4_18_0_193_14_3_el8_2_x86_64
    • symap_custom_4_18_0_193_14_3_el8_2_x86_64

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content