Server hung with "list_del corruption" and "list_add double add" warnings on ertp_insert_data_to_cache [eset_rtp] module

Issue

• The server got hung up after the large number of list_del corruption and list_add double add warnings.

RHEL7.6

[172661.714817] ------------[ cut here ]------------
[172661.714818] WARNING: CPU: 0 PID: 6960 at lib/list_debug.c:62 __list_del_entry+0x82/0xd0
[172661.714819] list_del corruption. next->prev should be ffffa06302b57e18, but was ffffffffc07361e0
[172661.714819] Modules linked in: veeamsnap(OE) eset_rtp(OE) binfmt_misc tcp_diag udp_diag inet_diag af_packet_diag netlink_diag unix_diag sv_hti(OE) smbus(OE) mptctl mptbase ipt_REJECT nf_reject_ipv4 bonding xt_conntrack ebtable_nat ebtable_broute bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6_tables iptable_filter sunrpc vfat fat intel_pmc_core intel_powerclamp coretemp intel_rapl iosf_mbi iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr i2c_i801 sg i2c_designware_platform idma64 i2c_designware_core virt_dma wmi ipmi_si ipmi_devintf ipmi_msghandler video pinctrl_cannonlake
[172661.714833]  pinctrl_intel acpi_power_meter acpi_pad ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci igb libata megaraid_sas crct10dif_pclmul crct10dif_common crc32c_intel ptp pps_core dca drm_panel_orientation_quirks i2c_algo_bit i2c_hid dm_mirror dm_region_hash dm_log dm_mod [last unloaded: eset_rtp]
[172661.714839] CPU: 0 PID: 6960 Comm: munin-graph Kdump: loaded Tainted: G        W  OE  ------------   3.10.0-957.el7.x86_64 #1
[172661.714840] Hardware name: XXXXX, BIOS V5.0.0.13 R1.13.0 for D3675-A1x                    11/14/2019
[172661.714840] Call Trace:
[172661.714842]  [<ffffffffbe961dc1>] dump_stack+0x19/0x1b
[172661.714843]  [<ffffffffbe297648>] __warn+0xd8/0x100
[172661.714844]  [<ffffffffbe2976cf>] warn_slowpath_fmt+0x5f/0x80
[172661.714845]  [<ffffffffbe594a42>] __list_del_entry+0x82/0xd0
[172661.714846]  [<ffffffffc0732b33>] ertp_insert_data_to_cache+0x133/0x230 [eset_rtp]
[172661.714848]  [<ffffffffc072f97b>] ertp_process_request+0x19b/0x290 [eset_rtp]
[172661.714849]  [<ffffffffc07314fd>] ertp_check_file+0x11d/0x290 [eset_rtp]
[172661.714851]  [<ffffffffc07316a3>] ertp_check_close+0x13/0x20 [eset_rtp]
[172661.714852]  [<ffffffffc073064f>] check_close+0x1f/0x30 [eset_rtp]
[172661.714853]  [<ffffffffc073082d>] sys_ertp_close+0x1d/0x80 [eset_rtp]
[172661.714855]  [<ffffffffbe974ddb>] system_call_fastpath+0x22/0x27
[172661.714855] ---[ end trace 0b1e8beb113629bb ]---
[172661.714856] ------------[ cut here ]------------
[172661.714857] WARNING: CPU: 0 PID: 6960 at lib/list_debug.c:36 __list_add+0x8a/0xc0
[172661.714858] list_add double add: new=ffffa06302b57e18, prev=ffffffffc07361e0, next=ffffa06302b57e18.
[172661.715408] BUG: recent printk recursion!
[172661.714858] Modules linked in: veeamsnap(OE) eset_rtp(OE) binfmt_misc tcp_diag udp_diag inet_diag af_packet_diag netlink_diag unix_diag sv_hti(OE) smbus(OE) mptctl mptbase ipt_REJECT nf_reject_ipv4 bonding xt_conntrack ebtable_nat ebtable_broute bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6_tables iptable_filter sunrpc vfat fat intel_pmc_core intel_powerclamp coretemp intel_rapl iosf_mbi iTCO_wdt iTCO_vendor_support
[172661.715409] Kernel panic - not syncing: NMI: Not continuing
[172661.715411] CPU: 0 PID: 6960 Comm: munin-graph Kdump: loaded Tainted: G        W  OE  ------------   3.10.0-957.el7.x86_64 #1
[172661.715411] Hardware name: XXXXX, BIOS V5.0.0.13 R1.13.0 for D3675-A1x                    11/14/2019
[172661.715412] Call Trace:
[172661.715413]  <NMI>  [<ffffffffbe961dc1>] dump_stack+0x19/0x1b
[172661.715415]  [<ffffffffbe95b4d0>] panic+0xe8/0x21f
[172661.715416]  [<ffffffffbe29739f>] nmi_panic+0x3f/0x40
[172661.715417]  [<ffffffffbe96cdb6>] do_nmi+0x3f6/0x460
[172661.715418]  [<ffffffffbe96bd69>] end_repeat_nmi+0x1e/0x81
[172661.715421]  [<ffffffffbe58108c>] ? string.isra.7+0x6c/0xf0
[172661.715423]  [<ffffffffbe58108c>] ? string.isra.7+0x6c/0xf0
[172661.715424]  [<ffffffffbe58108c>] ? string.isra.7+0x6c/0xf0
[172661.715424]  <EOE>  [<ffffffffbe582771>] vsnprintf+0x201/0x6a0
[172661.715426]  [<ffffffffbe582d96>] ? sprintf+0x56/0x80
[172661.715428]  [<ffffffffbe582c1d>] vscnprintf+0xd/0x30
[172661.715429]  [<ffffffffbe29a24b>] vprintk_emit+0x11b/0x510
[172661.715431]  [<ffffffffbe59498a>] ? __list_add+0x8a/0xc0
[172661.715432]  [<ffffffffbe29a8a9>] vprintk_default+0x29/0x40
[172661.715433]  [<ffffffffbe95b667>] printk+0x60/0x77
[172661.715434]  [<ffffffffbe319d53>] print_modules+0xa3/0xd0
[172661.715436]  [<ffffffffbe59498a>] ? __list_add+0x8a/0xc0
[172661.715437]  [<ffffffffbe2975fe>] __warn+0x8e/0x100
[172661.715438]  [<ffffffffbe2976cf>] warn_slowpath_fmt+0x5f/0x80
[172661.715439]  [<ffffffffbe59498a>] __list_add+0x8a/0xc0
[172661.715441]  [<ffffffffc0732b42>] ertp_insert_data_to_cache+0x142/0x230 [eset_rtp]
[172661.715443]  [<ffffffffc072f97b>] ertp_process_request+0x19b/0x290 [eset_rtp]
[172661.715444]  [<ffffffffc07314fd>] ertp_check_file+0x11d/0x290 [eset_rtp]
[172661.715446]  [<ffffffffc07316a3>] ertp_check_close+0x13/0x20 [eset_rtp]
[172661.715447]  [<ffffffffc073064f>] check_close+0x1f/0x30 [eset_rtp]
[172661.715449]  [<ffffffffc073082d>] sys_ertp_close+0x1d/0x80 [eset_rtp]
[172661.715450]  [<ffffffffbe974ddb>] system_call_fastpath+0x22/0x27

RHEL8.6

[1026361.815855] list_add corruption. prev->next should be next (ffffa36c86b07e20), but was ffff91c4a0fbbd80. (prev=ffffffffc06d3220).
[1026361.824827] ------------[ cut here ]------------
[1026361.828766] kernel BUG at lib/list_debug.c:28!
[1026361.832669] invalid opcode: 0000 [#1] SMP PTI
[1026361.836645] CPU: 2 PID: 766420 Comm: ERAAgent Kdump: loaded Tainted: G           OE  X --------- -  - 4.18.0-372.19.1.el8_6.x86_64 #1
[1026361.845965] Hardware name: XXXXX, BIOS 1.0 10/16/2017
[1026361.850910] RIP: 0010:__list_add_valid.cold.0+0x26/0x28
[1026361.855317] Code: 00 00 00 c3 48 89 d1 48 c7 c7 c0 66 12 88 48 89 c2 e8 9e e7 c9 ff 0f 0b 48 89 c1 4c 89 c6 48 c7 c7 18 67 12 88 e8 8a e7 c9 ff <0f> 0b 48 89 fe 48 89 c2 48 c7 c7 a8 67 12 88 e8 76 e7 c9 ff 0f 0b
[1026361.869410] RSP: 0018:ffffa36c86b07e18 EFLAGS: 00010246
[1026361.873797] RAX: 0000000000000075 RBX: 0000000000000000 RCX: 0000000000000000
[1026361.880557] RDX: 0000000000000000 RSI: ffff91c79d516758 RDI: ffff91c79d516758
[1026361.887072] RBP: ffffa36c86b07e20 R08: 0000000000000000 R09: c0000000ffff7fff
[1026361.893403] R10: 0000000000000001 R11: ffffa36c86b07c38 R12: ffffffffc06d3af8
[1026361.899453] R13: 0000000000000002 R14: ffffa36c86b07e20 R15: ffffffffc06d3220
[1026361.905504] FS:  00007f4d20fd1700(0000) GS:ffff91c79d500000(0000) knlGS:0000000000000000
[1026361.912082] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1026361.916079] CR2: 0000555ec5513980 CR3: 0000000103f84003 CR4: 00000000007706e0
[1026361.922135] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1026361.928213] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[1026361.934299] PKRU: 55555554
[1026361.937091] Call Trace:
[1026361.939793]  ertp_insert_data_to_cache+0x178/0x280 [eset_rtp]
[1026361.943849]  ertp_process_request+0x20c/0x2d0 [eset_rtp]
[1026361.947670]  ? free_unref_page_commit+0x91/0x110
[1026361.951202]  ? ertp_check_file.part.4+0x96/0x1a0 [eset_rtp]
[1026361.955093]  ? ertp_get_data+0x19/0x30 [eset_rtp]
[1026361.958662]  ertp_check_file.part.4+0x96/0x1a0 [eset_rtp]
[1026361.962619]  check_close+0x1b/0x30 [eset_rtp]
[1026361.966169]  __x64_sys_ertp_close+0x1c/0x60 [eset_rtp]
[1026361.969967]  do_syscall_64+0x5b/0x1a0
[1026361.973179]  entry_SYSCALL_64_after_hwframe+0x65/0xca
[1026361.976972] RIP: 0033:0x7f4d2ef9b6eb
[1026361.980139] Code: c3 48 8b 15 9f 97 29 00 f7 d8 64 89 02 b8 ff ff ff ff eb b8 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 69 97 29 00 f7 d8
[1026361.992299] RSP: 002b:00007f4d20fcfd78 EFLAGS: 00000206 ORIG_RAX: 0000000000000003
[1026361.998548] RAX: ffffffffffffffda RBX: 0000555ec5636a00 RCX: 00007f4d2ef9b6eb
[1026362.004727] RDX: 00007f4d2f231880 RSI: 0000000000000001 RDI: 000000000000001f
[1026362.011093] RBP: 00007f4d2f2323c0 R08: 00007f4d20fd1700 R09: 0000000000000000
[1026362.017516] R10: 00007f4d20fcf802 R11: 0000000000000206 R12: 0000000000000000
[1026362.023920] R13: 00007f4d20fcff90 R14: 00007f4d20fcff08 R15: 00007f4d20fcff10
[1026362.030339] Modules linked in: ext4 mbcache jbd2 unix_diag eset_rtp(OE) drbd_transport_tcp(OE) drbd(OE) intel_rapl_msr intel_rapl_common isst_if_common nfit libnvdimm ppdev rapl parport_pc pcspkr parport i2c_piix4 binfmt_misc xfs libcrc32c crct10dif_pclmul nvme_tcp(X) crc32_pclmul nvme_fabrics nvme crc32c_intel ghash_clmulni_intel ena nvme_core serio_raw t10_pi sunrpc dm_mirror dm_region_hash dm_log dm_mod