RHEL-9.5: kernel panic in open_cached_dir_by_dentry or in find_or_create_cached_dir

Issue

• Kernel panic in find_or_create_cached_dir.

[257573.790647] BUG: kernel NULL pointer dereference, address: 0000000000000000
[257573.790652] #PF: supervisor read access in kernel mode
[257573.790654] #PF: error_code(0x0000) - not-present page
[257573.790656] PGD 0 P4D 0 
[257573.790659] Oops: 0000 [#1] PREEMPT SMP NOPTI
[257573.790663] CPU: 16 PID: 3482236 Comm: ls Kdump: loaded Not tainted 5.14.0-503.21.1.el9_5.x86_64 #1
[257573.790666] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.21100432.B64.2301110304 01/11/2023
[257573.790668] RIP: 0010:strcmp+0xc/0x30
[257573.790679] Code: 01 45 84 c0 75 ee c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 31 c0 eb 08 48 83 c0 01 84 d2 74 13 <0f> b6 14 07 3a 14 06 74 ef 19 c0 83 c8 01 c3 cc cc cc cc 31 c0 c3
[257573.790682] RSP: 0018:ffffa2e293cafa90 EFLAGS: 00010246
[257573.790684] RAX: 0000000000000000 RBX: ffff889d5273fa08 RCX: 0000000000000010
[257573.790686] RDX: 0000000000000001 RSI: ffff889d4dcc1fe9 RDI: 0000000000000000
[257573.790688] RBP: ffff889d4dcc1fe9 R08: 0000000000000001 R09: 0000000000000001
[257573.790690] R10: 0000000000000065 R11: ffffa2e293caf9c4 R12: ffff88a223f9b600
[257573.790692] R13: ffff889d5273fa00 R14: 0000000000000000 R15: 0000000000000010
[257573.790694] FS:  00007f1c810c6c40(0000) GS:ffff88b41fa00000(0000) knlGS:0000000000000000
[257573.790696] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[257573.790699] CR2: 0000000000000000 CR3: 000000012dbc4005 CR4: 00000000007706f0
[257573.790731] PKRU: 55555554
[257573.790733] Call Trace:
[257573.790735]  <TASK>
[257573.790739]  ? show_trace_log_lvl+0x1c4/0x2df
[257573.790747]  ? show_trace_log_lvl+0x1c4/0x2df
[257573.790754]  ? find_or_create_cached_dir+0x45/0x200 [cifs]
[257573.790978]  ? __die_body.cold+0x8/0xd
[257573.790983]  ? page_fault_oops+0x134/0x170
[257573.790988]  ? cifsConvertToUTF16.part.0+0x4d/0x3b0 [cifs]
[257573.791122]  ? exc_page_fault+0x62/0x150
[257573.791132]  ? asm_exc_page_fault+0x22/0x30
[257573.791140]  ? strcmp+0xc/0x30
[257573.791145]  find_or_create_cached_dir+0x45/0x200 [cifs]
[257573.791267]  open_cached_dir+0x153/0x9f0 [cifs]
[257573.791405]  ? vm_mmap_pgoff+0xfd/0x180
[257573.791412]  ? rmqueue+0x426/0x10d0
[257573.791417]  ? avc_has_perm_noaudit+0x94/0x110
[257573.791424]  ? selinux_file_open+0xad/0xd0
[257573.791434]  ? __dentry_path+0xda/0x130
[257573.791451]  ? dentry_path_raw+0x4a/0x70
[257573.791454]  ? __build_path_from_dentry_optional_prefix+0x85/0x250 [cifs]
[257573.791588]  ? cifs_readdir+0xdf/0xbb0 [cifs]
[257573.791731]  cifs_readdir+0xdf/0xbb0 [cifs]
[257573.791888]  ? selinux_file_permission+0x108/0x150
[257573.791894]  iterate_dir+0x179/0x1c0
[257573.791901]  __x64_sys_getdents64+0x80/0x120
[257573.791905]  ? __pfx_filldir64+0x10/0x10
[257573.791908]  do_syscall_64+0x5c/0xf0
[257573.791914]  ? __count_memcg_events+0x4f/0xb0
[257573.791918]  ? mm_account_fault+0x6c/0x100
[257573.791925]  ? handle_mm_fault+0x116/0x270
[257573.791928]  ? do_user_addr_fault+0x1d6/0x6a0
[257573.791932]  ? clear_bhb_loop+0x25/0x80
[257573.791936]  ? clear_bhb_loop+0x25/0x80
[257573.791939]  ? clear_bhb_loop+0x25/0x80
[257573.791942]  ? clear_bhb_loop+0x25/0x80
[257573.791944]  ? clear_bhb_loop+0x25/0x80
[257573.791947]  entry_SYSCALL_64_after_hwframe+0x78/0x80
[257573.791951] RIP: 0033:0x7f1c80ed4da7
[257573.791977] Code: 1d fb ff 4c 89 e0 5b 5d 41 5c c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 ff ff ff 7f 48 39 c2 48 0f 47 d0 b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 41 40 12 00 f7 d8 64 89 02 48
[257573.791980] RSP: 002b:00007fff1037f4f8 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9
[257573.791984] RAX: ffffffffffffffda RBX: 00007f1c80cff040 RCX: 00007f1c80ed4da7
[257573.791986] RDX: 0000000000100000 RSI: 00007f1c80cff040 RDI: 0000000000000003
[257573.791988] RBP: 00007f1c80cff014 R08: 00007f1c80cff010 R09: 0000000000000000
[257573.791990] R10: 0000000000000022 R11: 0000000000000293 R12: fffffffffffffea0
[257573.791991] R13: 0000000000000000 R14: 00007f1c80cff010 R15: 0000000000000000
[257573.791995]  </TASK>

• Kernel panic in open_cached_dir_by_dentry:

[24258.785837] general protection fault, probably for non-canonical address 0x6b6b6b6b6b6b6c03: 0000 [#1] PREEMPT SMP NOPTI
[24258.785842] CPU: 8 PID: 1160274 Comm: irisdb Kdump: loaded Not tainted 5.14.0-503.21.1.el9_5.x86_64 #1
[24258.785845] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.21100432.B64.2301110304 01/11/2023
[24258.785846] RIP: 0010:open_cached_dir_by_dentry+0x3d/0x100 [cifs]
[24258.786029] Code: 85 e4 0f 84 d5 00 00 00 4c 89 e7 48 89 f5 49 89 d5 e8 57 d1 47 c6 49 8b 5c 24 08 49 8d 44 24 08 48 39 c3 74 16 48 85 ed 74 09 <48> 39 ab 98 00 00 00 74 20 48 8b 1b 48 39 c3 75 ea 4c 89 e7 e8 1a
[24258.786031] RSP: 0018:ffffb4108734f868 EFLAGS: 00010282
[24258.786033] RAX: ffff9171b9631b88 RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000038
[24258.786034] RDX: 0000000000000001 RSI: ffff9171b18ff8c8 RDI: ffff9171b9631b80
[24258.786035] RBP: ffff9171b18ff8c8 R08: 0000000000000001 R09: 0000000000000007
[24258.786036] R10: ffff9171b18ff8c8 R11: d0868b968d9e93bc R12: ffff9171b9631b80
[24258.786037] R13: ffffb4108734f890 R14: d0d0d0d0d0d0d0d0 R15: 2f2f2f2f2f2f2f2f
[24258.786038] FS:  00007fc5c6775200(0000) GS:ffff91885f800000(0000) knlGS:0000000000000000
[24258.786039] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[24258.786040] CR2: 00007f8519755214 CR3: 000000017f1ac001 CR4: 00000000007706f0
[24258.786065] PKRU: 55555554
[24258.786066] Call Trace:
[24258.786067]  <TASK>
[24258.786068]  ? show_trace_log_lvl+0x1c4/0x2df
[24258.786077]  ? show_trace_log_lvl+0x1c4/0x2df
[24258.786079]  ? cifs_dentry_needs_reval+0xb3/0x1a0 [cifs]
[24258.786152]  ? __die_body.cold+0x8/0xd
[24258.786155]  ? die_addr+0x39/0x60
[24258.786177]  ? exc_general_protection+0x1c8/0x420
[24258.786185]  ? asm_exc_general_protection+0x22/0x30
[24258.786192]  ? open_cached_dir_by_dentry+0x3d/0x100 [cifs]
[24258.786266]  ? open_cached_dir_by_dentry+0x29/0x100 [cifs]
[24258.786347]  cifs_dentry_needs_reval+0xb3/0x1a0 [cifs]
[24258.786418]  cifs_revalidate_dentry_attr+0x3e/0x3a0 [cifs]
[24258.786489]  cifs_revalidate_dentry+0xf/0x30 [cifs]
[24258.786557]  cifs_d_revalidate+0x57/0x160 [cifs]
[24258.786628]  lookup_fast+0xda/0x160
[24258.786632]  walk_component+0x41/0x1d0
[24258.786635]  link_path_walk.part.0.constprop.0+0x24e/0x3c0
[24258.786637]  ? path_init+0x2c5/0x3f0
[24258.786639]  path_parentat+0x3c/0x90
[24258.786641]  __filename_parentat+0xd8/0x1e0
[24258.786643]  ? __pfx_stack_trace_consume_entry+0x10/0x10
[24258.786648]  ? __pfx_kfree_link+0x10/0x10
[24258.786653]  ? init_object+0xa4/0xd0
[24258.786656]  do_unlinkat+0x58/0x2e0
[24258.786659]  __x64_sys_unlink+0x3e/0x60
[24258.786662]  do_syscall_64+0x5c/0xf0
[24258.786663]  ? fpregs_restore_userregs+0x47/0xd0
[24258.786669]  ? exit_to_user_mode_prepare+0xef/0x100
[24258.786672]  ? syscall_exit_to_user_mode+0x19/0x40
[24258.786678]  ? do_syscall_64+0x6b/0xf0
[24258.786679]  ? stack_depot_save_flags+0x41/0x4c0
[24258.786685]  ? check_bytes_and_report+0x52/0xc0
[24258.786687]  ? check_object+0x146/0x350
[24258.786689]  ? init_object+0xa4/0xd0
[24258.786691]  ? free_debug_processing+0x82/0x2e0
[24258.786693]  ? do_unlinkat+0x89/0x2e0
[24258.786696]  ? free_to_partial_list+0x80/0x280
[24258.786698]  ? do_unlinkat+0x89/0x2e0
[24258.786700]  ? do_unlinkat+0x89/0x2e0
[24258.786703]  ? syscall_exit_work+0x103/0x130
[24258.786704]  ? syscall_exit_to_user_mode+0x19/0x40
[24258.786706]  ? do_syscall_64+0x6b/0xf0
[24258.786708]  ? syscall_exit_work+0x103/0x130
[24258.786709]  ? syscall_exit_to_user_mode+0x19/0x40
[24258.786711]  ? clear_bhb_loop+0x25/0x80
[24258.786714]  ? clear_bhb_loop+0x25/0x80
[24258.786715]  ? clear_bhb_loop+0x25/0x80
[24258.786717]  ? clear_bhb_loop+0x25/0x80
[24258.786719]  ? clear_bhb_loop+0x25/0x80
[24258.786721]  entry_SYSCALL_64_after_hwframe+0x78/0x80
[24258.786723] RIP: 0033:0x7fc5c5aff37b
[24258.786743] Code: f0 ff ff 73 01 c3 48 8b 0d 9a 9a 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 6d 9a 0f 00 f7 d8 64 89 01 48
[24258.786745] RSP: 002b:00007ffd86da25f8 EFLAGS: 00000206 ORIG_RAX: 0000000000000057
[24258.786747] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007fc5c5aff37b
[24258.786748] RDX: 0000000000c99290 RSI: 00007fc57da81802 RDI: 00007fc57e9fe010
[24258.786749] RBP: 0000000000000003 R08: 0000000000000001 R09: 00007fc57e9fe010
[24258.786750] R10: 00007fc57da2c018 R11: 0000000000000206 R12: 0000000000000030
[24258.786751] R13: 0000000000000005 R14: ffffffffffffffff R15: 0000000000000081
[24258.786753]  </TASK>

• another panic log with list_del corruption:

[295468.230412] list_del corruption, ffff9f1c5aaa8400->next is LIST_POISON1 (dead000000000100)
[295468.230460] ------------[ cut here ]------------
[295468.230461] kernel BUG at lib/list_debug.c:45!
[295468.230477] invalid opcode: 0000 [#1] PREEMPT SMP PTI
[295468.230479] CPU: 9 PID: 109820 Comm: Thread-21 (Acti Kdump: loaded Not tainted 5.14.0-503.22.1.el9_5.x86_64 #1
[295468.230482] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[295468.230483] RIP: 0010:__list_del_entry_valid.cold+0xf/0x47
[295468.230514] Code: fe ff 0f 0b 48 89 d1 4c 89 c6 4c 89 ca 48 c7 c7 d8 b5 a7 92 e8 51 77 fe ff 0f 0b 48 89 fe 48 c7 c7 68 b6 a7 92 e8 40 77 fe ff <0f> 0b 48 c7 c7 18 b7 a7 92 e8 32 77 fe ff 0f 0b 48 89 f2 48 89 fe
[295468.230516] RSP: 0018:ffffafdc82cefd18 EFLAGS: 00010246
[295468.230518] RAX: 000000000000004e RBX: ffff9f1c5aaa8430 RCX: 0000000000000000
[295468.230520] RDX: 0000000000000000 RSI: ffff9f1d760608c0 RDI: ffff9f1d760608c0
[295468.230521] RBP: ffff9f1c5aaa8400 R08: 0000000000000000 R09: ffffafdc82cefbd8
[295468.230522] R10: ffffafdc82cefbd0 R11: ffffffff935e93e8 R12: 00000000004b4794
[295468.230524] R13: ffff9f1c43b4bffb R14: ffff9f1c45acf900 R15: 0000000000000000
[295468.230525] FS:  00007f4332f9f640(0000) GS:ffff9f1d76040000(0000) knlGS:0000000000000000
[295468.230526] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[295468.230527] CR2: 00007f43a230c000 CR3: 0000000127638001 CR4: 00000000003706f0
[295468.230549] Call Trace:
[295468.230551]  <TASK>
[295468.230553]  ? show_trace_log_lvl+0x1c4/0x2df
[295468.230557]  ? show_trace_log_lvl+0x1c4/0x2df
[295468.230564]  ? smb2_close_cached_fid+0x4f/0xb0 [cifs]
[295468.231025]  ? __die_body.cold+0x8/0xd
[295468.231030]  ? die+0x2b/0x50
[295468.231033]  ? do_trap+0xce/0x120
[295468.231036]  ? __list_del_entry_valid.cold+0xf/0x47
[295468.231041]  ? do_error_trap+0x65/0x80
[295468.231042]  ? __list_del_entry_valid.cold+0xf/0x47
[295468.231045]  ? exc_invalid_op+0x4e/0x70
[295468.231048]  ? __list_del_entry_valid.cold+0xf/0x47
[295468.231051]  ? asm_exc_invalid_op+0x16/0x20
[295468.231056]  ? __list_del_entry_valid.cold+0xf/0x47
[295468.231059]  ? __list_del_entry_valid.cold+0xf/0x47
[295468.231062]  smb2_close_cached_fid+0x4f/0xb0 [cifs]
[295468.231149]  cifs_readdir+0x7f5/0xbb0 [cifs]
[295468.231235]  ? update_load_avg+0x7e/0x7a0
[295468.231239]  ? selinux_file_permission+0x108/0x150
[295468.231244]  iterate_dir+0x17c/0x1c0
[295468.231247]  __x64_sys_getdents64+0x80/0x120
[295468.231250]  ? __pfx_filldir64+0x10/0x10
[295468.231252]  do_syscall_64+0x5f/0xf0
[295468.231254]  ? __hrtimer_run_queues+0x141/0x2b0
[295468.231260]  ? __pfx_read_tsc+0x10/0x10
[295468.231264]  ? ktime_get+0x38/0xa0
[295468.231266]  ? __pfx_lapic_next_deadline+0x10/0x10
[295468.231269]  ? clockevents_program_event+0x96/0x100
[295468.231273]  ? hrtimer_interrupt+0x126/0x210
[295468.231275]  ? sched_clock+0xc/0x30
[295468.231278]  ? sched_clock_cpu+0xb/0x190
[295468.231281]  ? irqtime_account_irq+0x3c/0xb0
[295468.231285]  ? __irq_exit_rcu+0x46/0xc0
[295468.231289]  ? sysvec_apic_timer_interrupt+0x3c/0x90
[295468.231293]  entry_SYSCALL_64_after_hwframe+0x78/0x80
[295468.231295] RIP: 0033:0x7f43a1cd4da7
[295468.231327] Code: 1d fb ff 4c 89 e0 5b 5d 41 5c c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 ff ff ff 7f 48 39 c2 48 0f 47 d0 b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 41 40 12 00 f7 d8 64 89 02 48
[295468.231328] RSP: 002b:00007f4332f9d0e8 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9
[295468.231330] RAX: ffffffffffffffda RBX: 0000556a1fcaff70 RCX: 00007f43a1cd4da7
[295468.231332] RDX: 0000000000100000 RSI: 0000556a1fcaff70 RDI: 000000000000056a
[295468.231333] RBP: 0000556a1fcaff44 R08: 0000556a1693b380 R09: 0000000000000000
[295468.231335] R10: 0000000000000100 R11: 0000000000000293 R12: ffffffffffffff88
[295468.231336] R13: 0000000000000002 R14: 0000556a1fcaff40 R15: 0000556a15432150
[295468.231338]  </TASK>
[295468.231338] Modules linked in: tls binfmt_misc rpcsec_gss_krb5 auth_rpcgss nfsv3 nls_utf8 nfs_acl cifs nfsv4 cifs_arc4 rdma_cm iw_cm nfs ib_cm lockd ib_core grace fscache cifs_md4 dns_resolver netfs nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock sunrpc intel_rapl_msr intel_rapl_common intel_uncore_frequency_common vmwgfx vmw_balloon rapl drm_ttm_helper ttm pcspkr drm_kms_helper i2c_piix4 vmw_vmci joydev drm xfs libcrc32c sr_mod cdrom sd_mod ata_generic t10_pi sg crct10dif_pclmul ata_piix crc32_pclmul crc32c_intel libata ghash_clmulni_intel vmxnet3 vmw_pvscsi serio_raw dm_mirror dm_region_hash dm_log dm_mod fuse