RHEL 8.6: kernel crashed at dfs_cache_free_tgts during cifs_reconnect

Issue

• Kernel crashed with below logs:

[2427411.033222] CIFS: VFS: cifs_mount failed w/return code = -115
[2427412.989636] CIFS: Attempting to mount \\cifsmount
[2427419.981293] CIFS: VFS: \\cifsmount Cancelling wait for mid 0 cmd: 0
[2427419.981330] CIFS: VFS: cifs_mount failed w/return code = -512
[2427419.999169] CIFS: Attempting to mount \\cifsmount
[2427420.313326] Unable to handle kernel paging request for data at address 0x00000000
[2427420.313341] Faulting instruction address: 0xd000000005ee439c
[2427420.313345] Oops: Kernel access of bad area, sig: 11 [#1]
[2427420.313347] LE SMP NR_CPUS=2048 NUMA pSeries
[2427420.313353] Modules linked in: md4 cmac nls_utf8 cifs libarc4 dns_resolver rpadlpar_io rpaphp mptcp_diag xsk_diag tcp_diag udp_diag raw_diag inet_diag unix_diag af_packet_diag netlink_diag nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_tables_set nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink rpcrdma sunrpc rdma_ucm ib_srpt ib_isert iscsi_target_mod target_core_mod ib_iser libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm ib_cm mlx5_ib pseries_rng ib_uverbs ib_core xts vmx_crypto uio_pdrv_genirq uio xfs libcrc32c dm_service_time sr_mod cdrom sd_mod t10_pi sg mlx5_core ibmvfc ibmvscsi scsi_transport_fc scsi_transport_srp mlxfw tls psample dm_multipath dm_mirror dm_region_hash dm_log dm_mod fuse
[2427420.313417] CPU: 2 PID: 93278 Comm: cifsd Kdump: loaded Not tainted 4.18.0-348.23.1.el8_5.ppc64le #1
[2427420.313421] NIP:  d000000005ee439c LR: d000000005ee3a44 CTR: c000000000f10000
[2427420.313426] REGS: c000000502c47870 TRAP: 0300   Not tainted  (4.18.0-348.23.1.el8_5.ppc64le)
[2427420.313429] MSR:  800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  CR: 28008288  XER: 20040000
[2427420.313436] CFAR: c000000000008934 DAR: 0000000000000000 DSISR: 40000000 IRQMASK: 0 
                 GPR00: d000000005ee3a44 c000000502c47b00 d000000005fea800 0000000000000000 
                 GPR04: c000000502c47ae8 c000000502c47ae8 0024a18c638fd590 c000000007fcde80 
                 GPR08: c0000007ff80c280 0000000000000000 0000000000000000 d000000005f7a898 
                 GPR12: c000000000f10000 c000000007fcde80 c0000000001a1288 0000000000000000 
                 GPR16: 0000000000000001 0000000000000001 0000000000000000 0000000000000001 
                 GPR20: 0000000000000000 c0000003fcb05860 0000000000000000 0000000000000000 
                 GPR24: 0000000000000000 d00000000600c17c 0000000000000000 c000000502c47b98 
                 GPR28: ffffffffffffffea d00000000600c104 0000000000000000 c0000003fcb05800 
[2427420.313486] NIP [d000000005ee439c] cifs_reconnect+0xa44/0x1590 [cifs]
[2427420.313503] LR [d000000005ee3a44] cifs_reconnect+0xec/0x1590 [cifs]
[2427420.313506] Call Trace:
[2427420.313519] [c000000502c47b00] [d000000005ee3a08] cifs_reconnect+0xb0/0x1590 [cifs] (unreliable)
[2427420.313534] [c000000502c47c40] [d000000005ee5e84] cifs_demultiplex_thread+0x68c/0x1148 [cifs]
[2427420.313540] [c000000502c47db0] [c0000000001a1430] kthread+0x1b0/0x1c0
[2427420.313546] [c000000502c47e20] [c00000000000b7d8] ret_from_kernel_thread+0x5c/0x64
[2427420.313549] Instruction dump:
[2427420.313555] a1490cf2 394affff b1490cf2 7c2004ac 39200000 91390000 e9210098 3b610098 
[2427420.313560] 7fbb4800 419e00ec ebc10098 7fc3f378 <ebbe0000> 3bdefff0 7fa3d800 3bbdfff0 
[2427420.313566] ---[ end trace 9b60a4fdb8c95490 ]---

• Another logs:

[16874.490487] CIFS: Attempting to mount \\cifsmount
[16874.491657] CIFS: VFS: cifs_mount failed w/return code = -13
[16874.512866] CIFS: Attempting to mount \\cifsmount
[16874.514225] CIFS: VFS: cifs_mount failed w/return code = -13
[16875.928304] CIFS: VFS:  BAD_NETWORK_NAME: \\cifsmount
[16875.965597] CIFS: VFS:  BAD_NETWORK_NAME: \\cifsmount
[16878.517321] CIFS: VFS:  BAD_NETWORK_NAME: \\cifsmount
[17010.289052] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[17010.289707] PGD 0 P4D 0 
[17010.290300] Oops: 0000 [#1] SMP NOPTI
[17010.290943] CPU: 3 PID: 4047 Comm: cifsd Kdump: loaded Tainted: P                 --------- -  - 4.18.0-372.19.1.el8_6.x86_64 #1
[17010.292117] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.18227214.B64.2106252220 06/25/2021
[17010.293012] RIP: 0010:cifs_reconnect+0x983/0xc50 [cifs]
[17010.293580] Code: 54 24 30 48 8d 44 24 30 48 39 d0 74 7a 48 8b 7c 24 30 48 89 04 24 49 be 00 01 00 00 00 00 ad de 49 bd 00 02 00 00 00 00 ad de <48> 8b 37 4c 8d 7f f0 48 8d 6e f0 48 39 c7 74 45 e8 28 f5 b0 e4 84
[17010.294762] RSP: 0018:ffffbadb11f37d18 EFLAGS: 00010286
[17010.295324] RAX: ffffbadb11f37d48 RBX: ffff969762b0c800 RCX: 0000000000000000
[17010.295908] RDX: 0000000000000000 RSI: ffffbadb11f37cf8 RDI: 0000000000000000
[17010.296497] RBP: ffff969762b0c9bc R08: 0000000000000000 R09: ffff969783bec0dc
[17010.297098] R10: ffff969783bebb48 R11: 0000000000000001 R12: 00000000ffffffea
[17010.297727] R13: dead000000000200 R14: dead000000000100 R15: 0000000000000000
[17010.298438] FS:  0000000000000000(0000) GS:ffff96aa2db80000(0000) knlGS:0000000000000000
[17010.299089] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[17010.299937] CR2: 0000000000000000 CR3: 0000000c3e810004 CR4: 00000000007706e0
[17010.300818] PKRU: 55555554
[17010.301501] Call Trace:
[17010.301994]  cifs_readv_from_socket+0x1ad/0x260 [cifs]
[17010.302382]  cifs_read_from_socket+0x4a/0x70 [cifs]
[17010.302764]  ? cifs_small_buf_get+0x16/0x20 [cifs]
[17010.303130]  ? allocate_buffers+0x66/0x120 [cifs]
[17010.303498]  cifs_demultiplex_thread+0xf6/0xbf0 [cifs]
[17010.303856]  ? cifs_handle_standard+0x190/0x190 [cifs]
[17010.304225]  kthread+0x10a/0x120
[17010.304574]  ? set_kthread_struct+0x40/0x40
[17010.304908]  ret_from_fork+0x1f/0x40
[17010.305267] Modules linked in: cmac md4 nls_utf8 cifs rdma_cm iw_cm ib_cm ib_core libarc4 falcon_lsm_serviceable(P) falcon_nf_netcontain(P) falcon_kal falcon_lsm_pinned_14108 nfsv3 nfs_acl rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache nft_redir nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_tables_set nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vfat fat intel_rapl_msr intel_rapl_common isst_if_mbox_msr isst_if_common nfit libnvdimm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl vmw_balloon joydev pcspkr vmw_vmci i2c_piix4 auth_rpcgss binfmt_misc sunrpc xfs libcrc32c sd_mod sr_mod cdrom t10_pi sg ata_generic vmwgfx ttm drm_kms_helper syscopyarea sysfillrect sysimgblt ahci fb_sys_fops libahci ata_piix drm crc32c_intel
[17010.305324]  libata serio_raw vmxnet3 vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod fuse ipmi_devintf ipmi_msghandler
[17010.309119] CR2: 0000000000000000