When using CephFS client, kernel panic (RIP) in __ceph_remove_cap
Issue
When using CephFS client, kernel panic (RIP) in __ceph_remove_cap
The issue was observed in RHEL 7.9, kernel 3.10.0-1160.80.1.el7.x86_64
Crash Details:
[1172681.442506] BUG: unable to handle kernel NULL pointer dereference at 0000000000000368
[1172681.443001] IP: [<ffffffffc080595a>] __ceph_remove_cap+0x2a/0x230 [ceph]
[1172681.443386] PGD 8000001fb0174067 PUD 1fb006f067 PMD 0
[1172681.443733] Oops: 0000 [#1] SMP
[1172681.444074] Modules linked in: iptable_filter af_packet_diag netlink_diag vsock_diag tcp_diag udp_diag inet_diag unix_diag falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_14306(E) nfsv3 nfs_acl ceph nfs lockd libceph grace fscache dns_resolver vmw_vsock_vmci_transport vsock sunrpc vmwgfx iosf_mbi crc32_pclmul ttm ghash_clmulni_intel drm_kms_helper ppdev syscopyarea vmw_balloon aesni_intel lrw sysfillrect gf128mul glue_helper ablk_helper cryptd sysimgblt fb_sys_fops joydev drm pcspkr sg vmw_vmci i2c_piix4 drm_panel_orientation_quirks parport_pc parport binfmt_misc ip_tables xfs libcrc32c ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic ata_piix nfit libata crct10dif_pclmul crct10dif_common crc32c_intel libnvdimm serio_raw vmxnet3 vmw_pvscsi dm_mirror dm_region_hash
[1172681.446282] dm_log dm_mod fuse
[1172681.446736] CPU: 3 PID: 18092 Comm: kworker/3:0 Kdump: loaded Tainted: P E ------------ T 3.10.0-1160.80.1.el7.x86_64 #1 <<<< 7.9
[1172681.447229] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[1172681.447749] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[1172681.448274] task: ffff96315f99a100 ti: ffff962d45f90000 task.ti: ffff962d45f90000
[1172681.448827] RIP: 0010:[<ffffffffc080595a>] [<ffffffffc080595a>] __ceph_remove_cap+0x2a/0x230 [ceph] <--- Here
[1172681.449402] RSP: 0018:ffff962d45f93ae8 EFLAGS: 00010246
[1172681.449939] RAX: 00000000fffff2aa RBX: ffff962f7faeb078 RCX: 0000000000000004
[1172681.450500] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff962f7faeb078
[1172681.451072] RBP: ffff962d45f93b20 R08: 0000000000000000 R09: 0000000000000000
[1172681.451671] R10: 0000000000000000 R11: 0000000000000d55 R12: 0000000000000000
[1172681.452238] R13: ffff962ee7d90000 R14: 0000000000000001 R15: ffff963160e29000
[1172681.452903] FS: 0000000000000000(0000) GS:ffff96317f2c0000(0000) knlGS:0000000000000000
[1172681.453647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1172681.454428] CR2: 0000000000000368 CR3: 0000001fab4ae000 CR4: 00000000003607e0
[1172681.455181] Call Trace:
[1172681.455974] [<ffffffffc0811113>] trim_caps_cb+0x103/0x320 [ceph]
[1172681.456711] [<ffffffff9c07a51b>] ? destroy_inode+0x3b/0x70
[1172681.457462] [<ffffffff9c07a665>] ? evict+0x115/0x180
[1172681.458240] [<ffffffffc081047d>] iterate_session_caps+0xbd/0x250 [ceph]
[1172681.459035] [<ffffffffc0811010>] ? parse_reply_info_in+0x310/0x310 [ceph]
[1172681.459735] [<ffffffffc08133cc>] ceph_trim_caps+0x4c/0xd0 [ceph]
[1172681.460433] [<ffffffffc08184f8>] dispatch+0x2f8/0xb00 [ceph]
[1172681.461245] [<ffffffff9c45e6da>] ? kernel_recvmsg+0x3a/0x50
[1172681.462040] [<ffffffffc06581d4>] try_read+0x544/0x1300 [libceph]
[1172681.462757] [<ffffffff9bee8b74>] ? update_curr+0x164/0x1f0
[1172681.463647] [<ffffffff9bee711e>] ? account_entity_dequeue+0xae/0xd0
[1172681.464351] [<ffffffff9bee908c>] ? dequeue_entity+0x11c/0x5d0
[1172681.465128] [<ffffffffc0659194>] ceph_con_workfn+0xe4/0x1540 [libceph]
[1172681.465897] [<ffffffff9bec319f>] process_one_work+0x17f/0x440
[1172681.466623] [<ffffffff9bec42e6>] worker_thread+0x126/0x3c0
[1172681.467262] [<ffffffff9bec41c0>] ? manage_workers.isra.26+0x2b0/0x2b0
[1172681.467872] [<ffffffff9becb4d1>] kthread+0xd1/0xe0
[1172681.468539] [<ffffffff9becb400>] ? insert_kthread_work+0x40/0x40
[1172681.469205] [<ffffffff9c5c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[1172681.469855] [<ffffffff9becb400>] ? insert_kthread_work+0x40/0x40
[1172681.470528] Code: 00 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 89 f6 41 55 41 54 53 48 89 fb 48 83 ec 10 4c 8b 27 f6 05 e4 ef 02 00 04 4c 8b 7f 20 <49> 8b 84 24 68 03 00 00 48 8b 80 50 03 00 00 48 8b 40 28 48 89
[1172681.472240] RIP [<ffffffffc080595a>] __ceph_remove_cap+0x2a/0x230 [ceph] <--- Here
[1172681.473230] RSP <ffff962d45f93ae8>
[1172681.474213] CR2: 0000000000000368
Environment
Red Hat Enterprise Linux (RHEL) 7.9
Red Hat Enterprise Linux (RHEL) 8.x
Red Hat Enterprise Linux (RHEL) 9.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.