When using CephFS client, kernel panic (RIP) in __ceph_remove_cap

Solution Verified - Updated -

Issue

When using CephFS client, kernel panic (RIP) in __ceph_remove_cap
The issue was observed in RHEL 7.9, kernel 3.10.0-1160.80.1.el7.x86_64
Crash Details:

[1172681.442506] BUG: unable to handle kernel NULL pointer dereference at 0000000000000368
[1172681.443001] IP: [<ffffffffc080595a>] __ceph_remove_cap+0x2a/0x230 [ceph]
[1172681.443386] PGD 8000001fb0174067 PUD 1fb006f067 PMD 0 
[1172681.443733] Oops: 0000 [#1] SMP 
[1172681.444074] Modules linked in: iptable_filter af_packet_diag netlink_diag vsock_diag tcp_diag udp_diag inet_diag unix_diag falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_14306(E) nfsv3 nfs_acl ceph nfs lockd libceph grace fscache dns_resolver vmw_vsock_vmci_transport vsock sunrpc vmwgfx iosf_mbi crc32_pclmul ttm ghash_clmulni_intel drm_kms_helper ppdev syscopyarea vmw_balloon aesni_intel lrw sysfillrect gf128mul glue_helper ablk_helper cryptd sysimgblt fb_sys_fops joydev drm pcspkr sg vmw_vmci i2c_piix4 drm_panel_orientation_quirks parport_pc parport binfmt_misc ip_tables xfs libcrc32c ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic ata_piix nfit libata crct10dif_pclmul crct10dif_common crc32c_intel libnvdimm serio_raw vmxnet3 vmw_pvscsi dm_mirror dm_region_hash
[1172681.446282]  dm_log dm_mod fuse
[1172681.446736] CPU: 3 PID: 18092 Comm: kworker/3:0 Kdump: loaded Tainted: P            E  ------------ T 3.10.0-1160.80.1.el7.x86_64 #1 <<<< 7.9
[1172681.447229] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[1172681.447749] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[1172681.448274] task: ffff96315f99a100 ti: ffff962d45f90000 task.ti: ffff962d45f90000
[1172681.448827] RIP: 0010:[<ffffffffc080595a>]  [<ffffffffc080595a>] __ceph_remove_cap+0x2a/0x230 [ceph]     <--- Here
[1172681.449402] RSP: 0018:ffff962d45f93ae8  EFLAGS: 00010246
[1172681.449939] RAX: 00000000fffff2aa RBX: ffff962f7faeb078 RCX: 0000000000000004
[1172681.450500] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff962f7faeb078
[1172681.451072] RBP: ffff962d45f93b20 R08: 0000000000000000 R09: 0000000000000000
[1172681.451671] R10: 0000000000000000 R11: 0000000000000d55 R12: 0000000000000000
[1172681.452238] R13: ffff962ee7d90000 R14: 0000000000000001 R15: ffff963160e29000
[1172681.452903] FS:  0000000000000000(0000) GS:ffff96317f2c0000(0000) knlGS:0000000000000000
[1172681.453647] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1172681.454428] CR2: 0000000000000368 CR3: 0000001fab4ae000 CR4: 00000000003607e0
[1172681.455181] Call Trace:
[1172681.455974]  [<ffffffffc0811113>] trim_caps_cb+0x103/0x320 [ceph]
[1172681.456711]  [<ffffffff9c07a51b>] ? destroy_inode+0x3b/0x70
[1172681.457462]  [<ffffffff9c07a665>] ? evict+0x115/0x180
[1172681.458240]  [<ffffffffc081047d>] iterate_session_caps+0xbd/0x250 [ceph]
[1172681.459035]  [<ffffffffc0811010>] ? parse_reply_info_in+0x310/0x310 [ceph]
[1172681.459735]  [<ffffffffc08133cc>] ceph_trim_caps+0x4c/0xd0 [ceph]
[1172681.460433]  [<ffffffffc08184f8>] dispatch+0x2f8/0xb00 [ceph]
[1172681.461245]  [<ffffffff9c45e6da>] ? kernel_recvmsg+0x3a/0x50
[1172681.462040]  [<ffffffffc06581d4>] try_read+0x544/0x1300 [libceph]
[1172681.462757]  [<ffffffff9bee8b74>] ? update_curr+0x164/0x1f0
[1172681.463647]  [<ffffffff9bee711e>] ? account_entity_dequeue+0xae/0xd0
[1172681.464351]  [<ffffffff9bee908c>] ? dequeue_entity+0x11c/0x5d0
[1172681.465128]  [<ffffffffc0659194>] ceph_con_workfn+0xe4/0x1540 [libceph]
[1172681.465897]  [<ffffffff9bec319f>] process_one_work+0x17f/0x440
[1172681.466623]  [<ffffffff9bec42e6>] worker_thread+0x126/0x3c0
[1172681.467262]  [<ffffffff9bec41c0>] ? manage_workers.isra.26+0x2b0/0x2b0
[1172681.467872]  [<ffffffff9becb4d1>] kthread+0xd1/0xe0
[1172681.468539]  [<ffffffff9becb400>] ? insert_kthread_work+0x40/0x40
[1172681.469205]  [<ffffffff9c5c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[1172681.469855]  [<ffffffff9becb400>] ? insert_kthread_work+0x40/0x40
[1172681.470528] Code: 00 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 89 f6 41 55 41 54 53 48 89 fb 48 83 ec 10 4c 8b 27 f6 05 e4 ef 02 00 04 4c 8b 7f 20 <49> 8b 84 24 68 03 00 00 48 8b 80 50 03 00 00 48 8b 40 28 48 89 
[1172681.472240] RIP  [<ffffffffc080595a>] __ceph_remove_cap+0x2a/0x230 [ceph]     <--- Here
[1172681.473230]  RSP <ffff962d45f93ae8>
[1172681.474213] CR2: 0000000000000368

Environment

Red Hat Enterprise Linux (RHEL) 7.9
Red Hat Enterprise Linux (RHEL) 8.x
Red Hat Enterprise Linux (RHEL) 9.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content