Ceph - CephFS kernel client crashed with 'PANIC: "kernel BUG at fs/ceph/mds_client.c:1229!"'

Solution In Progress - Updated -

Issue

  • CephFS kernel client crashed with 'PANIC: "kernel BUG at fs/ceph/mds_client.c:1229!"'
[3277706.270816] kernel BUG at fs/ceph/mds_client.c:1229!
[3277706.277518] invalid opcode: 0000 [#1] SMP 
[3277706.283460] Modules linked in: ceph libceph rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache veth xt_multiport iptable_raw iptabl
e_mangle ip_set_hash_ip ip_set_hash_net ipip tunnel4 ip_tunnel xt_set ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipport ip
_set_hash_ipportip ip_set dummy ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark ip_vs_sh ip_vs_wrr ip_v
s_rr ip_vs ip6table_filter ip6_tables sctp_diag sctp dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_dia
g binfmt_misc ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_n
at_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge 8021q garp mrp stp llc overlay(T) bonding ex
t4 mbcache jbd2 vfat
[3277706.372796]  fat iTCO_wdt iTCO_vendor_support skx_edac edac_core coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pcl
mul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses enclosure scsi_transport_sas joydev sg me
i_me mei i2c_i801 i2c_core lpc_ich shpchp nfit libnvdimm acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs 
libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci libata megaraid_sas i40e(O
E) ptp pps_core dm_mirror dm_region_hash dm_log dm_mod
[3277706.446050] CPU: 7 PID: 16308 Comm: kworker/7:0 Tainted: G        W  OE  ------------ T 3.10.0-693.17.1.el7.x86_64 #1
[3277706.462360] Hardware name: Huawei 2288H V5/BC11SPSCA0, BIOS 0.99 11/14/2018
[3277706.475210] Workqueue: ceph-msgr ceph_con_workfn [libceph]
[3277706.486696] task: ffff88015205cf10 ti: ffff88112427c000 task.ti: ffff88112427c000
[3277706.500393] RIP: 0010:[<ffffffffc0a09653>]  [<ffffffffc0a09653>] remove_session_caps+0x153/0x160 [ceph]
[3277706.516304] RSP: 0018:ffff88112427fc48  EFLAGS: 00010202
[3277706.528221] RAX: 0000000000000001 RBX: ffff881ef0f10540 RCX: 0000000000000400
[3277706.542165] RDX: 0000000000000289 RSI: ffff881ebc6eca08 RDI: ffff88112427fc08
[3277706.556198] RBP: ffff88112427fc88 R08: ffff8834f6b98d88 R09: 0000000000000000
[3277706.570350] R10: 0000000000000000 R11: 0000000000000000 R12: ffff881ef0f10000
[3277706.584633] R13: ffff8834f6b98e88 R14: ffff881ef0f10548 R15: ffff881994bf8000
[3277706.599016] FS:  0000000000000000(0000) GS:ffff881ffe7c0000(0000) knlGS:0000000000000000
[3277706.614575] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3277706.627887] CR2: 000055f301942fe0 CR3: 00000019a2f52000 CR4: 00000000003607e0
[3277706.642769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3277706.657733] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3277706.672726] Call Trace:
[3277706.683083]  [<ffffffffc0a1054e>] dispatch+0x3fe/0xb00 [ceph]
[3277706.696939]  [<ffffffff8156c90a>] ? kernel_recvmsg+0x3a/0x50
[3277706.710811]  [<ffffffffc094af2f>] try_read+0x4df/0x1240 [libceph]
[3277706.725222]  [<ffffffff810d043e>] ? dequeue_task_fair+0x41e/0x660
[3277706.739742]  [<ffffffff810c95d5>] ? sched_clock_cpu+0x85/0xc0
[3277706.753996]  [<ffffffff8102954d>] ? __switch_to+0xcd/0x500
[3277706.768059]  [<ffffffffc094bd49>] ceph_con_workfn+0xb9/0x650 [libceph]
[3277706.783295]  [<ffffffff810aa59a>] process_one_work+0x17a/0x440
[3277706.797940]  [<ffffffff810ab266>] worker_thread+0x126/0x3c0
[3277706.812406]  [<ffffffff810ab140>] ? manage_workers.isra.24+0x2a0/0x2a0
[3277706.827965]  [<ffffffff810b270f>] kthread+0xcf/0xe0
[3277706.841932]  [<ffffffff810b2640>] ? insert_kthread_work+0x40/0x40
[3277706.857235]  [<ffffffff816b8798>] ret_from_fork+0x58/0x90
[3277706.871909]  [<ffffffff810b2640>] ? insert_kthread_work+0x40/0x40
[3277706.887386] Code: 5c 41 5d 41 5e 41 5f 5d c3 48 89 fa 48 c7 c6 10 f6 a1 c0 48 c7 c7 30 cc a2 c0 e8 49 35 94 c0 e9 e9 fe ff ff e
8 ef fc 67 c0 0f 0b <0f> 0b 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 49 89 
[3277706.927141] RIP  [<ffffffffc0a09653>] remove_session_caps+0x153/0x160 [ceph]
[3277706.944188]  RSP <ffff88112427fc48>

Environment

  • Red Hat Enterprise Linux 7.4
  • kernel-3.10.0-693.17.1.el7.x86_64
  • CephFS

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In