Ceph/ODF: MDS Crashing (CLBO), the crash backtrace shows "_unlink_" or "_unlink_local".

Issue

MDS Crashing (CLBO), the crash backtrace shows _unlink_ or _unlink_local.

The Ceph metadata daemon is crashing frequently and unlink is seen in the backtrace of the crash

/builddir/build/BUILD/ceph-14.2.11/src/mds/Server.cc: In function 'void Server::_unlink_local(MDRequestRef&, CDentry*, CDentry*)' thread 7f7848acc700 time 2022-02-23 08:48:31.877094
/builddir/build/BUILD/ceph-14.2.11/src/mds/Server.cc: 7023: FAILED ceph_assert(in->first <= straydn->first)
 ceph version 14.2.11-208.el8cp (6738ba96f296a41c24357c12e8d594fbde457abc) nautilus (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x156) [0x7f78578a9308]
 2: (()+0x275522) [0x7f78578a9522]
 3: (Server::_unlink_local(boost::intrusive_ptr<MDRequestImpl>&, CDentry*, CDentry*)+0xfbc) [0x558e8d56654c]
 4: (Server::handle_client_unlink(boost::intrusive_ptr<MDRequestImpl>&)+0xd4c) [0x558e8d56b73c]
 5: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0xaab) [0x558e8d58122b]
 6: (Server::handle_client_request(boost::intrusive_ptr<MClientRequest const> const&)+0x402) [0x558e8d5819a2]
 7: (Server::dispatch(boost::intrusive_ptr<Message const> const&)+0x12a) [0x558e8d58e44a]
 8: (MDSRank::handle_deferrable_message(boost::intrusive_ptr<Message const> const&)+0xa94) [0x558e8d4f7344]
 9: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x80f) [0x558e8d4f975f]
 10: (MDSRank::retry_dispatch(boost::intrusive_ptr<Message const> const&)+0x16) [0x558e8d4f9d66]
 11: (MDSContext::complete(int)+0x7f) [0x558e8d79b5df]
 12: (MDSRank::_advance_queues()+0xac) [0x558e8d4f86ec]
 13: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x1ed) [0x558e8d4f913d]
 14: (MDSRank::retry_dispatch(boost::intrusive_ptr<Message const> const&)+0x16) [0x558e8d4f9d66]
 15: (MDSContext::complete(int)+0x7f) [0x558e8d79b5df]
 16: (MDSRank::_advance_queues()+0xac) [0x558e8d4f86ec]
 17: (MDSRank::ProgressThread::entry()+0x45) [0x558e8d4f8e25]
 18: (()+0x817a) [0x7f785568917a]
 19: (clone()+0x43) [0x7f78541a0dc3]
debug 2022-02-23 08:48:31.877 7f7848acc700 -1 /builddir/build/BUILD/ceph-14.2.11/src/mds/Server.cc: In function 'void Server::_unlink_local(MDRequestRef&, CDentry*, CDentry*)' thread 7f7848acc700 time 2022-02-23 08:48:31.877094
/builddir/build/BUILD/ceph-14.2.11/src/mds/Server.cc: 7023: FAILED ceph_assert(in->first <= straydn->first)

To view current crashes utilize the following command ceph crash ls. To view more information regarding the crash ceph crash info <crashid>. For additional information see Ceph Crash Module