Ceph: A subset of OSDs are in a crash loop: "bluestore/BlueFS.cc: 926: FAILED ceph_assert(bl.length() <= get_super_length())".

Issue

A subset of OSDs are in a crash loop: "bluestore/BlueFS.cc: 926: FAILED ceph_assert(bl.length() <= get_super_length())".

For Ceph Clusters where large OSDs are in use and much of the data written to cluster is small files, the BlueFS may grow extremely large and may result in the ceph_assert above.

Full example:

janv. 13 13:05:58 pve14 systemd1: Starting Ceph object storage daemon osd.7...
janv. 13 13:05:58 pve14 systemd1: Started Ceph object storage daemon osd.7.
janv. 13 13:06:14 pve14 ceph-osd2821431: ./src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_write_super(int)' thread 7fa9df456f00 time 2022-01-13T13:06:14.347694+0100
janv. 13 13:06:14 pve14 ceph-osd2821431: ./src/os/bluestore/BlueFS.cc: 926: FAILED ceph_assert(bl.length() <= get_super_length())
janv. 13 13:06:14 pve14 ceph-osd2821431: ceph version 16.2.7 (f9aa029788115b5df5eeee328f584156565ee5b7) pacific (stable)
janv. 13 13:06:14 pve14 ceph-osd2821431: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x5597005b0ae6]
janv. 13 13:06:14 pve14 ceph-osd2821431: 2: /usr/bin/ceph-osd(+0xabcc71) [0x5597005b0c71]
janv. 13 13:06:14 pve14 ceph-osd2821431: 3: (BlueFS::_write_super(int)+0x586) [0x559700c9ff86]
janv. 13 13:06:14 pve14 ceph-osd2821431: 4: (BlueFS::_compact_log_async(std::unique_lock<std::mutex>&)+0xb33) [0x559700caa363]
janv. 13 13:06:14 pve14 ceph-osd2821431: 5: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock<std::mutex>&)+0x67) [0x559700cab497]
janv. 13 13:06:14 pve14 ceph-osd2821431: 6: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x100) [0x559700cc37d0]
janv. 13 13:06:14 pve14 ceph-osd2821431: 7: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x48) [0x55970118a24e]
janv. 13 13:06:14 pve14 ceph-osd2821431: 8: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x338) [0x559701364d18]
janv. 13 13:06:14 pve14 ceph-osd2821431: 9: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x5d7) [0x55970136329b]
janv. 13 13:06:14 pve14 ceph-osd2821431: 10: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0x11d) [0x55970152d2d7]
janv. 13 13:06:14 pve14 ceph-osd2821431: 11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x7d0) [0x55970152d0be]
janv. 13 13:06:14 pve14 ceph-osd2821431: 12: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x48) [0x55970152c8da]
janv. 13 13:06:14 pve14 ceph-osd2821431: 13: (rocksdb::BlockBasedTableBuilder::Flush()+0x9a) [0x55970152c88a]
janv. 13 13:06:14 pve14 ceph-osd2821431: 14: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x197) [0x55970152c3bf]
janv. 13 13:06:14 pve14 ceph-osd2821431: 15: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocks>
janv. 13 13:06:14 pve14 ceph-osd2821431: 16: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x5ea) [0x559701228226]
janv. 13 13:06:14 pve14 ceph-osd2821431: 17: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0x1ad1) [0x559701226e9d]
janv. 13 13:06:14 pve14 ceph-osd2821431: 18: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0x159e) [0x5597012243d4]
janv. 13 13:06:14 pve14 ceph-osd2821431: 19: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std:>
janv. 13 13:06:14 pve14 ceph-osd2821431: 20: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::all>
janv. 13 13:06:14 pve14 ceph-osd2821431: 21: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x10a6) [0x5597011398b6]
janv. 13 13:06:14 pve14 ceph-osd2821431: 22: (BlueStore::_open_db(bool, bool, bool)+0xa19) [0x559700bb7b19]
janv. 13 13:06:14 pve14 ceph-osd2821431: 23: (BlueStore::_open_db_and_around(bool, bool)+0x332) [0x559700bfcb92]
janv. 13 13:06:14 pve14 ceph-osd2821431: 24: (BlueStore::_mount()+0x191) [0x559700bff531]
janv. 13 13:06:14 pve14 ceph-osd2821431: 25: (OSD::init()+0x58d) [0x5597006a65ed]
janv. 13 13:06:14 pve14 ceph-osd2821431: 26: main()
janv. 13 13:06:14 pve14 ceph-osd2821431: 27: __libc_start_main()
janv. 13 13:06:14 pve14 ceph-osd2821431: 28: _start()
janv. 13 13:06:14 pve14 ceph-osd2821431: ** Caught signal (Aborted) *