Ceph: OSD fails to start with "bluefs mount failed to replay log: (5) Input/output error" after OSD crashed with "bluestore/AvlAllocator.cc: 173: FAILED ceph_assert(rs->start <= start)".

Issue

An OSD fails to start with "bluefs mount failed to replay log: (5) Input/output error" after the same OSD crashed with "bluestore/AvlAllocator.cc: 173: FAILED ceph_assert(rs->start <= start)".

OSD failing to start:

Mar 21 20:02:53 osmes22cep002 ceph-osd[448006]: bluefs _replay 0x683000: txn(seq 139428395 len 0x23 crc 0x49cd37d2)
Mar 21 20:02:53 osmes22cep002 ceph-osd[448006]: bluefs _replay 0x683000:  op_file_update_inc  delta(ino 2762118 size 0x8762a5 mtime 2025-03-20T14:44:32.276785+0000 offset 11a0000 extents [])
Mar 21 20:02:53 osmes22cep002 ceph-osd[448006]: bluefs _replay 0x683000:  op_file_update_inc produced  file(ino 2762118 size 0x8762a5 mtime 2025-03-20T14:44:32.276785+0000 allocated 11a0000 alloc_commit 0 extents [1:0x5336d9ac000~1d0000,1:0x17f09cdc000~140000,1:0x4e5dd390000~130000,1:0x59a5c548000~130000,1:0x59a6dcb4000~130000,1:0x38e02a18000~120000,1:0x55dd0390000~120000,1:0x5741ba18000~120000,1:0x5c583850000~120000,1:0x3b36d250000~110000,1:0x3bf3210c000~110000,1:0x391d52b8000~f0000,1:0x41f7c020000~f0000,1:0x4311cd3c000~f0000,1:0x4313bd34000~f0000,1:0x335bc800000~a0000])
Mar 21 20:02:53 osmes22cep002 ceph-osd[448006]: bluefs _read h 0x5635e2e59e80 0x684000~1000 from file(ino 1 size 0x684000 mtime 2025-03-20T14:44:27.958475+0000 allocated 840000 alloc_commit 10000 extents [1:0x5154e8e4000~10000,1:0x5154e8b4000~30000,1:0x39f2e4c0000~f0000,1:0x3dfd904c000~f0000,1:0x41956f88000~f0000,1:0x41bb428c000~f0000,1:0x29918534000~40000,1:0x49f77818000~110000,1:0x15bfd3c000~f0000,1:0x466fd100000~f0000,1:0x4912eee0000~f0000,1:0x335bc8a0000~20000])
Mar 21 20:02:53 osmes22cep002 ceph-osd[448006]: bluefs _read left 0xac000 len 0x1000
Mar 21 20:02:53 osmes22cep002 ceph-osd[216344]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #5844367. Immutable memtables: 2.
Mar 21 20:02:53 osmes22cep002 ceph-osd[216344]: rocksdb: (Original Log Time 2025/03/21-19:02:53.379369) [db/db_impl/db_impl_compaction_flush.cc:3275] Compaction nothing to do
Mar 21 20:02:53 osmes22cep002 ceph-osd[448006]: bluefs _replay file with link count 0: file(ino 2762119 size 0xf71553 mtime 2025-03-20T14:44:36.965304+0000 allocated f80000 alloc_commit f80000 extents [1:0x370c3cb0000~180000,1:0x33f7e81c000~160000,1:0x2f4a7718000~140000,1:0x13031fe8000~f0000,1:0x532b1a3c000~f0000,1:0x571a1e1c000~f0000,1:0x4d808770000~f0000,1:0x5a6c7460000~f0000,1:0x595382c000~e0000,1:0xd473f00000~e0000,1:0xd520748000~e0000,1:0xf41e720000~e0000,1:0xfbdaba0000~e0000,1:0x11bf2230000~e0000,1:0x11f3fbe4000~e0000,1:0x12ce0e98000~90000])
Mar 21 20:02:53 osmes22cep002 ceph-osd[448006]: bluefs mount failed to replay log: (5) Input/output error
Mar 21 20:02:53 osmes22cep002 ceph-osd[448006]: bluestore(/var/lib/ceph/osd/ceph-9) _open_bluefs failed bluefs mount: (5) Input/output error
Mar 21 20:02:53 osmes22cep002 ceph-osd[448006]: bluestore(/var/lib/ceph/osd/ceph-9) _open_db failed to prepare db environment:
Mar 21 20:02:53 osmes22cep002 ceph-osd[448006]: bdev(0x5635e2e04e00 /var/lib/ceph/osd/ceph-9/block) close

The ceph_assert (crash) which occurred before the "bluefs _replay" failure:

Mar 21 12:02:22 osmes22cep002 ceph-osd[232728]: bluestore.MempoolThread(0x55d05b173b60) _resize_shards cache_size: 2845415832 kv_alloc: 956301312 kv_used: 954422112 kv_onode_alloc: 128849018 kv_onode_used: -22 meta_alloc: 603979776 meta_used: 555123438 data_alloc: 318767104 data_used: 213504000
Mar 21 12:02:22 osmes22cep002 ceph-68e6e06a-a970-11ea-a1c1-9440c9867480-osd-36[232721]:  -9999> 2025-03-21T11:02:21.457+0000 7fdfe18a6640 -1 /builddir/build/BUILD/ceph-18.2.1/src/os/bluestore/AvlAllocator.cc: In function 'void AvlAllocator::_remove_from_tree(uint64_t, uint64_t)' thread 7fdfe18a6640 time 2025-03-21T11:02:21.447809+0000
Mar 21 12:02:22 osmes22cep002 ceph-68e6e06a-a970-11ea-a1c1-9440c9867480-osd-36[232721]: /builddir/build/BUILD/ceph-18.2.1/src/os/bluestore/AvlAllocator.cc: 173: FAILED ceph_assert(rs->start <= start)
Mar 21 12:02:22 osmes22cep002 ceph-68e6e06a-a970-11ea-a1c1-9440c9867480-osd-36[232721]:  ceph version 18.2.1-262.el9cp (4857b2aad4c3aaa8ff58e0b60396fa6ab731f9ff) reef (stable)
Mar 21 12:02:22 osmes22cep002 ceph-68e6e06a-a970-11ea-a1c1-9440c9867480-osd-36[232721]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x55d0590de297]
Mar 21 12:02:22 osmes22cep002 ceph-68e6e06a-a970-11ea-a1c1-9440c9867480-osd-36[232721]:  2: /usr/bin/ceph-osd(+0x488453) [0x55d0590de453]
Mar 21 12:02:22 osmes22cep002 ceph-68e6e06a-a970-11ea-a1c1-9440c9867480-osd-36[232721]:  3: /usr/bin/ceph-osd(+0x481ec0) [0x55d0590d7ec0]