RHCS3 - Bluestore OSD crash with "BlueFS.cc: 1525: FAILED assert(0 == "bluefs enospc")" but there is still enough free space on the device

Solution Verified - Updated -

Issue

We see Bluestore OSD crashing with " assert(0 == "bluefs enospc")" , but the device has plenty of free space:

From ceph-osd.<ID>.log:
    -3> 2019-02-14 13:04:01.288640 7f3937c97700  1 bluefs _allocate failed to allocate 0x4200000 on bdev 1, free 0x1600000; fallback to bdev 2
    -2> 2019-02-14 13:04:01.288652 7f3937c97700 -1 bluefs _allocate failed to allocate 0x4200000 on bdev 2, dne
    -1> 2019-02-14 13:04:01.288654 7f3937c97700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x4184c95
     0> 2019-02-14 13:04:01.293854 7f3937c97700 -1 /builddir/build/BUILD/ceph-12.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f3937c9770
0 time 2019-02-14 13:04:01.288664
/builddir/build/BUILD/ceph-12.2.8/src/os/bluestore/BlueFS.cc: 1525: FAILED assert(0 == "bluefs enospc")

 ceph version 12.2.8-52.el7cp (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x565444530fa0]
 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1b1b) [0x5654444aec3b]
 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x188) [0x5654444af1a8]
 4: (BlueRocksWritableFile::Flush()+0x3c) [0x5654444cad9c]
 5: (rocksdb::WritableFileWriter::Flush()+0xaf) [0x5654448a253f]
 6: (rocksdb::WritableFileWriter::Sync(bool)+0x33) [0x5654448a2753]
 7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::RangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0xa1b)
 [0x5654449688bb]
 8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0xea2) [0x56544496b332]
 9: (rocksdb::CompactionJob::Run()+0x1de) [0x56544496bdee]
 10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, void*)+0x9c7) [0x5654448f76a7]
 11: (rocksdb::DBImpl::BackgroundCallCompaction(void*)+0x28a) [0x5654448facaa]
 12: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x271) [0x565444961f41]
 13: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x3d) [0x56544496212d]
 14: (()+0xb5070) [0x7f3949ee7070]
 15: (()+0x7dd5) [0x7f394a55add5]
 16: (clone()+0x6d) [0x7f394964bb3d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

OSD usage :
ID  CLASS WEIGHT    REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS TYPE NAME          
<ID>   ssd   0.21829  1.00000  224GiB 51.5GiB  172GiB 23.02 0.36  82         osd.<ID>

Environment

Red Hat Ceph Storage 3 - up to version RHCS 3.2z1 - 12.2.8-89.el7cp included
Bluestore OSDs only

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In