RHCS3 - Bluestore OSD crash with "BlueFS.cc: 1525: FAILED assert(0 == "bluefs enospc")" but there is still enough free space on the device

Solution Verified - Updated -

Issue

  • We see Bluestore OSD crashing with assert(0 == "bluefs enospc"), but the device has plenty of free space:

    From ceph-osd.<ID>.log:
        -3> 2019-02-14 13:04:01.288640 7f3937c97700  1 bluefs _allocate failed to allocate 0x4200000 on bdev 1, free 0x1600000; fallback to bdev 2
        -2> 2019-02-14 13:04:01.288652 7f3937c97700 -1 bluefs _allocate failed to allocate 0x4200000 on bdev 2, dne
        -1> 2019-02-14 13:04:01.288654 7f3937c97700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x4184c95
         0> 2019-02-14 13:04:01.293854 7f3937c97700 -1 /builddir/build/BUILD/ceph-12.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f3937c9770
    0 time 2019-02-14 13:04:01.288664
    /builddir/build/BUILD/ceph-12.2.8/src/os/bluestore/BlueFS.cc: 1525: FAILED assert(0 == "bluefs enospc")
    
     ceph version 12.2.8-52.el7cp (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)
     1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x565444530fa0]
     2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1b1b) [0x5654444aec3b]
     3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x188) [0x5654444af1a8]
     4: (BlueRocksWritableFile::Flush()+0x3c) [0x5654444cad9c]
     5: (rocksdb::WritableFileWriter::Flush()+0xaf) [0x5654448a253f]
     6: (rocksdb::WritableFileWriter::Sync(bool)+0x33) [0x5654448a2753]
     7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::RangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0xa1b)
     [0x5654449688bb]
     8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0xea2) [0x56544496b332]
     9: (rocksdb::CompactionJob::Run()+0x1de) [0x56544496bdee]
     10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, void*)+0x9c7) [0x5654448f76a7]
     11: (rocksdb::DBImpl::BackgroundCallCompaction(void*)+0x28a) [0x5654448facaa]
     12: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x271) [0x565444961f41]
     13: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x3d) [0x56544496212d]
     14: (()+0xb5070) [0x7f3949ee7070]
     15: (()+0x7dd5) [0x7f394a55add5]
     16: (clone()+0x6d) [0x7f394964bb3d]
     NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
    
    OSD usage :
    ID  CLASS WEIGHT    REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS TYPE NAME
    <ID>   ssd   0.21829  1.00000  224GiB 51.5GiB  172GiB 23.02 0.36  82         osd.<ID>
    

Environment

  • Red Hat Ceph Storage (RHCS) 3
    • up to version RHCS 3.2z1 - 12.2.8-89.el7cp included
  • Bluestore OSDs only

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In