RHCS3 - Bluestore OSD crash with "BlueFS.cc: 1525: FAILED assert(0 == "bluefs enospc")" but there is still enough free space on the device

Issue

• We see Bluestore OSD crashing with assert(0 == "bluefs enospc"), but the device has plenty of free space:

  From ceph-osd.<ID>.log:
      -3> 2019-02-14 13:04:01.288640 7f3937c97700  1 bluefs _allocate failed to allocate 0x4200000 on bdev 1, free 0x1600000; fallback to bdev 2
      -2> 2019-02-14 13:04:01.288652 7f3937c97700 -1 bluefs _allocate failed to allocate 0x4200000 on bdev 2, dne
      -1> 2019-02-14 13:04:01.288654 7f3937c97700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x4184c95
       0> 2019-02-14 13:04:01.293854 7f3937c97700 -1 /builddir/build/BUILD/ceph-12.2.8/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f3937c9770
  0 time 2019-02-14 13:04:01.288664
  /builddir/build/BUILD/ceph-12.2.8/src/os/bluestore/BlueFS.cc: 1525: FAILED assert(0 == "bluefs enospc")
  
   ceph version 12.2.8-52.el7cp (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)
   1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x565444530fa0]
   2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1b1b) [0x5654444aec3b]
   3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x188) [0x5654444af1a8]
   4: (BlueRocksWritableFile::Flush()+0x3c) [0x5654444cad9c]
   5: (rocksdb::WritableFileWriter::Flush()+0xaf) [0x5654448a253f]
   6: (rocksdb::WritableFileWriter::Sync(bool)+0x33) [0x5654448a2753]
   7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::RangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0xa1b)
   [0x5654449688bb]
   8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0xea2) [0x56544496b332]
   9: (rocksdb::CompactionJob::Run()+0x1de) [0x56544496bdee]
   10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, void*)+0x9c7) [0x5654448f76a7]
   11: (rocksdb::DBImpl::BackgroundCallCompaction(void*)+0x28a) [0x5654448facaa]
   12: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x271) [0x565444961f41]
   13: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x3d) [0x56544496212d]
   14: (()+0xb5070) [0x7f3949ee7070]
   15: (()+0x7dd5) [0x7f394a55add5]
   16: (clone()+0x6d) [0x7f394964bb3d]
   NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  
  OSD usage :
  ID  CLASS WEIGHT    REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS TYPE NAME
  <ID>   ssd   0.21829  1.00000  224GiB 51.5GiB  172GiB 23.02 0.36  82         osd.<ID>