Ceph - After replacing OSD disk, new OSD disk will not start with fsid mismatch stating "invalid (someone else's?) journal"

Issue

• After replacing OSD disk, the new OSD disk will not start and the OSD log reports an fsid mismatch with "invalid (someone else's?) journal"

• The OSD log reports the following:

2018-07-03 14:54:37.610037 7f0ac49167c0  0 ceph version 0.94.9-4.el7cp (2a9afd7547bacdf5f1443ec8f5a16e2150ed1a69), process ceph-osd, pid 936825
2018-07-03 14:54:37.727970 7f0ac49167c0  0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
2018-07-03 14:54:37.728944 7f0ac49167c0  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is supported and appears to work
2018-07-03 14:54:37.728949 7f0ac49167c0  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2018-07-03 14:54:37.729226 7f0ac49167c0  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2018-07-03 14:54:37.729258 7f0ac49167c0  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_feature: extsize is disabled by conf
2018-07-03 14:54:37.745614 7f0ac49167c0  0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2018-07-03 14:54:37.760106 7f0ac49167c0  1 journal _open /var/lib/ceph/osd/ceph-2/journal fd 20: 80054583296 bytes, block size 4096 bytes, directio = 1, aio = 1
2018-07-03 14:54:37.760311 7f0ac49167c0 -1 journal FileJournal::open: ondisk fsid d8a93fc9-f57f-4085-b70c-aa00f98166e3 doesn't match expected 533acad2-59aa-491b-b6d3-bb970ecc4367, invalid (someone else's?) journal
2018-07-03 14:54:37.761768 7f0ac49167c0 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7f0ac49167c0 time 2018-07-03 14:54:37.760349
os/FileJournal.h: 406: FAILED assert(fd == -1)

 ceph version 0.94.9-4.el7cp (2a9afd7547bacdf5f1443ec8f5a16e2150ed1a69)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xb13535]
 2: (FileJournal::~FileJournal()+0x5dc) [0x8f3b5c]
 3: (FileJournal::~FileJournal()+0x9) [0x8f3bf9]
 4: (JournalingObjectStore::journal_replay(unsigned long)+0xba3) [0x9034c3]
 5: (FileStore::mount()+0x3c6f) [0x8d1eaf]
 6: (OSD::init()+0x26c) [0x67ec9c]
 7: (main()+0x2aec) [0x61a1bc]
 8: (__libc_start_main()+0xf5) [0x7f0ac22d9b15]
 9: /usr/bin/ceph-osd() [0x621189]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.