Ceph - After replacing OSD disk, new OSD disk will not start with fsid mismatch stating "invalid (someone else's?) journal"
Issue
-
After replacing OSD disk, the new OSD disk will not start and the OSD log reports an fsid mismatch with "invalid (someone else's?) journal"
-
The OSD log reports the following:
2018-07-03 14:54:37.610037 7f0ac49167c0 0 ceph version 0.94.9-4.el7cp (2a9afd7547bacdf5f1443ec8f5a16e2150ed1a69), process ceph-osd, pid 936825
2018-07-03 14:54:37.727970 7f0ac49167c0 0 filestore(/var/lib/ceph/osd/ceph-2) backend xfs (magic 0x58465342)
2018-07-03 14:54:37.728944 7f0ac49167c0 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is supported and appears to work
2018-07-03 14:54:37.728949 7f0ac49167c0 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2018-07-03 14:54:37.729226 7f0ac49167c0 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2018-07-03 14:54:37.729258 7f0ac49167c0 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-2) detect_feature: extsize is disabled by conf
2018-07-03 14:54:37.745614 7f0ac49167c0 0 filestore(/var/lib/ceph/osd/ceph-2) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2018-07-03 14:54:37.760106 7f0ac49167c0 1 journal _open /var/lib/ceph/osd/ceph-2/journal fd 20: 80054583296 bytes, block size 4096 bytes, directio = 1, aio = 1
2018-07-03 14:54:37.760311 7f0ac49167c0 -1 journal FileJournal::open: ondisk fsid d8a93fc9-f57f-4085-b70c-aa00f98166e3 doesn't match expected 533acad2-59aa-491b-b6d3-bb970ecc4367, invalid (someone else's?) journal
2018-07-03 14:54:37.761768 7f0ac49167c0 -1 os/FileJournal.h: In function 'virtual FileJournal::~FileJournal()' thread 7f0ac49167c0 time 2018-07-03 14:54:37.760349
os/FileJournal.h: 406: FAILED assert(fd == -1)
ceph version 0.94.9-4.el7cp (2a9afd7547bacdf5f1443ec8f5a16e2150ed1a69)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xb13535]
2: (FileJournal::~FileJournal()+0x5dc) [0x8f3b5c]
3: (FileJournal::~FileJournal()+0x9) [0x8f3bf9]
4: (JournalingObjectStore::journal_replay(unsigned long)+0xba3) [0x9034c3]
5: (FileStore::mount()+0x3c6f) [0x8d1eaf]
6: (OSD::init()+0x26c) [0x67ec9c]
7: (main()+0x2aec) [0x61a1bc]
8: (__libc_start_main()+0xf5) [0x7f0ac22d9b15]
9: /usr/bin/ceph-osd() [0x621189]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Environment
- Red Hat Ceph Storage
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
