Ceph - recreate containerized MON using CLI after monstore.db corruption for a single MON failure scenario
Issue
Due to encountering issue Ceph Monitor down with FAILED assert in AuthMonitor::update_from_paxos we need to re-deploy Ceph MON in containerized environment using CLI.
The MON assert looks like:
Feb 20 11:45:16 mons-1 docker: debug 0> 2021-02-20 11:45:16.111768 7f0eff2de000 -1 /builddir/build/BUILD/ceph-12.2.12/src/mon/AuthMonitor.cc: In function 'virtual void AuthMonitor::update_from_paxos(bool*)' thread 7f0eff2de000 time 2021-02-20 11:45:16.109760
Feb 20 11:45:16 mons-1 docker: /builddir/build/BUILD/ceph-12.2.12/src/mon/AuthMonitor.cc: 160: FAILED assert(ret == 0)
Feb 20 11:45:16 mons-1 docker: ceph version 12.2.12-101.el7cp (20a4945f2321019ed50c1844b413059c07304074) luminous (stable)
Feb 20 11:45:16 mons-1 docker: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55d18d3e4150]
Feb 20 11:45:16 mons-1 docker: 2: (AuthMonitor::update_from_paxos(bool*)+0x159d) [0x55d18d1e3acd]
Feb 20 11:45:16 mons-1 docker: 3: (PaxosService::refresh(bool*)+0x1ae) [0x55d18d2caebe]
Feb 20 11:45:16 mons-1 docker: 4: (Monitor::refresh_from_paxos(bool*)+0x1a3) [0x55d18d177203]
Feb 20 11:45:16 mons-1 docker: 5: (Monitor::init_paxos()+0x115) [0x55d18d177645]
Feb 20 11:45:16 mons-1 docker: 6: (Monitor::preinit()+0xa9f) [0x55d18d17813f]
Feb 20 11:45:16 mons-1 docker: 7: (main()+0x4018) [0x55d18d0a56c8]
Feb 20 11:45:16 mons-1 docker: 8: (__libc_start_main()+0xf5) [0x7f0efb9a1545]
Feb 20 11:45:16 mons-1 docker: 9: (()+0x3d93ae) [0x55d18d14c3ae]
Environment
Red Hat Ceph Storage 3
Red Hat Ceph Storage 4
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.