Chapter 4. Troubleshooting Ceph Monitors

To Troubleshoot This Problem

1. Verify that the ceph-mon daemon is running. If not, start it:

   Syntax

   systemctl status ceph-FSID@DAEMON_NAME
   systemctl start ceph-FSID@DAEMON_NAME

   Example

   [root@mon ~]# systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@mon.host01.service
   [root@mon ~]# systemctl start ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@mon.host01.service

2. If you are not able to start ceph-mon, follow the steps in The ceph-mon daemon cannot start.
3. If you are able to start the ceph-mon daemon but is marked as down, follow the steps in The ceph-mon daemon is running, but marked as `down`.

The ceph-mon Daemon Cannot Start

1. Check the corresponding Ceph Monitor log located at /var/log/ceph/CLUSTER_FSID/ceph-mon.HOST_NAME.log by default.

   Note

   By default, the monitor logs are not present in the log folder. You need to enable logging to files for the logs to appear in the folder. See the Ceph daemon logs to enable logging to files.

2. If the log contains error messages similar to the following ones, the Ceph Monitor might have a corrupted store.

   Corruption: error in middle of record
   Corruption: 1 missing files; example: /var/lib/ceph/mon/mon.0/store.db/1234567.ldb

   To fix this problem, replace the Ceph Monitor. See Replacing a failed monitor.

3. If the log contains an error message similar to the following one, the /var/ partition might be full. Delete any unnecessary data from /var/.

   Caught signal (Bus error)

   Important

   Do not delete any data from the Monitor directory manually. Instead, use the ceph-monstore-tool to compact it. See Compacting the Ceph Monitor store for details.

4. If you see any other error messages, open a support ticket. See Contacting Red Hat Support for service for details.

The ceph-mon Daemon Is Running, but Still Marked as down

1. From the Ceph Monitor host that is out of the quorum, use the mon_status command to check its state:

   [root@mon ~]# ceph daemon ID mon_status

   Replace ID with the ID of the Ceph Monitor, for example:

   [ceph: root@host01 /]# ceph daemon mon.host01 mon_status

2. If the status is probing, verify the locations of the other Ceph Monitors in the mon_status output.

   1. If the addresses are incorrect, the Ceph Monitor has incorrect Ceph Monitor map (monmap). To fix this problem, see Injecting a Ceph Monitor map.
   2. If the addresses are correct, verify that the Ceph Monitor clocks are synchronized. See Clock skew for details. In addition, troubleshoot any networking issues, see Troubleshooting Networking issues for details.
3. If the status is electing, verify that the Ceph Monitor clocks are synchronized. See Clock skew for details.
4. If the status changes from electing to synchronizing, open a support ticket. See Contacting Red Hat Support for service for details.
5. If the Ceph Monitor is the leader or a peon, verify that the Ceph Monitor clocks are synchronized. See Clock skew for details. Open a support ticket if synchronizing the clocks does not solve the problem. See Contacting Red Hat Support for service for details.

To Troubleshoot This Problem

1. Verify that your network works correctly. For details, see Troubleshooting networking issues. If you use chrony for NTP, see Basic chrony NTP troubleshooting section for more information.
2. If you use a remote NTP server, consider deploying your own chrony NTP server on your network. For details, see the Using the Chrony Suite to Configure NTP chapter in the Configuring basic system settings for Red Hat Enterprise Linux 8.

To Troubleshoot This Problem

1. Check the size of the database:

   Syntax

   du -sch /var/lib/ceph/CLUSTER_FSID/mon.HOST_NAME/store.db/

   Specify the name of the cluster and the short host name of the host where the ceph-mon is running.

   Example

   [root@mon ~]# du -sh  /var/lib/ceph/b341e254-b165-11ed-a564-ac1f6bb26e8c/mon.host01/
   109M	/var/lib/ceph/b341e254-b165-11ed-a564-ac1f6bb26e8c/mon.host01/
   47G     /var/lib/ceph/mon/ceph-ceph1/store.db/
   47G     total

2. Compact the Ceph Monitor store. For details, see Compacting the Ceph Monitor Store.

Procedure

1. Mount all disks with Ceph data to a temporary location. Repeat this step for all OSD nodes.

   1. List the data partitions using the ceph-volume command:

      Example

      [ceph: root@host01 /]# ceph-volume lvm list

   2. Mount the data partitions to a temporary location:

      Syntax

      mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-$i

   3. Restore the SELinux context:

      Syntax

      for i in {OSD_ID}; do restorecon /var/lib/ceph/osd/ceph-$i; done

      Replace OSD_ID with a numeric, space-separated list of Ceph OSD IDs on the OSD node.

   4. Change the owner and group to ceph:ceph:

      Syntax

      for i in {OSD_ID}; do chown -R ceph:ceph /var/lib/ceph/osd/ceph-$i; done

      Replace OSD_ID with a numeric, space-separated list of Ceph OSD IDs on the OSD node.

      Important

      Due to a bug that causes the update-mon-db command to use additional db and db.slow directories for the Monitor database, you must also copy these directories. To do so:

      1. Prepare a temporary location outside the container to mount and access the OSD database and extract the OSD maps needed to restore the Ceph Monitor:

         Syntax

         ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev OSD-DATA --path /var/lib/ceph/osd/ceph-OSD-ID

         Replace OSD-DATA with the Volume Group (VG) or Logical Volume (LV) path to the OSD data and OSD-ID with the ID of the OSD.

      2. Create a symbolic link between the BlueStore database and block.db:

         Syntax

         ln -snf BLUESTORE DATABASE /var/lib/ceph/osd/ceph-OSD-ID/block.db

         Replace BLUESTORE-DATABASE with the Volume Group (VG) or Logical Volume (LV) path to the BlueStore database and OSD-ID with the ID of the OSD.

2. Use the following commands from the Ceph Monitor node with the corrupted store. Repeat them for all OSDs on all nodes.

   1. Collect the cluster map from all OSD nodes:

      Example

      [root@host01 ~]# cd /root/
      [root@host01 ~]# ms=/tmp/monstore/
      [root@host01 ~]# db=/root/db/
      [root@host01 ~]# db_slow=/root/db.slow/
      
      [root@host01 ~]# mkdir $ms
      [root@host01 ~]# for host in $osd_nodes; do
                      echo "$host"
                      rsync -avz $ms $host:$ms
                      rsync -avz $db $host:$db
                      rsync -avz $db_slow $host:$db_slow
      
                      rm -rf $ms
                      rm -rf $db
                      rm -rf $db_slow
      
                      sh -t $host <<EOF
                        for osd in /var/lib/ceph/osd/ceph-*; do
                          ceph-objectstore-tool --type bluestore --data-path \$osd --op update-mon-db --mon-store-path $ms
      
                         done
                      EOF
      
                            rsync -avz $host:$ms $ms
                            rsync -avz $host:$db $db
                            rsync -avz $host:$db_slow $db_slow
                      done

   2. Set the appropriate capabilities:

      Example

      [ceph: root@host01 /]# ceph-authtool /etc/ceph/ceph.client.admin.keyring -n mon. --cap mon 'allow *' --gen-key
      [ceph: root@host01 /]# cat /etc/ceph/ceph.client.admin.keyring
        [mon.]
          key = AQCleqldWqm5IhAAgZQbEzoShkZV42RiQVffnA==
          caps mon = "allow *"
        [client.admin]
          key = AQCmAKld8J05KxAArOWeRAw63gAwwZO5o75ZNQ==
          auid = 0
          caps mds = "allow *"
          caps mgr = "allow *"
          caps mon = "allow *"
          caps osd = "allow *"

   3. Move all sst file from the db and db.slow directories to the temporary location:

      Example

      [ceph: root@host01 /]# mv /root/db/*.sst /root/db.slow/*.sst /tmp/monstore/store.db

   4. Rebuild the Monitor store from the collected map:

      Example

      [ceph: root@host01 /]# ceph-monstore-tool /tmp/monstore rebuild -- --keyring /etc/ceph/ceph.client.admin

      Note

      After using this command, only keyrings extracted from the OSDs and the keyring specified on the ceph-monstore-tool command line are present in Ceph’s authentication database. You have to recreate or import all other keyrings, such as clients, Ceph Manager, Ceph Object Gateway, and others, so those clients can access the cluster.

   5. Back up the corrupted store. Repeat this step for all Ceph Monitor nodes:

      Syntax

      mv /var/lib/ceph/mon/ceph-HOSTNAME/store.db /var/lib/ceph/mon/ceph-HOSTNAME/store.db.corrupted

      Replace HOSTNAME with the host name of the Ceph Monitor node.

   6. Replace the corrupted store. Repeat this step for all Ceph Monitor nodes:

      Syntax

      scp -r /tmp/monstore/store.db HOSTNAME:/var/lib/ceph/mon/ceph-HOSTNAME/

      Replace HOSTNAME with the host name of the Monitor node.

   7. Change the owner of the new store. Repeat this step for all Ceph Monitor nodes:

      Syntax

      chown -R ceph:ceph /var/lib/ceph/mon/ceph-HOSTNAME/store.db

      Replace HOSTNAME with the host name of the Ceph Monitor node.

3. Unmount all the temporary mounted OSDs on all nodes:

   Example

   [root@host01 ~]# umount /var/lib/ceph/osd/ceph-*

4. Start all the Ceph Monitor daemons:

   Syntax

   systemctl start ceph-FSID@DAEMON_NAME

   Example

   [root@mon ~]# systemctl start ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@mon.host01.service

5. Ensure that the Monitors are able to form a quorum:

   Syntax

   ceph -s

   Replace HOSTNAME with the host name of the Ceph Monitor node.

6. Import the Ceph Manager keyring and start all Ceph Manager processes:

   Syntax

   ceph auth import -i /etc/ceph/ceph.mgr.HOSTNAME.keyring
   systemctl start ceph-FSID@DAEMON_NAME

   Example

   [root@host01 ~]# systemctl start ceph-b341e254-b165-11ed-a564-ac1f6bb26e8c@mgr.extensa003.exrqql.service

   Replace HOSTNAME with the host name of the Ceph Manager node.

7. Start all OSD processes across all OSD nodes. Repeat for all OSDs on the cluster:

   Syntax

   systemctl start ceph-FSID@osd.OSD_ID

   Example

   [root@host01 ~]# systemctl start ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service

8. Ensure that the OSDs are returning to service:

   Example

   [ceph: root@host01 /]# ceph -s