Chapter 7. Installing and Using ceph-medic to Diagnose a Ceph Storage Cluster

Use the ceph-medic utility to detect common issues with a Ceph Storage Cluster that prevents it from functioning properly.

7.1. Prerequisite

  • A working Red Hat Ceph Storage cluster

7.2. Installing the ceph-medic Utility

Prerequisite

  • Access to the Red Hat Ceph Storage 3 software repositories

Procedure

Do the following steps on the Ansible administration node, as the root user.

  1. Install the ceph-medic package: 

    [root@admin ~]# yum install ceph-medic

  2. Verify the installation of ceph-medic: 

    [root@admin ~]# ceph-medic --help

Additional Resources

7.3. Using the ceph-medic Utility

The ceph-medic utility performs checks against a running Ceph Storage Cluster to identify potential problems.

The ceph-medics utility for example checks:

  • The correct ownership of files and directories
  • If the fsid is the same for all nodes in the storage cluster
  • If the secret key in the keyring is different than other nodes in the storage cluster

7.3.1. Prerequisites

  • A working Red Hat Ceph Storage cluster
  • SSH and sudo access to storage nodes

7.3.2. Running a Diagnostic Check

A basic check for potential problems with a Ceph Storage Cluster.

Prerequisites
  • A working Red Hat Ceph Storage cluster
  • SSH and sudo access to storage nodes
Procedure

Do the following step from the Ansible administration node, as a normal user.

  • Use the ceph-medic check command: 

    [admin@admin ~]$ ceph-medic check
Host: mon0                  connection: [connected  ]
Host: mon1                  connection: [connected  ]
Host: mon2                  connection: [connected  ]
Host: osd0                  connection: [connected  ]
Host: osd1                  connection: [connected  ]
Host: osd2                  connection: [connected  ]
Collection completed!

=======================  Starting remote check session  ========================
Version: 1.0.2    Cluster Name: "example"
Total hosts: [6]
OSDs:    3    MONs:    3     Clients:    0
MDSs:    0    RGWs:    0     MGRs:       0

================================================================================

------------ osds ------------
 osd0
 osd1
 osd2

------------ mons ------------
 mon0
 mon1
 mon2

17 passed, 0 errors, on 6 hosts
Additional Resources

7.3.3. Using a Custom Inventory File

The ceph-medic utility must know the storage cluster topology. By default, ceph-medic uses the Ansible inventory file (/etc/ansible/hosts) to detect nodes.

Prerequisites
  • A working Red Hat Ceph Storage cluster
  • SSH and sudo access to storage nodes
Procedure

To use a custom inventory file, do the following steps on the Ansible administration node, as a user.

  1. Create a custom hosts file: 

    [admin@admin ~]$ touch ~/example/hosts

  2. Open the hosts file for editing. Add the nodes in the storage cluster under the appropriate node group type. The ceph-medic tool supports the following node group types: mons, osds, rgws, mdss, mgrs and clients.

    For example: 

    [mons]
mon0
mon1
mon2

[osds]
osd0
osd1
osd2

  3. When doing a diagnostic check, to specify a custom inventory file, use the --inventory option: 

    ceph-medic --inventory $PATH_TO_HOSTS_FILE check
    Replace

    • $PATH_TO_HOSTS_FILE with the full path to the hosts file.

      For example: 

      [admin@admin ~]$ ceph-medic --inventory ~/example/hosts check
Additional Resources

7.3.4. Configuring a Custom Logging Path

The ceph-medic log file contains more verbosity than the command output on the terminal. The ceph-medic tool by default writes logs to the current working directory.

Prerequisites
  • A working Red Hat Ceph Storage cluster
  • SSH and sudo access to storage nodes
Procedure

To change where these logs are written, do the following step on the Ansible administration node, as a normal user.

  1. Open the ~/.cephmedic.conf file for editing.

  2. Change the --log-path option, from a . to a custom log location.

    For example: 

    --log-path = /var/log/ceph-medic/
Additional Resources

7.3.5. Additional Resources