Chapter 9. Installing and Using ceph-medic to Diagnose a Ceph Storage Cluster

Use the ceph-medic utility to detect common issues with a Ceph Storage Cluster that prevents it from functioning properly.

9.1. Prerequisite

  • A working Red Hat Ceph Storage cluster

9.2. Installing the ceph-medic Utility

Prerequisite

  • Access to the Red Hat Ceph Storage 3 software repositories

Procedure

Do the following steps on the Ansible administration node, as the root user.

  1. Install the ceph-medic package:

    [root@admin ~]# yum install ceph-medic
  2. Verify the installation of ceph-medic:

    [root@admin ~]# ceph-medic --help

Additional Resources

9.3. Using the ceph-medic Utility

The ceph-medic utility performs checks against a running Ceph Storage Cluster to identify potential problems.

The ceph-medics utility for example checks:

  • The correct ownership of files and directories
  • If the fsid is the same for all nodes in the storage cluster
  • If the secret key in the keyring is different than other nodes in the storage cluster

9.3.1. Prerequisites

  • A working Red Hat Ceph Storage cluster
  • SSH and sudo access to storage nodes

9.3.2. Running a Diagnostic Check

A basic check for potential problems with a Ceph Storage Cluster.

Prerequisites
  • A working Red Hat Ceph Storage cluster
  • SSH and sudo access to storage nodes
Procedure

Do the following step from the Ansible administration node, as a normal user.

  • Use the ceph-medic check command:

    [admin@admin ~]$ ceph-medic check
    Host: mon0                  connection: [connected  ]
    Host: mon1                  connection: [connected  ]
    Host: mon2                  connection: [connected  ]
    Host: osd0                  connection: [connected  ]
    Host: osd1                  connection: [connected  ]
    Host: osd2                  connection: [connected  ]
    Collection completed!
    
    =======================  Starting remote check session  ========================
    Version: 1.0.2    Cluster Name: "example"
    Total hosts: [6]
    OSDs:    3    MONs:    3     Clients:    0
    MDSs:    0    RGWs:    0     MGRs:       0
    
    ================================================================================
    
    ------------ osds ------------
     osd0
     osd1
     osd2
    
    ------------ mons ------------
     mon0
     mon1
     mon2
    
    17 passed, 0 errors, on 6 hosts
Additional Resources

9.3.3. Using a Custom Inventory File

The ceph-medic utility must know the storage cluster topology. By default, ceph-medic uses the Ansible inventory file (/etc/ansible/hosts) to detect nodes.

Prerequisites
  • A working Red Hat Ceph Storage cluster
  • SSH and sudo access to storage nodes
Procedure

To use a custom inventory file, do the following steps on the Ansible administration node, as a user.

  1. Create a custom hosts file:

    [admin@admin ~]$ touch ~/example/hosts
  2. Open the hosts file for editing. Add the nodes in the storage cluster under the appropriate node group type. The ceph-medic tool supports the following node group types: mons, osds, rgws, mdss, mgrs and clients.

    For example:

    [mons]
    mon0
    mon1
    mon2
    
    [osds]
    osd0
    osd1
    osd2
  3. When doing a diagnostic check, to specify a custom inventory file, use the --inventory option:

    ceph-medic --inventory $PATH_TO_HOSTS_FILE check
    Replace
    • $PATH_TO_HOSTS_FILE with the full path to the hosts file.

      For example:

      [admin@admin ~]$ ceph-medic --inventory ~/example/hosts check
Additional Resources

9.3.4. Configuring a Custom Logging Path

The ceph-medic log file contains more verbosity than the command output on the terminal. The ceph-medic tool by default writes logs to the current working directory.

Prerequisites
  • A working Red Hat Ceph Storage cluster
  • SSH and sudo access to storage nodes
Procedure

To change where these logs are written, do the following step on the Ansible administration node, as a normal user.

  1. Open the ~/.cephmedic.conf file for editing.
  2. Change the --log-path option, from a . to a custom log location.

    For example:

    --log-path = /var/log/ceph-medic/
Additional Resources

9.3.5. Additional Resources