Chapter 9. Management of Ceph OSDs on the dashboard

As a storage administrator, you can monitor and manage OSDs on the Red Hat Ceph Storage Dashboard.

Some of the capabilities of the Red Hat Ceph Storage Dashboard are:

  • List OSDs, their status, statistics, information such as attributes, metadata, device health, performance counters and performance details.
  • Mark OSDs down, in, out, lost, purge, reweight, scrub, deep-scrub, destroy, delete, and select profiles to adjust backfilling activity.
  • List all drives associated with an OSD.
  • Set and change the device class of an OSD.
  • Deploy OSDs on new drives and hosts.

9.1. Prerequisites

  • A running Red Hat Ceph Storage cluster
  • cluster-manager level of access on the Red Hat Ceph Storage dashboard

9.2. Managing the OSDs on the Ceph dashboard

You can carry out the following actions on a Ceph OSD on the Red Hat Ceph Storage Dashboard:

  • Create a new OSD.
  • Edit the device class of the OSD.
  • Mark the Flags as No Up, No Down, No In, or No Out.
  • Scrub and deep-scrub the OSDs.
  • Reweight the OSDs.
  • Mark the OSDs Out, In, Down, or Lost.
  • Purge the OSDs.
  • Destroy the OSDs.
  • Delete the OSDs.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.
  • Hosts are added to the storage cluster.
  • All the services including OSDs are deployed on the storage cluster.

Procedure

  1. Log in to the Dashboard.
  2. From the Cluster drop-down menu, select OSDs.

Creating an OSD

  1. To create the OSD, click Create.

    1. In the Create OSDs window, click +Add for Primary devices.
    2. In the Primary devices dialog box, from the Hostname drop-down list, select any one filter. From Any drop-down list, select the respective option.

      Note

      You have to select the Hostname first and then at least one filter to add the devices.

      For example, from Hostname list, select Type and from Any list select hdd. Select Vendor and from Any list, select ATA

      Figure 9.1. Add device for OSDs

      Add device for OSDs
    3. Click Add.
    4. In the Create OSDs window, click the Preview button.
    5. In the OSD Creation Preview dialog box, Click Create.
    6. You get a notification that the OSD was created successfully.
    7. The OSD will change the status to in and down to in and up.

Editing an OSD

  1. To edit an OSD, select the row.

    1. From Edit drop-down menu, select Edit.
    2. Edit the device class.
    3. Click Edit OSD.

      Figure 9.2. Edit an OSD

      Edit an OSD
    4. You get a notification that the OSD was updated successfully.

Marking the Flags of OSDs

  1. To mark the flag of the OSD, select the row.

    1. From Edit drop-down menu, select Flags.
    2. Mark the Flags with No Up, No Down, No In, or No Out.
    3. Click Update.

      Figure 9.3. Marking Flags of an OSD

      Marking Flags of an OSD
    4. You get a notification that the flags of the OSD was updated successfully.

Scrubbing the OSDs

  1. To scrub the OSD, select the row.

    1. From Edit drop-down menu, select Scrub.
    2. In the OSDs Scrub dialog box, click Update.

      Figure 9.4. Scrubbing an OSD

      Scrubbing an OSD
    3. You get a notification that the scrubbing of the OSD was initiated successfully.

Deep-scrubbing the OSDs

  1. To deep-scrub the OSD, select the row.

    1. From Edit drop-down menu, select Deep scrub.
    2. In the OSDs Deep Scrub dialog box, click Update.

      Figure 9.5. Deep-scrubbing an OSD

      Deep-scrubbing an OSD
    3. You get a notification that the deep scrubbing of the OSD was initiated successfully.

Reweighting the OSDs

  1. To reweight the OSD, select the row.

    1. From Edit drop-down menu, select Reweight.
    2. In the Reweight OSD dialog box, enter a value between zero and one.
    3. Click Reweight.

      Figure 9.6. Reweighting an OSD

      Reweighting an OSD

Marking OSDs Out

  1. To mark the OSD out, select the row.

    1. From Edit drop-down menu, select Mark Out.
    2. In the Mark OSD out dialog box, click Mark Out.

      Figure 9.7. Marking OSDs out

      Marking OSDs out
    3. The status of the OSD will change to out.

Marking OSDs In

  1. To mark the OSD in, select the OSD row that is in out status.

    1. From Edit drop-down menu, select Mark In.
    2. In the Mark OSD in dialog box, click Mark In.

      Figure 9.8. Marking OSDs in

      Marking OSDs in
    3. The status of the OSD will change to in.

Marking OSDs Down

  1. To mark the OSD down, select the row.

    1. From Edit drop-down menu, select Mark Down.
    2. In the Mark OSD down dialog box, click Mark Down.

      Figure 9.9. Marking OSDs down

      Marking OSDs down
    3. The status of the OSD will change to down.

Marking OSDs Lost

  1. To mark the OSD lost, select the OSD in out and down status.

    1. From Edit drop-down menu, select Mark Lost.
    2. In the Mark OSD Lost dialog box, check Yes, I am sure option, and click Mark Lost.

      Figure 9.10. Marking OSDs Lost

      Marking OSDs lost

Purging OSDs

  1. To purge the OSD, select the OSD in down status.

    1. From Edit drop-down menu, select Purge.
    2. In the Purge OSDs dialog box, check Yes, I am sure option, and click Purge OSD.

      Figure 9.11. Purging OSDs

      Purging OSDs
    3. All the flags are reset and the OSD is back in in and up status.

Destroying OSDs

  1. To destroy the OSD, select the OSD in down status.

    1. From Edit drop-down menu, select Destroy.
    2. In the Destroy OSDs dialog box, check Yes, I am sure option, and click Destroy OSD.

      Figure 9.12. Destroying OSDs

      Destroying OSDs
    3. The status of the OSD changes to destroyed.

Deleting OSDs

  1. To delete the OSD, select the OSD in down status.

    1. From Edit drop-down menu, select Delete.
    2. In the Destroy OSDs dialog box, check Yes, I am sure option, and click Delete OSD.

      Note

      You can preserve the OSD_ID when you have to to replace the failed OSD.

      Figure 9.13. Deleting OSDs

      Deleting OSDs

9.3. Replacing the failed OSDs on the Ceph dashboard

You can replace the failed OSDs in a Red Hat Ceph Storage cluster with the cluster-manager level of access on the dashboard. One of the highlights of this feature on the dashboard is that the OSD IDs can be preserved while replacing the failed OSDs.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • At least cluster-manager level of access to the Ceph Dashboard.
  • At least one of the OSDs is down

Procedure

  1. On the dashboard, you can identify the failed OSDs in the following ways:

    • Dashboard AlertManager pop-up notifications.
    • Dashboard landing page showing HEALTH_WARN status.
    • Dashboard landing page showing failed OSDs.
    • Dashboard OSD page showing failed OSDs.

      Health status of OSDs

      In this example, you can see that one of the OSDs is down on the landing page of the dashboard.

      Apart from this, on the physical drive, you can view the LED lights blinking if one of the OSDs is down.

  2. Click OSDs.
  3. Select the out and down OSD:

    1. From the Edit drop-down menu, select Flags and select No Up and click Update.
    2. From the Edit drop-down menu, select Delete.
    3. In the Delete OSD dialog box, select the Preserve OSD ID(s) for replacement and Yes, I am sure check boxes.
    4. Click Delete OSD.
    5. Wait till the status of the OSD changes to out and destroyed status.
  4. Optional: If you want to change the No Up Flag for the entire cluster, in the Cluster-wide configuration drop-down menu, select Flags.

    1. In Cluster-wide OSDs Flags dialog box, select No Up and click Update.
  5. Optional: If the OSDs are down due to a hard disk failure, replace the physical drive:

    • If the drive is hot-swappable, replace the failed drive with a new one.
    • If the drive is not hot-swappable and the node contains multiple OSDs, you might have to shut down the whole node and replace the physical drive. Consider preventing the cluster from backfilling. See the Stopping and Starting Rebalancing chapter in the Red Hat Ceph Storage Troubleshooting Guide for details.
    • When the drive appears under the /dev/ directory, make a note of the drive path.
    • If you want to add the OSD manually, find the OSD drive and format the disk.
    • If the new disk has data, zap the disk:

      Syntax

      ceph orch device zap HOST_NAME PATH --force

      Example

      ceph orch device zap ceph-adm2 /dev/sdc --force

  6. From the Create drop-down menu, select Create.
  7. In the Create OSDs window, click +Add for Primary devices.

    1. In the Primary devices dialog box, from the Hostname drop-down list, select any one filter. From Any drop-down list, select the respective option.

      Note

      You have to select the Hostname first and then at least one filter to add the devices.

      For example, from Hostname list, select Type and from Any list select hdd. Select Vendor and from Any list, select ATA

      Add device for OSDs
    2. Click Add.
    3. In the Create OSDs window, click the Preview button.
    4. In the OSD Creation Preview dialog box, Click Create.
    5. You will get a notification that the OSD is created. The OSD will be in out and down status.
  8. Select the newly created OSD that has out and down status.

    1. In the Edit drop-down menu, select Mark-in.
    2. In the Mark OSD in window, select Mark in.
    3. In the Edit drop-down menu, select Flags.
    4. Uncheck No Up and click Update.
  9. Optional: If you have changed the No Up Flag before for cluster-wide configuration, in the Cluster-wide configuration menu, select Flags.

    1. In Cluster-wide OSDs Flags dialog box, uncheck No Up and click Update.

Verification

  1. Verify that the OSD that was destroyed is created on the device and the OSD ID is preserved.

    OSD is created

Additional Resources

  • For more information on Down OSDs, see the Down OSDs section in the Red Hat Ceph Storage Troubleshooting Guide.
  • For additional assistance see the Red Hat Support for service section in the Red Hat Ceph Storage Troubleshooting Guide.
  • For more information on system roles, see the System roles on the Ceph dashboard section in the Red Hat Ceph Storage Dashboard Guide.