Menu Close

Chapter 7. Managing the Cluster

7.1. Managing the Cluster

The management functions of the dashboard allow you to view and modify configuration settings, and manage cluster resources.

7.2. Viewing the CRUSH map

The CRUSH map contains a list of OSDs and related information. Together, the CRUSH map and CRUSH algorithm determine how and where data is stored. The Red Hat Ceph Storage dashboard allows you to view different aspects of the CRUSH map, including OSD hosts, OSD daemons, ID numbers, device class, and more.

The CRUSH map allows you to determine which node a specific OSD ID is running on. This is helpful if there is an issue with an OSD.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.

Procedure

  1. Log in to the Dashboard.
  2. On the navigation bar, click Cluster.
  3. Click CRUSH map.

    CRUSH Map initial view

    In the above example, you can see the default CRUSH map, three nodes, and OSDs running on two of the three nodes.

  4. Click on the CRUSH map name, nodes, or OSDs, to view details about each object.

    CRUSH Map detail view

    In the above example, you can see the values of variables related to an OSD running on the jb-rhel-osd3 node. In particular, note the id is 2.

Additional Resources

7.3. Configuring manager modules

The Red Hat Ceph Storage dashboard allows you to view and configure manager module parameters.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.

Procedure

  1. Log in to the Dashboard.
  2. On the navigation bar, click Cluster.
  3. Click Manager modules:

    Manager modules initial view

    The above screenshot shows the first of three pages of manager modules.

  4. Click on a row for a module you want to configure:

    Manage modules edit button

    Not all modules have configurable parameters. If a module is not configurable, the Edit button is disabled.

  5. Towards the upper left of the page, click the Edit button to load the page with the configurable parameters.

    Manager module balancer parameters

    The above screenshot shows parameters that you can change for the balancer module. To display a description of a parameter, click the question mark button.

  6. To change a parameter, modify the state of the parameter and click the Update button at the bottom of the page:

    Manager module update button

    A notification confirming the change appears in the upper-right corner of the page:

    Manager module updated options notification

Additional Resources

7.4. Filtering logs

The Red Hat Ceph Storage Dashboard allows you to view and filter logs based on several criteria. The criteria include priority, keyword, date, and time range.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • The Dashboard is installed.
  • Log entries have been generated since the Monitor was last started.
Note

The Dashboard logging feature only displays the thirty latest high level events. The events are stored in memory by the Monitor. The entries disappear after restarting the Monitor. If you need to review detailed or older logs, refer to the file based logs. See Additional Resources below for more information about file based logs.

Procedure

  1. Log in to the Dashboard.
  2. Click the Cluster drop-down menu in the top navigation bar.
  3. Click Logs in the drop-down menu.
  4. View the last thirty unfiltered log entries.

    Unfiltered cluster logs.
    1. To filter by priority, click the Priority drop-down menu and select either Info, Warning, or Error. The example below only shows log entries with the priority of Error.

      Filter by priority.
    2. To filter by keyword, enter text into the Keyword form. The example below only shows log entries that include the text osd.2.

      Filter by keyword.
    3. To filter by date, click the Date form and either use the date picker to select a date from the menu, or enter a date in the form of YYYY-MM-DD. The example below only shows log entries with the date of 2019-10-15.

      Filter by date.
    4. To filter by time, enter a range in the Time range fields using the HH:MM - HH:MM format. Hours must be entered using numbers 0 to 23. The example below only shows log entries from 12:14 to 12:23.

      Filter by time range.
    5. To combine filters, set two or more filters. The example below only shows entries that have both a Priority of Warning and the keyword of osd.

      Combine filters.

Additional Resources

7.5. Configuring OSD recovery settings

As a storage administrator, you can change the OSD recovery priority and customize how the cluster recovers. This allows you to influence your cluster’s rebuild performance or recovery speed.

Prerequisites

  • A Red Hat Ceph Storage cluster.
  • The dashboard is installed.

Procedure

  1. Log in to the dashboard.
  2. Click the Cluster drop-down menu in the top navigation bar.
  3. Click OSDs in the drop-down menu.
  4. Click the Cluster-Wide Flags drop-down menu.

    Click the Cluster-Wide Flags drop-down.
  5. Select Cluster-wide Recovery Priority in the drop-down.

    Select Cluster-wide Recovery Priority.
  6. Optional: Select Priority in the drop-down menu , and then click the Submit button.

    Note

    There are 3 predefined options: Low, Default, High

    Default priority values
  7. Optional: Click Customize priority values, make the required changes, and then click the Submit button.

    Customize priority values
  8. A notification towards the top right corner of the page pops up indicating the flags were updated successfully.

    OSD recovery speed priority updated notification

Additional Resources

7.6. Viewing and monitoring configuration

The Red Hat Ceph Storage Dashboard allows you to view the list of all configuration options for the Ceph cluster. You can also edit the configuration on the Dashboard.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.

Procedure

  1. Log in to the Dashboard.
  2. On the navigation bar, click Cluster and then click Configuration.
  3. To view the details of the configuration, click its row:

    Configuration details
    • You can search for the configuration using the Search box:

      Search Configuration
    • You can filter for the configuration using Level, Service or Source drop-down:

      Filter Configuration
  4. To edit a configuration, click its row and click the Edit button:

    Edit Configuration
    1. In the dialog window, edit the required parameters and Click the Save button:

      Edit Configuration window

      A notification confirming the change appears in the upper-right corner of the page.

      Edit Configuration notification

Additional Resources

7.7. Managing the Prometheus environment

To monitor a Ceph storage cluster with Prometheus you can configure and enable the Prometheus exporter so the metadata information about the Ceph storage cluster can be collected.

Prerequisites

  • A running Red Hat Ceph Storage 3.1 or higher cluster.
  • Installation of the Red Hat Ceph Storage Dashboard.
  • Root-level access to the Red Hat Ceph Storage Dashboard node.

Procedure

  1. Open and edit the /etc/prometheus/prometheus.yml file.

    1. Under the global section, set the scrape_interval and evaluation_interval options to 15 seconds.

      Example

      global:
        scrape_interval:     15s
        evaluation_interval: 15s

    2. Under the scrape_configs section, add the honor_labels: true option, and edit the targets, and instance options for each of the ceph-mgr nodes.

      Example

      scrape_configs:
        - job_name: 'node'
          honor_labels: true
          static_configs:
          - targets: [ 'node1.example.com:9100' ]
            labels:
              instance: "node1.example.com"
          - targets: ['node2.example.com:9100']
            labels:
              instance: "node2.example.com"

      Note

      Using the honor_labels option enables Ceph to output properly-labelled data relating to any node in the Ceph storage cluster. This allows Ceph to export the proper instance label without Prometheus overwriting it.

    3. To add a new node, simply add the targets, and instance options in the following format:

      Example

      - targets: [ 'new-node.example.com:9100' ]
        labels:
          instance: "new-node"

      Note

      The instance label has to match what appears in Ceph’s OSD metadata instance field, which is the short host name of the node. This helps to correlate Ceph stats with the node’s stats.

  2. Add Ceph targets to the /etc/prometheus/ceph_targets.yml file in the following format.

    Example

    [
        {
            "targets": [ "cephnode1.example.com:9283" ],
            "labels": {}
        }
    ]

  3. Enable the Prometheus module:

    [root@mon ~]# ceph mgr module enable prometheus

7.8. Restoring grafana-server and Prometheus

The grafana-server includes the Grafana UI, Prometheus, the containers, and the Red Hat Ceph Storage configuration. When the grafana-server crashes or is faulty, you can restore it by taking a back-up of the files and restoring it using the backed-up files. For Prometheus, you can take an external back-up and then restore the data.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the Grafana nodes.

Procedure

  1. Take the back-up of the Grafana database:

    1. On the grafana-server node, stop the Grafana service:

      Example

      [root@node04 ~]# systemctl stop grafana-server.service
      [root@node04 ~]# systemctl status grafana-server.service

    2. Take the back-up of the Grafana database:

      Example

      [root@node04 ~]# cp /var/lib/grafana/grafana.db /var/lib/grafana/grafana_backup.db

    3. On the grafana-server node, restart the Grafana service:

      Example

      [root@node04 ~]# systemctl restart grafana-server.service

  2. Restore the grafana-server:

    1. On the grafana-server node, if the Grafana service is running, stop the service:

      Example

      [root@node04 ~]# systemctl stop grafana-server.service
      [root@node04 ~]# systemctl status grafana-server.service

    2. Move the backed-up grafana.db file to /var/lib/grafana/ directory:

      Example

      [root@node04 ~]# mv /var/lib/grafana/grafana_backup.db /var/lib/grafana/

    3. On the grafana-server node, restart the Grafana service:

      Example

      [root@node04 ~]# systemctl restart grafana-server.service

  3. For the Prometheus alerts, you have to take external back-up of prometheus_data_dir directory, a Ceph-Ansible setting which by default is var/lib/prometheus directory and restore the service using the backed-up directory.

    1. On the grafana-server node, stop the Prometheus service:

      Example

      [root@node04 ~]# systemctl stop prometheus.service
      [root@node04 ~]# systemctl status prometheus.service

    2. Take the back-up of the default Prometheus directory:

      Example

      [root@node04 ~]# cp /var/lib/prometheus/ /var/lib/prometheus_backup/

    3. Replace the prometheus_data_dir directory with the backed-up directory:

      Example

      [root@node04 ~]# mv /var/lib/prometheus_backup/ /var/lib/prometheus_data_dir

    4. On the grafana-server node, restart the prometheus service:

      Example

      [root@node04 ~]# systemctl restart prometheus.service
      [root@node04 ~]# systemctl status prometheus.service

      Note

      If you have made changes to the Prometheus parameters in group_vars/all.yml file, then you have to rerun the playbook.

  4. Optional: If the changes do not reflect on the Red Hat Ceph Storage Dashboard, then you have to disable and then enable the dashboard:

    Example

    [root@node04 ~]# ceph mgr module disable dashboard
    [root@node04 ~]# ceph mgr module enable dashboard

7.9. Viewing and managing alerts

As a storage administrator, you can see the details of alerts and create silences for them on the Red Hat Ceph Storage dashboard. This includes the following pre-defined alerts:

  • OSD(s) Down
  • Ceph Health Error
  • Ceph Health Warning
  • Cluster Capacity Low
  • Disk(s) Near Full
  • MON(s) Down
  • Network Errors
  • OSD Host Loss Check
  • OSD Host(s) Down
  • OSD(s) with High PG Count
  • PG(s) Stuck
  • Pool Capacity Low
  • Slow OSD Responses

7.9.1. Viewing alerts

After an alert has fired, you can view it on the Red Hat Ceph Storage Dashboard. You can also enable the dashboard to send an email about the alert.

Prerequisite

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.
  • An alert fired.

Procedure

  1. Log in to the Dashboard.
  2. Customize the alerts module on the dashboard to get an email alert for the storage cluster:

    1. On the navigation bar, click Cluster.
    2. Select Manager modules.
    3. Select alerts module.
    4. In the Edit drop-down menu, select Edit.
    5. In the Edit Manager module, update the required parameters and click Update.

      Figure 7.1. Edit Manager module for alerts

      Edit Manager module for alerts
  3. On the navigation bar, click Cluster.
  4. Select Monitoring from the drop-down menu.
  5. To view details about the alert, click on its row:

    Figure 7.2. Alert Details

    Alert Details
  6. To view the source of an alert, click on its row, and then click Source.

7.9.2. Creating a silence

You can create a silence for an alert for a specified amount of time on the Red Hat Ceph Storage Dashboard.

Prerequisite

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.
  • An alert fired.

Procedure

  1. Log in to the Dashboard.
  2. On the navigation bar, click Cluster.
  3. Select Monitoring from the drop-down menu.
  4. Click on the row for the alert and then click +Create Silence.
  5. In the CreateSilence window, Add the details for the Duration and click Create Silence.

    Figure 7.3. Create Silence

    Create Silence
  6. You get a notification that the silence was created successfully.

7.9.3. Re-creating a silence

You can re-create a silence from an expired silence on the Red Hat Ceph Storage Dashboard.

Prerequisite

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.
  • An alert fired.
  • A silence created for the alert.

Procedure

  1. Log in to the Dashboard.
  2. Select Monitoring from the drop-down menu.
  3. Click the Silences tab.
  4. Click on the row for the expired silence.
  5. Click the Recreate button.
  6. In the RecreateSilence window, add the details and click RecreateSilence.

    Figure 7.4. Recreate silence

    Re-create Silence
  7. You get a notification that the silence was recreated successfully.

7.9.4. Editing a silence

You can edit an active silence, for example, to extend the time it is active on the Red Hat Ceph Storage Dashboard. If the silence has expired, you can either recreate a silence or create a new silence for the alert.

Prerequisite

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.
  • An alert fired.
  • A silence created for the alert.

Procedure

  1. Log in to the Dashboard.
  2. On the navigation bar, click Cluster.
  3. Select Monitoring from the drop-down menu.
  4. Click the Silences tab.
  5. Click on the row for the silence.
  6. In the Edit drop-down menu, select Edit.
  7. In the EditSilence window, update the details and click Edit Silence.

    Figure 7.5. Edit silence

    Edit Silence
  8. You get a notification that the silence was updated successfully.

7.9.5. Expiring a silence

You can expire a silence so any matched alerts will not be suppressed on the Red Hat Ceph Storage Dashboard.

Prerequisite

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.
  • An alert fired.
  • A silence created for the alert.

Procedure

  1. Log in to the Dashboard.
  2. On the navigation bar, click Cluster.
  3. Select Monitoring from the drop-down menu.
  4. Click the Silences tab.
  5. Click on the row for the silence.
  6. In the Edit drop-down menu, select Expire.
  7. In the Expire Silence dialog box, select Yes, I am sure, and then click Expire Silence.

    Figure 7.6. Expire Silence

    Expire Silence
  8. You get a notification that the silence was expired successfully.

7.9.6. Additional Resources

7.10. Managing pools

As a storage administrator, you can create, delete, and edit pools.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed

7.10.1. Creating pools

You can create pools to logically partition your storage objects.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.

Procedure

  1. Log in to the dashboard.
  2. On the navigation bar, click Pools.

    Click pools
  3. Click the Create button towards the top left corner of the page.

    Click create
  4. In the dialog window, set the name.

    Set name
  5. Select either replicated or Erasure Coded (EC) pool type.

    Select pool type
  6. Set the Placement Group (PG) number.

    Set PG number

    For assistance in choosing the PG number, use the PG calculator. Contact Red Hat Technical Support if unsure.

  7. Optional: If using a replicated pool type, set the replicated size.

    Replicated size
  8. Optional: If using an EC pool type configure the following additional settings.

    1. Optional: To see the settings for the currently selected EC profile, click the question mark.

      EC profile setttings
      1. A table of the settings for the selected EC profile is shown.

        EC profile table
    2. Optional: Add a new EC profile by clicking the plus symbol.

      Click the plus symbol
      1. Set the name of the new EC profile, at 1, click any question mark symbol for info about that setting, at 2, and after modifying all required settings, click Create EC Profile, at 3.

        EC profile dialog
      2. Select the new EC profile.

        Select new EC profile
    3. Optional: If EC overwrites are required, click its button.

      EC overwites
  9. Optional: Click the pencil symbol to select an application for the pool.

    Select appliation type
  10. Optional: If compression is required, select passive, aggressive, or force.

    Compression selection
  11. Click the Create Pool button.

    Click Create Pool
  12. Notifications towards the top right corner of the page indicate the pool was created successfully.

    Pool created notification

Additional Resources

7.10.2. Editing pools

The Red Hat Ceph Storage Dashboard allows editing of pools.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.
  • A pool is created.

Procedure

  1. Log in to the dashboard.
  2. On the navigation bar, click Pools.

    Click Pools menu
  3. To edit the pool, click its row:

    Click pool
  4. Select Edit In the Edit drop-down:

    Click Edit
  5. In the dialog window, edit the required parameters and click the EditPool button:

    Edit pool window
  6. A notifications towards the top right corner of the page indicates the pool was updated successfully.

    Pool edit notification

Additional Resources

  • See the Ceph pools in the Red Hat Ceph Storage Architecture Guide for more information.
  • See the Pool values in the Red Hat Ceph Storage Storage Strategies Guide for more information on Compression Modes.

7.10.3. Deleting pools

The Red Hat Ceph Storage Dashboard allows deletion of pools.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.
  • A pool is created.

Procedure

  1. Log in to the dashboard.
  2. Ensure the values of mon_allow_pool_delete is set to true:

    1. On the navigation bar, click Cluster and then click Configuration.
    2. In the Level drop-down menu, select Advanced:

      Click Levels drop-down
    3. Search for mon_allow_pool_delete and set the values to true

      Edit Configuration
  3. On the navigation bar, click Pools:

    Click Pools menu
  4. To delete the pool, click on its row:

    Click pool
  5. Select Delete in the Edit drop-down:

    Click Delete
  6. In the Delete Pool dialog window, Click the Yes, I am sure box and then Click Delete Pool to save the settings:

    Delete pool window
  7. A notification towards the top right corner of the page indicates the pool was deleted successfully.

    Pool delete notification

Additional Resources

  • See the Ceph pools in the Red Hat Ceph Storage Architecture Guide for more information.
  • See the Monitoring Configuration in the Red Hat Ceph Storage Dashboard Guide for more inforamtion.
  • See the Pool values in the Red Hat Ceph Storage Storage Strategies Guide for more information on Compression Modes.