Procedure

1. Check the overall status of the storage cluster:

   [root@mon ~]# ceph health detail

   If the command returns HEALTH_WARN or HEALTH_ERR see Understanding Ceph health for details.

2. Check the Ceph logs for any error messages listed in Understanding Ceph logs. The logs are located by default in the /var/log/ceph/ directory.
3. If the logs do not include sufficient amount of information, increase the debugging level and try to reproduce the action that failed. See Configuring logging for details.

1. To enable logging Ceph Monitors, Ceph Manager, Ceph Object Gateway, and any other daemons, set log_to_file to true under [global] settings.

   Example

   [ceph: root@host01 ~]# ceph config set global log_to_file true

2. To enable logging for Ceph Monitor cluster and audit logs, set mon_cluster_log_to_file to true.

   Example

   [ceph: root@host01 ~]# ceph config set mon mon_cluster_log_to_file true

To Troubleshoot This Problem

1. Verify that the ceph-mon daemon is running. If not, start it:

   [root@mon ~]# systemctl status ceph-mon@HOST_NAME
   [root@mon ~]# systemctl start ceph-mon@HOST_NAME

   Replace HOST_NAME with the short name of the host where the daemon is running. Use the hostname -s command when unsure.

2. If you are not able to start ceph-mon, follow the steps in The ceph-mon daemon cannot start.
3. If you are able to start the ceph-mon daemon but is is marked as down, follow the steps in The ceph-mon daemon is running, but marked as `down`.

The ceph-mon Daemon Cannot Start

1. Check the corresponding Ceph Monitor log, by default located at /var/log/ceph/ceph-mon.HOST_NAME.log.

2. If the log contains error messages similar to the following ones, the Ceph Monitor might have a corrupted store.

   Corruption: error in middle of record
   Corruption: 1 missing files; e.g.: /var/lib/ceph/mon/mon.0/store.db/1234567.ldb

   To fix this problem, replace the Ceph Monitor. See Replacing a failed monitor.

3. If the log contains an error message similar to the following one, the /var/ partition might be full. Delete any unnecessary data from /var/.

   Caught signal (Bus error)

   Important

   Do not delete any data from the Monitor directory manually. Instead, use the ceph-monstore-tool to compact it. See Compacting the Ceph Monitor store for details.

4. If you see any other error messages, open a support ticket. See Contacting Red Hat Support for service for details.

The ceph-mon Daemon Is Running, but Still Marked as down

1. From the Ceph Monitor host that is out of the quorum, use the mon_status command to check its state:

   [root@mon ~]# ceph daemon ID mon_status

   Replace ID with the ID of the Ceph Monitor, for example:

   [root@mon ~]# ceph daemon mon.a mon_status

2. If the status is probing, verify the locations of the other Ceph Monitors in the mon_status output.

   1. If the addresses are incorrect, the Ceph Monitor has incorrect Ceph Monitor map (monmap). To fix this problem, see Injecting a Ceph Monitor map.
   2. If the addresses are correct, verify that the Ceph Monitor clocks are synchronized. See Clock skew for details. In addition, troubleshoot any networking issues, see Troubleshooting Networking issues for details.
3. If the status is electing, verify that the Ceph Monitor clocks are synchronized. See Clock skew for details.
4. If the status changes from electing to synchronizing, open a support ticket. See Contacting Red Hat Support for service for details.
5. If the Ceph Monitor is the leader or a peon, verify that the Ceph Monitor clocks are synchronized. See Clock skew for details. Open a support ticket if synchronizing the clocks does not solve the problem. See Contacting Red Hat Support for service for details.

To Troubleshoot This Problem

1. Verify that your network works correctly. For details, see Troubleshooting networking issues. In particular, troubleshoot any problems with NTP clients if you use NTP.

   1. If you use chrony for NTP, see Basic chrony NTP troubleshooting section for more information.
   2. If you use ntpd, see Basic NTP troubleshooting.

2. If you use a remote NTP server, consider deploying your own NTP server on your network.

   1. For details, see the Using the Chrony suite to configure NTP chapter in the Configuring basic system settings for Red Hat Enterprise Linux 8.
   2. See the Configuring NTP Using ntpd chapter in the System Administrator’s Guide for Red Hat Enterprise Linux 7.

To Troubleshoot This Problem

1. Check the size of the database:

   du -sch /var/lib/ceph/mon/CLUSTER_NAME-SHORT_HOST_NAME/store.db

   Specify the name of the cluster and the short host name of the host where the ceph-mon is running.

   Example

   # du -sch /var/lib/ceph/mon/ceph-host1/store.db
   47G     /var/lib/ceph/mon/ceph-ceph1/store.db/
   47G     total

2. Compact the Ceph Monitor store. For details, see Compacting the Ceph Monitor Store.

Procedure

1. If you use Ceph in containers, mount all disk with Ceph data to a temporary location. Repeat this step for all OSD nodes.

   1. List the data partitions. Use ceph-volume or ceph-disk depending on which utility you used to set up the devices:

      [root@osd ~]# ceph-volume lvm list

      or

      [root@osd ~]# ceph-disk list

   2. Mount the data partitions to temporary location:

      mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-$i

   3. Restore the SELinux context:

      for i in {OSD_ID}; do restorecon /var/lib/ceph/osd/ceph-$i; done

      Replace OSD_ID with a numeric, space-separated list of Ceph OSD IDs on the OSD node.

   4. Change the owner and group to ceph:ceph:

      for i in {OSD_ID}; do chown -R ceph:ceph /var/lib/ceph/osd/ceph-$i; done

      Replace OSD_ID with a numeric, space-separated list of Ceph OSD IDs on the OSD node.

      Important

      Due to a bug that causes the update-mon-db command to use additional db and db.slow directories for the Monitor database, you must also copy these directories. To do so:

      1. Prepare a temporary location outside the container to mount and access the OSD database and extract the OSD maps needed to restore the Ceph Monitor:

         ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev OSD-DATA --path /var/lib/ceph/osd/ceph-OSD-ID

         Replace OSD-DATA with the Volume Group (VG) or Logical Volume (LV) path to the to the OSD data and OSD-ID with the ID of the OSD.

      2. Create a symbolic link between the BlueStore database and block.db:

         ln -snf BLUESTORE DATABASE /var/lib/ceph/osd/ceph-OSD-ID/block.db

         Replace BLUESTORE-DATABASE with the Volume Group (VG) or Logical Volume (LV) path to the BlueStore database and OSD-ID with the ID of the OSD.

2. Use the following commands from the Ceph Monitor node with the corrupted store. Repeat them for all OSDs on all nodes.

   1. Collect the cluster map from all OSD nodes:

      [root@ mon~]# cd /root/
      [root@mon ~]# ms=/tmp/monstore/
      [root@mon ~]# db=/root/db/
      [root@mon ~]# db_slow=/root/db.slow/
      
      [root@mon ~]# mkdir $ms
      [root@mon ~]# for host in $osd_nodes; do
                      echo "$host"
                      rsync -avz $ms $host:$ms
                      rsync -avz $db $host:$db
                      rsync -avz $db_slow $host:$db_slow
      
                      rm -rf $ms
                      rm -rf $db
                      rm -rf $db_slow
      
                      sh -t $host <<EOF
                        for osd in /var/lib/ceph/osd/ceph-*; do
                          ceph-objectstore-tool --type bluestore --data-path \$osd --op update-mon-db --mon-store-path $ms
      
                         done
                      EOF
      
                            rsync -avz $host:$ms $ms
                            rsync -avz $host:$db $db
                            rsync -avz $host:$db_slow $db_slow
                      done

   2. Set the appropriate capabilities:

      [root@mon ~]# ceph-authtool /etc/ceph/ceph.client.admin.keyring -n mon. --cap mon 'allow *' --gen-key
      [root@mon ~]# cat /etc/ceph/ceph.client.admin.keyring
        [mon.]
          key = AQCleqldWqm5IhAAgZQbEzoShkZV42RiQVffnA==
          caps mon = "allow *"
        [client.admin]
          key = AQCmAKld8J05KxAArOWeRAw63gAwwZO5o75ZNQ==
          auid = 0
          caps mds = "allow *"
          caps mgr = "allow *"
          caps mon = "allow *"
          caps osd = "allow *"

   3. Move all sst file from the db and db.slow directories to the temporary location:

      [root@mon ~]# mv /root/db/*.sst /root/db.slow/*.sst /tmp/monstore/store.db

   4. Rebuild the Monitor store from the collected map:

      [root@mon ~]# ceph-monstore-tool /tmp/monstore rebuild -- --keyring /etc/ceph/ceph.client.admin

      Note

      After using this command, only keyrings extracted from the OSDs and the keyring specified on the ceph-monstore-tool command line are present in Ceph’s authentication database. You have to recreate or import all other keyrings, such as clients, Ceph Manager, Ceph Object Gateway, and others, so those clients can access the cluster.

   5. Back up the corrupted store. Repeat this step for all Ceph Monitor nodes:

      mv /var/lib/ceph/mon/ceph-HOSTNAME/store.db /var/lib/ceph/mon/ceph-HOSTNAME/store.db.corrupted

      Replace HOSTNAME with the host name of the Ceph Monitor node.

   6. Replace the corrupted store. Repeat this step for all Ceph Monitor nodes:

      scp -r /tmp/monstore/store.db HOSTNAME:/var/lib/ceph/mon/ceph-HOSTNAME/

      Replace HOSTNAME with the host name of the Monitor node.

   7. Change the owner of the new store. Repeat this step for all Ceph Monitor nodes:

      chown -R ceph:ceph /var/lib/ceph/mon/ceph-HOSTNAME/store.db

      Replace HOSTNAME with the host name of the Ceph Monitor node.

3. If you use Ceph in containers, then unmount all the temporary mounted OSDs on all nodes:

   [root@osd ~]# umount /var/lib/ceph/osd/ceph-*

4. Start all the Ceph Monitor daemons:

   [root@mon ~]# systemctl start ceph-mon *

5. Ensure that the Monitors are able to form a quorum:

   • Bare-metal deployments

     [root@mon ~]# ceph -s

   • Containers

     [user@admin ~]$ docker exec ceph-mon-_HOSTNAME_ ceph -s

     Replace HOSTNAME with the host name of the Ceph Monitor node.

6. Import the Ceph Manager keyring and start all Ceph Manager processes:

   ceph auth import -i /etc/ceph/ceph.mgr.HOSTNAME.keyring
   systemctl start ceph-mgr@HOSTNAME

   Replace HOSTNAME with the host name of the Ceph Manager node.

7. Start all OSD processes across all OSD nodes:

   [root@osd ~]# systemctl start ceph-osd *

8. Ensure that the OSDs are returning to service:

   • Bare-metal deployments

     [root@mon ~]# ceph -s

   • Containers

     [user@admin ~]$ docker exec ceph-mon-_HOSTNAME_ ceph -s

     Replace HOSTNAME with the host name of the Ceph Monitor node.

To Troubleshoot This Problem:

1. Verify that the PG count is sufficient and increase it if needed.
2. Verify that you use CRUSH tunables optimal to the cluster version and adjust them if not.
3. Change the weight of OSDs by utilization.

4. Enable the Ceph Manager balancer module which optimizes the placement of placement groups (PGs) across OSDs in order to achieve a balanced distribution

   Example

   [root@mon ~]# ceph mgr module enable balancer

5. Determine how much space is left on the disks used by OSDs.

   1. To view how much space OSDs use in general:

      [root@mon ~]# ceph osd df

   2. To view how much space OSDs use on particular nodes. Use the following command from the node containing nearful OSDs:

      $ df

   3. If needed, add a new OSD node.

To Troubleshoot This Problem

1. Determine which OSD is down:

   [root@mon ~]# ceph health detail
   HEALTH_WARN 1/3 in osds are down
   osd.0 is down since epoch 23, last address 192.168.106.220:6800/11080

2. Try to restart the ceph-osd daemon:

   [root@mon ~]# systemctl restart ceph-osd@OSD_NUMBER

   Replace OSD_NUMBER with the ID of the OSD that is down, for example:

   [root@mon ~]# systemctl restart ceph-osd@0

   1. If you are not able start ceph-osd, follow the steps in The ceph-osd daemon cannot start.
   2. If you are able to start the ceph-osd daemon but it is marked as down, follow the steps in The ceph-osd daemon is running but still marked as `down`.

The ceph-osd daemon cannot start

1. If you have a node containing a number of OSDs (generally, more than twelve), verify that the default maximum number of threads (PID count) is sufficient. See Increasing the PID count for details.
2. Verify that the OSD data and journal partitions are mounted properly. You can use the ceph-volume lvm list command to list all devices and volumes associated with the Ceph Storage Cluster and then manually inspect if they are mounted properly. See the mount(8) manual page for details.
3. If you got the ERROR: missing keyring, cannot use cephx for authentication error message, the OSD is a missing keyring.

4. If you got the ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-1 error message, the ceph-osd daemon cannot read the underlying file system. See the following steps for instructions on how to troubleshoot and fix this error.

   Note

   If this error message is returned during boot time of the OSD host, open a support ticket as this might indicate a known issue tracked in the Red Hat Bugzilla 1439210.

5. Check the corresponding log file to determine the cause of the failure. By default, Ceph stores log files in the /var/log/ceph/ directory for bare-metal deployments.

   Note

   For container-based deployment, Ceph generates logs to journald. You can enable logging to files in /var/log/ceph by setting log_to_file parameter to true under [global] in the Ceph configuration file. See Understanding ceph logs for more details.

   1. An EIO error message similar to the following one indicates a failure of the underlying disk:

      To fix this problem replace the underlying OSD disk. See Replacing an OSD drive for details.

   2. If the log includes any other FAILED assert errors, such as the following one, open a support ticket. See Contacting Red Hat Support for service for details.

      FAILED assert(0 == "hit suicide timeout")

6. Check the dmesg output for the errors with the underlying file system or disk:

   $ dmesg

   1. If the dmesg output includes any SCSI error error messages, see the SCSI Error Codes Solution Finder solution on the Red Hat Customer Portal to determine the best way to fix the problem.
   2. Alternatively, if you are unable to fix the underlying file system, replace the OSD drive. See Replacing an OSD drive for details.

7. If the OSD failed with a segmentation fault, such as the following one, gather the required information and open a support ticket. See Contacting Red Hat Support for service for details.

   Caught signal (Segmentation fault)

The ceph-osd is running but still marked as down

1. Check the corresponding log file to determine the cause of the failure. By default, Ceph stores log files in the /var/log/ceph/ directory in bare-metal deployments.

   Note

   For container-based deployment, Ceph generates logs to journald. You can enable logging to files in /var/log/ceph by setting log_to_file parameter to true under [global] in the Ceph configuration file. See Understanding ceph logs for more details.

   1. If the log includes error messages similar to the following ones, see Flapping OSDs.

      wrongly marked me down
      heartbeat_check: no reply from osd.2 since back

   2. If you see any other errors, open a support ticket. See Contacting Red Hat Support for service for details.

To Troubleshoot This Problem

1. Check the output of the ceph health detail command again. If it includes the slow requests error message, see for details on how to troubleshoot this issue.

   # ceph health detail
   HEALTH_WARN 30 requests are blocked > 32 sec; 3 osds have slow requests
   30 ops are blocked > 268435 sec
   1 ops are blocked > 268435 sec on osd.11
   1 ops are blocked > 268435 sec on osd.18
   28 ops are blocked > 268435 sec on osd.39
   3 osds have slow requests

2. Determine which OSDs are marked as down and on what nodes they reside:

   # ceph osd tree | grep down

3. On the nodes containing the flapping OSDs, troubleshoot and fix any networking problems. For details, see Troubleshooting networking issues.

4. Alternatively, you can temporary force Monitors to stop marking the OSDs as down and up by setting the noup and nodown flags:

   # ceph osd set noup
   # ceph osd set nodown

   Important

   Using the noup and nodown flags does not fix the root cause of the problem but only prevents OSDs from flapping. To open a support ticket, see the Contacting Red Hat Support for service section for details.

To Troubleshoot This Problem

1. Determine if the OSDs with slow or block requests share a common piece of hardware, for example a disk drive, host, rack, or network switch.

2. If the OSDs share a disk:

   1. Use the smartmontools utility to check the health of the disk or the logs to determine any errors on the disk.

      Note

      The smartmontools utility is included in the smartmontools package.

   2. Use the iostat utility to get the I/O wait report (%iowai) on the OSD disk to determine if the disk is under heavy load.

      Note

      The iostat utility is included in the sysstat package.

3. If the OSDs share the node with another service:

   1. Check the RAM and CPU utilization
   2. Use the netstat utility to see the network statistics on the Network Interface Controllers (NICs) and troubleshoot any networking issues. See also Troubleshooting networking issues for further information.
4. If the OSDs share a rack, check the network switch for the rack. For example, if you use jumbo frames, verify that the NIC in the path has jumbo frames set.
5. If you are unable to determine a common piece of hardware shared by OSDs with slow requests, or to troubleshoot and fix hardware and networking problems, open a support ticket. See Contacting Red Hat support for service for details.

Procedure

1. Log in to the Ansible administration node.

2. Change to the /usr/share/ceph-ansible directory:

   Example

   [ansible@admin ~]$ cd /usr/share/ceph-ansible

3. Run the Ansible shrink-mds.yml playbook, and when prompted, type yes to confirm shrinking the cluster:

   Syntax

   ansible-playbook infrastructure-playbooks/shrink-mds.yml -e mds_to_kill=ID -i hosts

   Replace ID with the ID of the MDS node you want to remove. You can remove only one Ceph MDS each time the playbook runs.

   Example

   [ansible @admin ceph-ansible]$ ansible-playbook infrastructure-playbooks/shrink-mds.yml -e mds_to_kill=node02 -i hosts

4. As root or with sudo access, open and edit the /usr/share/ceph-ansible/hosts inventory file and remove the MDS node under the [mdss] section:

   Syntax

   [mdss]
   MDS_NODE_NAME
   MDS_NODE_NAME

   Example

   [mdss]
   node01
   node03

   In this example, node02 was removed from the [mdss] list.

Procedure

1. Log into the Ceph MDS node that you want to remove the MDS daemon from.

2. Stop the Ceph MDS service:

   Syntax

   sudo systemctl stop ceph-mds@HOST_NAME

   Replace HOST_NAME with the short name of the host where the daemon is running.

   Example

   [admin@node02 ~]$ sudo systemctl stop ceph-mds@node02

3. Disable the MDS service if you are not redeploying MDS to this node:

   Syntax

   sudo systemctl disable ceph-mds@HOST_NAME

   Replace HOST_NAME with the short name of the host to disable the daemon.

   Example

   [admin@node02 ~]$ sudo systemctl disable ceph-mds@node02

4. Remove the /var/lib/ceph/mds/ceph-MDS_ID directory on the MDS node:

   Syntax

   sudo rm -fr /var/lib/ceph/mds/ceph-MDS_ID

   Replace MDS_ID with the ID of the MDS node that you want to remove the MDS daemon from.

   Example

   [admin@node02 ~]$ sudo rm -fr /var/lib/ceph/mds/ceph-node02

Procedure

1. Log in to the Ansible administration node

2. Change to the /usr/share/ceph-ansible directory:

   Example

   [ansible@admin ~]$ cd /usr/share/ceph-ansible

3. As root or with sudo access, open and edit the /usr/share/ceph-ansible/hosts inventory file and add the MDS node under the [mdss] section:

   Syntax

   [mdss]
   MDS_NODE_NAME
   NEW_MDS_NODE_NAME

   Replace NEW_MDS_NODE_NAME with the host name of the node where you want to install the MDS server.

   Alternatively, you can colocate the MDS daemon with the OSD daemon on one node by adding the same node under the [osds] and [mdss] sections.

   Example

   [mdss]
   node01
   node03

4. As the ansible user, run the Ansible playbook to provision the MDS node:

   • Bare-metal deployments:

     [ansible@admin ceph-ansible]$ ansible-playbook site.yml --limit mdss -i hosts

   • Container deployments:

     [ansible@admin ceph-ansible]$ ansible-playbook site-container.yml --limit mdss -i hosts

     After the Ansible playbook has finished running, the new Ceph MDS node appears in the storage cluster.

Procedure

1. Add a new MDS node by logging into the node and creating an MDS mount point:

   Syntax

   sudo mkdir /var/lib/ceph/mds/ceph-MDS_ID

   Replace MDS_ID with the ID of the MDS node that you want to add the MDS daemon to.

   Example

   [admin@node03 ~]$ sudo mkdir /var/lib/ceph/mds/ceph-node03

2. If this is a new MDS node, create the authentication key if you are using Cephx authentication:

   Syntax

   sudo ceph auth get-or-create mds.MDS_ID mon 'profile mds' mgr 'profile mds' mds 'allow *' osd 'allow *' > /var/lib/ceph/mds/ceph-MDS_ID/keyring

   Replace MDS_ID with the ID of the MDS node to deploy the MDS daemon on.

   Example

   [admin@node03 ~]$ sudo ceph auth get-or-create mds.node03 mon 'profile mds' mgr 'profile mds' mds 'allow *' osd 'allow *' > /var/lib/ceph/mds/ceph-node03/keyring

   Note

   Cephx authentication is enabled by default. See the Cephx authentication link in the Additional Resources section for more information about Cephx authentication.

3. Start the MDS daemon:

   Syntax

   sudo systemctl start ceph-mds@HOST_NAME

   Replace HOST_NAME with the short name of the host to start the daemon.

   Example

   [admin@node03 ~]$ sudo systemctl start ceph-mds@node03

4. Enable the MDS service:

   Syntax

   systemctl enable ceph-mds@HOST_NAME

   Replace HOST_NAME with the short name of the host to enable the service.

   Example

   [admin@node03 ~]$ sudo systemctl enable ceph-mds@node03

To Troubleshoot This Problem

1. Identify which PGs are stale and on what OSDs they are stored. The error message will include information similar to the following example:

   Example

   # ceph health detail
   HEALTH_WARN 24 pgs stale; 3/300 in osds are down
   ...
   pg 2.5 is stuck stale+active+remapped, last acting [2,0]
   ...
   osd.10 is down since epoch 23, last address 192.168.106.220:6800/11080
   osd.11 is down since epoch 13, last address 192.168.106.220:6803/11539
   osd.12 is down since epoch 24, last address 192.168.106.220:6806/11861

2. Troubleshoot any problems with the OSDs that are marked as down. For details, see Down OSDs.

To Troubleshoot This Problem

1. Determine which placement group is in the inconsistent state:

   # ceph health detail
   HEALTH_ERR 1 pgs inconsistent; 2 scrub errors
   pg 0.6 is active+clean+inconsistent, acting [0,1,2]
   2 scrub errors

2. Determine why the placement group is inconsistent.

   1. Start the deep scrubbing process on the placement group:

      [root@mon ~]# ceph pg deep-scrub ID

      Replace ID with the ID of the inconsistent placement group, for example:

      [root@mon ~]# ceph pg deep-scrub 0.6
      instructing pg 0.6 on osd.0 to deep-scrub

   2. Search the output of the ceph -w for any messages related to that placement group:

      ceph -w | grep ID

      Replace ID with the ID of the inconsistent placement group, for example:

      [root@mon ~]# ceph -w | grep 0.6
      2015-02-26 01:35:36.778215 osd.106 [ERR] 0.6 deep-scrub stat mismatch, got 636/635 objects, 0/0 clones, 0/0 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 whiteouts, 1855455/1854371 bytes.
      2015-02-26 01:35:36.788334 osd.106 [ERR] 0.6 deep-scrub 1 errors

3. If the output includes any error messages similar to the following ones, you can repair the inconsistent placement group. See Repairing inconsistent placement groups for details.

   PG.ID shard OSD: soid OBJECT missing attr , missing attr _ATTRIBUTE_TYPE
   PG.ID shard OSD: soid OBJECT digest 0 != known digest DIGEST, size 0 != known size SIZE
   PG.ID shard OSD: soid OBJECT size 0 != known size SIZE
   PG.ID deep-scrub stat mismatch, got MISMATCH
   PG.ID shard OSD: soid OBJECT candidate had a read error, digest 0 != known digest DIGEST

4. If the output includes any error messages similar to the following ones, it is not safe to repair the inconsistent placement group because you can lose data. Open a support ticket in this situation. See Contacting Red Hat support for details.

   PG.ID shard OSD: soid OBJECT digest DIGEST != known digest DIGEST
   PG.ID shard OSD: soid OBJECT omap_digest DIGEST != known omap_digest DIGEST

To Troubleshoot This Problem

1. Determine which OSDs are down:

   # ceph osd tree

2. Troubleshoot and fix any problems with the OSDs. See Down OSDs for details.

To Troubleshoot This Problem

1. Determine which OSDs are down:

   # ceph osd tree

2. Troubleshoot and fix any problems with the OSDs.

1. osd.1 goes down.
2. osd.2 handles some write operations.
3. osd.1 comes up.
4. A peering process between osd.1 and osd.2 starts, and the objects missing on osd.1 are queued for recovery.
5. Before Ceph copies new objects, osd.2 goes down.

To Troubleshoot This Problem

1. Determine which placement group contain unfound objects:

   [root@mon ~]# ceph health detail
   HEALTH_WARN 1 pgs recovering; 1 pgs stuck unclean; recovery 5/937611 objects degraded (0.001%); 1/312537 unfound (0.000%)
   pg 3.8a5 is stuck unclean for 803946.712780, current state active+recovering, last acting [320,248,0]
   pg 3.8a5 is active+recovering, acting [320,248,0], 1 unfound
   recovery 5/937611 objects degraded (0.001%); **1/312537 unfound (0.000%)**

2. List more information about the placement group:

   [root@mon ~]# ceph pg ID query

   Replace ID with the ID of the placement group containing the unfound objects, for example:

   [root@mon ~]# ceph pg 3.8a5 query
   { "state": "active+recovering",
     "epoch": 10741,
     "up": [
           320,
           248,
           0],
     "acting": [
           320,
           248,
           0],
   <snip>
     "recovery_state": [
           { "name": "Started\/Primary\/Active",
             "enter_time": "2015-01-28 19:30:12.058136",
             "might_have_unfound": [
                   { "osd": "0",
                     "status": "already probed"},
                   { "osd": "248",
                     "status": "already probed"},
                   { "osd": "301",
                     "status": "already probed"},
                   { "osd": "362",
                     "status": "already probed"},
                   { "osd": "395",
                     "status": "already probed"},
                   { "osd": "429",
                     "status": "osd is down"}],
             "recovery_progress": { "backfill_targets": [],
                 "waiting_on_backfill": [],
                 "last_backfill_started": "0\/\/0\/\/-1",
                 "backfill_info": { "begin": "0\/\/0\/\/-1",
                     "end": "0\/\/0\/\/-1",
                     "objects": []},
                 "peer_backfill_info": [],
                 "backfills_in_flight": [],
                 "recovering": [],
                 "pg_backend": { "pull_from_peer": [],
                     "pushing": []}},
             "scrub": { "scrubber.epoch_start": "0",
                 "scrubber.active": 0,
                 "scrubber.block_writes": 0,
                 "scrubber.finalizing": 0,
                 "scrubber.waiting_on": 0,
                 "scrubber.waiting_on_whom": []}},
           { "name": "Started",
             "enter_time": "2015-01-28 19:30:11.044020"}],

   The might_have_unfound section includes OSDs where Ceph tried to locate the unfound objects:

   • The already probed status indicates that Ceph cannot locate the unfound objects in that OSD.
   • The osd is down status indicates that Ceph cannot contact that OSD.
3. Troubleshoot the OSDs that are marked as down. See Down OSDs for details.
4. If you are unable to fix the problem that causes the OSD to be down, open a support ticket. See Contacting Red Hat Support for service for details.

Procedure

1. Verify the appropriate OSD is down:

   [root@osd ~]# systemctl status ceph-osd@OSD_NUMBER

   Example

   [root@osd ~]# systemctl status ceph-osd@1

2. For containerized deployments, to access the bluestore tool, follow the below steps:

   1. Set noout flag on cluster.

      Example

      [root@mon ~]# ceph osd set noout

   2. Login to the node hosting the OSD container.

   3. Backup /etc/systemd/system/ceph-osd@.service unit file to /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.backup

   4. Move /run/ceph-osd@OSD_ID.service-cid file to /root.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /root

   5. Edit /etc/systemd/system/ceph-osd@.service unit file and add -it --entrypoint /bin/bash option to podman command.

      Example

      # Please do not change this file directly since it is managed by Ansible and will be overwritten
      [Unit]
      Description=Ceph OSD
      After=network.target
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
      ExecStartPre=-/usr/bin/podman rm -f ceph-osd-%i
      ExecStart=/usr/bin/podman run -it --entrypoint /bin/bash \
        -d --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
        --rm \
        --net=host \
        --privileged=true \
        --pid=host \
        --ipc=host \
        --cpus=2 \
        -v /dev:/dev \
        -v /etc/localtime:/etc/localtime:ro \
        -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
        -v /var/run/udev/:/var/run/udev/ \
        -v /var/log/ceph:/var/log/ceph:z \
        -e OSD_BLUESTORE=1 -e OSD_FILESTORE=0 -e OSD_DMCRYPT=0 \
        -e CLUSTER=ceph \
        -v /run/lvm/:/run/lvm/ \
        -e CEPH_DAEMON=OSD_CEPH_VOLUME_ACTIVATE \
        -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \
        -e OSD_ID=%i \
        -e DEBUG=stayalive \
        --name=ceph-osd-%i \
         \
        registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      ExecStop=-/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
      KillMode=none
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      Type=forking
      PIDFile=/%t/%n-pid
      
      [Install]
      WantedBy=multi-user.target

   6. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

   7. Restart the OSD service associated with the OSD_ID.

      Syntax

      systemctl restart ceph-osd@OSD_ID.service

      Replace OSD_ID with the ID of the OSD.

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

   8. Login to the container associated with the OSD_ID.

      Syntax

      podman exec -it ceph-osd-OSD_ID /bin/bash

      Example

      [root@osd ~]# podman exec -it ceph-osd-0 /bin/bash

   9. Get osd fsid and activate the OSD to mount OSD’s logical volume (LV).

      Syntax

      ceph-volume lvm list |grep -A15 "osd\.OSD_ID"|grep "osd fsid"
      ceph-volume lvm activate --bluestore OSD_ID OSD_FSID

      Example

      [root@osd ~]# ceph-volume lvm list |grep -A15 "osd\.0"|grep "osd fsid"
                    osd fsid                  087eee15-6561-40a3-8fe4-9583ba64a4ff
      [root@osd ~]# ceph-volume lvm activate --bluestore 0 087eee15-6561-40a3-8fe4-9583ba64a4ff
      Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc --path /var/lib/ceph/osd/ceph-0 --no-mon-config
      Running command: /usr/bin/ln -snf /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--41c69f8f--30e2--4685--9c5c--c605898c5537-osd--data--d073e8b3--0b89--4271--af5b--83045fd000dc
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff
       stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff.service → /usr/lib/systemd/system/ceph-volume@.service.
      Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
       stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph osd@0.service → /usr/lib/systemd/system/ceph-osd@.service.
      Running command: /usr/bin/systemctl start ceph-osd@0
       stderr: Running in chroot, ignoring request: start
      --> ceph-volume lvm activate successful for osd ID: 0

3. Identify all the objects within an OSD, regardless of their placement group:

   [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --op list

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op list

4. Identify all the objects within a placement group:

   [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID --op list

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c --op list

5. Identify the PG an object belongs to:

   [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --op list OBJECT_ID

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op list default.region

6. For containerized deployments, to revert the changes, follow the below steps:

   1. After exiting the container, copy /etc/systemd/system/ceph-osd@.service unit file from /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.modified
      [root@osd ~]# cp /root/ceph-osd@.service.backup /etc/systemd/system/ceph-osd@.service

   2. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

   3. Move /run/ceph-osd@OSD_ID.service-cid file to /tmp.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /tmp

   4. Restart the OSD service associated with the OSD_ID.

      Syntax

      [root@osd ~]# systemctl restart ceph-osd@OSD_ID.service

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

Procedure

1. Verify the appropriate OSD is down:

   [root@osd ~]# systemctl status ceph-osd@OSD_NUMBER

   Example

   [root@osd ~]# systemctl status ceph-osd@1

2. For containerized deployments, to access the bluestore tool, follow the below steps:

   1. Set noout flag on cluster.

      Example

      [root@mon ~]# ceph osd set noout

   2. Login to the node hosting the OSD container.

   3. Backup /etc/systemd/system/ceph-osd@.service unit file to /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.backup

   4. Move /run/ceph-osd@OSD_ID.service-cid file to /root.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /root

   5. Edit /etc/systemd/system/ceph-osd@.service unit file and add -it --entrypoint /bin/bash option to podman command.

      Example

      # Please do not change this file directly since it is managed by Ansible and will be overwritten
      [Unit]
      Description=Ceph OSD
      After=network.target
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
      ExecStartPre=-/usr/bin/podman rm -f ceph-osd-%i
      ExecStart=/usr/bin/podman run -it --entrypoint /bin/bash \
        -d --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
        --rm \
        --net=host \
        --privileged=true \
        --pid=host \
        --ipc=host \
        --cpus=2 \
        -v /dev:/dev \
        -v /etc/localtime:/etc/localtime:ro \
        -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
        -v /var/run/udev/:/var/run/udev/ \
        -v /var/log/ceph:/var/log/ceph:z \
        -e OSD_BLUESTORE=1 -e OSD_FILESTORE=0 -e OSD_DMCRYPT=0 \
        -e CLUSTER=ceph \
        -v /run/lvm/:/run/lvm/ \
        -e CEPH_DAEMON=OSD_CEPH_VOLUME_ACTIVATE \
        -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \
        -e OSD_ID=%i \
        -e DEBUG=stayalive \
        --name=ceph-osd-%i \
         \
        registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      ExecStop=-/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
      KillMode=none
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      Type=forking
      PIDFile=/%t/%n-pid
      
      [Install]
      WantedBy=multi-user.target

   6. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

   7. Restart the OSD service associated with the OSD_ID.

      Syntax

      systemctl restart ceph-osd@OSD_ID.service

      Replace OSD_ID with the ID of the OSD.

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

   8. Login to the container associated with the OSD_ID.

      Syntax

      podman exec -it ceph-osd-OSD_ID /bin/bash

      Example

      [root@osd ~]# podman exec -it ceph-osd-0 /bin/bash

   9. Get osd fsid and activate the OSD to mount OSD’s logical volume (LV).

      Syntax

      ceph-volume lvm list |grep -A15 "osd\.OSD_ID"|grep "osd fsid"
      ceph-volume lvm activate --bluestore OSD_ID OSD_FSID

      Example

      [root@osd ~]# ceph-volume lvm list |grep -A15 "osd\.0"|grep "osd fsid"
                    osd fsid                  087eee15-6561-40a3-8fe4-9583ba64a4ff
      [root@osd ~]# ceph-volume lvm activate --bluestore 0 087eee15-6561-40a3-8fe4-9583ba64a4ff
      Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc --path /var/lib/ceph/osd/ceph-0 --no-mon-config
      Running command: /usr/bin/ln -snf /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--41c69f8f--30e2--4685--9c5c--c605898c5537-osd--data--d073e8b3--0b89--4271--af5b--83045fd000dc
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff
       stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff.service → /usr/lib/systemd/system/ceph-volume@.service.
      Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
       stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph osd@0.service → /usr/lib/systemd/system/ceph-osd@.service.
      Running command: /usr/bin/systemctl start ceph-osd@0
       stderr: Running in chroot, ignoring request: start
      --> ceph-volume lvm activate successful for osd ID: 0

3. Use the ceph-objectstore-tool utility to list lost and unfound objects. Select the appropriate circumstance:

   1. To list all the lost objects:

      [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --op list-lost

      Example

      [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op list-lost

   2. To list all the lost objects within a placement group:

      [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID --op list-lost

      Example

      [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c --op list-lost

   3. To list a lost object by its identifier:

      [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --op list-lost OBJECT_ID

      Example

      [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op list-lost default.region

4. For containerized deployments, to revert the changes, follow the below steps:

   1. After exiting the container, copy /etc/systemd/system/ceph-osd@.service unit file from /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.modified
      [root@osd ~]# cp /root/ceph-osd@.service.backup /etc/systemd/system/ceph-osd@.service

   2. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

   3. Move /run/ceph-osd@OSD_ID.service-cid file to /tmp.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /tmp

   4. Restart the OSD service associated with the OSD_ID.

      Syntax

      [root@osd ~]# systemctl restart ceph-osd@OSD_ID.service

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

Procedure

1. Verify the appropriate OSD is down:

   Syntax

   [root@osd ~]# systemctl status ceph-osd@OSD_NUMBER

   Example

   [root@osd ~]# systemctl status ceph-osd@1

2. For containerized deployments, to access the bluestore tool, follow the below steps:

   1. Set noout flag on cluster.

      Example

      [root@mon ~]# ceph osd set noout

   2. Login to the node hosting the OSD container.

   3. Backup /etc/systemd/system/ceph-osd@.service unit file to /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.backup

   4. Move /run/ceph-osd@OSD_ID.service-cid file to /root.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /root

   5. Edit /etc/systemd/system/ceph-osd@.service unit file and add -it --entrypoint /bin/bash option to podman command.

      Example

      # Please do not change this file directly since it is managed by Ansible and will be overwritten
      [Unit]
      Description=Ceph OSD
      After=network.target
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
      ExecStartPre=-/usr/bin/podman rm -f ceph-osd-%i
      ExecStart=/usr/bin/podman run -it --entrypoint /bin/bash \
        -d --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
        --rm \
        --net=host \
        --privileged=true \
        --pid=host \
        --ipc=host \
        --cpus=2 \
        -v /dev:/dev \
        -v /etc/localtime:/etc/localtime:ro \
        -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
        -v /var/run/udev/:/var/run/udev/ \
        -v /var/log/ceph:/var/log/ceph:z \
        -e OSD_BLUESTORE=1 -e OSD_FILESTORE=0 -e OSD_DMCRYPT=0 \
        -e CLUSTER=ceph \
        -v /run/lvm/:/run/lvm/ \
        -e CEPH_DAEMON=OSD_CEPH_VOLUME_ACTIVATE \
        -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \
        -e OSD_ID=%i \
        -e DEBUG=stayalive \
        --name=ceph-osd-%i \
         \
        registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      ExecStop=-/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
      KillMode=none
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      Type=forking
      PIDFile=/%t/%n-pid
      
      [Install]
      WantedBy=multi-user.target

   6. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

   7. Restart the OSD service associated with the OSD_ID.

      Syntax

      systemctl restart ceph-osd@OSD_ID.service

      Replace OSD_ID with the ID of the OSD.

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

   8. Login to the container associated with the OSD_ID.

      Syntax

      podman exec -it ceph-osd-OSD_ID /bin/bash

      Example

      [root@osd ~]# podman exec -it ceph-osd-0 /bin/bash

   9. Get osd fsid and activate the OSD to mount OSD’s logical volume (LV).

      Syntax

      ceph-volume lvm list |grep -A15 "osd\.OSD_ID"|grep "osd fsid"
      ceph-volume lvm activate --bluestore OSD_ID OSD_FSID

      Example

      [root@osd ~]# ceph-volume lvm list |grep -A15 "osd\.0"|grep "osd fsid"
                    osd fsid                  087eee15-6561-40a3-8fe4-9583ba64a4ff
      [root@osd ~]# ceph-volume lvm activate --bluestore 0 087eee15-6561-40a3-8fe4-9583ba64a4ff
      Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc --path /var/lib/ceph/osd/ceph-0 --no-mon-config
      Running command: /usr/bin/ln -snf /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--41c69f8f--30e2--4685--9c5c--c605898c5537-osd--data--d073e8b3--0b89--4271--af5b--83045fd000dc
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff
       stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff.service → /usr/lib/systemd/system/ceph-volume@.service.
      Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
       stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph osd@0.service → /usr/lib/systemd/system/ceph-osd@.service.
      Running command: /usr/bin/systemctl start ceph-osd@0
       stderr: Running in chroot, ignoring request: start
      --> ceph-volume lvm activate successful for osd ID: 0

3. To list all the lost legacy objects:

   Syntax

   ceph-objectstore-tool --data-path PATH_TO_OSD --op fix-lost --dry-run

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op fix-lost --dry-run

4. Use the ceph-objectstore-tool utility to fix lost and unfound objects as a ceph user. Select the appropriate circumstance:

   • To fix all lost objects:

     Syntax

     su - ceph -c 'ceph-objectstore-tool --data-path PATH_TO_OSD --op fix-lost'

     Example

     [root@osd ~]# su - ceph -c 'ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op fix-lost'

   • To fix all the lost objects within a placement group:

     su - ceph -c 'ceph-objectstore-tool --data-path _PATH_TO_OSD_ --pgid _PG_ID_ --op fix-lost'

     Example

     [root@osd ~]# su - ceph -c 'ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c --op fix-lost'

   • To fix a lost object by its identifier:

     Syntax

     su - ceph -c 'ceph-objectstore-tool --data-path PATH_TO_OSD --op fix-lost OBJECT_ID'

     Example

     [root@osd ~]# su - ceph -c 'ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op fix-lost default.region'

5. For containerized deployments, to revert the changes, follow the below steps:

   1. After exiting the container, copy /etc/systemd/system/ceph-osd@.service unit file from /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.modified
      [root@osd ~]# cp /root/ceph-osd@.service.backup /etc/systemd/system/ceph-osd@.service

   2. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

   3. Move /run/ceph-osd@OSD_ID.service-cid file to /tmp.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /tmp

   4. Restart the OSD service associated with the OSD_ID.

      Syntax

      [root@osd ~]# systemctl restart ceph-osd@OSD_ID.service

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

Procedure

1. Verify the appropriate OSD is down:

   [root@osd ~]# systemctl status ceph-osd@$OSD_NUMBER

   Example

   [root@osd ~]# systemctl status ceph-osd@1

2. For containerized deployments, to access the bluestore tool, follow the below steps:

   1. Set noout flag on cluster.

      Example

      [root@mon ~]# ceph osd set noout

   2. Login to the node hosting the OSD container.

   3. Backup /etc/systemd/system/ceph-osd@.service unit file to /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.backup

   4. Move /run/ceph-osd@OSD_ID.service-cid file to /root.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /root

   5. Edit /etc/systemd/system/ceph-osd@.service unit file and add -it --entrypoint /bin/bash option to podman command.

      Example

      # Please do not change this file directly since it is managed by Ansible and will be overwritten
      [Unit]
      Description=Ceph OSD
      After=network.target
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
      ExecStartPre=-/usr/bin/podman rm -f ceph-osd-%i
      ExecStart=/usr/bin/podman run -it --entrypoint /bin/bash \
        -d --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
        --rm \
        --net=host \
        --privileged=true \
        --pid=host \
        --ipc=host \
        --cpus=2 \
        -v /dev:/dev \
        -v /etc/localtime:/etc/localtime:ro \
        -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
        -v /var/run/udev/:/var/run/udev/ \
        -v /var/log/ceph:/var/log/ceph:z \
        -e OSD_BLUESTORE=1 -e OSD_FILESTORE=0 -e OSD_DMCRYPT=0 \
        -e CLUSTER=ceph \
        -v /run/lvm/:/run/lvm/ \
        -e CEPH_DAEMON=OSD_CEPH_VOLUME_ACTIVATE \
        -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \
        -e OSD_ID=%i \
        -e DEBUG=stayalive \
        --name=ceph-osd-%i \
         \
        registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      ExecStop=-/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
      KillMode=none
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      Type=forking
      PIDFile=/%t/%n-pid
      
      [Install]
      WantedBy=multi-user.target

   6. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

   7. Restart the OSD service associated with the OSD_ID.

      Syntax

      systemctl restart ceph-osd@OSD_ID.service

      Replace OSD_ID with the ID of the OSD.

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

   8. Login to the container associated with the OSD_ID.

      Syntax

      podman exec -it ceph-osd-OSD_ID /bin/bash

      Example

      [root@osd ~]# podman exec -it ceph-osd-0 /bin/bash

   9. Get osd fsid and activate the OSD to mount OSD’s logical volume (LV).

      Syntax

      ceph-volume lvm list |grep -A15 "osd\.OSD_ID"|grep "osd fsid"
      ceph-volume lvm activate --bluestore OSD_ID OSD_FSID

      Example

      [root@osd ~]# ceph-volume lvm list |grep -A15 "osd\.0"|grep "osd fsid"
                    osd fsid                  087eee15-6561-40a3-8fe4-9583ba64a4ff
      [root@osd ~]# ceph-volume lvm activate --bluestore 0 087eee15-6561-40a3-8fe4-9583ba64a4ff
      Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc --path /var/lib/ceph/osd/ceph-0 --no-mon-config
      Running command: /usr/bin/ln -snf /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--41c69f8f--30e2--4685--9c5c--c605898c5537-osd--data--d073e8b3--0b89--4271--af5b--83045fd000dc
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff
       stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff.service → /usr/lib/systemd/system/ceph-volume@.service.
      Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
       stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph osd@0.service → /usr/lib/systemd/system/ceph-osd@.service.
      Running command: /usr/bin/systemctl start ceph-osd@0
       stderr: Running in chroot, ignoring request: start
      --> ceph-volume lvm activate successful for osd ID: 0

3. Find the object by listing the objects of the OSD or placement group (PG).

4. Before setting the bytes on an object, make a backup and a working copy of the object:

   [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID \
   OBJECT \
   get-bytes > OBJECT_FILE_NAME
   
   [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID \
   OBJECT \
   get-bytes > OBJECT_FILE_NAME

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
   '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
   get-bytes > zone_info.default.backup
   
   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
   '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
   get-bytes > zone_info.default.working-copy

5. Edit the working copy object file and modify the object contents accordingly.

6. Set the bytes of the object:

   [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID \
   OBJECT \
   set-bytes < OBJECT_FILE_NAME

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
   '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
   set-bytes < zone_info.default.working-copy

7. For containerized deployments, to revert the changes, follow the below steps:

   1. After exiting the container, copy /etc/systemd/system/ceph-osd@.service unit file from /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.modified
      [root@osd ~]# cp /root/ceph-osd@.service.backup /etc/systemd/system/ceph-osd@.service

   2. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

   3. Move /run/ceph-osd@OSD_ID.service-cid file to /tmp.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /tmp

   4. Restart the OSD service associated with the OSD_ID.

      Syntax

      [root@osd ~]# systemctl restart ceph-osd@OSD_ID.service

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

Procedure

1. Verify the appropriate OSD is down:

   [root@osd ~]# systemctl status ceph-osd@$OSD_NUMBER

   Example

   [root@osd ~]# systemctl status ceph-osd@1

2. For containerized deployments, to access the bluestore tool, follow the below steps:

   1. Set noout flag on cluster.

      Example

      [root@mon ~]# ceph osd set noout

   2. Login to the node hosting the OSD container.

   3. Backup /etc/systemd/system/ceph-osd@.service unit file to /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.backup

   4. Move /run/ceph-osd@OSD_ID.service-cid file to /root.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /root

   5. Edit /etc/systemd/system/ceph-osd@.service unit file and add -it --entrypoint /bin/bash option to podman command.

      Example

      # Please do not change this file directly since it is managed by Ansible and will be overwritten
      [Unit]
      Description=Ceph OSD
      After=network.target
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
      ExecStartPre=-/usr/bin/podman rm -f ceph-osd-%i
      ExecStart=/usr/bin/podman run -it --entrypoint /bin/bash \
        -d --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
        --rm \
        --net=host \
        --privileged=true \
        --pid=host \
        --ipc=host \
        --cpus=2 \
        -v /dev:/dev \
        -v /etc/localtime:/etc/localtime:ro \
        -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
        -v /var/run/udev/:/var/run/udev/ \
        -v /var/log/ceph:/var/log/ceph:z \
        -e OSD_BLUESTORE=1 -e OSD_FILESTORE=0 -e OSD_DMCRYPT=0 \
        -e CLUSTER=ceph \
        -v /run/lvm/:/run/lvm/ \
        -e CEPH_DAEMON=OSD_CEPH_VOLUME_ACTIVATE \
        -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \
        -e OSD_ID=%i \
        -e DEBUG=stayalive \
        --name=ceph-osd-%i \
         \
        registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      ExecStop=-/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
      KillMode=none
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      Type=forking
      PIDFile=/%t/%n-pid
      
      [Install]
      WantedBy=multi-user.target

   6. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

   7. Restart the OSD service associated with the OSD_ID.

      Syntax

      systemctl restart ceph-osd@OSD_ID.service

      Replace OSD_ID with the ID of the OSD.

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

   8. Login to the container associated with the OSD_ID.

      Syntax

      podman exec -it ceph-osd-OSD_ID /bin/bash

      Example

      [root@osd ~]# podman exec -it ceph-osd-0 /bin/bash

   9. Get osd fsid and activate the OSD to mount OSD’s logical volume (LV).

      Syntax

      ceph-volume lvm list |grep -A15 "osd\.OSD_ID"|grep "osd fsid"
      ceph-volume lvm activate --bluestore OSD_ID OSD_FSID

      Example

      [root@osd ~]# ceph-volume lvm list |grep -A15 "osd\.0"|grep "osd fsid"
                    osd fsid                  087eee15-6561-40a3-8fe4-9583ba64a4ff
      [root@osd ~]# ceph-volume lvm activate --bluestore 0 087eee15-6561-40a3-8fe4-9583ba64a4ff
      Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc --path /var/lib/ceph/osd/ceph-0 --no-mon-config
      Running command: /usr/bin/ln -snf /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--41c69f8f--30e2--4685--9c5c--c605898c5537-osd--data--d073e8b3--0b89--4271--af5b--83045fd000dc
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff
       stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff.service → /usr/lib/systemd/system/ceph-volume@.service.
      Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
       stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph osd@0.service → /usr/lib/systemd/system/ceph-osd@.service.
      Running command: /usr/bin/systemctl start ceph-osd@0
       stderr: Running in chroot, ignoring request: start
      --> ceph-volume lvm activate successful for osd ID: 0

3. Remove an object:

   Syntax

   ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID \
   OBJECT \
   remove

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
   '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
   remove

4. For containerized deployments, to revert the changes, follow the below steps:

   1. After exiting the container, copy /etc/systemd/system/ceph-osd@.service unit file from /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.modified
      [root@osd ~]# cp /root/ceph-osd@.service.backup /etc/systemd/system/ceph-osd@.service

   2. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

   3. Move /run/ceph-osd@OSD_ID.service-cid file to /tmp.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /tmp

   4. Restart the OSD service associated with the OSD_ID.

      Syntax

      [root@osd ~]# systemctl restart ceph-osd@OSD_ID.service

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

Procedure

1. Verify the appropriate OSD is down:

   [root@osd ~]# systemctl status ceph-osd@OSD_NUMBER

   Example

   [root@osd ~]# systemctl status ceph-osd@1

2. For containerized deployments, to access the bluestore tool, follow the below steps:

   1. Set noout flag on cluster.

      Example

      [root@mon ~]# ceph osd set noout

   2. Login to the node hosting the OSD container.

   3. Backup /etc/systemd/system/ceph-osd@.service unit file to /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.backup

   4. Move /run/ceph-osd@OSD_ID.service-cid file to /root.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /root

   5. Edit /etc/systemd/system/ceph-osd@.service unit file and add -it --entrypoint /bin/bash option to podman command.

      Example

      # Please do not change this file directly since it is managed by Ansible and will be overwritten
      [Unit]
      Description=Ceph OSD
      After=network.target
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
      ExecStartPre=-/usr/bin/podman rm -f ceph-osd-%i
      ExecStart=/usr/bin/podman run -it --entrypoint /bin/bash \
        -d --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
        --rm \
        --net=host \
        --privileged=true \
        --pid=host \
        --ipc=host \
        --cpus=2 \
        -v /dev:/dev \
        -v /etc/localtime:/etc/localtime:ro \
        -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
        -v /var/run/udev/:/var/run/udev/ \
        -v /var/log/ceph:/var/log/ceph:z \
        -e OSD_BLUESTORE=1 -e OSD_FILESTORE=0 -e OSD_DMCRYPT=0 \
        -e CLUSTER=ceph \
        -v /run/lvm/:/run/lvm/ \
        -e CEPH_DAEMON=OSD_CEPH_VOLUME_ACTIVATE \
        -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \
        -e OSD_ID=%i \
        -e DEBUG=stayalive \
        --name=ceph-osd-%i \
         \
        registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      ExecStop=-/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
      KillMode=none
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      Type=forking
      PIDFile=/%t/%n-pid
      
      [Install]
      WantedBy=multi-user.target

   6. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

   7. Restart the OSD service associated with the OSD_ID.

      Syntax

      systemctl restart ceph-osd@OSD_ID.service

      Replace OSD_ID with the ID of the OSD.

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

   8. Login to the container associated with the OSD_ID.

      Syntax

      podman exec -it ceph-osd-OSD_ID /bin/bash

      Example

      [root@osd ~]# podman exec -it ceph-osd-0 /bin/bash

   9. Get osd fsid and activate the OSD to mount OSD’s logical volume (LV).

      Syntax

      ceph-volume lvm list |grep -A15 "osd\.OSD_ID"|grep "osd fsid"
      ceph-volume lvm activate --bluestore OSD_ID OSD_FSID

      Example

      [root@osd ~]# ceph-volume lvm list |grep -A15 "osd\.0"|grep "osd fsid"
                    osd fsid                  087eee15-6561-40a3-8fe4-9583ba64a4ff
      [root@osd ~]# ceph-volume lvm activate --bluestore 0 087eee15-6561-40a3-8fe4-9583ba64a4ff
      Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc --path /var/lib/ceph/osd/ceph-0 --no-mon-config
      Running command: /usr/bin/ln -snf /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--41c69f8f--30e2--4685--9c5c--c605898c5537-osd--data--d073e8b3--0b89--4271--af5b--83045fd000dc
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff
       stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff.service → /usr/lib/systemd/system/ceph-volume@.service.
      Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
       stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph osd@0.service → /usr/lib/systemd/system/ceph-osd@.service.
      Running command: /usr/bin/systemctl start ceph-osd@0
       stderr: Running in chroot, ignoring request: start
      --> ceph-volume lvm activate successful for osd ID: 0

3. List the object map:

   [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID \
   OBJECT \
   list-omap

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
   '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
   list-omap

4. For containerized deployments, to revert the changes, follow the below steps:

   1. After exiting the container, copy /etc/systemd/system/ceph-osd@.service unit file from /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.modified
      [root@osd ~]# cp /root/ceph-osd@.service.backup /etc/systemd/system/ceph-osd@.service

   2. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

   3. Move /run/ceph-osd@OSD_ID.service-cid file to /tmp.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /tmp

   4. Restart the OSD service associated with the OSD_ID.

      Syntax

      [root@osd ~]# systemctl restart ceph-osd@OSD_ID.service

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service