Language:
Format:

Red Hat Training

A Red Hat training course is available for Red Hat Ceph Storage

Chapter 8. Troubleshooting objects

As a storage administrator, you can use the ceph-objectstore-tool utility to perform high-level or low-level object operations. The ceph-objectstore-tool utility can help you troubleshoot problems related to objects within a particular OSD or placement group.

Important

Manipulating objects can cause unrecoverable data loss. Contact Red Hat support before using the ceph-objectstore-tool utility.

8.1. Prerequisites

Verify there are no network related issues.

8.2. Troubleshooting high-level object operations

As a storage administrator, you can use the ceph-objectstore-tool utility to perform high-level object operations. The ceph-objectstore-tool utility supports the following high-level object operations:

List objects
List lost objects
Fix lost objects

Important

Manipulating objects can cause unrecoverable data loss. Contact Red Hat support before using the ceph-objectstore-tool utility.

8.2.1. Prerequisites

Having root access to the Ceph OSD nodes.

8.2.2. Listing objects

The OSD can contain zero to many placement groups, and zero to many objects within a placement group (PG). The ceph-objectstore-tool utility allows you to list objects stored within an OSD.

Prerequisites

Having root access to the Ceph OSD node.
Stopping the ceph-osd daemon.

Procedure

Verify the appropriate OSD is down:

Syntax

systemctl status ceph-osd@$OSD_NUMBER

Example

[root@osd ~]# systemctl status ceph-osd@1

Identify all the objects within an OSD, regardless of their placement group:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD --op list

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op list

Identify all the objects within a placement group:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD --pgid $PG_ID --op list

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c --op list

Identify the PG an object belongs to:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD --op list $OBJECT_ID

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op list default.region

Additional Resources

For more information on stopping an OSD, see the Starting, Stopping, and Restarting a Ceph Daemons by Instance section in the Red Hat Ceph Storage Administration Guide.

8.2.3. Fixing lost objects

You can use the ceph-objectstore-tool utility to list and fix lost and unfound objects stored within a Ceph OSD. This procedure applies only to legacy objects.

Prerequisites

Having root access to the Ceph OSD node.
Stopping the ceph-osd daemon.

Procedure

Verify the appropriate OSD is down:

Syntax

systemctl status ceph-osd@$OSD_NUMBER

Example

[root@osd ~]# systemctl status ceph-osd@1

To list all the lost legacy objects:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD --op fix-lost --dry-run

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op fix-lost --dry-run

Use the ceph-objectstore-tool utility to fix lost and unfound objects. Select the appropriate circumstance:

To fix all lost objects:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD --op fix-lost

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op fix-lost

To fix all the lost objects within a placement group:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD --pgid $PG_ID --op fix-lost

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c --op fix-lost

To fix a lost object by its identifier:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD --op fix-lost $OBJECT_ID

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op fix-lost default.region

Additional Resources

For more information on stopping an OSD, see the Starting, Stopping, and Restarting a Ceph Daemons by Instance section in the Red Hat Ceph Storage Administration Guide.

8.3. Troubleshooting low-level object operations

As a storage administrator, you can use the ceph-objectstore-tool utility to perform low-level object operations. The ceph-objectstore-tool utility supports the following low-level object operations:

Manipulate the object’s content
Remove an object
List the object map (OMAP)
Manipulate the OMAP header
Manipulate the OMAP key
List the object’s attributes
Manipulate the object’s attribute key

Important

Manipulating objects can cause unrecoverable data loss. Contact Red Hat support before using the ceph-objectstore-tool utility.

8.3.1. Prerequisites

Having root access to the Ceph OSD nodes.

8.3.2. Manipulating the object’s content

With the ceph-objectstore-tool utility, you can get or set bytes on an object.

Important

Setting the bytes on an object can cause unrecoverable data loss. To prevent data loss, make a backup copy of the object.

Prerequisites

Having root access to the Ceph OSD node.
Stopping the ceph-osd daemon.

Procedure

Verify the appropriate OSD is down:

Syntax

systemctl status ceph-osd@$OSD_NUMBER

Example

[root@osd ~]# systemctl status ceph-osd@1

Find the object by listing the objects of the OSD or placement group (PG).

Before setting the bytes on an object, make a backup and a working copy of the object:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD --pgid $PG_ID \
$OBJECT \
get-bytes > $OBJECT_FILE_NAME

ceph-objectstore-tool --data-path $PATH_TO_OSD --pgid $PG_ID \
$OBJECT \
get-bytes > $OBJECT_FILE_NAME

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
'{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
get-bytes > zone_info.default.backup

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
'{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
get-bytes > zone_info.default.working-copy

Edit the working copy object file and modify the object contents accordingly.

Set the bytes of the object:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD --pgid $PG_ID \
$OBJECT \
set-bytes < $OBJECT_FILE_NAME

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
'{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
set-bytes < zone_info.default.working-copy

Additional Resources

For more information on stopping an OSD, see the Starting, Stopping, and Restarting a Ceph Daemons by Instance section in the Red Hat Ceph Storage Administration Guide.

8.3.3. Removing an object

Use the ceph-objectstore-tool utility to remove an object. By removing an object, its contents and references are removed from the placement group (PG).

Important

You cannot recreate an object once it is removed.

Prerequisites

Having root access to the Ceph OSD node.
Stopping the ceph-osd daemon.

Procedure

Remove an object:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD --pgid $PG_ID \
$OBJECT \
remove

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
'{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
remove

Additional Resources

For more information on stopping an OSD, see the Starting, Stopping, and Restarting a Ceph Daemons by Instance section in the Red Hat Ceph Storage Administration Guide.

8.3.4. Listing the object map

Use the ceph-objectstore-tool utility to list the contents of the object map (OMAP). The output provides you a list of keys.

Prerequisites

Having root access to the Ceph OSD node.
Stopping the ceph-osd daemon.

Procedure

Verify the appropriate OSD is down:

Syntax

systemctl status ceph-osd@$OSD_NUMBER

Example

[root@osd ~]# systemctl status ceph-osd@1

List the object map:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD --pgid $PG_ID \
$OBJECT \
list-omap

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
'{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
list-omap

Additional Resources

For more information on stopping an OSD, see the Starting, Stopping, and Restarting a Ceph Daemons by Instance section in the Red Hat Ceph Storage Administration Guide.

8.3.5. Manipulating the object map header

The ceph-objectstore-tool utility will output the object map (OMAP) header with the values associated with the object’s keys.

Note

If using FileStore as the OSD backend object store, then add the --journal-path $PATH_TO_JOURNAL argument when getting or setting the object map header. Where the $PATH_TO_JOURNAL variable is the absolute path to the OSD journal, for example /var/lib/ceph/osd/ceph-0/journal.

Prerequisites

Having root access to the Ceph OSD node.
Stopping the ceph-osd daemon.

Procedure

Verify the appropriate OSD is down:

Syntax

systemctl status ceph-osd@$OSD_NUMBER

Example

[root@osd ~]# systemctl status ceph-osd@1

Get the object map header:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD \
--pgid $PG_ID $OBJECT \
get-omaphdr > $OBJECT_MAP_FILE_NAME

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
get-omaphdr > zone_info.default.omaphdr.txt

Set the object map header:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD \
--pgid $PG_ID $OBJECT \
get-omaphdr < $OBJECT_MAP_FILE_NAME

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
set-omaphdr < zone_info.default.omaphdr.txt

Additional Resources

For more information on stopping an OSD, see the Starting, Stopping, and Restarting a Ceph Daemons by Instance section in the Red Hat Ceph Storage Administration Guide.

8.3.6. Manipulating the object map key

Use the ceph-objectstore-tool utility to change the object map (OMAP) key. You need to provide the data path, the placement group identifier (PG ID), the object, and the key in the OMAP.

Note

If using FileStore as the OSD backend object store, then add the --journal-path $PATH_TO_JOURNAL argument when getting, setting or removing the object map key. Where the $PATH_TO_JOURNAL variable is the absolute path to the OSD journal, for example /var/lib/ceph/osd/ceph-0/journal.

Prerequisites

Having root access to the Ceph OSD node.
Stopping the ceph-osd daemon.

Procedure

Get the object map key:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD \
--pgid $PG_ID $OBJECT \
get-omap $KEY > $OBJECT_MAP_FILE_NAME

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
get-omap "" > zone_info.default.omap.txt

Set the object map key:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD \
--pgid $PG_ID $OBJECT \
set-omap $KEY < $OBJECT_MAP_FILE_NAME

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
set-omap "" < zone_info.default.omap.txt

Remove the object map key:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD \
--pgid $PG_ID $OBJECT \
rm-omap $KEY

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
rm-omap ""

Additional Resources

For more information on stopping an OSD, see the Starting, Stopping, and Restarting a Ceph Daemons by Instance section in the Red Hat Ceph Storage Administration Guide.

8.3.7. Listing the object’s attributes

Use the ceph-objectstore-tool utility to list an object’s attributes. The output provides you with the object’s keys and values.

Note

If using FileStore as the OSD backend object store, then add the --journal-path $PATH_TO_JOURNAL argument when listing an object’s attributes. Where the $PATH_TO_JOURNAL variable is the absolute path to the OSD journal, for example /var/lib/ceph/osd/ceph-0/journal.

Prerequisites

Having root access to the Ceph OSD node.
Stopping the ceph-osd daemon.

Procedure

Verify the appropriate OSD is down:

Syntax

systemctl status ceph-osd@$OSD_NUMBER

Example

[root@osd ~]# systemctl status ceph-osd@1

List the object’s attributes:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD \
--pgid $PG_ID $OBJECT \
list-attrs

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
list-attrs

Additional Resources

For more information on stopping an OSD, see the Starting, Stopping, and Restarting a Ceph Daemons by Instance section in the Red Hat Ceph Storage Administration Guide.

8.3.8. Manipulating the object attribute key

Use the ceph-objectstore-tool utility to change an object’s attributes. To manipulate the object’s attributes you need the data and journal paths, the placement group identifier (PG ID), the object, and the key in the object’s attribute.

Note

If using FileStore as the OSD backend object store, then add the --journal-path $PATH_TO_JOURNAL argument when getting, setting or removing the object’s attributes. Where the $PATH_TO_JOURNAL variable is the absolute path to the OSD journal, for example /var/lib/ceph/osd/ceph-0/journal.

Prerequisites

Having root access to the Ceph OSD node.
Stopping the ceph-osd daemon.

Procedure

Verify the appropriate OSD is down:

Syntax

systemctl status ceph-osd@$OSD_NUMBER

Example

[root@osd ~]# systemctl status ceph-osd@1

Get the object’s attributes:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD \
--pgid $PG_ID $OBJECT \
get-attrs $KEY > $OBJECT_ATTRS_FILE_NAME

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
get-attrs "oid" > zone_info.default.attr.txt

Set an object’s attributes:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD \
--pgid $PG_ID $OBJECT \
set-attrs $KEY < $OBJECT_ATTRS_FILE_NAME

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
set-attrs "oid" < zone_info.default.attr.txt

Remove an object’s attributes:

Syntax

ceph-objectstore-tool --data-path $PATH_TO_OSD \
--pgid $PG_ID $OBJECT \
rm-attrs $KEY

Example

[root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
rm-attrs "oid"

Additional Resources

For more information on stopping an OSD, see the Starting, Stopping, and Restarting a Ceph Daemons by Instance section in the Red Hat Ceph Storage Administration Guide.

8.4. Additional Resources

For Red Hat Ceph Storage support, see the Red Hat Customer Portal.

Select Your Language

Red Hat Training

Chapter 8. Troubleshooting objects

8.1. Prerequisites

8.2. Troubleshooting high-level object operations

8.2.1. Prerequisites

8.2.2. Listing objects

8.2.3. Fixing lost objects

8.3. Troubleshooting low-level object operations

8.3.1. Prerequisites

8.3.2. Manipulating the object’s content

8.3.3. Removing an object

8.3.4. Listing the object map

8.3.5. Manipulating the object map header

8.3.6. Manipulating the object map key

8.3.7. Listing the object’s attributes

8.3.8. Manipulating the object attribute key

8.4. Additional Resources

Quick Links

Help

Site Info

Related Sites

About

Red Hat legal and privacy links

Red Hat legal and privacy links

Language and Page Formatting Options

Red Hat Training

Chapter 8. Troubleshooting objects

8.1. Prerequisites

8.2. Troubleshooting high-level object operations

8.2.1. Prerequisites

8.2.2. Listing objects

8.2.3. Fixing lost objects

8.3. Troubleshooting low-level object operations

8.3.1. Prerequisites

8.3.2. Manipulating the object’s content

8.3.3. Removing an object

8.3.4. Listing the object map

8.3.5. Manipulating the object map header

8.3.6. Manipulating the object map key

8.3.7. Listing the object’s attributes

8.3.8. Manipulating the object attribute key

8.4. Additional Resources

Quick Links

Help

Site Info

Related Sites

Systems Status

About

Red Hat legal and privacy links

Red Hat legal and privacy links