Ceph performance data collection script V20

Solution Verified - Updated -

Environment

  • Red Hat Ceph Storage (RHCS) 5
  • Red Hat Ceph Storage (RHCS) 6
  • Red Hat Ceph Storage (RHCS) 7
  • Red Hat Ceph Storage (RHCS) 8
  • Red Hat Ceph Storage (RHCS) 9
  • Openshift Data Foundations

Issue

  • MDS, OSD, MGR or RGW Sync performance data needs to be collected for standalone Ceph or ODF environments.

Resolution

ⓘ Please note that this is not to be treated as a Red Hat official binary / software, and feel free to go through the source code to look for what's happening behind the scenes.

With the script in this KCS (Please download from the bottom of the page) common ceph performance data can be collected in a periodic fashion. These outputs can be very big (multiple gigabytes). This script will periodically compress the collected data. This helps to keep the local space usage down. By default the script deletes data older than 240 minutes and collects performance data every 5 seconds. The script also provides a progress bar (stuck commands will be seen easily) and print's the local space usage for convenience during the collection run.

Execution

  1. ODF Environment

    ⓘ When using a TOKEN to login (copying the cli login with the token from the console for example) might expire, once the token expires, the script keeps running but will be unable to execute ceph commands. Please make sure to use a long lived authentication method, for example a properly exported KUBECONFIG Environment variable pointing to a config file.

    Run the script with the --odf parameter. You need to be on a (bastion) node where oc is in the path and you can use oc to execute commands on the OCP cluster. Keep in mind that RGW perf dumps and RGW sync performance are not usefull in ODF environments.

  2. Standalone Ceph Environment
    Run the script from a node where you can execute ceph commands like ceph status.

Usage

Use the --help parameter to see what kind of data to collect.

Most important parameters:

Parameter Description
--collect=health Standard Health related data (ceph status, ceph health detail).
--collect=mds Standard MDS performance data.
--collect=mds-advanced Data like subtrees, mempools and objecter_requests which are need only on special occations.
--collect=mgr MGR related performance data.
--collect=osd OSD related performance data.
--collect=rgw RGW related perf dumps.
--collect=rgw-sync-performance RGW related performance data. Realm, Zonegroup and Zone can be specified if needed (see --help).
--collect=rgw-sync-stats RGW related performance data. Realm, Zonegroup and Zone can be specified if needed (see --help). Stats will be collected at the top of each hour as it is not needed with further granularity.
--delay=X The delay between collections (set to 5 seconds by default).
--keep-logs-for=X Keep only X minutes of data, by default 240 minutes of date (last 4 hours) are kept.
--filter_osds=X Restrict OSD data collection to a space separated list of OSDs. Defaults to ALL OSDs.
--no=X Exclude specific data sets (see --help for more info)
--replace=X Sanitize output, you can use this parameter as many times as you need, it works like sed, for example, use --replace="/192\.168\./10.1./" to replace all occurences of 192.168. with 10.1. See --help as well

 

RGW performance collection considerations

ⓘ RGW Perf collections are not usefull for ODF environments.

For RGW sync performance testing run the script on every server which contains RGW instances which have the sync thread enabled. On one of those servers (in both, primary and secondary clusters) run both --collect=rgw-sync-performance and --collect=rgw-sync-stats while on the other servers with RGW sync instances you only need to run --collect=rgw-sync-performance. The performance sync is data per sync instance, while the sync-stats are per cluster, so we only need that to run once per cluster participating in the multisite sync operation. To check the sync performance, you also have to configure the zone you want the performance to be gathered from, not configuring the zone will print an error message. Realm, Zonegroup and Zone can be specified, but will fall back to default. If you are only interested in the data collected by --collect=rgw-sync-stats then raise the --compress-interval value. Those bucket stats will only be collected on the op of every hour, making compression every actual 100 collection-runs unneccessary. If we collect for a couple of days setting a compression interval of one day (--compress-interval=86400 ) would be sufficient. When collecting both (rgw-sync-performance and rgw-sync-stats) or if you collect other perf data as well (e.g. You add --collect=health or any of the other --collect=XXX parameters) then leave the compression interval on it's default as data will accumulate rapidly.

Stopping, Restarting the Collection

The script can be interrupted via CTRL+C (It can also be restarted, with the same or changed parameters, by default it will again add to what has been collected already). This is useful if after starting collection you discover that you need some changes in how or what to collect. When the script is stopped, it creates a last compression batch of the current data.

Analysing the Data.

Because we use compression to minimize the impact of the collected data to analyze you have to uncompress it. An uncompress script is utomatically created inside the collection directory. cd inside the collection directory and run the uncompress script which will put all the data in it's correct folder structure. If you do not uncompress the data, you only will see the empty folder structure instead.

Providing the performance data to Red Hat Support

When finished with the collection tar up the created ceph_performance_collection directory (or the directory you have configured to be your collection directory) and upload that file to the support case.

Examples

Run in an OpenShift inernal ODF deployment and collect MDS performance data:

# ./ceph-performance-data-collection.sh --odf --collect=mds

 *********************************************************************************************
  Please note that this is not to be treated as a Red Hat official binary / software,
  and feel free to go through the source code to look for what's happening behind the scenes.
 *********************************************************************************************

./ceph-performance-data-collection.sh - Version: 9



Performance data which will be collected with current parameters:
    ceph tell ${MDS} ops
    ceph tell ${MDS} session ls
    ceph fs dump
    ceph tell ${MDS} perf dump
    ceph tell ${MDS} dump_blocked_ops
    ceph health detail
    ceph tell ${MDS} dump_historic_ops_by_duration
    ceph status

Current batch size: 16K Free space: 142G (Uncompressed size: 1.1K)
Run: 6 Timestamp: 2025-04-01T17:40:36+02:00 Step: ceph_mds_dump_blocked_ops

Run on a standalone ceph installation and collect RGW performance data with a concrete log directory:

# ./ceph-performance-data-collection.sh --collect=rgw-sync-stats --collect=rgw-sync-performance --delay=15 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west


 *********************************************************************************************
  Please note that this is not to be treated as a Red Hat official binary / software,
  and feel free to go through the source code to look for what's happening behind the scenes.
 *********************************************************************************************

./ceph-performance-data-collection.sh - Version: 14

For details please see the KCS https://access.redhat.com/solutions/7114164


Performance data collected with current parameters:
    RGW Sync perf dumps from zone 'us-west'
    RGW Sync bucket stats (hourly)
    RGW Sync error list (hourly)


Current batch size: 1.8M Free space: 41G (Uncompressed size: 7.4M)
Run: 224 Timestamp: 2025-10-07T06:08:34-04:00 Step: Delaying for 1s, press [CTRL+C]

Uncompress the data

log-collection-perf-dump-server-1 is the directory we configured for the collection:

[log-collection-perf-dump-server-1]# ./extract_perf_data.sh
./
./rgw_perf_data/
./rgw_perf_data/perf_dump/
./rgw_perf_data/perf_dump/rgw_lgqbso_perf_dump_2025-10-06T16:06:49-04:00.json
./rgw_perf_data/perf_dump/rgw_lgqbso_queues.tsv
./rgw_perf_data/perf_dump/rgw_tytwny_perf_dump_2025-10-06T16:06:49-04:00.json
./rgw_perf_data/perf_dump/rgw_tytwny_queues.tsv
./rgw_perf_data/perf_dump/rgw_lgqbso_perf_dump_2025-10-06T16:06:53-04:00.json
./rgw_perf_data/perf_dump/rgw_tytwny_perf_dump_2025-10-06T16:06:53-04:00.json
./rgw_perf_data/perf_dump/rgw_lgqbso_perf_dump_2025-10-06T16:06:58-04:00.json
./rgw_perf_data/perf_dump/rgw_tytwny_perf_dump_2025-10-06T16:06:58-04:00.json
./rgw_perf_data/perf_dump/rgw_lgqbso_perf_dump_2025-10-06T16:07:02-04:00.json
./rgw_perf_data/perf_dump/rgw_tytwny_perf_dump_2025-10-06T16:07:02-04:00.json
[...]

Review when the script has started and what parameters have been used

[log-collection-perf-dump]# cat script_parameters.txt
2025-10-06T16:06:49-04:00 --collect=rgw-sync-performance --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west
2025-10-06T16:07:40-04:00 --collect=rgw-sync-performance --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --replace=/tytwny/INSTANCE1/
2025-10-06T16:13:12-04:00 --collect=rgw-sync-performance --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T02:54:32-04:00 --collect=rgw-sync-performance --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T02:56:01-04:00 --collect=rgw-sync-performance --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T02:56:33-04:00 --collect=rgw-sync-performance --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T03:07:52-04:00 --collect=rgw-sync-performance --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T03:09:07-04:00 --collect=rgw-sync-performance --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T03:10:36-04:00 --collect=rgw-sync-performance --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T03:11:50-04:00 --collect=rgw-sync-performance --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T03:15:59-04:00 --collect=rgw-sync-performance --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T03:18:10-04:00 --collect=rgw-sync-performance --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T03:19:38-04:00 --collect=rgw-sync-performance --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T03:21:38-04:00 --collect=rgw-sync-performance --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T03:22:59-04:00 --collect=rgw-sync-performance --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T03:27:40-04:00 --collect=rgw-sync-performance --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T03:40:09-04:00 --collect=rgw-sync-performance --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T03:40:48-04:00 --collect=rgw-sync-performance --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T03:42:32-04:00 --collect=rgw-sync-performance --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T03:42:54-04:00 --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --collect=health
2025-10-07T03:43:22-04:00 --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west
2025-10-07T03:45:50-04:00 --collect=rgw-sync-stats --delay=1 --keep-logs-for=-1 --logpath=/tmp/log-collection-perf-dump --rgw_data_sync_from=us-west --compress-interval=86400

Root Cause

Performance related data is needed to investigate issues. This script provides an easy way to collect various performance related data.

Attachments

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments