-
Language:
English
-
Language:
English
Red Hat Training
A Red Hat training course is available for Red Hat Ceph Storage
Chapter 17. Access
The performance counters are available through a socket interface for the Ceph Monitors and the OSDs. The socket file for each respective daemon is located under /var/run/ceph
, by default. The performance counters are grouped together into collection names. These collections names represent a subsystem or an instance of a subsystem. For example, the throttle
mechanism reports statistics on how it is throttling.
Example output of the throttle
collection:
throttle-filestore_bytes .. throttle-filestore_ops .. throttle-msgr_dispatch_throttler-client .. throttle-msgr_dispatch_throttler-cluster .. throttle-msgr_dispatch_throttler-hb_back_server .. throttle-msgr_dispatch_throttler-hb_front_server .. throttle-msgr_dispatch_throttler-hbclient .. throttle-msgr_dispatch_throttler-ms_objecter .. throttle-objecter_bytes .. throttle-objecter_ops .. throttle-osd_client_bytes .. throttle-osd_client_messages ..
17.1. Schema
The ceph daemon .. perf schema
command outputs the available metrics. Each metric has an associated bit field value type.
To view the metric’s schema:
ceph daemon {daemon-name} perf schema
You must run the ceph daemon
command from the node running the daemon.
Executing ceph daemon .. perf schema
command from the Monitor node:
sudo ceph daemon mon.`hostname -s` perf schema
Example output:
{ "cluster": { "num_mon": { "type": 2 }, "num_mon_quorum": { "type": 2 }, "num_osd": { "type": 2 }, "num_osd_up": { "type": 2 }, "num_osd_in": { "type": 2 }, ...
Executing the ceph daemon .. perf schema
command from the OSD node:
sudo ceph daemon osd.0 perf schema
Example output:
... "filestore": { "journal_queue_max_ops": { "type": 2 }, "journal_queue_ops": { "type": 2 }, "journal_ops": { "type": 10 }, "journal_queue_max_bytes": { "type": 2 }, "journal_queue_bytes": { "type": 2 }, "journal_bytes": { "type": 10 }, "journal_latency": { "type": 5 }, ...
Table 17.1. The bit field value definitions
Bit | Meaning |
---|---|
1 | Floating point value |
2 | Unsigned 64-bit integer value |
4 | Average (Sum + Count) |
8 | Counter |
Each value will have bit 1 or 2 set to indicate the type, either a floating point or an integer value. When bit 4 (average) is set, there will be two values to read, a sum and a count. Please see the Average Count and Sum section below for more details. When bit 8 (counter) is set, the average for the previous interval would be the sum delta (since the previous read) divided by the count delta. Alternatively, dividing the values outright would provide the lifetime average value. Typically these are used to measure latencies, the number of requests and a sum of request latencies. Some bit values are combined, for example 5, 6 and 10. A bit value of 5 is a combination of bit 1 and bit 4. This means the average will be a floating point value. A bit value of 6 is a combination of bit 2 and bit 4. This means the average value will be an integer. A bit value of 10 is a combination of bit 2 and bit 8. This means the counter value will be an integer value.
17.2. Dump
The ceph daemon .. perf dump
command outputs the current values and groups the metrics under the collection name for each subsystem.
To view the current metric data:
ceph daemon {daemon-name} perf dump
You must run the ceph daemon
command from the node running the daemon.
Executing ceph daemon .. perf dump
command from the Monitor node:
sudo ceph daemon mon.`hostname -s` perf dump
Example output:
{ "cluster": { "num_mon": 1, "num_mon_quorum": 1, "num_osd": 2, "num_osd_up": 2, "num_osd_in": 2, ...
To view a short description of each Monitor metric available, please see the table below.
Executing the ceph daemon .. perf dump
command from the OSD node:
sudo ceph daemon osd.0 perf dump
Example output:
... "filestore": { "journal_queue_max_ops": 300, "journal_queue_ops": 0, "journal_ops": 992, "journal_queue_max_bytes": 33554432, "journal_queue_bytes": 0, "journal_bytes": 934537, "journal_latency": { "avgcount": 992, "sum": 254.975925772 }, ...
17.2.1. Average Count and Sum
All latency numbers have a bit field value of 5. This field contains floating point values for the average count and sum. The avgcount
is the number of operations within this range and the sum
is the total latency in seconds. When dividing the sum
by the avgcount
this will provide you with an idea of the latency per operation.
To view a short description of each OSD metric available, please see the OSD table below.
17.2.2. Monitor Metrics Description Tables
Table 17.2. Cluster Metrics Table
Collection Name | Metric Name | Bit Field Value | Short Description |
---|---|---|---|
|
| 2 | Number of monitors |
| 2 | Number of monitors in quorum | |
| 2 | Total number of OSD | |
| 2 | Number of OSDs that are up | |
| 2 | Number of OSDs that are in cluster | |
| 2 | Current epoch of OSD map | |
| 2 | Total capacity of cluster in bytes | |
| 2 | Number of used bytes on cluster | |
| 2 | Number of available bytes on cluster | |
| 2 | Number of pools | |
| 2 | Total number of placement groups | |
| 2 | Number of placement groups in active+clean state | |
| 2 | Number of placement groups in active state | |
| 2 | Number of placement groups in peering state | |
| 2 | Total number of objects on cluster | |
| 2 | Number of degraded (missing replicas) objects | |
| 2 | Number of misplaced (wrong location in the cluster) objects | |
| 2 | Number of unfound objects | |
| 2 | Total number of bytes of all objects | |
| 2 | Number of MDSs that are up | |
| 2 | Number of MDS that are in cluster | |
| 2 | Number of failed MDS | |
| 2 | Current epoch of MDS map |
Table 17.3. Level Database Metrics Table
Collection Name | Metric Name | Bit Field Value | Short Description |
---|---|---|---|
|
| 10 | Gets |
| 10 | Transactions | |
| 10 | Compactions | |
| 10 | Compactions by range | |
| 10 | Mergings of ranges in compaction queue | |
| 2 | Length of compaction queue |
Table 17.4. General Monitor Metrics Table
Collection Name | Metric Name | Bit Field Value | Short Description |
---|---|---|---|
|
| 2 | Current number of opened monitor sessions |
| 10 | Number of created monitor sessions | |
| 10 | Number of remove_session calls in monitor | |
| 10 | Number of trimed monitor sessions | |
| 10 | Number of elections monitor took part in | |
| 10 | Number of elections started by monitor | |
| 10 | Number of elections won by monitor | |
| 10 | Number of elections lost by monitor |
Table 17.5. Paxos Metrics Table
Collection Name | Metric Name | Bit Field Value | Short Description |
---|---|---|---|
|
| 10 | Starts in leader role |
| 10 | Starts in peon role | |
| 10 | Restarts | |
| 10 | Refreshes | |
| 5 | Refresh latency | |
| 10 | Started and handled begins | |
| 6 | Keys in transaction on begin | |
| 6 | Data in transaction on begin | |
| 5 | Latency of begin operation | |
| 10 | Commits | |
| 6 | Keys in transaction on commit | |
| 6 | Data in transaction on commit | |
| 5 | Commit latency | |
| 10 | Peon collects | |
| 6 | Keys in transaction on peon collect | |
| 6 | Data in transaction on peon collect | |
| 5 | Peon collect latency | |
| 10 | Uncommitted values in started and handled collects | |
| 10 | Collect timeouts | |
| 10 | Accept timeouts | |
| 10 | Lease acknowledgement timeouts | |
| 10 | Lease timeouts | |
| 10 | Store a shared state on disk | |
| 6 | Keys in transaction in stored state | |
| 6 | Data in transaction in stored state | |
| 5 | Storing state latency | |
| 10 | Sharings of state | |
| 6 | Keys in shared state | |
| 6 | Data in shared state | |
| 10 | New proposal number queries | |
| 5 | New proposal number getting latency |
Table 17.6. Throttle Metrics Table
Collection Name | Metric Name | Bit Field Value | Short Description |
---|---|---|---|
|
| 10 | Currently available throttle |
| 10 | Max value for throttle | |
| 10 | Gets | |
| 10 | Got data | |
| 10 | Get blocked during get_or_fail | |
| 10 | Successful get during get_or_fail | |
| 10 | Takes | |
| 10 | Taken data | |
| 10 | Puts | |
| 10 | Put data | |
| 5 | Waiting latency |
17.2.3. OSD Metrics Description Tables
Table 17.7. Write Back Throttle Table
Collection Name | Metric Name | Bit Field Value | Short Description |
---|---|---|---|
|
| 2 | Dirty data |
| 2 | Written data | |
| 2 | Dirty operations | |
| 2 | Written operations | |
| 2 | Entries waiting for write | |
| 2 | Written entries |
Table 17.8. File Store Table
Collection Name | Metric Name | Bit Field Value | Short Description |
---|---|---|---|
|
| 2 | Max operations in journal queue |
| 2 | Operations in journal queue | |
| 10 | Total journal entries written | |
| 2 | Max data in journal queue | |
| 2 | Size of journal queue | |
| 10 | Total operations size in journal | |
| 5 | Average journal queue completing latency | |
| 10 | Journal write IOs | |
| 6 | Journal data written | |
| 10 | Journal writes while full | |
| 2 | Is currently committing | |
| 10 | Commit cycles | |
| 5 | Average interval between commits | |
| 5 | Average latency of commit | |
| 2 | Max operations count in queue | |
| 2 | Operations count in queue | |
| 10 | Operations | |
| 2 | Max size of queue | |
| 2 | Size of queue | |
| 10 | Data written to store | |
| 5 | Apply latency | |
| 5 | Store operation queue latency |
Table 17.9. Level Database Table
Collection Name | Metric Name | Bit Field Value | Short Description |
---|---|---|---|
|
| 10 | Gets |
| 10 | Transactions | |
| 10 | Compactions | |
| 10 | Compactions by range | |
| 10 | Mergings of ranges in compaction queue | |
| 2 | Length of compaction queue |
Table 17.10. Objecter Table
Collection Name | Metric Name | Bit Field Value | Short Description |
---|---|---|---|
|
| 2 | Active operations |
| 2 | Laggy operations | |
| 10 | Sent operations | |
| 10 | Sent data | |
| 10 | Resent operations | |
| 10 | Commit callbacks | |
| 10 | Operation commits | |
| 10 | Operation | |
| 10 | Read operations | |
| 10 | Write operations | |
| 10 | Read-modify-write operations | |
| 10 | PG operation | |
| 10 | Stat operations | |
| 10 | Create object operations | |
| 10 | Read operations | |
| 10 | Write operations | |
| 10 | Write full object operations | |
| 10 | Append operation | |
| 10 | Set object to zero operations | |
| 10 | Truncate object operations | |
| 10 | Delete object operations | |
| 10 | Map extent operations | |
| 10 | Sparse read operations | |
| 10 | Clone range operations | |
| 10 | Get xattr operations | |
| 10 | Set xattr operations | |
| 10 | Xattr comparison operations | |
| 10 | Remove xattr operations | |
| 10 | Reset xattr operations | |
| 10 | TMAP update operations | |
| 10 | TMAP put operations | |
| 10 | TMAP get operations | |
| 10 | Call (execute) operations | |
| 10 | Watch by object operations | |
| 10 | Notify about object operations | |
| 10 | Extended attribute comparison in multi operations | |
| 10 | Other operations | |
| 2 | Active lingering operations | |
| 10 | Sent lingering operations | |
| 10 | Resent lingering operations | |
| 10 | Sent pings to lingering operations | |
| 2 | Active pool operations | |
| 10 | Sent pool operations | |
| 10 | Resent pool operations | |
| 2 | Active get pool stat operations | |
| 10 | Pool stat operations sent | |
| 10 | Resent pool stats | |
| 2 | Statfs operations | |
| 10 | Sent FS stats | |
| 10 | Resent FS stats | |
| 2 | Active commands | |
| 10 | Sent commands | |
| 10 | Resent commands | |
| 2 | OSD map epoch | |
| 10 | Full OSD maps received | |
| 10 | Incremental OSD maps received | |
| 2 | Open sessions | |
| 10 | Sessions opened | |
| 10 | Sessions closed | |
| 2 | Laggy OSD sessions |
Table 17.11. Read and Write Operations Table
Collection Name | Metric Name | Bit Field Value | Short Description |
---|---|---|---|
|
| 2 | Replication operations currently being processed (primary) |
| 10 | Client operations total write size | |
| 10 | Client operations total read size | |
| 5 | Latency of client operations (including queue time) | |
| 5 | Latency of client operations (excluding queue time) | |
| 10 | Client read operations | |
| 10 | Client data read | |
| 5 | Latency of read operation (including queue time) | |
| 5 | Latency of read operation (excluding queue time) | |
| 10 | Client write operations | |
| 10 | Client data written | |
| 5 | Client write operation readable/applied latency | |
| 5 | Latency of write operation (including queue time) | |
| 5 | Latency of write operation (excluding queue time) | |
| 10 | Client read-modify-write operations | |
| 10 | Client read-modify-write operations write in | |
| 10 | Client read-modify-write operations read out | |
| 5 | Client read-modify-write operation readable/applied latency | |
| 5 | Latency of read-modify-write operation (including queue time) | |
| 5 | Latency of read-modify-write operation (excluding queue time) | |
| 10 | Suboperations | |
| 10 | Suboperations total size | |
| 5 | Suboperations latency | |
| 10 | Replicated writes | |
| 10 | Replicated written data size | |
| 5 | Replicated writes latency | |
| 10 | Suboperations pull requests | |
| 5 | Suboperations pull latency | |
| 10 | Suboperations push messages | |
| 10 | Suboperations pushed size | |
| 5 | Suboperations push latency | |
| 10 | Pull requests sent | |
| 10 | Push messages sent | |
| 10 | Pushed size | |
| 10 | Inbound push messages | |
| 10 | Inbound pushed size | |
| 10 | Started recovery operations | |
| 2 | CPU load | |
| 2 | Total allocated buffer size | |
| 2 | Placement groups | |
| 2 | Placement groups for which this osd is primary | |
| 2 | Placement groups for which this osd is replica | |
| 2 | Placement groups ready to be deleted from this osd | |
| 2 | Heartbeat (ping) peers we send to | |
| 2 | Heartbeat (ping) peers we recv from | |
| 10 | OSD map messages | |
| 10 | OSD map epochs | |
| 10 | OSD map duplicates | |
| 2 | OSD size | |
| 2 | Used space | |
| 2 | Available space | |
| 10 | Rados copy-from operations | |
| 10 | Tier promotions | |
| 10 | Tier flushes | |
| 10 | Failed tier flushes | |
| 10 | Tier flush attempts | |
| 10 | Failed tier flush attempts | |
| 10 | Tier evictions | |
| 10 | Tier whiteouts | |
| 10 | Dirty tier flag set | |
| 10 | Dirty tier flag cleaned | |
| 10 | Tier delays (agent waiting) | |
| 10 | Tier proxy reads | |
| 10 | Tiering agent wake up | |
| 10 | Objects skipped by agent | |
| 10 | Tiering agent flushes | |
| 10 | Tiering agent evictions | |
| 10 | Object context cache hits | |
| 10 | Object context cache lookups |
Table 17.12. Recovery State Table
Collection Name | Metric Name | Bit Field Value | Short Description |
---|---|---|---|
|
| 5 | Initial recovery state latency |
| 5 | Started recovery state latency | |
| 5 | Reset recovery state latency | |
| 5 | Start recovery state latency | |
| 5 | Primary recovery state latency | |
| 5 | Peering recovery state latency | |
| 5 | Backfilling recovery state latency | |
| 5 | Wait remote backfill reserved recovery state latency | |
| 5 | Wait local backfill reserved recovery state latency | |
| 5 | Notbackfilling recovery state latency | |
| 5 | Repnotrecovering recovery state latency | |
| 5 | Rep wait recovery reserved recovery state latency | |
| 5 | Rep wait backfill reserved recovery state latency | |
| 5 | RepRecovering recovery state latency | |
| 5 | Activating recovery state latency | |
| 5 | Wait local recovery reserved recovery state latency | |
| 5 | Wait remote recovery reserved recovery state latency | |
| 5 | Recovering recovery state latency | |
| 5 | Recovered recovery state latency | |
| 5 | Clean recovery state latency | |
| 5 | Active recovery state latency | |
| 5 | Replicaactive recovery state latency | |
| 5 | Stray recovery state latency | |
| 5 | Getinfo recovery state latency | |
| 5 | Getlog recovery state latency | |
| 5 | Waitactingchange recovery state latency | |
| 5 | Incomplete recovery state latency | |
| 5 | Getmissing recovery state latency | |
| 5 | Waitupthru recovery state latency |
Table 17.13. OSD Throttle Table
Collection Name | Metric Name | Bit Field Value | Short Description |
---|---|---|---|
|
| 10 | Currently available throttle |
| 10 | Max value for throttle | |
| 10 | Gets | |
| 10 | Got data | |
| 10 | Get blocked during get_or_fail | |
| 10 | Successful get during get_or_fail | |
| 10 | Takes | |
| 10 | Taken data | |
| 10 | Puts | |
| 10 | Put data | |
| 5 | Waiting latency |