Appendix G. Object Storage Daemon (OSD) configuration options

The following are Ceph Object Storage Daemon (OSD) configuration options that can be set during deployment.

You can set these configuration options with the ceph config set osd CONFIGURATION_OPTION VALUE command.

osd_uuid
Description
The universally unique identifier (UUID) for the Ceph OSD.
Type
UUID
Default
The UUID.
Note
The osd uuid applies to a single Ceph OSD. The fsid applies to the entire cluster.
osd_data
Description

The path to the OSD’s data. You must create the directory when deploying Ceph. Mount a drive for OSD data at this mount point. 

IMPORTANT: Red Hat does not recommend changing the default.
Type
String
Default
/var/lib/ceph/osd/$cluster-$id
osd_max_write_size
Description
The maximum size of a write in megabytes.
Type
32-bit Integer
Default
90
osd_client_message_size_cap
Description
The largest client data message allowed in memory.
Type
64-bit Integer Unsigned
Default
500MB default. 500*1024L*1024L
osd_class_dir
Description
The class path for RADOS class plug-ins.
Type
String
Default
$libdir/rados-classes
osd_max_scrubs
Description
The maximum number of simultaneous scrub operations for a Ceph OSD.
Type
32-bit Int
Default
1
osd_scrub_thread_timeout
Description
The maximum time in seconds before timing out a scrub thread.
Type
32-bit Integer
Default
60
osd_scrub_finalize_thread_timeout
Description
The maximum time in seconds before timing out a scrub finalize thread.
Type
32-bit Integer
Default
60*10
osd_scrub_begin_hour
Description
This restricts scrubbing to this hour of the day or later. Use osd_scrub_begin_hour = 0 and osd_scrub_end_hour = 0 to allow scrubbing the entire day. Along with osd_scrub_end_hour, they define a time window, in which the scrubs can happen. But a scrub is performed no matter whether the time window allows or not, as long as the placement group’s scrub interval exceeds osd_scrub_max_interval.
Type
Integer
Default
0
Allowed range
[0,23]
osd_scrub_end_hour
Description
This restricts scrubbing to the hour earlier than this. Use osd_scrub_begin_hour = 0 and osd_scrub_end_hour = 0 to allow scrubbing for the entire day. Along with osd_scrub_begin_hour, they define a time window, in which the scrubs can happen. But a scrub is performed no matter whether the time window allows or not, as long as the placement group’s scrub interval exceeds osd_scrub_max_interval.
Type
Integer
Default
0
Allowed range
[0,23]
osd_scrub_load_threshold
Description
The maximum load. Ceph will not scrub when the system load (as defined by the getloadavg() function) is higher than this number. Default is 0.5.
Type
Float
Default
0.5
osd_scrub_min_interval
Description
The minimum interval in seconds for scrubbing the Ceph OSD when the Red Hat Ceph Storage cluster load is low.
Type
Float
Default
Once per day. 60*60*24
osd_scrub_max_interval
Description
The maximum interval in seconds for scrubbing the Ceph OSD irrespective of cluster load.
Type
Float
Default
Once per week. 7*60*60*24
osd_scrub_interval_randomize_ratio
Description
Takes the ratio and randomizes the scheduled scrub between osd scrub min interval and osd scrub max interval.
Type
Float
Default
0.5.
mon_warn_not_scrubbed
Description
Number of seconds after osd_scrub_interval to warn about any PGs that were not scrubbed.
Type
Integer
Default
0 (no warning).
osd_scrub_chunk_min
Description
The object store is partitioned into chunks which end on hash boundaries. For chunky scrubs, Ceph scrubs objects one chunk at a time with writes blocked for that chunk. The osd scrub chunk min setting represents the minimum number of chunks to scrub.
Type
32-bit Integer
Default
5
osd_scrub_chunk_max
Description
The maximum number of chunks to scrub.
Type
32-bit Integer
Default
25
osd_scrub_sleep
Description
The time to sleep between deep scrub operations.
Type
Float
Default
0 (or off).
osd_scrub_during_recovery
Description
Allows scrubbing during recovery.
Type
Bool
Default
false
osd_scrub_invalid_stats
Description
Forces extra scrub to fix stats marked as invalid.
Type
Bool
Default
true
osd_scrub_priority
Description
Controls queue priority of scrub operations versus client I/O.
Type
Unsigned 32-bit Integer
Default
5
osd_requested_scrub_priority
Description
The priority set for user requested scrub on the work queue. If this value were to be smaller than osd_client_op_priority, it can be boosted to the value of osd_client_op_priority when scrub is blocking client operations.
Type
Unsigned 32-bit Integer
Default
120
osd_scrub_cost
Description
Cost of scrub operations in megabytes for queue scheduling purposes.
Type
Unsigned 32-bit Integer
Default
52428800
osd_deep_scrub_interval
Description
The interval for deep scrubbing, that is fully reading all data. The osd scrub load threshold parameter does not affect this setting.
Type
Float
Default
Once per week. 60*60*24*7
osd_deep_scrub_stride
Description
Read size when doing a deep scrub.
Type
32-bit Integer
Default
512 KB. 524288
mon_warn_not_deep_scrubbed
Description
Number of seconds after osd_deep_scrub_interval to warn about any PGs that were not scrubbed.
Type
Integer
Default
0 (no warning)
osd_deep_scrub_randomize_ratio
Description
The rate at which scrubs will randomly become deep scrubs (even before osd_deep_scrub_interval has passed).
Type
Float
Default
0.15 or 15%
osd_deep_scrub_update_digest_min_age
Description
How many seconds old objects must be before scrub updates the whole-object digest.
Type
Integer
Default
7200 (120 hours)
osd_deep_scrub_large_omap_object_key_threshold
Description
Warning when you encounter an object with more OMAP keys than this.
Type
Integer
Default
200000
osd_deep_scrub_large_omap_object_value_sum_threshold
Description
Warning when you encounter an object with more OMAP key bytes than this.
Type
Integer
Default
1 G
osd_op_num_shards
Description
The number of shards for client operations.
Type
32-bit Integer
Default
0
osd_op_num_threads_per_shard
Description
The number of threads per shard for client operations.
Type
32-bit Integer
Default
0
osd_op_num_shards_hdd
Description
The number of shards for HDD operations.
Type
32-bit Integer
Default
5
osd_op_num_threads_per_shard_hdd
Description
The number of threads per shard for HDD operations.
Type
32-bit Integer
Default
1
osd_op_num_shards_ssd
Description
The number of shards for SSD operations.
Type
32-bit Integer
Default
8
osd_op_num_threads_per_shard_ssd
Description
The number of threads per shard for SSD operations.
Type
32-bit Integer
Default
2
osd_client_op_priority
Description
The priority set for client operations. It is relative to osd recovery op priority.
Type
32-bit Integer
Default
63
Valid Range
1-63
osd_recovery_op_priority
Description
The priority set for recovery operations. It is relative to osd client op priority.
Type
32-bit Integer
Default
3
Valid Range
1-63
osd_op_thread_timeout
Description
The Ceph OSD operation thread timeout in seconds.
Type
32-bit Integer
Default
15
osd_op_complaint_time
Description
An operation becomes complaint worthy after the specified number of seconds have elapsed.
Type
Float
Default
30
osd_disk_threads
Description
The number of disk threads, which are used to perform background disk intensive OSD operations such as scrubbing and snap trimming.
Type
32-bit Integer
Default
1
osd_op_history_size
Description
The maximum number of completed operations to track.
Type
32-bit Unsigned Integer
Default
20
osd_op_history_duration
Description
The oldest completed operation to track.
Type
32-bit Unsigned Integer
Default
600
osd_op_log_threshold
Description
How many operations logs to display at once.
Type
32-bit Integer
Default
5
osd_op_timeout
Description
The time in seconds after which running OSD operations time out.
Type
Integer
Default
0
Important

Do not set the osd op timeout option unless your clients can handle the consequences. For example, setting this parameter on clients running in virtual machines can lead to data corruption because the virtual machines interpret this timeout as a hardware failure.

osd_max_backfills
Description
The maximum number of backfill operations allowed to or from a single OSD.
Type
64-bit Unsigned Integer
Default
1
osd_backfill_scan_min
Description
The minimum number of objects per backfill scan.
Type
32-bit Integer
Default
64
osd_backfill_scan_max
Description
The maximum number of objects per backfill scan.
Type
32-bit Integer
Default
512
osd_backfill_full_ratio
Description
Refuse to accept backfill requests when the Ceph OSD’s full ratio is above this value.
Type
Float
Default
0.85
osd_backfill_retry_interval
Description
The number of seconds to wait before retrying backfill requests.
Type
Double
Default
30.000000
osd_map_dedup
Description
Enable removing duplicates in the OSD map.
Type
Boolean
Default
true
osd_map_cache_size
Description
The size of the OSD map cache in megabytes.
Type
32-bit Integer
Default
50
osd_map_cache_bl_size
Description
The size of the in-memory OSD map cache in OSD daemons.
Type
32-bit Integer
Default
50
osd_map_cache_bl_inc_size
Description
The size of the in-memory OSD map cache incrementals in OSD daemons.
Type
32-bit Integer
Default
100
osd_map_message_max
Description
The maximum map entries allowed per MOSDMap message.
Type
32-bit Integer
Default
40
osd_snap_trim_thread_timeout
Description
The maximum time in seconds before timing out a snap trim thread.
Type
32-bit Integer
Default
60*60*1
osd_pg_max_concurrent_snap_trims
Description
The max number of parallel snap trims/PG. This controls how many objects per PG to trim at once.
Type
32-bit Integer
Default
2
osd_snap_trim_sleep
Description
Insert a sleep between every trim operation a PG issues.
Type
32-bit Integer
Default
0
osd_max_trimming_pgs
Description
The max number of trimming PGs
Type
32-bit Integer
Default
2
osd_backlog_thread_timeout
Description
The maximum time in seconds before timing out a backlog thread.
Type
32-bit Integer
Default
60*60*1
osd_default_notify_timeout
Description
The OSD default notification timeout (in seconds).
Type
32-bit Integer Unsigned
Default
30
osd_check_for_log_corruption
Description
Check log files for corruption. Can be computationally expensive.
Type
Boolean
Default
false
osd_remove_thread_timeout
Description
The maximum time in seconds before timing out a remove OSD thread.
Type
32-bit Integer
Default
60*60
osd_command_thread_timeout
Description
The maximum time in seconds before timing out a command thread.
Type
32-bit Integer
Default
10*60
osd_command_max_records
Description
Limits the number of lost objects to return.
Type
32-bit Integer
Default
256
osd_auto_upgrade_tmap
Description
Uses tmap for omap on old objects.
Type
Boolean
Default
true
osd_tmapput_sets_users_tmap
Description
Uses tmap for debugging only.
Type
Boolean
Default
false
osd_preserve_trimmed_log
Description
Preserves trimmed log files, but uses more disk space.
Type
Boolean
Default
false
osd_recovery_delay_start
Description
After peering completes, Ceph delays for the specified number of seconds before starting to recover objects.
Type
Float
Default
0
osd_recovery_max_active
Description
The number of active recovery requests per OSD at one time. More requests will accelerate recovery, but the requests place an increased load on the cluster.
Type
32-bit Integer
Default
0
osd_recovery_max_chunk
Description
The maximum size of a recovered chunk of data to push.
Type
64-bit Integer Unsigned
Default
8388608
osd_recovery_threads
Description
The number of threads for recovering data.
Type
32-bit Integer
Default
1
osd_recovery_thread_timeout
Description
The maximum time in seconds before timing out a recovery thread.
Type
32-bit Integer
Default
30
osd_recover_clone_overlap
Description
Preserves clone overlap during recovery. Should always be set to true.
Type
Boolean
Default
true
rados_osd_op_timeout
Description
Number of seconds that RADOS waits for a response from the OSD before returning an error from a RADOS operation. A value of 0 means no limit.
Type
Double
Default
0