Red Hat Training

A Red Hat training course is available for Red Hat Ceph Storage

Chapter 5. Pool, PG, and CRUSH Configuration Reference

When you create pools and set the number of placement groups for the pool, Ceph uses default values when you do not specifically override the defaults. Red Hat recommends overriding some of the defaults. Specifically, set a pool’s replica size and override the default number of placement groups. You can set these values when running pool commands. You can also override the defaults by adding new ones in the [global] section of the Ceph configuration file.

[global]

# By default, Ceph makes 3 replicas of objects. If you want to set 4
# copies of an object as the default value--a primary copy and three replica
# copies--reset the default values as shown in 'osd pool default size'.
# If you want to allow Ceph to write a lesser number of copies in a degraded
# state, set 'osd pool default min size' to a number less than the
# 'osd pool default size' value.

osd_pool_default_size = 4  # Write an object 4 times.
osd_pool_default_min_size = 1 # Allow writing one copy in a degraded state.

# Ensure you have a realistic number of placement groups. We recommend
# approximately 100 per OSD. E.g., total number of OSDs multiplied by 100
# divided by the number of replicas (i.e., osd pool default size). So for
# 10 OSDs and osd pool default size = 4, we'd recommend approximately
# (100 * 10) / 4 = 250.

osd_pool_default_pg_num = 250
osd_pool_default_pgp_num = 250

5.1. Settings

mon_max_pool_pg_num
Description
The maximum number of placement groups per pool.
Type
Integer
Default
65536
mon_pg_create_interval
Description
Number of seconds between PG creation in the same Ceph OSD Daemon.
Type
Float
Default
30.0
mon_pg_stuck_threshold
Description
Number of seconds after which PGs can be considered as being stuck.
Type
32-bit Integer
Default
300
osd_pg_bits
Description
Placement group bits per Ceph OSD Daemon.
Type
32-bit Integer
Default
6
osd_pgp_bits
Description
The number of bits per Ceph OSD Daemon for Placement Groups for Placement purpose (PGPs).
Type
32-bit Integer
Default
6
osd_crush_chooseleaf_type
Description
The bucket type to use for chooseleaf in a CRUSH rule. Uses ordinal rank rather than name.
Type
32-bit Integer
Default
1. Typically a host containing one or more Ceph OSD Daemons.
osd_pool_default_crush_replicated_ruleset
Description
The default CRUSH ruleset to use when creating a replicated pool.
Type
8-bit Integer
Default
0
osd_pool_erasure_code_stripe_width
Description
Sets the desired size, in bytes, of an object stripe on every erasure coded pools. Every object if size S will be stored as N stripes and each stripe will be encoded/decoded individually.
Type
Unsigned 32-bit Integer
Default
4096
osd_pool_default_size
Description
Sets the number of replicas for objects in the pool. The default value is the same as ceph osd pool set {pool-name} size {size}.
Type
32-bit Integer
Default
3
osd_pool_default_min_size
Description
Sets the minimum number of written replicas for objects in the pool in order to acknowledge a write operation to the client. If minimum is not met, Ceph will not acknowledge the write to the client. This setting ensures a minimum number of replicas when operating in degraded mode.
Type
32-bit Integer
Default
0, which means no particular minimum. If 0, minimum is size - (size / 2).
osd_pool_default_pg_num
Description
The default number of placement groups for a pool. The default value is the same as pg_num with mkpool.
Type
32-bit Integer
Default
8
osd_pool_default_pgp_num
Description
The default number of placement groups for placement for a pool. The default value is the same as pgp_num with mkpool. PG and PGP should be equal (for now).
Type
32-bit Integer
Default
8
osd_pool_default_flags
Description
The default flags for new pools.
Type
32-bit Integer
Default
0
osd_max_pgls
Description
The maximum number of placement groups to list. A client requesting a large number can tie up the Ceph OSD Daemon.
Type
Unsigned 64-bit Integer
Default
1024
Note
Default should be fine.
osd_min_pg_log_entries
Description
The minimum number of placement group logs to maintain when trimming log files.
Type
32-bit Int Unsigned
Default
1000
osd_default_data_pool_replay_window
Description
The time (in seconds) for an OSD to wait for a client to replay a request.
Type
32-bit Integer
Default
45