Red Hat Training

A Red Hat training course is available for Red Hat Ceph Storage

Chapter 8. File Store Configuration Reference

8.1. Extended Attributes

Extended attributes (XATTRs) are an important aspect in the CephFS configuration. Some file systems have limits on the number of bytes stored in extended attributes. Additionally, in some cases, the file system might not be as fast as an alternative method of storing extended attributes. The following settings improve CephFS performance by using a method of storing extended attributes that is extrinsic to the underlying file system.

Ceph extended attributes are stored as inline xattr, using the extended attributes provided by the underlying file system, if it does not impose a size limit. If there is a size limit (4KB total on ext4, for instance), some Ceph extended attributes will be stored in an key-value database called omap when the filestore max inline xattr size or filestore max inline xattrs threshold are reached.

filestore_xattr_use_omap
Description
Use object map for XATTRS. Set to true for ext4 file systems.
Type
Boolean
Required
No
Default
false
filestore_omap_header_cache_size
Description
Determines the size of the LRU used to cache object omap headers. Larger values use more memory but can reduce lookups on omap. (Experts only).
Type
Integer
Default
1024
filestore_omap_backend
Description
Used to determine which back end is used for the omap. Can be set to leveldb or rocksdb. (Experts only. rocksdb is experimental.)
Type
String
Default
leveldb
filestore_debug_omap_check
Description
Debugging check on synchronization. Expensive. For debugging only.
Type
Boolean
Required
No
Default
0
filestore_max_inline_xattr_size
Description
The maximum size of an extended attribute stored in a file system (that is, XFS, btrfs, ext4, and others) per object. Should not be larger than the file system can handle.
Type
Unsigned 32-bit Integer
Required
No
Default
512
filestore_max_inline_xattrs
Description
The maximum number of extended attributes stored in the file system per object.
Type
32-bit Integer
Required
No
Default
2
filestore_max_inline_xattr_size_xfs
Description
The maximum size of an extended attribute stored in the file system for XFS file systems per object. Should not be larger than the file system can handle.
Type
Unsigned 32-bit Integer
Default
65536
filestore_max_inline_xattr_size_btrfs
Description
The maximum size of an extended attribute stored in the file system for btrfs per object. Should not be larger than the file system can handle.
Type
Unsigned 32-bit Integer
Default
2048
filestore_max_inline_xattr_size_other
Description
The maximum size of an extended attribute stored in the file system for file systems other than btrfs or XFS per object. Should not be larger than the file system can handle.
Type
Unsigned 32-bit Integer
Default
512
filestore_max_inline_xattrs
Description
The maximum number of extended attributes stored in the file system per object. Overrides fine-grained settings.
Type
Unsigned 32-bit Integer
Default
0
filestore_max_inline_xattrs_xfs
Description
The maximum number of extended attributes stored in an XFS file system per object.
Type
Unsigned 32-bit Integer
Default
10
filestore_max_inline_xattrs_btrfs
Description
The maximum number of extended attributes stored in a btrfs file system per object.
Type
Unsigned 32-bit Integer
Default
10
filestore_max_inline_xattrs_other
Description
The maximum number of extended attributes stored in file systems other than btrfs or XFS per object.
Type
Unsigned 32-bit Integer
Default
2

8.2. Synchronization Intervals

Periodically, the file store needs to quiesce write operations and synchronize the file system, which creates a consistent commit point. It can then free journal entries up to the commit point. Synchronizing more frequently tends to reduce the time required to perform synchronization, and reduces the amount of data that needs to remain in the journal. Less frequent synchronization allows the backing file system to coalesce small writes and metadata updates more optimally—​potentially resulting in more efficient synchronization.

filestore_max_sync_interval
Description
The maximum interval in seconds for synchronizing the file store.
Type
Double
Required
No
Default
5
filestore_min_sync_interval
Description
The minimum interval in seconds for synchronizing the file store.
Type
Double
Required
No
Default
.01

8.3. Flusher

The file store flusher forces data from large write operations to be written out using the sync file range option before the synchronization in order to reduce the cost of the eventual synchronization. In practice, disabling the file store flusher seems to improve performance in some cases.

filestore_flusher
Description
Enables the file store flusher.
Type
Boolean
Required
No
Default
false
filestore_flusher_max_fds
Description
Sets the maximum number of file descriptors for the flusher.
Type
Integer
Required
No
Default
512
filestore_sync_flush
Description
Enables the synchronization flusher.
Type
Boolean
Required
No
Default
false
filestore_fsync_flushes_journal_data
Description
Flush journal data during file system synchronization.
Type
Boolean
Required
No
Default
false

8.4. Queue

The following settings provide limits on the size of the file store queue.

filestore_queue_max_ops
Description
Defines the maximum number of operations in progress that the file store accepts before blocking on queuing new operations.
Type
Integer
Required
No. Minimal impact on performance.
Default
500
filestore_queue_max_bytes
Description
The maximum number of bytes for an operation.
Type
Integer
Required
No
Default
100 << 20
filestore_queue_committing_max_ops
Description
The maximum number of operations that the file store can commit.
Type
Integer
Required
No
Default
500
filestore_queue_committing_max_bytes
Description
The maximum number of bytes that the file store can commit.
Type
Integer
Required
No
Default
100 << 20

8.5. Writeback Throttle

Ceph replicates some of the write-back behavior in the kernel, because the page cache tends to keep dirty data round too long.

filestore_wbthrottle_enable
Description
Enables the file store write-back throttle. The file store write-back throttle is used to prevent large amounts of uncommitted data from building up before each file store sync. (Experts only).
Type
Boolean
Default
true
filestore_wbthrottle_btrfs_bytes_start_flusher
Description
Dirty bytes threshold at which Ceph begins background flushing for the btrfs file system.
Type
64-bit Unsigned Integer
Default
41943040
filestore_wbthrottle_btrfs_bytes_hard_limit
Description
Dirty bytes threshold at which Ceph begins to throttle I/O until the flusher catches up for btrfs.
Type
64-bit Unsigned Integer
Default
419430400
filestore_wbthrottle_btrfs_ios_start_flusher
Description
Dirty I/Os threshold at which Ceph begins background flushing for btrfs.
Type
64-bit Unsigned Integer
Default
500
filestore_wbthrottle_btrfs_ios_hard_limit
Description
Dirty I/Os threshold at which Ceph begins to throttle IO until the flusher catches up for btrfs.
Type
64-bit Unsigned Integer
Default
5000
filestore_wbthrottle_btrfs_inodes_start_flusher
Description
Dirty inodes threshold at which Ceph begins background flushing for btrfs.
Type
64-bit Unsigned Integer
Default
500
filestore_wbthrottle_btrfs_inodes_hard_limit
Description
Dirty inodes threshold at which Ceph begins to throttle IO until the flusher catches up for btrfs. Must be less than the fd limit.
Type
64-bit Unsigned Integer
Default
5000
filestore_wbthrottle_xfs_bytes_start_flusher
Description
Dirty bytes threshold at which Ceph begins background flushing for the XFS file system.
Type
64-bit Unsigned Integer
Default
41943040
filestore_wbthrottle_xfs_bytes_hard_limit
Description
Dirty bytes threshold at which Ceph begins to throttle IO until the flusher catches up for XFS.
Type
64-bit Unsigned Integer
Default
419430400
filestore_wbthrottle_xfs_ios_start_flusher
Description
Dirty I/Os threshold at which Ceph begins background flushing for XFS.
Type
64-bit Unsigned Integer
Default
500
filestore_wbthrottle_xfs_ios_hard_limit
Description
Dirty I/Os threshold at which Ceph begins to throttle IO until the flusher catches up for XFS.
Type
64-bit Unsigned Integer
Default
5000
filestore_wbthrottle_xfs_inodes_start_flusher
Description
Dirty inodes threshold at which Ceph begins background flushing for XFS.
Type
64-bit Unsigned Integer
Default
500
filestore_wbthrottle_xfs_inodes_hard_limit
Description
Dirty inodes threshold at which Ceph begins to throttle IO until the flusher catches up for XFS. Must be less than the fd limit.
Type
64-bit Unsigned Integer
Default
5000

8.6. Timeouts

filestore_op_threads
Description
The number of file system operation threads that execute in parallel.
Type
Integer
Required
No
Default
2
filestore_op_thread_timeout
Description
The timeout for a file system operation thread (in seconds).
Type
Integer
Required
No
Default
60
filestore_op_thread_suicide_timeout
Description
The timeout for a commit operation before canceling the commit (in seconds).
Type
Integer
Required
No
Default
180

8.7. B-Tree File System

filestore_btrfs_snap
Description
Enable snapshots for a btrfs file store.
Type
Boolean
Required
No. Only used for btrfs.
Default
true
filestore_btrfs_clone_range
Description
Enable cloning ranges for a btrfs file store.
Type
Boolean
Required
No. Only used for btrfs.
Default
true

8.8. Journal

filestore_journal_parallel
Description
Enables parallel journaling, default for btrfs.
Type
Boolean
Required
No
Default
false
filestore_journal_writeahead
Description
Enables write-ahead journaling, default for XFS.
Type
Boolean
Required
No
Default
false
filestore_journal_trailing
Description
Deprecated, never use.
Type
Boolean
Required
No
Default
false

8.9. Miscellaneous

filestore_merge_threshold
Description
Minimum number of files in a subdirectory before merging into parent NOTE: A negative value means to disable subdirectory merging.
Type
Integer
Required
No
Default
10
filestore_split_multiple
Description
filestore_split_multiple * abs(filestore_merge_threshold) * 16 is the maximum number of files in a subdirectory before splitting into child directories.
Type
Integer
Required
No
Default
2
filestore_update_to
Description
Limits file store auto upgrade to specified version.
Type
Integer
Required
No
Default
1000
filestore_blackhole
Description
Drop any new transactions on the floor.
Type
Boolean
Required
No
Default
false
filestore_dump_file
Description
File onto which store transaction dumps.
Type
Boolean
Required
No
Default
false
filestore_kill_at
Description
Inject a failure at the n’th opportunity.
Type
String
Required
No
Default
false
filestore_fail_eio
Description
Fail or terminate unexpectedly on EIO.
Type
Boolean
Required
No
Default
true