Chapter 4. Administering Ceph File Systems

This chapter describes common Ceph File System administrative tasks.

To map a directory to a particular MDS rank, see Section 4.2, “Mapping Directory Trees to MDS Ranks”.
To disassociate a directory from a MDS rank, see Section 4.3, “Disassociating Directory Trees from MDS Ranks”.
To work with files and directory layouts, see Section 4.4, “Working with File and Directory Layouts”.
To add a new data pool, see Section 4.5, “Adding Data Pools”.

4.1. Prerequisites

Deploy a Ceph Storage Cluster if you do not have one. For details, see the Installation Guide for Red Hat Enterprise Linux or Ubuntu.
Install and configure Ceph Metadata Server daemons (ceph-mds). For details, see the Installation Guide for Red Hat Enterprise Linux or Ubuntu Chapter 2, Configuring Metadata Server Daemons.
Create and mount the Ceph File System. For details, see Chapter 3, Deploying Ceph File Systems.

4.2. Mapping Directory Trees to MDS Ranks

This section describes how to map a directory and its subdirectories to a particular active Metadata Server (MDS) rank so that its metadata is only managed by the MDS daemon holding that rank. This approach enables you to evenly spread application load or limit impact of users' metadata requests to the entire cluster.

Important

Note that an internal balancer already dynamically spreads the application load. Therefore, map directory trees to ranks only for certain carefully chosen applications. In addition, when a directory is mapped to a rank, the balancer cannot split it. Consequently, a large number of operations within the mapped directory can overload the rank and the MDS daemon that manages it.

Prerequisites

Configure multiple active MDS daemons. See Section 2.6, “Configuring Multiple Active Metadata Server Daemons” for details.
Ensure that the attr package is installed on the client node with mounted Ceph File System.

Procedure

Set the ceph.dir.pin extended attribute on a directory.
```
setfattr -n ceph.dir.pin -v <rank> <directory>
```
For example, to assign the /home/ceph-user/ directory all of its subdirectories to rank 2:
```
[user@client ~]$ setfattr -n ceph.dir.pin -v 2 /home/ceph-user
```

Additional Resources

Section 4.3, “Disassociating Directory Trees from MDS Ranks”

4.3. Disassociating Directory Trees from MDS Ranks

This section describes how to disassociate a directory from a particular active Metadata Server (MDS) rank.

Prerequisites

Ensure that the attr package is installed on the client node with mounted Ceph File System.

Procedure

Set the ceph.dir.pin extended attribute to -1 on a directory.
```
setfattr -n ceph.dir.pin -v -1 <directory>
```
For example, to disassociate the /home/ceph-user/ directory from a MDS rank:
```
[user@client ~]$ serfattr -n ceph.dir.pin -v -1 /home/ceph-user
```
Note that any separately mapped subdirectories of /home/ceph-user/ are not affected.

Additional Resources

Section 4.2, “Mapping Directory Trees to MDS Ranks”

4.4. Working with File and Directory Layouts

This section describes how to:

4.4.1. Prerequisites

Make sure that the attr package is installed.

4.4.2. Understanding File and Directory Layouts

This section explains what file and directory layouts are in the context for the Ceph File System.

A layout of a file or directory controls how its content is mapped to Ceph RADOS objects. The directory layouts serves primarily for setting an inherited layout for new files in that directory. See Layouts Inheritance for more details.

To view and set a file or directory layout, use virtual extended attributes or extended file attributes (xattrs). The name of the layout attributes depends on whether a file is a regular file or a directory:

Regular files layout attributes are called ceph.file.layout
Directories layout attributes are called ceph.dir.layout

The File and Directory Layout Fields table lists available layout fields that you can set on files and directories.

Table 4.1. File and Directory Layout Fields

Field	Description	Type
`pool`	ID or name of the pool to store file’s data objects. Note that the pool must part of the set of data pools of the Ceph file system. See Section 4.5, “Adding Data Pools” for details.	string
`pool_namespace`	Namespace to write objects to. Empty by default, that means the default namespace is used.	string
`stripe_unit`	The size in bytes of a block of data used in the RAID 0 distribution of a file. All stripe units for a file have equal size. The last stripe unit is typically incomplete. That means it represents the data at the end of the file as well as unused space beyond it up to the end of the fixed stripe unit size.	integer
`stripe_count`	The number of consecutive stripe units that constitute a RAID 0 “stripe” of file data.	integer
`object_size`	Size of RADOS objects in bytes in which file data are chunked.	integer

Layouts Inheritance

Files inherit the layout of their parent directory when you create them. However, subsequent changes to the parent directory layout do not affect children. If a directory does not have any layouts set, files inherit the layout from the closest directory with layout in the directory structure.

4.4.3. Setting File and Directory Layouts

Use the setfattr command to set layout fields on a file or directory.

Important

When you modify the layout fields of a file, the file must be empty, otherwise an error occurs.

Procedure

To modify layout fields on a file or directory:
```
setfattr -n ceph.<type>.layout.<field> -v <value> <path>
```
Replace:
- <type> with file or dir
- <field> with the name of the field, see the File and Directory Layouts Fields table for details.
- <value> with the new value of the field
- <path> with the path to the file or directory
For example, to set the stripe_unit field to 1048576 on the test file:
```
$ setfattr -n ceph.file.layout.stripe_unit -v 1048576 test
```

Additional Resources

The setfattr(1) manual page

4.4.4. Viewing File and Directory Layouts

This section describes how to use the getfattr command to view layout fields on a file or directory.

Procedure

To view layout fields on a file or directory as a single string:
```
getfattr -n ceph.<type>.layout <path>
```
Replace * <path> with the path to the file or directory * <type> with file or dir
For example, to view file layouts on the /home/test/ file:
```
$ getfattr -n ceph.dir.layout /home/test
ceph.dir.layout="stripe_unit=4194304 stripe_count=2 object_size=4194304 pool=cephfs_data"
```
Note
Directories do not have an explicit layout until you set it (see Section 4.4.3, “Setting File and Directory Layouts”). Consequently, an attempt to view the layout fails if you never modified the layout.
To view individual layout fields on a file or directory:
```
getfattr -n ceph.<type>.layout.<field> <path>
```
Replace:
- <type> with file or dir
- <field> with the name of the field, see the File and Directory Layouts Fields table for details.
- <path> with the path to the file or directory
For example, to view the pool field of the test file:
```
$ getfattr -n ceph.file.layout.pool test
ceph.file.layout.pool="cephfs_data"
```
Note
When viewing the pool field, the pool is usually indicated by its name. However, when you just created the pool, it can be indicated by its ID.

Additional Resources

The getfattr(1) manual page

4.4.5. Removing Directory Layouts

This section describes how to use the setfattr command to remove layouts from a directory.

Note

When you set a file layout, you cannot change or remove it.

Procedure

To remove a layout from a directory:
```
setfattr -x ceph.dir.layout <path>
```
Replace:
- <path> with the path to the directory
For example:
```
$ setfattr -x ceph.dir.layout /home/cephfs
```
To remove the pool_namespace field:
```
$ setfattr -x ceph.dir.layout.pool_namespace <directory>
```
Replace:
- <path> with the path to the directory
For example:
```
$ setfattr -x ceph.dir.layout.pool_namespace /home/cephfs
```
Note
The pool_namespace field is the only field you can remove separately.

Additional Resources

The setfattr(1) manual page

4.5. Adding Data Pools

The Ceph File System (CephFS) supports adding more than one pool to be used for storing data. This can be useful for:

Storing log data on reduced redundancy pools
Storing user home directories on an SSD or NVMe pool
Basic data segregation.

Before using another data pool in the Ceph File System, you must add it as described in this section.

By default, for storing file data, CephFS uses the initial data pool that was specified during its creation. To use a secondary data pool, you must also configure a part of the file system hierarchy to store file data in that pool (and optionally, within a namespace of that pool) using file and directory layouts. See Section 4.4, “Working with File and Directory Layouts” for details.

Procedure

Use the following commands from a Monitor host and as the root user.

Create a new data pool.
```
ceph osd pool create <name> <pg_num>
```
Replace:
- <name> with the name of the pool
- <pg_num> with the number of placement groups (PGs)
For example:
```
[root@monitor]# ceph osd pool create cephfs_data_ssd 64
pool 'cephfs_data_ssd' created
```
Add the newly created pool under the control of the Metadata Servers.
```
ceph mds add_data_pool <name>
```
Replace:
- <name> with the name of the pool
For example:
```
[root@monitor]# ceph mds add_data_pool cephfs_data_ssd
added data pool 6 to fsmap
```

Verify that the pool was successfully added:

[root@monitor]# ceph fs ls
name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ssd]

If you use the cephx authentication, make sure that clients can access the new pool. See Section 3.3, “Creating Ceph File System Client Users” for details.

Where did the comment section go?

Red Hat's documentation publication system recently went through an upgrade to enable speedier, more mobile-friendly content. We decided to re-evaluate our commenting platform to ensure that it meets your expectations and serves as an optimal feedback mechanism. During this redesign, we invite your input on providing feedback on Red Hat documentation via the discussion platform.