Procedure

1. Check the current state of a Ceph File System:

   Example

   [root@mon ~]# ceph fs dump
   dumped fsmap epoch 399
   ...
   Filesystem 'cephfs01' (27)
   ...
   e399
   max_mds 1
   in      0
   up      {0=20384}
   failed
   damaged
   stopped
   ...
   [mds.a{0:20384} state up:active seq 239 addr [v2:127.0.0.1:6854/966242805,v1:127.0.0.1:6855/966242805]]
   
   Standby daemons:
   
   [mds.b{-1:10420} state up:standby seq 2 addr [v2:127.0.0.1:6856/2745199145,v1:127.0.0.1:6857/2745199145]]

2. Set the file system affinity:

   Syntax

   ceph config set STANDBY_DAEMON mds_join_fs FILE_SYSTEM_NAME

   Example

   [root@mon ~]# ceph config set mds.b mds_join_fs cephfs01

   After a Ceph MDS failover event, the file system favors the standby daemon for which the affinity is set.

   Example

   [root@mon ~]# ceph fs dump
   dumped fsmap epoch 405
   e405
   ...
   Filesystem 'cephfs01' (27)
   ...
   max_mds 1
   in      0
   up      {0=10420}
   failed
   damaged
   stopped
   ...
   [mds.b{0:10420} state up:active seq 274 join_fscid=27 addr [v2:127.0.0.1:6856/2745199145,v1:127.0.0.1:6857/2745199145]] 1
   
   Standby daemons:
   
   [mds.a{-1:10720} state up:standby seq 2 addr [v2:127.0.0.1:6854/1340357658,v1:127.0.0.1:6855/1340357658]]

   1

   The mds.b daemon now has the join_fscid=27 in the file system dump output.

   Important

   If a file system is in a degraded or undersized state, then no failover will occur to enforce the file system affinity.

Procedure

1. Set the max_mds parameter to the desired number of active MDS daemons:

   Syntax

   ceph fs set NAME max_mds NUMBER

   Example

   [root@mon ~]# ceph fs set cephfs max_mds 2

   This example increases the number of active MDS daemons to two in the CephFS called cephfs

   Note

   Ceph only increases the actual number of ranks in the CephFS if a spare MDS daemon is available to take the new rank.

2. Verify the number of active MDS daemons:

   Syntax

   ceph fs status NAME

   Example

   [root@mon ~]# ceph fs status cephfs
   cephfs - 0 clients
   ======
   +------+--------+-------+---------------+-------+-------+--------+--------+
   | RANK | STATE  |  MDS  |    ACTIVITY   |  DNS  |  INOS |  DIRS  |  CAPS  |
   +------+--------+-------+---------------+-------+-------+--------+--------+
   |  0   | active | node1 | Reqs:    0 /s |   10  |   12  |   12   |   0    |
   |  1   | active | node2 | Reqs:    0 /s |   10  |   12  |   12   |   0    |
   +------+--------+-------+---------------+-------+-------+--------+--------+
   +-----------------+----------+-------+-------+
   |       POOL      |   TYPE   |  USED | AVAIL |
   +-----------------+----------+-------+-------+
   | cephfs_metadata | metadata | 4638  | 26.7G |
   |   cephfs_data   |   data   |    0  | 26.7G |
   +-----------------+----------+-------+-------+
   
   +-------------+
   | STANDBY MDS |
   +-------------+
   |    node3    |
   +-------------+

Procedure

1. Set the same number of MDS daemons as specified by the max_mds parameter:

   Syntax

   ceph fs status NAME

   Example

   [root@mon ~]# ceph fs status cephfs
   cephfs - 0 clients
   
   +------+--------+-------+---------------+-------+-------+--------+--------+
   | RANK | STATE  |  MDS  |    ACTIVITY   |  DNS  |  INOS |  DIRS  |  CAPS  |
   +------+--------+-------+---------------+-------+-------+--------+--------+
   |  0   | active | node1 | Reqs:    0 /s |   10  |   12  |   12   |   0    |
   |  1   | active | node2 | Reqs:    0 /s |   10  |   12  |   12   |   0    |
   +------+--------+-------+---------------+-------+-------+--------+--------+
   +-----------------+----------+-------+-------+
   |       POOL      |   TYPE   |  USED | AVAIL |
   +-----------------+----------+-------+-------+
   | cephfs_metadata | metadata | 4638  | 26.7G |
   |   cephfs_data   |   data   |    0  | 26.7G |
   +-----------------+----------+-------+-------+
   
   +-------------+
   | Standby MDS |
   +-------------+
   |    node3    |
   +-------------+

2. On a node with administration capabilities, change the max_mds parameter to the desired number of active MDS daemons:

   Syntax

   ceph fs set NAME max_mds NUMBER

   Example

   [root@mon ~]# ceph fs set cephfs max_mds 1

3. Wait for the storage cluster to stabilize to the new max_mds value by watching the Ceph File System status.

4. Verify the number of active MDS daemons:

   Syntax

   ceph fs status NAME

   Example

   [root@mon ~]# ceph fs status cephfs
   cephfs - 0 clients
   
   +------+--------+-------+---------------+-------+-------+--------+--------+
   | RANK | STATE  |  MDS  |    ACTIVITY   |  DNS  |  INOS |  DIRS  |  CAPS  |
   +------+--------+-------+---------------+-------+-------+--------+--------+
   |  0   | active | node1 | Reqs:    0 /s |   10  |   12  |   12   |   0    |
   +------+--------+-------+---------------+-------+-------+--------|--------+
   +-----------------+----------+-------+-------+
   |       POOl      |   TYPE   |  USED | AVAIL |
   +-----------------+----------+-------+-------+
   | cephfs_metadata | metadata | 4638  | 26.7G |
   |   cephfs_data   |   data   |    0  | 26.7G |
   +-----------------+----------+-------+-------+
   
   +-------------+
   | Standby MDS |
   +-------------+
   |    node3    |
   |    node2    |
   +-------------+

Procedure

1. Configure the client node to use the Ceph storage cluster.

   1. Enable the Red Hat Ceph Storage Tools repository:

      Red Hat Enterprise Linux 8

      [root@client ~]# subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms

      Red Hat Enterprise Linux 9

      [root@client ~]# subscription-manager repos --enable=rhceph-5-tools-for-rhel-9-x86_64-rpms

   2. Install the ceph-fuse package:

      [root@client ~]# dnf install ceph-fuse

   3. Copy the Ceph client keyring from the Ceph Monitor node to the client node:

      Syntax

      scp root@MONITOR_NODE_NAME:/etc/ceph/KEYRING_FILE /etc/ceph/

      Replace MONITOR_NODE_NAME with the Ceph Monitor host name or IP address.

      Example

      [root@client ~]# scp root@192.168.0.1:/etc/ceph/ceph.client.1.keyring /etc/ceph/

   4. Copy the Ceph configuration file from a Ceph Monitor node to the client node:

      Syntax

      scp root@MONITOR_NODE_NAME:/etc/ceph/ceph.conf /etc/ceph/ceph.conf

      Replace MONITOR_NODE_NAME with the Ceph Monitor host name or IP address.

      Example

      [root@client ~]# scp root@192.168.0.1:/etc/ceph/ceph.conf /etc/ceph/ceph.conf

   5. Set the appropriate permissions for the configuration file:

      [root@client ~]# chmod 644 /etc/ceph/ceph.conf

2. Create a Ceph File System:

   Syntax

   ceph fs volume create FILE_SYSTEM_NAME

   Example

   [root@mon ~]# ceph fs volume create cephfs01

   Repeat this step to create additional file systems.

   Note

   By running this command, Ceph automatically creates the new pools, and deploys a new Ceph Metadata Server (MDS) daemon to support the new file system. This also configures the MDS affinity accordingly.

3. Verify access to the new Ceph File System from a Ceph client.

   1. Authorize a Ceph client to access the new file system:

      Syntax

      ceph fs authorize FILE_SYSTEM_NAME CLIENT_NAME DIRECTORY PERMISSIONS

      Important

      The supported values for PERMISSIONS are r (read) and rw (read/write).

      Example

      [root@mon ~]# ceph fs authorize cephfs01 client.1 / rw
      [client.1]
          key = BQAmthpf81M+JhAAiHDYQkMiCq3x+J0n9e8REK==
      
      [root@mon ~]# ceph auth get client.1
      exported keyring for client.1
      [client.1]
          key = BQAmthpf81M+JhAAiHDYQkMiCq3x+J0n9e8REK==
          caps mds = "allow rw fsname=cephfs01"
          caps mon = "allow r fsname=cephfs01"
          caps osd = "allow rw tag cephfs data=cephfs01"

      Note

      Optionally, you can add a safety measure by specifying the root_squash option. This prevents accidental deletion scenarios by disallowing clients with a uid=0 or gid=0 to do write operations, but still allows read operations.

      Example

      [root@mon ~]# ceph fs authorize cephfs01 client.1 / rw root_squash /volumes rw
      [client.1]
          key = BQAmthpf81M+JhAAiHDYQkMiCq3x+J0n9e8REK==
      
      [root@mon ~]# ceph auth get client.1
      [client.1]
          key = BQAmthpf81M+JhAAiHDYQkMiCq3x+J0n9e8REK==
          caps mds = "allow rw fsname=cephfs01 root_squash, allow rw fsname=cephfs01 path=/volumes"
          caps mon = "allow r fsname=cephfs01"
          caps osd = "allow rw tag cephfs data=cephfs01"

      In this example, root_squash is enabled for the file system cephfs01, except within the /volumes directory tree.

      Important

      The Ceph client can only see the CephFS it is authorized for.

   2. Copy the Ceph user’s keyring to the Ceph client node:

      Syntax

      ceph auth get CLIENT_NAME > OUTPUT_FILE_NAME
      scp OUTPUT_FILE_NAME TARGET_NODE_NAME:/etc/ceph

      Example

      [root@mon ~]# ceph auth get client.1 > ceph.client.1.keyring
      exported keyring for client.1
      [root@mon ~]# scp ceph.client.1.keyring client:/etc/ceph
      root@client's password:
      ceph.client.1.keyring                     100%  178   333.0KB/s   00:00

   3. On the Ceph client node, create a new directory:

      Syntax

      mkdir PATH_TO_NEW_DIRECTORY_NAME

      Example

      [root@client ~]# mkdir /mnt/mycephfs

   4. On the Ceph client node, mount the new Ceph File System:

      Syntax

      ceph-fuse PATH_TO_NEW_DIRECTORY_NAME -n CEPH_USER_NAME --client-fs=_FILE_SYSTEM_NAME

      Example

      [root@client ~]# ceph-fuse /mnt/mycephfs/ -n client.1 --client-fs=cephfs01
      ceph-fuse[555001]: starting ceph client
      2022-05-09T07:33:27.158+0000 7f11feb81200 -1 init, newargv = 0x55fc4269d5d0 newargc=15
      ceph-fuse[555001]: starting fuse

   5. On the Ceph client node, list the directory contents of the new mount point, or create a file on the new mount point.

Procedure

1. Create an erasure-coded data pool for CephFS:

   Syntax

   ceph osd pool create DATA_POOL_NAME erasure

   Example

   [root@mon ~]# ceph osd pool create cephfs-data-ec01 erasure
   pool 'cephfs-data-ec01' created

2. Verify the pool was added:

   Example

   [root@mon ~]# ceph osd lspools

3. Enable overwrites on the erasure-coded pool:

   Syntax

   ceph osd pool set DATA_POOL_NAME allow_ec_overwrites true

   Example

   [root@mon ~]# ceph osd pool set cephfs-data-ec01 allow_ec_overwrites true
   set pool 15 allow_ec_overwrites to true

4. Verify the status of the Ceph File System:

   Syntax

   ceph fs status FILE_SYSTEM_NAME

   Example

   [root@mon ~]# ceph fs status cephfs-ec
   cephfs-ec - 14 clients
   =========
   RANK  STATE              MDS                ACTIVITY     DNS    INOS   DIRS   CAPS
    0    active  cephfs-ec.example.ooymyq  Reqs:    0 /s  8231   8233    891    921
          POOL           TYPE     USED  AVAIL
   cephfs-metadata-ec  metadata   787M  8274G
     cephfs-data-ec      data    2360G  12.1T
   
          STANDBY MDS
   cephfs-ec.example.irsrql
   cephfs-ec.example.cauuaj

5. Add the erasure-coded data pool to the existing CephFS:

   Syntax

   ceph fs add_data_pool FILE_SYSTEM_NAME DATA_POOL_NAME

   Example

   [root@mon ~]# ceph fs add_data_pool cephfs-ec cephfs-data-ec01

   This example adds the new data pool, cephfs-data-ec01, to the existing erasure-coded file system, cephfs-ec.

6. Verify that the erasure-coded pool was added to the Ceph File System:

   Syntax

   ceph fs status FILE_SYSTEM_NAME

   Example

   [root@mon ~]# ceph fs status cephfs-ec
   cephfs-ec - 14 clients
   =========
   RANK  STATE              MDS                ACTIVITY     DNS    INOS   DIRS   CAPS
    0    active  cephfs-ec.example.ooymyq  Reqs:    0 /s  8231   8233    891    921
          POOL           TYPE     USED  AVAIL
   cephfs-metadata-ec  metadata   787M  8274G
     cephfs-data-ec      data    2360G  12.1T
    cephfs-data-ec01     data       0   12.1T
   
          STANDBY MDS
   cephfs-ec.example.irsrql
   cephfs-ec.example.cauuaj

7. Set the file layout on a new directory:

   Syntax

   mkdir PATH_TO_DIRECTORY
   setfattr -n ceph.dir.layout.pool -v DATA_POOL_NAME PATH_TO_DIRECTORY

   Example

   [root@mon ~]# mkdir /mnt/cephfs/newdir
   [root@mon ~]# setfattr -n ceph.dir.layout.pool -v cephfs-data-ec01 /mnt/cephfs/newdir

   In this example, all new files created in the /mnt/cephfs/newdir directory inherit the directory layout and places the data in the newly added erasure-coded pool.

Procedure

1. Log into the Cephadm shell on the monitor node:

   Example

   [root@host01 ~]# cephadm shell

2. On a Ceph Monitor node, create a client user:

   Syntax

   ceph fs authorize FILE_SYSTEM_NAME client.CLIENT_NAME /DIRECTORY CAPABILITY [/DIRECTORY CAPABILITY] ...

   • To restrict the client to only writing in the temp directory of filesystem cephfs_a:

     Example

     [ceph: root@host01 /]# ceph fs authorize cephfs_a client.1 / r /temp rw
     
     client.1
       key = AQBSdFhcGZFUDRAAcKhG9Cl2HPiDMMRv4DC43A==

   • To completely restrict the client to the temp directory, remove the root (/) directory:

     Example

     [ceph: root@host01 /]# ceph fs authorize cephfs_a client.1 /temp rw

   Note

   Supplying all or asterisk as the file system name grants access to every file system. Typically, it is necessary to quote the asterisk to protect it from the shell.

3. Verify the created key:

   Syntax

   ceph auth get client.ID

   Example

   [ceph: root@host01 /]# ceph auth get client.1
   
   client.1
     key = AQBSdFhcGZFUDRAAcKhG9Cl2HPiDMMRv4DC43A==
     caps mds = "allow r, allow rw path=/temp"
     caps mon = "allow r"
     caps osd = "allow rw tag cephfs data=cephfs_a"

4. Copy the keyring to the client.

   1. On the Ceph Monitor node, export the keyring to a file:

      Syntax

      ceph auth get client.ID -o ceph.client.ID.keyring

      Example

      [ceph: root@host01 /]# ceph auth get client.1 -o ceph.client.1.keyring
      exported keyring for client.1

   2. Copy the client keyring from the Ceph Monitor node to the /etc/ceph/ directory on the client node:

      Syntax

      scp /ceph.client.ID.keyring root@CLIENT_NODE_NAME:/etc/ceph/ceph.client.ID.keyring

      Replace CLIENT_NODE_NAME with the Ceph client node name or IP.

      Example

      [ceph: root@host01 /]# scp /ceph.client.1.keyring root@client01:/etc/ceph/ceph.client.1.keyring

5. From the client node, set the appropriate permissions for the keyring file:

   Syntax

   chmod 644 ceph.client.ID.keyring

   Example

   [root@client01 ~]# chmod 644 /etc/ceph/ceph.client.1.keyring

Procedure

1. Configure the client node to use the Ceph storage cluster.

   1. Enable the Red Hat Ceph Storage Tools repository:

      Red Hat Enterprise Linux 8

      [root@client ~]# subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms

      Red Hat Enterprise Linux 9

      [root@client ~]# subscription-manager repos --enable=rhceph-5-tools-for-rhel-9-x86_64-rpms

   2. Install the ceph-common package:

      [root@client01 ~]# dnf install ceph-common

   3. Log into the Cephadm shell on the monitor node:

      Example

      [root@host01 ~]# cephadm shell

   4. Copy the Ceph client keyring from the Ceph Monitor node to the client node:

      Syntax

      scp /ceph.client.ID.keyring root@CLIENT_NODE_NAME:/etc/ceph/ceph.client.ID.keyring

      Replace CLIENT_NODE_NAME with the Ceph client host name or IP address.

      Example

      [ceph: root@host01 /]# scp /ceph.client.1.keyring root@client01:/etc/ceph/ceph.client.1.keyring

   5. Copy the Ceph configuration file from a Ceph Monitor node to the client node:

      Syntax

      scp /etc/ceph/ceph.conf root@CLIENT_NODE_NAME:/etc/ceph/ceph.conf

      Replace CLIENT_NODE_NAME with the Ceph client host name or IP address.

      Example

      [ceph: root@host01 /]# scp /etc/ceph/ceph.conf root@client01:/etc/ceph/ceph.conf

   6. From the client node, set the appropriate permissions for the configuration file:

      [root@client01 ~]# chmod 644 /etc/ceph/ceph.conf

   7. Choose either automatically or manually mounting.

Manually Mounting

2. Create a mount directory on the client node:

   Syntax

   mkdir -p MOUNT_POINT

   Example

   [root@client01 ~]# mkdir -p /mnt/cephfs

3. Mount the Ceph File System. To specify multiple Ceph Monitor addresses, separate them with commas in the mount command, specify the mount point, and set the client name:

   Note

   As of Red Hat Ceph Storage 4.1, mount.ceph can read keyring files directly. As such, a secret file is no longer necessary. Just specify the client ID with name=CLIENT_ID, and mount.ceph will find the right keyring file.

   Syntax

   mount -t ceph MONITOR-1_NAME:6789,MONITOR-2_NAME:6789,MONITOR-3_NAME:6789:/ MOUNT_POINT -o name=CLIENT_ID,fs=FILE_SYSTEM_NAME

   Example

   [root@client01 ~]# mount -t ceph mon1:6789,mon2:6789,mon3:6789:/ /mnt/cephfs -o name=1,fs=cephfs01

   Note

   You can configure a DNS server so that a single host name resolves to multiple IP addresses. Then you can use that single host name with the mount command, instead of supplying a comma-separated list.

   Note

   You can also replace the Monitor host names with the string :/ and mount.ceph will read the Ceph configuration file to determine which Monitors to connect to.

   Note

   You can set the nowsync option to asynchronously execute file creation and removal on the Red Hat Ceph Storage clusters. This improves the performance of some workloads by avoiding round-trip latency for these system calls without impacting consistency. The nowsync option requires kernel clients with Red Hat Enterprise Linux 8.4 or later.

   Example

   [root@client01 ~]# mount -t ceph mon1:6789,mon2:6789,mon3:6789:/ /mnt/cephfs -o nowsync,name=1,fs=cephfs01

4. Verify that the file system is successfully mounted:

   Syntax

   stat -f MOUNT_POINT

   Example

   [root@client01 ~]# stat -f /mnt/cephfs

Automatically Mounting

2. On the client host, create a new directory for mounting the Ceph File System.

   Syntax

   mkdir -p MOUNT_POINT

   Example

   [root@client01 ~]# mkdir -p /mnt/cephfs

3. Edit the /etc/fstab file as follows:

   Syntax

   #DEVICE                                              PATH                      TYPE               OPTIONS
   MON_0_HOST:PORT,            MOUNT_POINT          ceph         name=CLIENT_ID,
   MON_1_HOST:PORT,                                                         ceph.client_mountpoint=/VOL/SUB_VOL_GROUP/SUB_VOL/UID_SUB_VOL,                                                       fs=FILE_SYSTEM_NAME,
   MON_2_HOST:PORT:/q[_VOL_]/SUB_VOL/UID_SUB_VOL,            [ADDITIONAL_OPTIONS]

   The first column sets the Ceph Monitor host names and the port number.

   The second column sets the mount point

   The third column sets the file system type, in this case, ceph, for CephFS.

   The fourth column sets the various options, such as, the user name and the secret file using the name and secretfile options. You can also set specific volumes, sub-volume groups, and sub-volumes using the ceph.client_mountpoint option.

   Set the _netdev option to ensure that the file system is mounted after the networking subsystem starts to prevent hanging and networking issues. If you do not need access time information, then setting the noatime option can increase performance.

   Set the fifth and sixth columns to zero.

   Example

   #DEVICE         PATH                   TYPE    OPTIONS         DUMP  FSCK
   mon1:6789,      /mnt/cephfs            ceph    name=1,            0     0
   mon2:6789,                                     ceph.client_mountpoint=/my_vol/my_sub_vol_group/my_sub_vol/0,
   mon3:6789:/                                    fs=cephfs01,
                                                  _netdev,noatime

   The Ceph File System will be mounted on the next system boot.

   Note

   As of Red Hat Ceph Storage 4.1, mount.ceph can read keyring files directly. As such, a secret file is no longer necessary. Just specify the client ID with name=CLIENT_ID, and mount.ceph will find the right keyring file.

   Note

   You can also replace the Monitor host names with the string :/ and mount.ceph will read the Ceph configuration file to determine which Monitors to connect to.

Procedure

1. Configure the client node to use the Ceph storage cluster.

   1. Enable the Red Hat Ceph Storage Tools repository:

      Red Hat Enterprise Linux 8

      [root@client ~]# subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms

      Red Hat Enterprise Linux 9

      [root@client ~]# subscription-manager repos --enable=rhceph-5-tools-for-rhel-9-x86_64-rpms

   2. Install the ceph-fuse package:

      [root@client01 ~]# dnf install ceph-fuse

   3. Log into the Cephadm shell on the monitor node:

      Example

      [root@host01 ~]# cephadm shell

   4. Copy the Ceph client keyring from the Ceph Monitor node to the client node:

      Syntax

      scp /ceph.client.ID.keyring root@CLIENT_NODE_NAME:/etc/ceph/ceph.client.ID.keyring

      Replace CLIENT_NODE_NAME with the Ceph client host name or IP address.

      Example

      [ceph: root@host01 /]# scp /ceph.client.1.keyring root@client01:/etc/ceph/ceph.client.1.keyring

   5. Copy the Ceph configuration file from a Ceph Monitor node to the client node:

      Syntax

      scp /etc/ceph/ceph.conf root@CLIENT_NODE_NAME:/etc/ceph/ceph.conf

      Replace CLIENT_NODE_NAME with the Ceph client host name or IP address.

      Example

      [ceph: root@host01 /]# scp /etc/ceph/ceph.conf root@client01:/etc/ceph/ceph.conf

   6. From the client node, set the appropriate permissions for the configuration file:

      [root@client01 ~]# chmod 644 /etc/ceph/ceph.conf

   7. Choose either automatically or manually mounting.

Manually Mounting

2. On the client node, create a directory for the mount point:

   Syntax

   mkdir PATH_TO_MOUNT_POINT

   Example

   [root@client01 ~]# mkdir /mnt/mycephfs

   Note

   If you used the path option with MDS capabilities, then the mount point must be within what is specified by the path.

3. Use the ceph-fuse utility to mount the Ceph File System.

   Syntax

   ceph-fuse -n client.CLIENT_ID --client_fs FILE_SYSTEM_NAME MOUNT_POINT

   Example

   [root@client01 ~]# ceph-fuse -n client.1 --client_fs cephfs01 /mnt/mycephfs

   Note

   If you do not use the default name and location of the user keyring, that is /etc/ceph/ceph.client.CLIENT_ID.keyring, then use the --keyring option to specify the path to the user keyring, for example:

   Example

   [root@client01 ~]# ceph-fuse -n client.1 --keyring=/etc/ceph/client.1.keyring /mnt/mycephfs

   Note

   Use the -r option to instruct the client to treat that path as its root:

   Syntax

   ceph-fuse -n client.CLIENT_ID MOUNT_POINT -r PATH

   Example

   [root@client01 ~]# ceph-fuse -n client.1 /mnt/cephfs -r /home/cephfs

   Note

   If you want to automatically reconnect an evicted Ceph client, then add the --client_reconnect_stale=true option.

   Example

   [root@client01 ~]# ceph-fuse -n client.1 /mnt/cephfs --client_reconnect_stale=true

4. Verify that the file system is successfully mounted:

   Syntax

   stat -f MOUNT_POINT

   Example

   [root@client01 ~]# stat -f /mnt/cephfs

Automatically Mounting

2. On the client node, create a directory for the mount point:

   Syntax

   mkdir PATH_TO_MOUNT_POINT

   Example

   [root@client01 ~]# mkdir /mnt/mycephfs

   Note

   If you used the path option with MDS capabilities, then the mount point must be within what is specified by the path.

3. Edit the /etc/fstab file as follows:

   Syntax

   #DEVICE                 PATH           TYPE          OPTIONS                  DUMP  FSCK
   HOST_NAME:PORT,     MOUNT_POINT fuse.ceph     ceph.id=CLIENT_ID,        0     0
   HOST_NAME:PORT,                                  ceph.client_mountpoint=/VOL/SUB_VOL_GROUP/SUB_VOL/UID_SUB_VOL,
   HOST_NAME:PORT:/                                 ceph.client_fs=FILE_SYSTEM_NAME,ceph.name=USERNAME,ceph.keyring=/etc/ceph/KEYRING_FILE,
                                                        [ADDITIONAL_OPTIONS]

   The first column sets the Ceph Monitor host names and the port number.

   The second column sets the mount point

   The third column sets the file system type, in this case, fuse.ceph, for CephFS.

   The fourth column sets the various options, such as the user name and the keyring using the ceph.name and ceph.keyring options. You can also set specific volumes, sub-volume groups, and sub-volumes using the ceph.client_mountpoint option. To specify which Ceph File System to access, use the ceph.client_fs option. Set the _netdev option to ensure that the file system is mounted after the networking subsystem starts to prevent hanging and networking issues. If you do not need access time information, then setting the noatime option can increase performance. If you want to automatically reconnect after an eviction, then set the client_reconnect_stale=true option.

   Set the fifth and sixth columns to zero.

   Example

   #DEVICE         PATH              TYPE         OPTIONS         DUMP  FSCK
   mon1:6789,      /mnt/mycephfs       fuse.ceph    ceph.id=1,         0     0
   mon2:6789,                                     ceph.client_mountpoint=/my_vol/my_sub_vol_group/my_sub_vol/0,
   mon3:6789:/                                    ceph.client_fs=cephfs01,ceph.name=client.1,ceph.keyring=/etc/ceph/client1.keyring,
                                                  _netdev,defaults

   The Ceph File System will be mounted on the next system boot.

Procedure

1. Enable the Red Hat Ceph Storage 5 tools repository, if it is not already enabled:

   Red Hat Enterprise Linux 8

   [root@client ~]# subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms

   Red Hat Enterprise Linux 9

   [root@client ~]# subscription-manager repos --enable=rhceph-5-tools-for-rhel-9-x86_64-rpms

2. Install the cephfs-top package:

   Example

   [root@client ~]# dnf install cephfs-top

3. Enable the Ceph Manager stats plugin:

   Example

   [root@client ~]# ceph mgr module enable stats

4. Create the client.fstop Ceph user:

   Example

   [root@client ~]# ceph auth get-or-create client.fstop mon 'allow r' mds 'allow r' osd 'allow r' mgr 'allow r' > /etc/ceph/ceph.client.fstop.keyring

   Note

   Optionally, use the --id argument to specify a different Ceph user, other than client.fstop.

5. Start the cephfs-top utility:

   Example

   [root@client ~]# cephfs-top
   cephfs-top - Thu Jun 15 16:27:14 2023
   
   All Filesystem Info
   Total Client(s): 4 - 3 FUSE, 1 kclient, 0 libcephfs
   COMMANDS: m - select a filesystem | s - sort menu | l - limit number of clients | r -reset to default | q - quit
   
   
      client_id  mount_root  chit(%)  dlease(%)  ofiles  oicaps  oinodes  rtio(MB)  raio(MB)  rsp(ms)  wtio(ms)  waio(MB)  wsp(MB/s)  rlatavg(ms)  rlatsd(ms)  wlatavg(ms)  wlatsd(ms)  mlatavg(ms)  mlatsd(ms)  mount_point@host/addr
   
      Filesystem: cephfs1 - 2 clients(s)
   
      4482            /       100.0       0.0       0       1       0       0.0      0.0        0.0      0.0       0.0        0.0        0.0         0.0      0.0            0.0           27.36        0.0       N/A@example/192.168.1.4
      4488            /       100.0       0.0       0       1       0       0.0      0.0        0.0      0.0       0.0        0.0        0.0         0.0      0.0            0.0           0.34        0.0        /mnt/cephfs2@example/192.168.1.4
   
      Filesystem: cephfs2 - 2 clients(s)
   
      4494            /       100.0       0.0       0       1       0       0.0      0.0        0.0      0.0       0.0        0.0        0.0         0.0      0.0            0.0           0.37        0.0        /mnt/cephfs3@example/192.168.1.4
      4500            /       100.0       0.0       0       1       0       0.0      0.0        0.0      0.0       0.0        0.0        0.0         0.0      0.0            0.0           0.29        0.0        /mnt/cephfs4@example/192.168.1.4

   Note

   By default, cephfs-top connects to the storage cluster name ceph. To use a non-default storage cluster name, you can use the --cluster NAME option with the cephfs-top utility.

Procedure

1. Add the p flag to the Ceph user’s capabilities:

   Syntax

   ceph fs authorize FILE_SYSTEM_NAME client.CLIENT_NAME /DIRECTORY CAPABILITY [/DIRECTORY CAPABILITY] ...

   Example

   [user@client ~]$ ceph fs authorize cephfs_a client.1 /temp rwp
   
   client.1
     key: AQBSdFhcGZFUDRAAcKhG9Cl2HPiDMMRv4DC43A==
     caps: [mds] allow r, allow rwp path=/temp
     caps: [mon] allow r
     caps: [osd] allow rw tag cephfs data=cephfs_a

2. Set the ceph.dir.pin extended attribute on a directory:

   Syntax

   setfattr -n ceph.dir.pin -v RANK DIRECTORY

   Example

   [user@client ~]$ setfattr -n ceph.dir.pin -v 2 /temp

   This example assigns the /temp directory and all of its subdirectories to rank 2.

Procedure

1. Create a new data pool:

   Syntax

   ceph osd pool create POOL_NAME

   Replace:

   • POOL_NAME with the name of the pool.

   Example

   [ceph: root@host01 /]# ceph osd pool create cephfs_data_ssd
   
   pool 'cephfs_data_ssd' created

2. Add the newly created pool under the control of the Metadata Servers:

   Syntax

   ceph fs add_data_pool FS_NAME POOL_NAME

   Replace:

   • FS_NAME with the name of the file system.
   • POOL_NAME with the name of the pool.

   Example:

   [ceph: root@host01 /]# ceph fs add_data_pool cephfs cephfs_data_ssd
   
   added data pool 6 to fsmap

3. Verify that the pool was successfully added:

   Example

   [ceph: root@host01 /]# ceph fs ls
   
   name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs.cephfs.data cephfs_data_ssd]

4. Optional: Remove a data pool from the file system:

   Syntax

   ceph fs rm_data_pool FS_NAME POOL_NAME

   Example:

   [ceph: root@host01 /]# ceph fs rm_data_pool cephfs cephfs_data_ssd
   
   removed data pool 6 from fsmap

   1. Verify that the pool was successfully removed:

      Example

      [ceph: root@host01 /]# ceph fs ls
      
      name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs.cephfs.data]

5. If you use the cephx authentication, make sure that clients can access the new pool.

Procedure

1. To mark the CephFS cluster down:

   Syntax

   ceph fs set FS_NAME down true

   Example

   [ceph: root@host01 /]# ceph fs set cephfs down true

   1. To bring the CephFS cluster back up:

      Syntax

      ceph fs set FS_NAME down false

      Example

      [ceph: root@host01 /]# ceph fs set cephfs down false

   or

1. To quickly take down a CephFS cluster:

   Syntax

   ceph fs fail FS_NAME

   Example

   [ceph: root@host01 /]# ceph fs fail cephfs

   Note

   To get the CephFS cluster back up, set cephfs to joinable:

   Syntax

   ceph fs set FS_NAME joinable true

   Example

   [ceph: root@host01 /]# ceph fs set cephfs joinable true
   cephfs marked joinable; MDS may join as newly active.

Procedure

1. Mark the storage cluster as down:

   Syntax

   ceph fs set FS_NAME down true

   Replace

   • FS_NAME with the name of the Ceph File System you want to remove.

   Example

   [ceph: root@host01 /]# ceph fs set cephfs down true
   cephfs marked down.

2. Display the status of the Ceph File System:

   ceph fs status

   Example

   [ceph: root@host01 /]# ceph fs status
   cephfs - 0 clients
   ======
   +-------------------+----------+-------+-------+
   |       POOL        |   TYPE   |  USED | AVAIL |
   +-----------------+------------+-------+-------+
   |cephfs.cephfs.meta | metadata | 31.5M |  52.6G|
   |cephfs.cephfs.data |   data   |    0  |  52.6G|
   +-----------------+----------+-------+---------+
                  STANDBY MDS
   cephfs.ceph-host01
   cephfs.ceph-host02
   cephfs.ceph-host03

3. Remove the Ceph File System:

   Syntax

   ceph fs rm FS_NAME --yes-i-really-mean-it

   Replace

   • FS_NAME with the name of the Ceph File System you want to remove.

   Example

   [ceph: root@host01 /]# ceph fs rm cephfs --yes-i-really-mean-it

4. Verify that the file system has been successfully removed:

   [ceph: root@host01 /]# ceph fs ls

5. Optional. Remove data and metadata pools associated with the removed file system.

Procedure

1. Review the client list:

   Syntax

   ceph tell DAEMON_NAME client ls

   Example

   [ceph: root@host01 /]# ceph tell mds.0 client ls
   [
       {
           "id": 4305,
           "num_leases": 0,
           "num_caps": 3,
           "state": "open",
           "replay_requests": 0,
           "completed_requests": 0,
           "reconnecting": false,
           "inst": "client.4305 172.21.9.34:0/422650892",
           "client_metadata": {
               "ceph_sha1": "79f0367338897c8c6d9805eb8c9ad24af0dcd9c7",
               "ceph_version": "ceph version 16.2.8-65.el8cp (79f0367338897c8c6d9805eb8c9ad24af0dcd9c7)",
               "entity_id": "0",
               "hostname": "senta04",
               "mount_point": "/tmp/tmpcMpF1b/mnt.0",
               "pid": "29377",
               "root": "/"
           }
       }
   ]

2. Evict the specified CephFS client:

   Syntax

   ceph tell DAEMON_NAME client evict id=ID_NUMBER

   Example

   [ceph: root@host01 /]# ceph tell mds.0 client evict id=4305

Procedure

1. Review the blocklist:

   Example

   [ceph: root@host01 /]# ceph osd blocklist ls
   listed 1 entries
   127.0.0.1:0/3710147553 2022-05-09 11:32:24.716146

2. Remove the CephFS client from the blocklist:

   Syntax

   ceph osd blocklist rm CLIENT_NAME_OR_IP_ADDR

   Example

   [ceph: root@host01 /]# ceph osd blocklist rm 127.0.0.1:0/3710147553
   un-blocklisting 127.0.0.1:0/3710147553

3. Optionally, you can have kernel-based CephFS clients automatically reconnect when removing them from the blocklist. On the kernel-based CephFS client, set the following option to clean either when doing a manual mount, or automatically mounting with an entry in the /etc/fstab file:

   recover_session=clean

4. Optionally, you can have FUSE-based CephFS clients automatically reconnect when removing them from the blocklist. On the FUSE client, set the following option to true either when doing a manual mount, or automatically mounting with an entry in the /etc/fstab file:

   client_reconnect_stale=true

Procedure

1. Log into the Cephadm shell:

   Example

   [root@mds ~]# cephadm shell

2. Enable the Ceph Manager NFS module:

   Example

   [ceph: root@host01 /]# ceph mgr module enable nfs

3. Create an NFS Ganesha cluster:

   Syntax

   ceph nfs cluster create CLUSTER_NAME [PLACEMENT] [--ingress] [--virtual_ip IP_ADDRESS] [--ingress-mode {default|keepalive-only|haproxy-standard|haproxy-protocol}] [--port PORT]

   Example

   [ceph: root@host01 /]# ceph nfs cluster create nfs-cephfs "host01 host02"
   NFS Cluster Created Successfully

   In this example, the NFS Ganesha cluster name is nfs-cephfs and the daemon containers are deployed to host01, and host02.

   Important

   Red Hat only supports one NFS Ganesha daemon running per host.

4. Verify the NFS Ganesha cluster information:

   Syntax

   ceph nfs cluster info [CLUSTER_NAME]

   Example

   [ceph: root@host01 /]# ceph nfs cluster info nfs-cephfs
   {
       "nfs-cephfs": [
           {
               "hostname": "host01",
               "ip": "10.74.179.124",
               "port": 2049
           },
           {
               "hostname": "host02",
               "ip": "10.74.180.160",
               "port": 2049
           }
       ]
   }

   Note

   Specifying the CLUSTER_NAME is optional.

Procedure

1. Create a configuration file:

   Example

   [ceph: root@host01 /]# touch nfs-cephfs.conf

2. Enable logging in the configuration file with the following block:

   Example

   [ceph: root@host01 /]# vi nfs-cephfs.conf
   
   LOG {
    COMPONENTS {
        ALL = FULL_DEBUG;
    }
   }

3. Set the new configuration:

   Syntax

   ceph nfs cluster config set CLUSTER_NAME -i PATH_TO_CONFIG_FILE

   Example

   [ceph: root@host01 /]# ceph nfs cluster config set nfs-cephfs -i nfs-cephfs.conf
   
   NFS-Ganesha Config Set Successfully

4. View the customized NFS Ganesha configuration:

   Syntax

   ceph nfs cluster config get CLUSTER_NAME

   Example

   [ceph: root@host01 /]#  ceph nfs cluster config get nfs-cephfs
   
   LOG {
       COMPONENTS {
           ALL = FULL_DEBUG;
       }
   }

   This provides the output, if any, of the user-defined configuration.

5. Optional: If you want to remove the user-defined configuration, run the following command:

   Syntax

   ceph nfs cluster config reset CLUSTER_NAME

   Example

   [ceph: root@host01 /]# ceph nfs cluster config reset nfs-cephfs
   
   NFS-Ganesha Config Reset Successfully

Procedure

1. Create a CephFS export:

   • For Red Hat Ceph Storage 5.0, create the export as follows:

     Syntax

     ceph nfs export create cephfs FILE_SYSTEM_NAME CLUSTER_NAME BINDING [--readonly] [--path=PATH_WITHIN_CEPHFS]

     Example

     [ceph: root@host01 /]# ceph nfs export create cephfs cephfs01 nfs-cephfs /ceph  --path=/

   • For Red Hat Ceph Storage 5.1 and later, create the export as follows:

     Syntax

     ceph nfs export create cephfs CLUSTER_NAME BINDING FILE_SYSTEM_NAME [--readonly] [--path=PATH_WITHIN_CEPHFS]

     Example

     [ceph: root@host01 /]# ceph nfs export create cephfs nfs-cephfs /ceph cephfs01 --path=/

     In this example, the BINDING (/ceph) is the pseudo root path, which must be unique, and an absolute path.

     Note

     The --readonly option exports a path with the read-only permission, the default being read, and write permissions.

     Note

     The PATH_WITHIN_CEPHFS can be a subvolume. You can get the absolute subvolume path by using the following command:

     Syntax

     ceph fs subvolume getpath VOLUME_NAME SUBVOLUME_NAME [--group_name SUBVOLUME_GROUP_NAME]

     Example

     [ceph: root@host01 /]# ceph fs subvolume getpath cephfs sub0

2. View the export block based on the pseudo root name:

   Syntax

   ceph nfs export get CLUSTER_NAME BINDING

   Example

   [ceph: root@host01 /]# ceph nfs export get nfs-cephfs /ceph
   {
     "export_id": 1,
     "path": "/",
     "cluster_id": "nfs-cephfs",
     "pseudo": "/ceph",
     "access_type": "RW",
     "squash": "no_root_squash",
     "security_label": true,
     "protocols": [
       4
     ],
     "transports": [
       "TCP"
     ],
     "fsal": {
       "name": "CEPH",
       "user_id": "cephnfs11",
       "fs_name": "cephfs",
       "sec_label_xattr": ""
     },
     "clients": []
   }

3. List the NFS exports:

   Syntax

   ceph nfs export ls CLUSTER_NAME [--detailed]

   Example

   [ceph: root@host01 /]# ceph nfs export ls nfs-cephfs
   [
     "/ceph/"
   ]
   [ceph: root@host01 /]# ceph nfs export ls nfs-cephfs --detailed
   [
     {
       "export_id": 100,
       "path": "/",
       "cluster_id": "nfs-cephfs",
       "pseudo": "/ceph/",
       "access_type": "RW",
       "squash": "no_root_squash",
       "security_label": true,
       "protocols": [
         4
       ],
       "transports": [
         "TCP"
       ],
       "fsal": {
         "name": "CEPH",
         "user_id": "nfstest01",
         "fs_name": "cephfs",
         "sec_label_xattr": ""
       },
       "clients": []
     }
   ]

4. Get the information of the NFS export:

   Syntax

   ceph nfs export info CLUSTER_NAME [PSEUDO_PATH]

   Example

   [ceph: root@host01 /]# ceph nfs export info nfs-cephfs /ceph
   
   {
     "export_id": 1,
     "path": "/",
     "cluster_id": "nfs-cephfs",
     "pseudo": "/ceph",
     "access_type": "RW",
     "squash": "none",
     "security_label": true,
     "protocols": [
       4
     ],
     "transports": [
       "TCP"
     ],
     "fsal": {
       "name": "CEPH",
       "user_id": "nfs.nfs-cephfs.1",
       "fs_name": "cephfs"
     },
     "clients": []
   }

5. On a client host, mount the exported Ceph File System:

   Syntax

   mount -t nfs -o port=GANESHA_PORT HOST_NAME:BINDING LOCAL_MOUNT_POINT

   Example

   [root@client01 ~]# mount -t nfs -o port=2049 host01:/ceph/ /mnt/nfs-cephfs

   1. To automatically mount on boot, open and edit the /etc/fstab file by adding a new line:

      Syntax

      HOST_NAME:BINDING   LOCAL_MOUNT_POINT   nfs4   defaults,seclabel,vers=4.2,proto=tcp,port=2049  0  0

      Example

      host01:/ceph/   /mnt/nfs-cephfs   nfs4   defaults,seclabel,vers=4.2,proto=tcp,port=2049  0  0

6. On a client host, to mount a exported NFS Ceph File System created with an ingress service:

   Syntax

   mount -t nfs VIRTUAL_IP_ADDRESS:BINDING LOCAL_MOUNT_POINT

   • Replace VIRTUAL_IP_ADDRESS with --ingress --virtual-ip IP address used to create the NFS cluster.
   • Replace BINDING with the pseudo root path.

   • Replace LOCAL_MOUNT_POINT with the mount point to mount the export on.

     Example

     [root@client01 ~]# mount -t nfs 10.10.128.75:/nfs-cephfs /mnt

     This example mounts the export nfs-cephfs that exists on a NFS cluster created with --ingress --virtual-ip 10.10.128.75 on the mount point /mnt.

Procedure

1. View the export block based on the pseudo root name:

   Syntax

   ceph nfs export get CLUSTER_NAME BINDING

   Example

   [ceph: root@host01 /]# ceph nfs export get nfs-cephfs /ceph
   {
     "export_id": 1,
     "path": "/",
     "cluster_id": "nfs-cephfs",
     "pseudo": "/ceph",
     "access_type": "RO",
     "squash": "none",
     "security_label": true,
     "protocols": [
       4
     ],
     "transports": [
       "TCP"
     ],
     "fsal": {
       "name": "CEPH",
       "user_id": "cephnfs11",
       "fs_name": "cephfs",
       "sec_label_xattr": ""
     },
     "clients": []
   }

2. Export the configuration file:

   Example

   [ceph: root@host01 /]# ceph nfs export get nfs-cephfs /ceph > export.conf

3. Edit the export information:

   Syntax

   {
     "export_id": EXPORT_ID,
     "path": "/",
     "cluster_id": "CLUSTER_NAME",
     "pseudo": "CLUSTER_PSEUDO_PATH",
     "access_type": "RW/RO",
     "squash": "SQUASH",
     "security_label": SECURITY_LABEL,
     "protocols": [
       PROTOCOL_ID_
     ],
     "transports": [
       "TCP"
     ],
     "fsal": {
       "name": "NAME",
       "user_id": "USER_ID",
       "fs_name": "FILE_SYSTEM_NAME",
       "sec_label_xattr": ""
     },
     "clients": []
   }

   Example

   [ceph: root@host01 /]# vi export.conf
   
   {
     "export_id": 1,
     "path": "/",
     "cluster_id": "nfs-cephfs",
     "pseudo": "/ceph",
     "access_type": "RW",
     "squash": "none",
     "security_label": true,
     "protocols": [
       4
     ],
     "transports": [
       "TCP"
     ],
     "fsal": {
       "name": "CEPH",
       "user_id": "cephnfs11",
       "fs_name": "cephfs",
       "sec_label_xattr": ""
     },
     "clients": []
   }

   In the above example, the access_type is modified from RO to RW.

4. Apply the specification:

   Syntax

   ceph nfs export apply CLUSTER_NAME PATH_TO_EXPORT_FILE

   Example

   [ceph: root@host01 /]# ceph nfs export apply nfs-cephfs -i export.conf
   Added export /ceph

5. Get the updated export information:

   Syntax

   ceph nfs export get CLUSTER_NAME BINDING

   Example

   [ceph: root@host01 /]# ceph nfs export get nfs-cephfs /ceph
   {
     "export_id": 1,
     "path": "/",
     "cluster_id": "nfs-cephfs",
     "pseudo": "/ceph",
     "access_type": "RW",
     "squash": "none",
     "security_label": true,
     "protocols": [
       4
     ],
     "transports": [
       "TCP"
     ],
     "fsal": {
       "name": "CEPH",
       "user_id": "cephnfs11",
       "fs_name": "cephfs",
       "sec_label_xattr": ""
     },
     "clients": []
   }

Procedure

1. Create a custom file:

   Example

   [ceph: root@host01 /]# touch export_new.conf

2. Create an export using the custom file:

   Syntax

   EXPORT {
     Export_Id = EXPORT_ID;
     Transports = TCP/UDP;
     Path = PATH;
     Pseudo = PSEUDO_PATH;
     Protocols = NFS_PROTOCOLS;
     Access_Type = ACCESS_TYPE;
     Attr_Expiration_Time = EXPIRATION_TIME;
     Squash = SQUASH;
     FSAL {
       Name = NAME;
       Filesystem = "CEPH_FILE_SYSTEM_NAME";
       User_Id = "USER_ID";
     }
   }

   Example

   [ceph: root@host01 /]# cat export_new.conf
   EXPORT {
     Export_Id = 2;
     Transports = TCP;
     Path = /;
     Pseudo = /ceph1/;
     Protocols = 4;
     Access_Type = RW;
     Attr_Expiration_Time = 0;
     Squash = None;
     FSAL {
       Name = CEPH;
       Filesystem = "cephfs";
       User_Id = "nfs.nfs-cephfs.2";
     }
   }

3. Apply the specification:

   Syntax

   ceph nfs export apply CLUSTER_NAME -i PATH_TO_EXPORT_FILE

   Example

   [ceph: root@host01 /]# ceph nfs export apply nfs-cephfs -i new_export.conf
   Added export /ceph1

4. Get the updated export information:

   Syntax

   ceph nfs export get CLUSTER_NAME BINDING

   Example

   [ceph: root@host01 /]# ceph nfs export get nfs-cephfs /ceph1
   {
     "export_id": 1,
     "path": "/",
     "cluster_id": "nfs-cephfs",
     "pseudo": "/ceph1",
     "access_type": "RW",
     "squash": "None",
     "security_label": true,
     "protocols": [
       4
     ],
     "transports": [
       "TCP"
     ],
     "fsal": {
       "name": "CEPH",
       "user_id": "nfs.nfs-cephfs.2",
       "fs_name": "cephfs",
       "sec_label_xattr": ""
     },
     "clients": []
   }

Procedure

1. Log into the Cephadm shell:

   Example

   [root@mds ~]# cephadm shell

2. Remove an NFS Ganesha cluster:

   Syntax

   ceph nfs cluster rm CLUSTER_NAME

   Example

   [ceph: root@host01 /]# ceph nfs cluster rm nfs-cephfs

Procedure

1. To view CephFS quotas.

   1. Using a byte-limit quota:

      Syntax

      getfattr -n ceph.quota.max_bytes DIRECTORY

      Example

      [root@client ~]# getfattr -n ceph.quota.max_bytes /mnt/cephfs/
      getfattr: Removing leading '/' from absolute path names
      # file: mnt/cephfs/
      ceph.quota.max_bytes="100000000"

      In this example, 100000000 equals 100 MB.

   2. Using a file-limit quota:

      Syntax

      getfattr -n ceph.quota.max_files DIRECTORY

      Example

      [root@client ~]# getfattr -n ceph.quota.max_files /mnt/cephfs/
      getfattr: Removing leading '/' from absolute path names
      # file: mnt/cephfs/
      ceph.quota.max_files="10000"

      In this example, 10000 equals 10,000 files.

Procedure

1. To set CephFS quotas.

   1. Using a byte-limit quota:

      Syntax

      setfattr -n ceph.quota.max_bytes -v 100000000 DIRECTORY

      Example

      [root@client ~]# setfattr -n ceph.quota.max_bytes -v 100000000 /cephfs/

      In this example, 100000000 bytes equals 100 MB.

   2. Using a file-limit quota:

      Syntax

      setfattr -n ceph.quota.max_files -v 10000 DIRECTORY

      Example

      [root@client ~]# setfattr -n ceph.quota.max_files -v 10000 /cephfs/

      In this example, 10000 equals 10,000 files.

Procedure

1. To remove CephFS quotas.

   1. Using a byte-limit quota:

      Syntax

      setfattr -n ceph.quota.max_bytes -v 0 DIRECTORY

      Example

      [root@client ~]# setfattr -n ceph.quota.max_bytes -v 0 /mnt/cephfs/

   2. Using a file-limit quota:

      Syntax

      setfattr -n ceph.quota.max_files -v 0 DIRECTORY

      Example

      [root@client ~]# setfattr -n ceph.quota.max_files -v 0 /mnt/cephfs/

Procedure

1. To remove a layout from a directory:

   Syntax

   setfattr -x ceph.dir.layout DIRECTORY_PATH

   Example

   [user@client ~]$ setfattr -x ceph.dir.layout /home/cephfs

2. To remove the pool_namespace field:

   Syntax

   setfattr -x ceph.dir.layout.pool_namespace DIRECTORY_PATH

   Example

   [user@client ~]$ setfattr -x ceph.dir.layout.pool_namespace /home/cephfs

   Note

   The pool_namespace field is the only field you can remove separately.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@mds ~]# cephadm shell

2. For existing Ceph File Systems, enable the snapshotting feature:

   Syntax

   ceph fs set FILE_SYSTEM_NAME allow_new_snaps true

   Example

   [ceph: root@mds ~]# ceph fs set cephfs01 allow_new_snaps true

3. Create a new snapshot subdirectory under the .snap directory:

   Syntax

   mkdir NEW_DIRECTORY_PATH

   Example

   [ceph: root@mds ~]# mkdir /.snap/new-snaps

   This example creates the new-snaps subdirectory, and this informs the Ceph Metadata Server (MDS) to start making snapshots.

   1. To delete snapshots:

      Syntax

      rmdir NEW_DIRECTORY_PATH

      Example

      [ceph: root@mds ~]# rmdir /.snap/new-snaps

      Important

      Attempting to delete root-level snapshots, which might contain underlying snapshots, will fail.

Procedure

1. Log into the Cephadm shell on a Ceph Manager node:

   Example

   [root@host01 ~]# cephadm shell

2. Enable the snap_schedule module:

   Example

   [ceph: root@host01 /]# ceph mgr module enable snap_schedule

3. Log into the client node:

   Example

   [root@host02 ~]# cephadm shell

4. Add a new schedule for a Ceph File System:

   Syntax

   ceph fs snap-schedule add FILE_SYSTEM_VOLUME_PATH REPEAT_INTERVAL [START_TIME]

   Example

   [ceph: root@host02 /]# ceph fs snap-schedule add /cephfs_kernelf739cwtus2/pmo9axbwsi 1h 2022-06-27T21:50:00

   Note

   START_TIME is represented in ISO 8601 format.

   This example creates a snapshot schedule for the path /cephfs within the filesystem mycephfs, snapshotting every hour, and starts on 27 June 2022 9:50 PM.

5. Add a new retention policy for snapshots of a CephFS volume path:

   Syntax

   ceph fs snap-schedule retention add FILE_SYSTEM_VOLUME_PATH [COUNT_TIME_PERIOD_PAIR] TIME_PERIOD COUNT

   Example

   [ceph: root@host02 /]# ceph fs snap-schedule retention add /cephfs h 14 1
   [ceph: root@host02 /]# ceph fs snap-schedule retention add /cephfs d 4 2
   [ceph: root@host02 /]# ceph fs snap-schedule retention add /cephfs 14h4w 3

   1

   This example keeps 14 snapshots at least one hour apart.

   2

   This example keeps 4 snapshots at least one day apart.

   3

   This example keeps 14 hourly, and 4 weekly snapshots.

6. List the snapshot schedules to verify the new schedule is created.

   Syntax

   ceph fs snap-schedule list FILE_SYSTEM_VOLUME_PATH [--format=plain|json] [--recursive=true]

   Example

   [ceph: root@host02 /]# ceph fs snap-schedule list /cephfs --recursive=true

   This example lists all schedules in the directory tree.

7. Check the status of a snapshot schedule:

   Syntax

   ceph fs snap-schedule status FILE_SYSTEM_VOLUME_PATH [--format=plain|json]

   Example

   [ceph: root@host02 /]# ceph fs snap-schedule status /cephfs --format=json

   This example displays the status of the snapshot schedule for the CephFS /cephfs path in JSON format. The default format is plain text, if not specified.

Procedure

1. Get the subvolume path:

   Syntax

   ceph fs subvolume getpath VOLUME_NAME SUBVOLUME_NAME SUBVOLUME_GROUP_NAME

   Example

   [ceph: root@host02 /]# ceph fs subvolume getpath cephfs subvol_1 subvolgroup_1

2. Add a new schedule for a Ceph File System subvolume path:

   Syntax

   ceph fs snap-schedule add /.. SNAP_SCHEDULE [START_TIME] --fs CEPH_FILE_SYSTEM_NAME  --subvol SUBVOLUME_NAME

   Example

   [ceph: root@host02 /]# ceph fs snap-schedule add /cephfs_kernelf739cwtus2/pmo9axbwsi 1h 2022-06-27T21:50:00 --fs cephfs --subvol subvol_1
   Schedule set for path /..

   Note

   START_TIME is represented in ISO 8601 format.

   This example creates a snapshot schedule for the subvolume path, snapshotting every hour, and starts on 27 June 2022 9:50 PM.

3. Add a new retention policy for snapshot schedules of a CephFS subvolume:

   Syntax

   ceph fs snap-schedule retention add SUBVOLUME_VOLUME_PATH [COUNT_TIME_PERIOD_PAIR] TIME_PERIOD COUNT

   Example

   [ceph: root@host02 /]# ceph fs snap-schedule retention add /volumes/_nogroup/subvol_1/85a615da-e8fa-46c1-afc3-0eb8ae64a954/.. h 14 1
   [ceph: root@host02 /]# ceph fs snap-schedule retention add /volumes/_nogroup/subvol_1/85a615da-e8fa-46c1-afc3-0eb8ae64a954/.. d 4 2
   [ceph: root@host02 /]# ceph fs snap-schedule retention add /volumes/_nogroup/subvol_1/85a615da-e8fa-46c1-afc3-0eb8ae64a954/.. 14h4w 3
   
   Retention added to path /volumes/_nogroup/subvol_1/85a615da-e8fa-46c1-afc3-0eb8ae64a954/..

Procedure

1. Remove a specific snapshot schedule:

   Syntax

   ceph fs snap-schedule remove FILE_SYSTEM_VOLUME_PATH [REPEAT_INTERVAL] [START_TIME]

   Example

   [ceph: root@host02 /]# ceph fs snap-schedule remove /cephfs 4h 2022-05-16T14:00:00

   This example removes the specific snapshot schedule for the /cephfs volume, that is snapshotting every four hours, and started on 16 May 2022 2:00 PM.

2. Remove all snapshot schedules for a specific CephFS volume path:

   Syntax

   ceph fs snap-schedule remove FILE_SYSTEM_VOLUME_PATH

   Example

   [ceph: root@host02 /]# ceph fs snap-schedule remove /cephfs

   This example removes all the snapshot schedules for the /cephfs volume path.

Procedure

1. On the source storage cluster, deploy the CephFS mirroring daemon:

   Syntax

   ceph orch apply cephfs-mirror ["NODE_NAME"]

   Example

   [root@mon ~]# ceph orch apply cephfs-mirror "node1.example.com"
   Scheduled cephfs-mirror update...

   This command creates a Ceph user called, cephfs-mirror, and deploys the cephfs-mirror daemon on the given node.

2. On the target storage cluster, create a user for each CephFS peer:

   Syntax

   ceph fs authorize FILE_SYSTEM_NAME CLIENT_NAME / rwps

   Example

   [root@mon ~]# ceph fs authorize cephfs client.mirror_remote / rwps
   [client.mirror_remote]
           key = AQCjZ5Jg739AAxAAxduIKoTZbiFJ0lgose8luQ==

3. On the source storage cluster, enable the CephFS mirroring module:

   Example

   [root@mon ~]# ceph mgr module enable mirroring

4. On the source storage cluster, enable mirroring on a Ceph File System:

   Syntax

   ceph fs snapshot mirror enable FILE_SYSTEM_NAME

   Example

   [root@mon ~]# ceph fs snapshot mirror enable cephfs

   1. Optional. To disable snapshot mirroring, use the following command:

      Syntax

      ceph fs snapshot mirror disable FILE_SYSTEM_NAME

      Example

      [root@mon ~]# ceph fs snapshot mirror disable cephfs

      Warning

      Disabling snapshot mirroring on a file system removes the configured peers. You have to import the peers again by bootstrapping them.

5. Prepare the target peer storage cluster.

   1. On a target node, enable the mirroring Ceph Manager module:

      Example

      [root@mon ~]# ceph mgr module enable mirroring

   2. On the same target node, create the peer bootstrap:

      Syntax

      ceph fs snapshot mirror peer_bootstrap create FILE_SYSTEM_NAME CLIENT_NAME SITE_NAME

      The SITE_NAME is a user-defined string to identify the target storage cluster.

      Example

      [root@mon ~]# ceph fs snapshot mirror peer_bootstrap create cephfs client.mirror_remote remote-site
      {"token": "eyJmc2lkIjogIjBkZjE3MjE3LWRmY2QtNDAzMC05MDc5LTM2Nzk4NTVkNDJlZiIsICJmaWxlc3lzdGVtIjogImJhY2t1cF9mcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcGVlcl9ib290c3RyYXAiLCAic2l0ZV9uYW1lIjogInNpdGUtcmVtb3RlIiwgImtleSI6ICJBUUFhcDBCZ0xtRmpOeEFBVnNyZXozai9YYUV0T2UrbUJEZlJDZz09IiwgIm1vbl9ob3N0IjogIlt2MjoxOTIuMTY4LjAuNTo0MDkxOCx2MToxOTIuMTY4LjAuNTo0MDkxOV0ifQ=="}

      Copy the token string between the double quotes for use in the next step.

6. On the source storage cluster, import the bootstrap token from the target storage cluster:

   Syntax

   ceph fs snapshot mirror peer_bootstrap import FILE_SYSTEM_NAME TOKEN

   Example

   [root@mon ~]# ceph fs snapshot mirror peer_bootstrap import cephfs eyJmc2lkIjogIjBkZjE3MjE3LWRmY2QtNDAzMC05MDc5LTM2Nzk4NTVkNDJlZiIsICJmaWxlc3lzdGVtIjogImJhY2t1cF9mcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcGVlcl9ib290c3RyYXAiLCAic2l0ZV9uYW1lIjogInNpdGUtcmVtb3RlIiwgImtleSI6ICJBUUFhcDBCZ0xtRmpOeEFBVnNyZXozai9YYUV0T2UrbUJEZlJDZz09IiwgIm1vbl9ob3N0IjogIlt2MjoxOTIuMTY4LjAuNTo0MDkxOCx2MToxOTIuMTY4LjAuNTo0MDkxOV0ifQ==

7. On the source storage cluster, list the CephFS mirror peers:

   Syntax

   ceph fs snapshot mirror peer_list FILE_SYSTEM_NAME

   Example

   [root@mon ~]# ceph fs snapshot mirror peer_list cephfs

   1. Optional. To remove a snapshot peer, use the following command:

      Syntax

      ceph fs snapshot mirror peer_remove FILE_SYSTEM_NAME PEER_UUID

      Example

      [root@mon ~]# ceph fs snapshot mirror peer_remove cephfs a2dc7784-e7a1-4723-b103-03ee8d8768f8

      Note

      See the Viewing the mirror status for a Ceph File System link in the Additional Resources section of this procedure on how to find the peer UUID value.

8. On the source storage cluster, configure a directory for snapshot mirroring:

   Syntax

   ceph fs snapshot mirror add FILE_SYSTEM_NAME PATH

   Example

   [root@mon ~]# ceph fs snapshot mirror add cephfs /volumes/_nogroup/subvol_1

   Important

   Only absolute paths inside the Ceph File System are valid.

   Note

   The Ceph Manager mirroring module normalizes the path. For example, the /d1/d2/../dN directories are equivalent to /d1/d2. Once a directory has been added for mirroring, its ancestor directories and subdirectories are prevented from being added for mirroring.

   1. Optional. To stop snapshot mirroring for a directory, use the following command:

      Syntax

      ceph fs snapshot mirror remove FILE_SYSTEM_NAME PATH

      Example

      [root@mon ~]# ceph fs snapshot mirror remove cephfs /home/user1