Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Use one of the following methods to deploy the daemons on the hosts:

   • Method 1: Specify the number of daemons and the host names:

     Syntax

     ceph orch apply SERVICE_NAME --placement="NUMBER_OF_DAEMONS HOST_NAME_1 HOST_NAME_2 HOST_NAME_3"

     Example

     [ceph: root@host01 /]# ceph orch apply mon --placement="3 host01 host02 host03"

   • Method 2: Add the labels to the hosts and then deploy the daemons using the labels:

     1. Add the labels to the hosts:

        Syntax

        ceph orch host label add HOSTNAME_1 LABEL

        Example

        [ceph: root@host01 /]# ceph orch host label add host01 mon

     2. Deploy the daemons with labels:

        Syntax

        ceph orch apply DAEMON_NAME label:LABEL

        Example

        ceph orch apply mon label:mon

   • Method 3: Add the labels to the hosts and deploy using the --placement argument:

     1. Add the labels to the hosts:

        Syntax

        ceph orch host label add HOSTNAME_1 LABEL

        Example

        [ceph: root@host01 /]# ceph orch host label add host01 mon

     2. Deploy the daemons using the label placement specification:

        Syntax

        ceph orch apply DAEMON_NAME --placement="label:LABEL"

        Example

        ceph orch apply mon --placement="label:mon"

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. List the hosts on which you want to deploy the Ceph daemons:

   Example

   [ceph: root@host01 /]# ceph orch host ls

3. Deploy the daemons:

   Syntax

   ceph orch apply SERVICE_NAME --placement="NUMBER_OF_DAEMONS HOST_NAME_1 _HOST_NAME_2 HOST_NAME_3"

   Example

   ceph orch apply mgr --placement="2 host01 host02 host03"

   In this example, the mgr daemons are deployed only on two hosts.

Procedure

1. Create the yaml file:

   Example

   [root@host01 ~]# touch mon.yaml

2. This file can be configured in two different ways:

   • Edit the file to include the host details in placement specification:

     Syntax

     service_type: SERVICE_NAME
     placement:
       hosts:
         - HOST_NAME_1
         - HOST_NAME_2

     Example

     service_type: mon
     placement:
       hosts:
         - host01
         - host02
         - host03

   • Edit the file to include the label details in placement specification:

     Syntax

     service_type: SERVICE_NAME
     placement:
       label: "LABEL_1"

     Example

     service_type: mon
     placement:
       label: "mon"

3. Optional: You can also use extra container arguments in the service specification files such as CPUs, CA certificates, and other files while deploying services:

   Example

   extra_container_args:
     - "-v"
     - "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro"
     - "--security-opt"
     - "label=disable"
     - "cpus=2"

4. Mount the YAML file under a directory in the container:

   Example

   [root@host01 ~]# cephadm shell --mount mon.yaml:/var/lib/ceph/mon/mon.yaml

5. Navigate to the directory:

   Example

   [ceph: root@host01 /]# cd /var/lib/ceph/mon/

6. Deploy the Ceph daemons using service specification:

   Syntax

   ceph orch apply -i FILE_NAME.yaml

   Example

   [ceph: root@host01 mon]# ceph orch apply -i mon.yaml

Procedure

1. From the Ceph administration node, log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Extract the cluster’s public SSH keys to a folder:

   Syntax

   ceph cephadm get-pub-key > ~/PATH

   Example

   [ceph: root@host01 /]# ceph cephadm get-pub-key > ~/ceph.pub

3. Copy Ceph cluster’s public SSH keys to the root user’s authorized_keys file on the new host:

   Syntax

   ssh-copy-id -f -i ~/PATH root@HOST_NAME_2

   Example

   [ceph: root@host01 /]# ssh-copy-id -f -i ~/ceph.pub root@host02

4. From the Ansible administration node, add the new host to the Ansible inventory file. The default location for the file is /usr/share/cephadm-ansible/hosts. The following example shows the structure of a typical inventory file:

   Example

   host01
   host02
   host03
   
   [admin]
   host00

   Note

   If you have previously added the new host to the Ansible inventory file and run the preflight playbook on the host, skip to step 6.

5. Run the preflight playbook with the --limit option:

   Syntax

   ansible-playbook -i INVENTORY_FILE cephadm-preflight.yml --extra-vars "ceph_origin=rhcs" --limit NEWHOST

   Example

   [ceph-admin@admin cephadm-ansible]$ ansible-playbook -i hosts cephadm-preflight.yml --extra-vars "ceph_origin=rhcs" --limit host02

   The preflight playbook installs podman, lvm2, chronyd, and cephadm on the new host. After installation is complete, cephadm resides in the /usr/sbin/ directory.

6. From the Ceph administration node, log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

7. Use the cephadm orchestrator to add hosts to the storage cluster:

   Syntax

   ceph orch host add HOST_NAME IP_ADDRESS_OF_HOST [--label=LABEL_NAME_1,LABEL_NAME_2]

   The --label option is optional and this adds the labels when adding the hosts. You can add multiple labels to the host.

   Example

   [ceph: root@host01 /]# ceph orch host add host02 10.10.128.70 --labels=mon,mgr

Procedure

1. Edit the hosts.yaml file to include the following details:

   Example

   service_type: host
   hostname: host01
   addr: 192.168.0.11
   location:
     rack: rack1

2. Mount the YAML file under a directory in the container:

   Example

   [root@host01 ~]# cephadm shell --mount hosts.yaml:/var/lib/ceph/hosts.yaml

3. Navigate to the directory:

   Example

   [ceph: root@host01 /]# cd /var/lib/ceph/

4. Deploy the hosts using service specification:

   Syntax

   ceph orch apply -i FILE_NAME.yaml

   Example

   [ceph: root@host01 ceph]# ceph orch apply -i hosts.yaml

Procedure

1. Create the hosts.yaml file:

   Example

   [root@host01 ~]# touch hosts.yaml

2. Edit the hosts.yaml file to include the following details:

   Example

   service_type: host
   addr: host01
   hostname: host01
   labels:
   - mon
   - osd
   - mgr
   ---
   service_type: host
   addr: host02
   hostname: host02
   labels:
   - mon
   - osd
   - mgr
   ---
   service_type: host
   addr: host03
   hostname: host03
   labels:
   - mon
   - osd

3. Mount the YAML file under a directory in the container:

   Example

   [root@host01 ~]# cephadm shell --mount hosts.yaml:/var/lib/ceph/hosts.yaml

4. Navigate to the directory:

   Example

   [ceph: root@host01 /]# cd /var/lib/ceph/

5. Deploy the hosts using service specification:

   Syntax

   ceph orch apply -i FILE_NAME.yaml

   Example

   [ceph: root@host01 hosts]# ceph orch apply -i hosts.yaml

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. List the hosts of the cluster:

   Example

   [ceph: root@host01 /]# ceph orch host ls

   You will see that the STATUS of the hosts is blank which is expected.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Add labels to the hosts:

   Syntax

   ceph orch host label add HOST_NAME LABEL_NAME

   Example

   [ceph: root@host01 /]# ceph orch host label add host02 mon

Procedure

1. Launch the cephadm shell:

   [root@host01 ~]# cephadm shell
   [ceph: root@host01 /]#

2. Remove the label.

   Syntax

   ceph orch host label rm HOSTNAME LABEL

   Example

   [ceph: root@host01 /]# ceph orch host label rm host02 mon

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Fetch the host details:

   Example

   [ceph: root@host01 /]# ceph orch host ls

3. Drain all the daemons from the host:

   Syntax

   ceph orch host drain HOSTNAME

   Example

   [ceph: root@host01 /]# ceph orch host drain host02

   The _no_schedule label is automatically applied to the host which blocks deployment.

4. Check the status of OSD removal:

   Example

   [ceph: root@host01 /]# ceph orch osd rm status

   When no placement groups (PG) are left on the OSD, the OSD is decommissioned and removed from the storage cluster.

5. Check if all the daemons are removed from the storage cluster:

   Syntax

   ceph orch ps HOSTNAME

   Example

   [ceph: root@host01 /]# ceph orch ps host02

6. Remove the host:

   Syntax

   ceph orch host rm HOSTNAME

   Example

   [ceph: root@host01 /]# ceph orch host rm host02

1. Confirms the removal of hosts does not impact data availability by running the orch host ok-to-stop command.
2. If the host has Ceph OSD daemons, it applies noout to the host subtree to prevent data migration from triggering during the planned maintenance slot.
3. Stops the Ceph target, thereby, stopping all the daemons.
4. Disables the ceph target on the host, to prevent a reboot from automatically starting Ceph services.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. You can either place the host in maintenance mode or place it out of the maintenance mode:

   • Place the host in maintenance mode:

     Syntax

     ceph orch host maintenance enter HOST_NAME [--force]

     Example

     [ceph: root@host01 /]# ceph orch host maintenance enter host02 --force

     The --force flag allows the user to bypass warnings, but not alerts.

   • Place the host out of the maintenance mode:

     Syntax

     ceph orch host maintenance exit HOST_NAME

     Example

     [ceph: root@host01 /]# ceph orch host maintenance exit host02

1. classic - This is the default mode in which the lowest ranked monitor is voted based on the elector module between the two sites.
2. disallow - This mode lets you mark monitors as disallowed, in which case they will participate in the quorum and serve clients, but cannot be an elected leader. This lets you add monitors to a list of disallowed leaders. If a monitor is in the disallowed list, it will always defer to another monitor.
3. connectivity - This mode is mainly used to resolve network discrepancies. It evaluates connection scores, based on pings that check liveness, provided by each monitor for its peers and elects the most connected and reliable monitor to be the leader. This mode is designed to handle net splits, which may happen if your cluster is stretched across multiple data centers or otherwise susceptible. This mode incorporates connection score ratings and elects the monitor with the best score. If a specific monitor is desired to be the leader, configure the election strategy so that the specific monitor is the first monitor in the list with a rank is 0.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. There are four different ways of deploying Ceph monitor daemons:

Procedure

1. Create the mon.yaml file:

   Example

   [root@host01 ~]# touch mon.yaml

2. Edit the mon.yaml file to include the following details:

   Syntax

   service_type: mon
   placement:
     hosts:
       - HOST_NAME_1
       - HOST_NAME_2

   Example

   service_type: mon
   placement:
     hosts:
       - host01
       - host02

3. Mount the YAML file under a directory in the container:

   Example

   [root@host01 ~]# cephadm shell --mount mon.yaml:/var/lib/ceph/mon/mon.yaml

4. Navigate to the directory:

   Example

   [ceph: root@host01 /]# cd /var/lib/ceph/mon/

5. Deploy the monitor daemons:

   Syntax

   ceph orch apply -i FILE_NAME.yaml

   Example

   [ceph: root@host01 mon]# ceph orch apply -i mon.yaml

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Disable automated monitor deployment:

   Example

   [ceph: root@host01 /]# ceph orch apply mon --unmanaged

3. Deploy monitors on hosts on specific network:

   Syntax

   ceph orch daemon add mon HOST_NAME_1:IP_OR_NETWORK

   Example

   [ceph: root@host01 /]# ceph orch daemon add mon host03:10.1.2.123

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Run the ceph orch apply command to deploy the required monitor daemons:

   Syntax

   ceph orch apply mon “NUMBER_OF_DAEMONS HOST_NAME_1 HOST_NAME_3”

   If you want to remove monitor daemons from host02, then you can redeploy the monitors on other hosts.

   Example

   [ceph: root@host01 /]# ceph orch apply mon “2 host01 host03”

Procedure

1. Identify a surviving monitor and log into the host:

   Syntax

   ssh root@MONITOR_ID

   Example

   [root@admin ~]# ssh root@host00

2. Log in to each Ceph Monitor host and stop all the Ceph Monitors:

   Syntax

   cephadm unit --name DAEMON_NAME.HOSTNAME stop

   Example

   [root@host00 ~]# cephadm unit --name mon.host00 stop

3. Set up the environment suitable for extended daemon maintenance and to run the daemon interactively:

   Syntax

   cephadm shell --name DAEMON_NAME.HOSTNAME

   Example

   [root@host00 ~]# cephadm shell --name mon.host00

4. Extract a copy of the monmap file:

   Syntax

   ceph-mon -i HOSTNAME --extract-monmap TEMP_PATH

   Example

   [ceph: root@host00 /]# ceph-mon -i host01 --extract-monmap /tmp/monmap
   
   2022-01-05T11:13:24.440+0000 7f7603bd1700 -1 wrote monmap to /tmp/monmap

5. Remove the non-surviving Ceph Monitor(s):

   Syntax

   monmaptool TEMPORARY_PATH --rm HOSTNAME

   Example

   [ceph: root@host00 /]# monmaptool /tmp/monmap --rm host01

6. Inject the surviving monitor map with the removed monitor(s) into the surviving Ceph Monitor:

   Syntax

   ceph-mon -i HOSTNAME --inject-monmap TEMP_PATH

   Example

   [ceph: root@host00 /]# ceph-mon -i host00 --inject-monmap /tmp/monmap

7. Start only the surviving monitors:

   Syntax

   cephadm unit --name DAEMON_NAME.HOSTNAME start

   Example

   [root@host00 ~]# cephadm unit --name mon.host00 start

8. Verify the monitors form a quorum:

   Example

   [ceph: root@host00 /]# ceph -s

9. Optional: Archive the removed Ceph Monitor’s data directory in /var/lib/ceph/CLUSTER_FSID/mon.HOSTNAME directory.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. You can deploy manager daemons in two different ways:

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Run the ceph orch apply command to redeploy the required manager daemons:

   Syntax

   ceph orch apply mgr "NUMBER_OF_DAEMONS HOST_NAME_1 HOST_NAME_3"

   If you want to remove manager daemons from host02, then you can redeploy the manager daemons on other hosts.

   Example

   [ceph: root@host01 /]# ceph orch apply mgr "2 host01 host03"

Procedure

1. Ensure the balancer module is enabled:

   Example

   [ceph: root@host01 /]# ceph mgr module enable balancer

2. Turn on the balancer module:

   Example

   [ceph: root@host01 /]# ceph balancer on

3. The default mode is upmap. The mode can be changed with:

   Example

   [ceph: root@host01 /]# ceph balancer mode crush-compact

   or

   Example

   [ceph: root@host01 /]# ceph balancer mode upmap

1. Building a plan.
2. Evaluating the quality of the data distribution, either for the current PG distribution, or the PG distribution that would result after executing a plan.

3. Executing the plan.

   • To evaluate and score the current distribution:

     Example

     [ceph: root@host01 /]# ceph balancer eval

   • To evaluate the distribution for a single pool:

     Syntax

     ceph balancer eval POOL_NAME

     Example

     [ceph: root@host01 /]# ceph balancer eval rbd

   • To see greater detail for the evaluation:

     Example

     [ceph: root@host01 /]# ceph balancer eval-verbose ...

   • To generate a plan using the currently configured mode:

     Syntax

     ceph balancer optimize PLAN_NAME

     Replace PLAN_NAME with a custom plan name.

     Example

     [ceph: root@host01 /]# ceph balancer optimize rbd_123

   • To see the contents of a plan:

     Syntax

     ceph balancer show PLAN_NAME

     Example

     [ceph: root@host01 /]# ceph balancer show rbd_123

   • To discard old plans:

     Syntax

     ceph balancer rm PLAN_NAME

     Example

     [ceph: root@host01 /]# ceph balancer rm rbd_123

   • To see currently recorded plans use the status command:

     [ceph: root@host01 /]# ceph balancer status

   • To calculate the quality of the distribution that would result after executing a plan:

     Syntax

     ceph balancer eval PLAN_NAME

     Example

     [ceph: root@host01 /]# ceph balancer eval rbd_123

   • To execute the plan:

     Syntax

     ceph balancer execute PLAN_NAME

     Example

     [ceph: root@host01 /]# ceph balancer execute rbd_123

     Note

     Only execute the plan if it is expected to improve the distribution. After execution, the plan will be discarded.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Enable the alerts module:

   Example

   [ceph: root@host01 /]# ceph mgr module enable alerts

3. Ensure the alerts module is enabled:

   Example

   [ceph: root@host01 /]# ceph mgr module ls | more
   {
       "always_on_modules": [
           "balancer",
           "crash",
           "devicehealth",
           "orchestrator",
           "pg_autoscaler",
           "progress",
           "rbd_support",
           "status",
           "telemetry",
           "volumes"
       ],
       "enabled_modules": [
           "alerts",
           "cephadm",
           "dashboard",
           "iostat",
           "nfs",
           "prometheus",
           "restful"
       ]

4. Configure the Simple Mail Transfer Protocol (SMTP):

   Syntax

   ceph config set mgr mgr/alerts/smtp_host SMTP_SERVER
   ceph config set mgr mgr/alerts/smtp_destination RECEIVER_EMAIL_ADDRESS
   ceph config set mgr mgr/alerts/smtp_sender SENDER_EMAIL_ADDRESS

   Example

   [ceph: root@host01 /]# ceph config set mgr mgr/alerts/smtp_host smtp.example.com
   [ceph: root@host01 /]# ceph config set mgr mgr/alerts/smtp_destination example@example.com
   [ceph: root@host01 /]# ceph config set mgr mgr/alerts/smtp_sender example2@example.com

5. Optional: Change the port to 465.

   Syntax

   ceph config set mgr mgr/alerts/smtp_port PORT_NUMBER

   Example

   [ceph: root@host01 /]# ceph config set mgr mgr/alerts/smtp_port 587

   Important

   SSL is not supported in Red Hat Ceph Storage 5 cluster. Do not set the smtp_ssl parameter while configuring alerts.

6. Authenticate to the SMTP server:

   Syntax

   ceph config set mgr mgr/alerts/smtp_user USERNAME
   ceph config set mgr mgr/alerts/smtp_password PASSWORD

   Example

   [ceph: root@host01 /]# ceph config set mgr mgr/alerts/smtp_user admin1234
   [ceph: root@host01 /]# ceph config set mgr mgr/alerts/smtp_password admin1234

7. Optional: By default, SMTP From name is Ceph. To change that, set the smtp_from_name parameter:

   Syntax

   ceph config set mgr mgr/alerts/smtp_from_name CLUSTER_NAME

   Example

   [ceph: root@host01 /]# ceph config set mgr mgr/alerts/smtp_from_name 'Ceph Cluster Test'

8. Optional: By default, the alerts module checks the storage cluster’s health every minute, and sends a message when there is a change in the cluster health status. To change the frequency, set the interval parameter:

   Syntax

   ceph config set mgr mgr/alerts/interval INTERVAL

   Example

   [ceph: root@host01 /]# ceph config set mgr mgr/alerts/interval "5m"

   In this example, the interval is set to 5 minutes.

9. Optional: Send an alert immediately:

   Example

   [ceph: root@host01 /]# ceph alerts send

Procedure

1. Ensure the crash module is enabled:

   Example

   [ceph: root@host01 /]# ceph mgr module ls | more
   {
       "always_on_modules": [
           "balancer",
           "crash",
           "devicehealth",
           "orchestrator_cli",
           "progress",
           "rbd_support",
           "status",
           "volumes"
       ],
       "enabled_modules": [
           "dashboard",
           "pg_autoscaler",
           "prometheus"
       ]

2. Save a crash dump: The metadata file is a JSON blob stored in the crash dir as meta. You can invoke the ceph command -i - option, which reads from stdin.

   Example

   [ceph: root@host01 /]# ceph crash post -i meta

3. List the timestamp or the UUID crash IDs for all the new and archived crash info:

   Example

   [ceph: root@host01 /]# ceph crash ls

4. List the timestamp or the UUID crash IDs for all the new crash information:

   Example

   [ceph: root@host01 /]# ceph crash ls-new

5. List the timestamp or the UUID crash IDs for all the new crash information:

   Example

   [ceph: root@host01 /]# ceph crash ls-new

6. List the summary of saved crash information grouped by age:

   Example

   [ceph: root@host01 /]# ceph crash stat
   8 crashes recorded
   8 older than 1 days old:
   2022-05-20T08:30:14.533316Z_4ea88673-8db6-4959-a8c6-0eea22d305c2
   2022-05-20T08:30:14.590789Z_30a8bb92-2147-4e0f-a58b-a12c2c73d4f5
   2022-05-20T08:34:42.278648Z_6a91a778-bce6-4ef3-a3fb-84c4276c8297
   2022-05-20T08:34:42.801268Z_e5f25c74-c381-46b1-bee3-63d891f9fc2d
   2022-05-20T08:34:42.803141Z_96adfc59-be3a-4a38-9981-e71ad3d55e47
   2022-05-20T08:34:42.830416Z_e45ed474-550c-44b3-b9bb-283e3f4cc1fe
   2022-05-24T19:58:42.549073Z_b2382865-ea89-4be2-b46f-9a59af7b7a2d
   2022-05-24T19:58:44.315282Z_1847afbc-f8a9-45da-94e8-5aef0738954e

7. View the details of the saved crash:

   Syntax

   ceph crash info CRASH_ID

   Example

   [ceph: root@host01 /]# ceph crash info 2022-05-24T19:58:42.549073Z_b2382865-ea89-4be2-b46f-9a59af7b7a2d
   {
       "assert_condition": "session_map.sessions.empty()",
       "assert_file": "/builddir/build/BUILD/ceph-16.1.0-486-g324d7073/src/mon/Monitor.cc",
       "assert_func": "virtual Monitor::~Monitor()",
       "assert_line": 287,
       "assert_msg": "/builddir/build/BUILD/ceph-16.1.0-486-g324d7073/src/mon/Monitor.cc: In function 'virtual Monitor::~Monitor()' thread 7f67a1aeb700 time 2022-05-24T19:58:42.545485+0000\n/builddir/build/BUILD/ceph-16.1.0-486-g324d7073/src/mon/Monitor.cc: 287: FAILED ceph_assert(session_map.sessions.empty())\n",
       "assert_thread_name": "ceph-mon",
       "backtrace": [
           "/lib64/libpthread.so.0(+0x12b30) [0x7f679678bb30]",
           "gsignal()",
           "abort()",
           "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f6798c8d37b]",
           "/usr/lib64/ceph/libceph-common.so.2(+0x276544) [0x7f6798c8d544]",
           "(Monitor::~Monitor()+0xe30) [0x561152ed3c80]",
           "(Monitor::~Monitor()+0xd) [0x561152ed3cdd]",
           "main()",
           "__libc_start_main()",
           "_start()"
       ],
       "ceph_version": "16.2.8-65.el8cp",
       "crash_id": "2022-07-06T19:58:42.549073Z_b2382865-ea89-4be2-b46f-9a59af7b7a2d",
       "entity_name": "mon.ceph-adm4",
       "os_id": "rhel",
       "os_name": "Red Hat Enterprise Linux",
       "os_version": "8.5 (Ootpa)",
       "os_version_id": "8.5",
       "process_name": "ceph-mon",
       "stack_sig": "957c21d558d0cba4cee9e8aaf9227b3b1b09738b8a4d2c9f4dc26d9233b0d511",
       "timestamp": "2022-07-06T19:58:42.549073Z",
       "utsname_hostname": "host02",
       "utsname_machine": "x86_64",
       "utsname_release": "4.18.0-240.15.1.el8_3.x86_64",
       "utsname_sysname": "Linux",
       "utsname_version": "#1 SMP Wed Jul 06 03:12:15 EDT 2022"
   }

8. Remove saved crashes older than KEEP days: Here, KEEP must be an integer.

   Syntax

   ceph crash prune KEEP

   Example

   [ceph: root@host01 /]# ceph crash prune 60

9. Archive a crash report so that it is no longer considered for the RECENT_CRASH health check and does not appear in the crash ls-new output. It appears in the crash ls.

   Syntax

   ceph crash archive CRASH_ID

   Example

   [ceph: root@host01 /]# ceph crash archive 2022-05-24T19:58:42.549073Z_b2382865-ea89-4be2-b46f-9a59af7b7a2d

10. Archive all crash reports:

    Example

    [ceph: root@host01 /]# ceph crash archive-all

11. Remove the crash dump:

    Syntax

    ceph crash rm CRASH_ID

    Example

    [ceph: root@host01 /]# ceph crash rm 2022-05-24T19:58:42.549073Z_b2382865-ea89-4be2-b46f-9a59af7b7a2d

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. List the available devices to deploy OSDs:

   Syntax

   ceph orch device ls [--hostname=HOSTNAME_1 HOSTNAME_2] [--wide] [--refresh]

   Example

   [ceph: root@host01 /]# ceph orch device ls --wide --refresh

   Using the --wide option provides all details relating to the device, including any reasons that the device might not be eligible for use as an OSD. This option does not support NVMe devices.

3. Optional: To enable Health, Ident, and Fault fields in the output of ceph orch device ls, run the following commands:

   Note

   These fields are supported by libstoragemgmt library and currently supports SCSI, SAS, and SATA devices.

   1. As root user outside the Cephadm shell, check your hardware’s compatibility with libstoragemgmt library to avoid unplanned interruption to services:

      Example

      [root@host01 ~]# cephadm shell lsmcli ldl

      In the output, you see the Health Status as Good with the respective SCSI VPD 0x83 ID.

      Note

      If you do not get this information, then enabling the fields might cause erratic behavior of devices.

   2. Log back into the Cephadm shell and enable libstoragemgmt support:

      Example

      [root@host01 ~]# cephadm shell
      [ceph: root@host01 /]# ceph config set mgr mgr/cephadm/device_enhanced_scan true

      Once this is enabled, ceph orch device ls gives the output of Health field as Good.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. List the available devices to deploy OSDs:

   Syntax

   ceph orch device ls [--hostname=HOSTNAME_1 HOSTNAME_2] [--wide] [--refresh]

   Example

   [ceph: root@host01 /]# ceph orch device ls --wide --refresh

3. Clear the data of a device:

   Syntax

   ceph orch device zap HOSTNAME FILE_PATH --force

   Example

   [ceph: root@host01 /]# ceph orch device zap host02 /dev/sdb --force

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. List the available devices to deploy OSDs:

   Syntax

   ceph orch device ls [--hostname=HOSTNAME_1 HOSTNAME_2] [--wide] [--refresh]

   Example

   [ceph: root@host01 /]# ceph orch device ls --wide --refresh

3. Deploy OSDs on all available devices:

   Example

   [ceph: root@host01 /]# ceph orch apply osd --all-available-devices

   The effect of ceph orch apply is persistent which means that the Orchestrator automatically finds the device, adds it to the cluster, and creates new OSDs. This occurs under the following conditions:

   • New disks or drives are added to the system.
   • Existing disks or drives are zapped.

   • An OSD is removed and the devices are zapped.

     You can disable automatic creation of OSDs on all the available devices by using the --unmanaged parameter.

     Example

     [ceph: root@host01 /]# ceph orch apply osd --all-available-devices --unmanaged=true

     Setting the parameter --unmanaged to true disables the creation of OSDs and also there is no change if you apply a new OSD service.

     Note

     The command ceph orch daemon add creates new OSDs, but does not add an OSD service.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. List the available devices to deploy OSDs:

   Syntax

   ceph orch device ls [--hostname=HOSTNAME_1 HOSTNAME_2] [--wide] [--refresh]

   Example

   [ceph: root@host01 /]# ceph orch device ls --wide --refresh

3. Deploy OSDs on specific devices and hosts:

   Syntax

   ceph orch daemon add osd HOSTNAME:DEVICE_PATH

   Example

   [ceph: root@host01 /]# ceph orch daemon add osd host02:/dev/sdb

   To deploy ODSs on a raw physical device, without an LVM layer, use the --method raw option.

   Syntax

   ceph orch daemon add osd --method raw HOSTNAME:DEVICE_PATH

   Example

   [ceph: root@host01 /]# ceph orch daemon add osd --method raw host02:/dev/sdb

   Note

   If you have separate DB or WAL devices, the ratio of block to DB or WAL devices MUST be 1:1.

Procedure

1. On the monitor node, create the osd_spec.yaml file:

   Example

   [root@host01 ~]# touch osd_spec.yaml

2. Edit the osd_spec.yaml file to include the following details:

   Syntax

   service_type: osd
   service_id: SERVICE_ID
   placement:
     host_pattern: '*' # optional
   data_devices: # optional
     model: DISK_MODEL_NAME # optional
     paths:
     - /DEVICE_PATH
   osds_per_device: NUMBER_OF_DEVICES # optional
   db_devices: # optional
     size: # optional
     all: true # optional
     paths:
      - /DEVICE_PATH
   encrypted: true

   1. Simple scenarios: In these cases, all the nodes have the same set-up.

      Example

      service_type: osd
      service_id: osd_spec_default
      placement:
        host_pattern: '*'
      data_devices:
        all: true
        paths:
        - /dev/sdb
      encrypted: true

      Example

      service_type: osd
      service_id: osd_spec_default
      placement:
        host_pattern: '*'
      data_devices:
        size: '80G'
      db_devices:
        size: '40G:'
        paths:
         - /dev/sdc

   2. Simple scenario: In this case, all the nodes have the same setup with OSD devices created in raw mode, without an LVM layer.

      Example

      service_type: osd
      service_id: all-available-devices
      encrypted: "true"
      method: raw
      placement:
        host_pattern: "*"
      data_devices:
        all: "true"

   3. Advanced scenario: This would create the desired layout by using all HDDs as data_devices with two SSD assigned as dedicated DB or WAL devices. The remaining SSDs are data_devices that have the NVMEs vendors assigned as dedicated DB or WAL devices.

      Example

      service_type: osd
      service_id: osd_spec_hdd
      placement:
        host_pattern: '*'
      data_devices:
        rotational: 0
      db_devices:
        model: Model-name
        limit: 2
      ---
      service_type: osd
      service_id: osd_spec_ssd
      placement:
        host_pattern: '*'
      data_devices:
        model: Model-name
      db_devices:
        vendor: Vendor-name

   4. Advanced scenario with non-uniform nodes: This applies different OSD specs to different hosts depending on the host_pattern key.

      Example

      service_type: osd
      service_id: osd_spec_node_one_to_five
      placement:
        host_pattern: 'node[1-5]'
      data_devices:
        rotational: 1
      db_devices:
        rotational: 0
      ---
      service_type: osd
      service_id: osd_spec_six_to_ten
      placement:
        host_pattern: 'node[6-10]'
      data_devices:
        model: Model-name
      db_devices:
        model: Model-name

   5. Advanced scenario with dedicated WAL and DB devices:

      Example

      service_type: osd
      service_id: osd_using_paths
      placement:
        hosts:
          - host01
          - host02
      data_devices:
        paths:
          - /dev/sdb
      db_devices:
        paths:
          - /dev/sdc
      wal_devices:
        paths:
          - /dev/sdd

   6. Advanced scenario with multiple OSDs per device:

      Example

      service_type: osd
      service_id: multiple_osds
      placement:
        hosts:
          - host01
          - host02
      osds_per_device: 4
      data_devices:
        paths:
          - /dev/sdb

   7. For pre-created volumes, edit the osd_spec.yaml file to include the following details:

      Syntax

      service_type: osd
      service_id: SERVICE_ID
      placement:
        hosts:
          - HOSTNAME
      data_devices: # optional
        model: DISK_MODEL_NAME # optional
        paths:
        - /DEVICE_PATH
      db_devices: # optional
        size: # optional
        all: true # optional
        paths:
         - /DEVICE_PATH

      Example

      service_type: osd
      service_id: osd_spec
      placement:
        hosts:
          - machine1
      data_devices:
        paths:
          - /dev/vg_hdd/lv_hdd
      db_devices:
        paths:
          - /dev/vg_nvme/lv_nvme

   8. For OSDs by ID, edit the osd_spec.yaml file to include the following details:

      Note

      This configuration is applicable for Red Hat Ceph Storage 5.3z1 and later releases. For earlier releases, use pre-created lvm.

      Syntax

      service_type: osd
      service_id: OSD_BY_ID_HOSTNAME
      placement:
        hosts:
          - HOSTNAME
      data_devices: # optional
        model: DISK_MODEL_NAME # optional
        paths:
        - /DEVICE_PATH
      db_devices: # optional
        size: # optional
        all: true # optional
        paths:
         - /DEVICE_PATH

      Example

      service_type: osd
      service_id: osd_by_id_host01
      placement:
        hosts:
          - host01
      data_devices:
        paths:
          - /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-5
      db_devices:
        paths:
          - /dev/disk/by-id/nvme-nvme.1b36-31323334-51454d55204e564d65204374726c-00000001

   9. For OSDs by path, edit the osd_spec.yaml file to include the following details:

      Note

      This configuration is applicable for Red Hat Ceph Storage 5.3z1 and later releases. For earlier releases, use pre-created lvm.

      Syntax

      service_type: osd
      service_id: OSD_BY_PATH_HOSTNAME
      placement:
        hosts:
          - HOSTNAME
      data_devices: # optional
        model: DISK_MODEL_NAME # optional
        paths:
        - /DEVICE_PATH
      db_devices: # optional
        size: # optional
        all: true # optional
        paths:
         - /DEVICE_PATH

      Example

      service_type: osd
      service_id: osd_by_path_host01
      placement:
        hosts:
          - host01
      data_devices:
        paths:
          - /dev/disk/by-path/pci-0000:0d:00.0-scsi-0:0:0:4
      db_devices:
        paths:
          - /dev/disk/by-path/pci-0000:00:02.0-nvme-1

3. Mount the YAML file under a directory in the container:

   Example

   [root@host01 ~]# cephadm shell --mount osd_spec.yaml:/var/lib/ceph/osd/osd_spec.yaml

4. Navigate to the directory:

   Example

   [ceph: root@host01 /]# cd /var/lib/ceph/osd/

5. Before deploying OSDs, do a dry run:

   Note

   This step gives a preview of the deployment, without deploying the daemons.

   Example

   [ceph: root@host01 osd]# ceph orch apply -i osd_spec.yaml --dry-run

6. Deploy OSDs using service specification:

   Syntax

   ceph orch apply -i FILE_NAME.yml

   Example

   [ceph: root@host01 osd]# ceph orch apply -i osd_spec.yaml

1. Evacuates all placement groups (PGs) from the cluster.
2. Removes the PG-free OSDs from the cluster.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Check the device and the node from which the OSD has to be removed:

   Example

   [ceph: root@host01 /]# ceph osd tree

3. Remove the OSD:

   Syntax

   ceph orch osd rm OSD_ID [--replace] [--force] --zap

   Example

   [ceph: root@host01 /]# ceph orch osd rm 0 --zap

   Note

   If you remove the OSD from the storage cluster without an option, such as --replace, the device is removed from the storage cluster completely. If you want to use the same device for deploying OSDs, you have to first zap the device before adding it to the storage cluster.

4. Optional: To remove multiple OSDs from a specific node, run the following command:

   Syntax

   ceph orch osd rm OSD_ID OSD_ID --zap

   Example

   [ceph: root@host01 /]# ceph orch osd rm 2 5 --zap

5. Check the status of the OSD removal:

   Example

   [ceph: root@host01 /]# ceph orch osd rm status
   
   OSD  HOST   STATE                    PGS  REPLACE  FORCE  ZAP   DRAIN STARTED AT
   9    host01 done, waiting for purge    0  False    False  True  2023-06-06 17:50:50.525690
   10   host03 done, waiting for purge    0  False    False  True  2023-06-06 17:49:38.731533
   11   host02 done, waiting for purge    0  False    False  True  2023-06-06 17:48:36.641105

   When no PGs are left on the OSD, it is decommissioned and removed from the cluster.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Ensure to dump and save a mapping of your OSD configurations for future references:

   Example

   [ceph: root@node /]# ceph osd metadata -f plain | grep device_paths
   "device_paths": "sde=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:1,sdi=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:1",
   "device_paths": "sde=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:1,sdf=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:1",
   "device_paths": "sdd=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:2,sdg=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:2",
   "device_paths": "sdd=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:2,sdh=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:2",
   "device_paths": "sdd=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:2,sdk=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:2",
   "device_paths": "sdc=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:3,sdl=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:3",
   "device_paths": "sdc=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:3,sdj=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:3",
   "device_paths": "sdc=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:3,sdm=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:3",
   [.. output omitted ..]

3. Check the device and the node from which the OSD has to be replaced:

   Example

   [ceph: root@host01 /]# ceph osd tree

4. Remove the OSD from the cephadm managed cluster:

   Important

   If the storage cluster has health_warn or other errors associated with it, check and try to fix any errors before replacing the OSD to avoid data loss.

   Syntax

   ceph orch osd rm OSD_ID --replace [--force]

   The --force option can be used when there are ongoing operations on the storage cluster.

   Example

   [ceph: root@host01 /]# ceph orch osd rm 0 --replace

5. Recreate the new OSD by applying the following OSD specification:

   Example

   service_type: osd
   service_id: osd
   placement:
     hosts:
     - myhost
   data_devices:
     paths:
     - /path/to/the/device

6. Check the status of the OSD replacement:

   Example

   [ceph: root@host01 /]# ceph orch osd rm status

7. Stop the orchestrator to apply any existing OSD specification:

   Example

   [ceph: root@node /]# ceph orch pause
   [ceph: root@node /]# ceph orch status
   Backend: cephadm
   Available: Yes
   Paused: Yes

8. Zap the OSD devices that have been removed:

   Example

   [ceph: root@node /]# ceph orch device zap node.example.com /dev/sdi --force
   zap successful for /dev/sdi on node.example.com
   
   [ceph: root@node /]# ceph orch device zap node.example.com /dev/sdf --force
   zap successful for /dev/sdf on node.example.com

9. Resume the Orcestrator from pause mode

   Example

   [ceph: root@node /]# ceph orch resume

10. Check the status of the OSD replacement:

    Example

    [ceph: root@node /]# ceph osd tree
    ID  CLASS  WEIGHT   TYPE NAME      STATUS  REWEIGHT  PRI-AFF
    -1         0.77112  root default
    -3         0.77112      host node
     0    hdd  0.09639          osd.0      up   1.00000  1.00000
     1    hdd  0.09639          osd.1      up   1.00000  1.00000
     2    hdd  0.09639          osd.2      up   1.00000  1.00000
     3    hdd  0.09639          osd.3      up   1.00000  1.00000
     4    hdd  0.09639          osd.4      up   1.00000  1.00000
     5    hdd  0.09639          osd.5      up   1.00000  1.00000
     6    hdd  0.09639          osd.6      up   1.00000  1.00000
     7    hdd  0.09639          osd.7      up   1.00000  1.00000
     [.. output omitted ..]

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Remove the OSD:

   Syntax

   ceph orch osd rm OSD_ID [--replace]

   Example

   [ceph: root@host01 /]# ceph orch osd rm 8 --replace
   Scheduled OSD(s) for removal

3. Verify the OSD is destroyed:

   Example

   [ceph: root@host01 /]# ceph osd tree
   
   ID   CLASS  WEIGHT   TYPE NAME        STATUS     REWEIGHT  PRI-AFF
    -1         0.32297  root default
    -9         0.05177      host host10
     3    hdd  0.01520          osd.3           up   1.00000  1.00000
    13    hdd  0.02489          osd.13          up   1.00000  1.00000
    17    hdd  0.01169          osd.17          up   1.00000  1.00000
   -13         0.05177      host host11
     2    hdd  0.01520          osd.2           up   1.00000  1.00000
    15    hdd  0.02489          osd.15          up   1.00000  1.00000
    19    hdd  0.01169          osd.19          up   1.00000  1.00000
    -7         0.05835      host host12
    20    hdd  0.01459          osd.20          up   1.00000  1.00000
    21    hdd  0.01459          osd.21          up   1.00000  1.00000
    22    hdd  0.01459          osd.22          up   1.00000  1.00000
    23    hdd  0.01459          osd.23          up   1.00000  1.00000
    -5         0.03827      host host04
     1    hdd  0.01169          osd.1           up   1.00000  1.00000
     6    hdd  0.01129          osd.6           up   1.00000  1.00000
     7    hdd  0.00749          osd.7           up   1.00000  1.00000
     9    hdd  0.00780          osd.9           up   1.00000  1.00000
    -3         0.03816      host host05
     0    hdd  0.01169          osd.0           up   1.00000  1.00000
     8    hdd  0.01129          osd.8    destroyed         0  1.00000
    12    hdd  0.00749          osd.12          up   1.00000  1.00000
    16    hdd  0.00769          osd.16          up   1.00000  1.00000
   -15         0.04237      host host06
     5    hdd  0.01239          osd.5           up   1.00000  1.00000
    10    hdd  0.01540          osd.10          up   1.00000  1.00000
    11    hdd  0.01459          osd.11          up   1.00000  1.00000
   -11         0.04227      host host07
     4    hdd  0.01239          osd.4           up   1.00000  1.00000
    14    hdd  0.01529          osd.14          up   1.00000  1.00000
    18    hdd  0.01459          osd.18          up   1.00000  1.00000

4. Zap and remove the OSD using the ceph-volume command:

   Syntax

   ceph-volume lvm zap --osd-id OSD_ID

   Example

   [ceph: root@host01 /]# ceph-volume lvm zap --osd-id 8
   
   Zapping: /dev/vg1/data-lv2
   Closing encrypted path /dev/mapper/l4D6ql-Prji-IzH4-dfhF-xzuf-5ETl-jNRcXC
   Running command: /usr/sbin/cryptsetup remove /dev/mapper/l4D6ql-Prji-IzH4-dfhF-xzuf-5ETl-jNRcXC
   Running command: /usr/bin/dd if=/dev/zero of=/dev/vg1/data-lv2 bs=1M count=10 conv=fsync
    stderr: 10+0 records in
   10+0 records out
    stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.034742 s, 302 MB/s
   Zapping successful for OSD: 8

5. Check the OSD topology:

   Example

   [ceph: root@host01 /]# ceph-volume lvm list

6. Recreate the OSD with a specification file corresponding to that specific OSD topology:

   Example

   [ceph: root@host01 /]# cat osd.yml
   service_type: osd
   service_id: osd_service
   placement:
     hosts:
     - host03
   data_devices:
     paths:
     - /dev/vg1/data-lv2
   db_devices:
     paths:
      - /dev/vg1/db-lv1

7. Apply the updated specification file:

   Example

   [ceph: root@host01 /]# ceph orch apply -i osd.yml
   Scheduled osd.osd_service update...

8. Verify the OSD is back:

   Example

   [ceph: root@host01 /]# ceph -s
   [ceph: root@host01 /]# ceph osd tree

Procedure

1. Identify the devices in the cluster:

   Example

   [root@host01 ~]# lsblk
   
   NAME                                                                                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
   sda                                                                                                     8:0    0   20G  0 disk
   ├─sda1                                                                                                  8:1    0    1G  0 part /boot
   └─sda2                                                                                                  8:2    0   19G  0 part
     ├─rhel-root                                                                                         253:0    0   17G  0 lvm  /
     └─rhel-swap                                                                                         253:1    0    2G  0 lvm  [SWAP]
   sdb                                                                                                     8:16   0   10G  0 disk
   └─ceph--5726d3e9--4fdb--4eda--b56a--3e0df88d663f-osd--block--3ceb89ec--87ef--46b4--99c6--2a56bac09ff0 253:2    0   10G  0 lvm
   sdc                                                                                                     8:32   0   10G  0 disk
   └─ceph--d7c9ab50--f5c0--4be0--a8fd--e0313115f65c-osd--block--37c370df--1263--487f--a476--08e28bdbcd3c 253:4    0   10G  0 lvm
   sdd                                                                                                     8:48   0   10G  0 disk
   ├─ceph--1774f992--44f9--4e78--be7b--b403057cf5c3-osd--db--31b20150--4cbc--4c2c--9c8f--6f624f3bfd89    253:7    0  2.5G  0 lvm
   └─ceph--1774f992--44f9--4e78--be7b--b403057cf5c3-osd--db--1bee5101--dbab--4155--a02c--e5a747d38a56    253:9    0  2.5G  0 lvm
   sde                                                                                                     8:64   0   10G  0 disk
   sdf                                                                                                     8:80   0   10G  0 disk
   └─ceph--412ee99b--4303--4199--930a--0d976e1599a2-osd--block--3a99af02--7c73--4236--9879--1fad1fe6203d 253:6    0   10G  0 lvm
   sdg                                                                                                     8:96   0   10G  0 disk
   └─ceph--316ca066--aeb6--46e1--8c57--f12f279467b4-osd--block--58475365--51e7--42f2--9681--e0c921947ae6 253:8    0   10G  0 lvm
   sdh                                                                                                     8:112  0   10G  0 disk
   ├─ceph--d7064874--66cb--4a77--a7c2--8aa0b0125c3c-osd--db--0dfe6eca--ba58--438a--9510--d96e6814d853    253:3    0    5G  0 lvm
   └─ceph--d7064874--66cb--4a77--a7c2--8aa0b0125c3c-osd--db--26b70c30--8817--45de--8843--4c0932ad2429    253:5    0    5G  0 lvm
   sr0

2. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

3. Identify the OSDs and their DB device:

   Example

   [ceph: root@host01 /]# ceph-volume lvm list /dev/sdh
   
   
   ====== osd.2 =======
   
     [db]          /dev/ceph-d7064874-66cb-4a77-a7c2-8aa0b0125c3c/osd-db-0dfe6eca-ba58-438a-9510-d96e6814d853
   
         block device              /dev/ceph-5726d3e9-4fdb-4eda-b56a-3e0df88d663f/osd-block-3ceb89ec-87ef-46b4-99c6-2a56bac09ff0
         block uuid                GkWLoo-f0jd-Apj2-Zmwj-ce0h-OY6J-UuW8aD
         cephx lockbox secret
         cluster fsid              fa0bd9dc-e4c4-11ed-8db4-001a4a00046e
         cluster name              ceph
         crush device class
         db device                 /dev/ceph-d7064874-66cb-4a77-a7c2-8aa0b0125c3c/osd-db-0dfe6eca-ba58-438a-9510-d96e6814d853
         db uuid                   6gSPoc-L39h-afN3-rDl6-kozT-AX9S-XR20xM
         encrypted                 0
         osd fsid                  3ceb89ec-87ef-46b4-99c6-2a56bac09ff0
         osd id                    2
         osdspec affinity          non-colocated
         type                      db
         vdo                       0
         devices                   /dev/sdh
   
   ====== osd.5 =======
   
     [db]          /dev/ceph-d7064874-66cb-4a77-a7c2-8aa0b0125c3c/osd-db-26b70c30-8817-45de-8843-4c0932ad2429
   
         block device              /dev/ceph-d7c9ab50-f5c0-4be0-a8fd-e0313115f65c/osd-block-37c370df-1263-487f-a476-08e28bdbcd3c
         block uuid                Eay3I7-fcz5-AWvp-kRcI-mJaH-n03V-Zr0wmJ
         cephx lockbox secret
         cluster fsid              fa0bd9dc-e4c4-11ed-8db4-001a4a00046e
         cluster name              ceph
         crush device class
         db device                 /dev/ceph-d7064874-66cb-4a77-a7c2-8aa0b0125c3c/osd-db-26b70c30-8817-45de-8843-4c0932ad2429
         db uuid                   mwSohP-u72r-DHcT-BPka-piwA-lSwx-w24N0M
         encrypted                 0
         osd fsid                  37c370df-1263-487f-a476-08e28bdbcd3c
         osd id                    5
         osdspec affinity          non-colocated
         type                      db
         vdo                       0
         devices                   /dev/sdh

4. In the osds.yaml file, set unmanaged parameter to true, else cephadm redeploys the OSDs:

   Example

   [ceph: root@host01 /]# cat osds.yml
   service_type: osd
   service_id: non-colocated
   unmanaged: true
   placement:
     host_pattern: 'ceph*'
   data_devices:
     paths:
      - /dev/sdb
      - /dev/sdc
      - /dev/sdf
      - /dev/sdg
   db_devices:
     paths:
      - /dev/sdd
      - /dev/sdh

5. Apply the updated specification file:

   Example

   [ceph: root@host01 /]# ceph orch apply -i osds.yml
   
   Scheduled osd.non-colocated update...

6. Check the status:

   Example

   [ceph: root@host01 /]# ceph orch ls
   
   NAME           PORTS        RUNNING  REFRESHED  AGE  PLACEMENT
   alertmanager   ?:9093,9094      1/1  9m ago     4d   count:1
   crash                           3/4  4d ago     4d   *
   grafana        ?:3000           1/1  9m ago     4d   count:1
   mgr                             1/2  4d ago     4d   count:2
   mon                             3/5  4d ago     4d   count:5
   node-exporter  ?:9100           3/4  4d ago     4d   *
   osd.non-colocated                 8  4d ago     5s   <unmanaged>
   prometheus     ?:9095           1/1  9m ago     4d   count:1

7. Remove the OSDs. Ensure to use the --zap option to remove hte backend services and the --replace option to retain the OSD IDs:

   Example

   [ceph: root@host01 /]# ceph orch osd rm 2 5 --zap --replace
   Scheduled OSD(s) for removal

8. Check the status:

   Example

   [ceph: root@host01 /]# ceph osd df tree | egrep -i "ID|host02|osd.2|osd.5"
   
   ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP  META     AVAIL   %USE   VAR   PGS  STATUS     TYPE NAME
   -5         0.04877         -   55 GiB   15 GiB  4.1 MiB   0 B   60 MiB  40 GiB  27.27  1.17    -                 host02
    2    hdd  0.01219   1.00000   15 GiB  5.0 GiB  996 KiB   0 B   15 MiB  10 GiB  33.33  1.43    0  destroyed          osd.2
    5    hdd  0.01219   1.00000   15 GiB  5.0 GiB  1.0 MiB   0 B   15 MiB  10 GiB  33.33  1.43    0  destroyed          osd.5

9. Edit the osds.yaml specification file to change unmanaged parameter to false and replace the path to the DB device if it has changed after the device got physically replaced:

   Example

   [ceph: root@host01 /]# cat osds.yml
   service_type: osd
   service_id: non-colocated
   unmanaged: false
   placement:
     host_pattern: 'ceph01*'
   data_devices:
     paths:
      - /dev/sdb
      - /dev/sdc
      - /dev/sdf
      - /dev/sdg
   db_devices:
     paths:
      - /dev/sdd
      - /dev/sde

   In the above example, /dev/sdh is replaced with /dev/sde.

   Important

   If you use the same host specification file to replace the faulty DB device on a single OSD node, modify the host_pattern option to specify only the OSD node, else the deployment fails and you cannot find the new DB device on other hosts.

10. Reapply the specification file with the --dry-run option to ensure the OSDs shall be deployed with the new DB device:

    Example

    [ceph: root@host01 /]# ceph orch apply -i osds.yml --dry-run
    WARNING! Dry-Runs are snapshots of a certain point in time and are bound
    to the current inventory setup. If any of these conditions change, the
    preview will be invalid. Please make sure to have a minimal
    timeframe between planning and applying the specs.
    ####################
    SERVICESPEC PREVIEWS
    ####################
    +---------+------+--------+-------------+
    |SERVICE  |NAME  |ADD_TO  |REMOVE_FROM  |
    +---------+------+--------+-------------+
    +---------+------+--------+-------------+
    ################
    OSDSPEC PREVIEWS
    ################
    +---------+-------+-------+----------+----------+-----+
    |SERVICE  |NAME   |HOST   |DATA      |DB        |WAL  |
    +---------+-------+-------+----------+----------+-----+
    |osd      |non-colocated  |host02  |/dev/sdb  |/dev/sde  |-    |
    |osd      |non-colocated  |host02  |/dev/sdc  |/dev/sde  |-    |
    +---------+-------+-------+----------+----------+-----+

11. Apply the specification file:

    Example

    [ceph: root@host01 /]# ceph orch apply -i osds.yml
    Scheduled osd.non-colocated update...

12. Check the OSDs are redeployed:

    Example

    [ceph: root@host01 /]# ceph osd df tree | egrep -i "ID|host02|osd.2|osd.5"
    
    ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP  META     AVAIL   %USE   VAR   PGS  STATUS  TYPE NAME
    -5         0.04877         -   55 GiB   15 GiB  4.5 MiB   0 B   60 MiB  40 GiB  27.27  1.17    -              host host02
     2    hdd  0.01219   1.00000   15 GiB  5.0 GiB  1.1 MiB   0 B   15 MiB  10 GiB  33.33  1.43    0      up          osd.2
     5    hdd  0.01219   1.00000   15 GiB  5.0 GiB  1.1 MiB   0 B   15 MiB  10 GiB  33.33  1.43    0      up          osd.5

Verification

1. From the OSD host where the OSDS are redeployed, verify if they are on the new DB device:

   Example

   [ceph: root@host01 /]# ceph-volume lvm list /dev/sde
   
   ====== osd.2 =======
   
     [db]          /dev/ceph-15ce813a-8a4c-46d9-ad99-7e0845baf15e/osd-db-1998a02e-5e67-42a9-b057-e02c22bbf461
   
         block device              /dev/ceph-a4afcb78-c804-4daf-b78f-3c7ad1ed0379/osd-block-564b3d2f-0f85-4289-899a-9f98a2641979
         block uuid                ITPVPa-CCQ5-BbFa-FZCn-FeYt-c5N4-ssdU41
         cephx lockbox secret
         cluster fsid              fa0bd9dc-e4c4-11ed-8db4-001a4a00046e
         cluster name              ceph
         crush device class
         db device                 /dev/ceph-15ce813a-8a4c-46d9-ad99-7e0845baf15e/osd-db-1998a02e-5e67-42a9-b057-e02c22bbf461
         db uuid                   HF1bYb-fTK7-0dcB-CHzW-xvNn-dCym-KKdU5e
         encrypted                 0
         osd fsid                  564b3d2f-0f85-4289-899a-9f98a2641979
         osd id                    2
         osdspec affinity          non-colocated
         type                      db
         vdo                       0
         devices                   /dev/sde
   
   ====== osd.5 =======
   
     [db]          /dev/ceph-15ce813a-8a4c-46d9-ad99-7e0845baf15e/osd-db-6c154191-846d-4e63-8c57-fc4b99e182bd
   
         block device              /dev/ceph-b37c8310-77f9-4163-964b-f17b4c29c537/osd-block-b42a4f1f-8e19-4416-a874-6ff5d305d97f
         block uuid                0LuPoz-ao7S-UL2t-BDIs-C9pl-ct8J-xh5ep4
         cephx lockbox secret
         cluster fsid              fa0bd9dc-e4c4-11ed-8db4-001a4a00046e
         cluster name              ceph
         crush device class
         db device                 /dev/ceph-15ce813a-8a4c-46d9-ad99-7e0845baf15e/osd-db-6c154191-846d-4e63-8c57-fc4b99e182bd
         db uuid                   SvmXms-iWkj-MTG7-VnJj-r5Mo-Moiw-MsbqVD
         encrypted                 0
         osd fsid                  b42a4f1f-8e19-4416-a874-6ff5d305d97f
         osd id                    5
         osdspec affinity          non-colocated
         type                      db
         vdo                       0
         devices                   /dev/sde

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Check the device and the node from which the OSD was initiated to be removed:

   Example

   [ceph: root@host01 /]# ceph osd tree

3. Stop the removal of the queued OSD:

   Syntax

   ceph orch osd rm stop OSD_ID

   Example

   [ceph: root@host01 /]# ceph orch osd rm stop 0

4. Check the status of the OSD removal:

   Example

   [ceph: root@host01 /]# ceph orch osd rm status

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. After the operating system of the host is reinstalled, activate the OSDs:

   Syntax

   ceph cephadm osd activate HOSTNAME

   Example

   [ceph: root@host01 /]# ceph cephadm osd activate host03

Procedure

1. Enable the prometheus module in the Ceph Manager daemon. This exposes the internal Ceph metrics so that Prometheus can read them:

   Example

   [ceph: root@host01 /]# ceph mgr module enable prometheus

   Important

   Ensure this command is run before Prometheus is deployed. If the command was not run before the deployment, you must redeploy Prometheus to update the configuration:

   ceph orch redeploy prometheus

2. Navigate to the following directory:

   Syntax

   cd /var/lib/ceph/DAEMON_PATH/

   Example

   [ceph: root@host01 mds/]# cd /var/lib/ceph/monitoring/

   Note

   If the directory monitoring does not exist, create it.

3. Create the monitoring.yml file:

   Example

   [ceph: root@host01 monitoring]# touch monitoring.yml

4. Edit the specification file with a content similar to the following example:

   Example

   service_type: prometheus
   service_name: prometheus
   placement:
     hosts:
     - host01
   networks:
   - 192.169.142.0/24
   ---
   service_type: node-exporter
   ---
   service_type: alertmanager
   service_name: alertmanager
   placement:
     hosts:
     - host01
   networks:
   - 192.169.142.0/24
   ---
   service_type: grafana
   service_name: grafana
   placement:
     hosts:
     - host01
   networks:
   - 192.169.142.0/24
   ---
   service_type: ceph-exporter

   Note

   Ensure the monitoring stack components alertmanager, prometheus, and grafana are deployed on the same host. The node-exporter and ceph-exporter components should be deployed on all the hosts.

5. Apply monitoring services:

   Example

   [ceph: root@host01 monitoring]# ceph orch apply -i monitoring.yml

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Use the ceph orch rm command to remove the monitoring stack:

   Syntax

   ceph orch rm SERVICE_NAME --force

   Example

   [ceph: root@host01 /]# ceph orch rm grafana
   [ceph: root@host01 /]# ceph orch rm prometheus
   [ceph: root@host01 /]# ceph orch rm node-exporter
   [ceph: root@host01 /]# ceph orch rm alertmanager
   [ceph: root@host01 /]# ceph orch rm ceph-exporter
   [ceph: root@host01 /]# ceph mgr module disable prometheus

3. Check the status of the process:

   Example

   [ceph: root@host01 /]# ceph orch status

Procedure

1. On the node where you want to set up the files, create a directory ceph in the /etc folder:

   Example

   [root@host01 ~]# mkdir /etc/ceph/

2. Navigate to /etc/ceph directory:

   Example

   [root@host01 ~]# cd /etc/ceph/

3. Generate the configuration file in the ceph directory:

   Example

   [root@host01 ceph]# ceph config generate-minimal-conf
   
   # minimal ceph.conf for 417b1d7a-a0e6-11eb-b940-001a4a000740
   [global]
   	fsid = 417b1d7a-a0e6-11eb-b940-001a4a000740
   	mon_host = [v2:10.74.249.41:3300/0,v1:10.74.249.41:6789/0]

   The contents of this file should be installed in /etc/ceph/ceph.conf path. You can use this configuration file to reach the Ceph monitors.

Procedure

1. On the node where you want to set up the keyring, create a directory ceph in the /etc folder:

   Example

   [root@host01 ~]# mkdir /etc/ceph/

2. Navigate to /etc/ceph directory in the ceph directory:

   Example

   [root@host01 ~]# cd /etc/ceph/

3. Generate the keyring for the client:

   Syntax

   ceph auth get-or-create client.CLIENT_NAME -o /etc/ceph/NAME_OF_THE_FILE

   Example

   [root@host01 ceph]# ceph auth get-or-create client.fs -o /etc/ceph/ceph.keyring

4. Verify the output in the ceph.keyring file:

   Example

   [root@host01 ceph]# cat ceph.keyring
   
   [client.fs]
   	key = AQAvoH5gkUCsExAATz3xCBLd4n6B6jRv+Z7CVQ==

   The resulting output should be put into a keyring file, for example /etc/ceph/ceph.keyring.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. There are two ways of deploying MDS daemons using placement specification:

Procedure

1. Create the mds.yaml file:

   Example

   [root@host01 ~]# touch mds.yaml

2. Edit the mds.yaml file to include the following details:

   Syntax

   service_type: mds
   service_id: FILESYSTEM_NAME
   placement:
     hosts:
     - HOST_NAME_1
     - HOST_NAME_2
     - HOST_NAME_3

   Example

   service_type: mds
   service_id: fs_name
   placement:
     hosts:
     - host01
     - host02

3. Mount the YAML file under a directory in the container:

   Example

   [root@host01 ~]# cephadm shell --mount mds.yaml:/var/lib/ceph/mds/mds.yaml

4. Navigate to the directory:

   Example

   [ceph: root@host01 /]# cd /var/lib/ceph/mds/

5. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

6. Navigate to the following directory:

   Example

   [ceph: root@host01 /]# cd /var/lib/ceph/mds/

7. Deploy MDS service using service specification:

   Syntax

   ceph orch apply -i FILE_NAME.yaml

   Example

   [ceph: root@host01 mds]# ceph orch apply -i mds.yaml

8. Once the MDS services is deployed and functional, create the CephFS:

   Syntax

   ceph fs new CEPHFS_NAME METADATA_POOL DATA_POOL

   Example

   [ceph: root@host01 /]# ceph fs new test metadata_pool data_pool

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. You can deploy the Ceph object gateway daemons in three different ways:

Procedure

1. As a root user, create a specification file:

   Example

   [root@host01 ~]# touch radosgw.yml

2. Edit the radosgw.yml file to include the following details for the default realm, zone, and zone group:

   Syntax

   service_type: rgw
   service_id: REALM_NAME.ZONE_NAME
   placement:
     hosts:
     - HOST_NAME_1
     - HOST_NAME_2
     count_per_host: NUMBER_OF_DAEMONS
   spec:
     rgw_realm: REALM_NAME
     rgw_zone: ZONE_NAME
     rgw_frontend_port: FRONT_END_PORT
   networks:
     -  NETWORK_CIDR # Ceph Object Gateway service binds to a specific network

   Note

   NUMBER_OF_DAEMONS controls the number of Ceph Object Gateways deployed on each host. To achieve the highest performance without incurring an additional cost, set this value to 2.

   Example

   service_type: rgw
   service_id: default
   placement:
     hosts:
     - host01
     - host02
     - host03
     count_per_host: 2
   spec:
     rgw_realm: default
     rgw_zone: default
     rgw_frontend_port: 1234
   networks:
     - 192.169.142.0/24

3. Optional: For custom realm, zone, and zone group, create the resources and then create the radosgw.yml file:

   1. Create the custom realm, zone, and zone group:

      Example

      [root@host01 ~]# radosgw-admin realm create --rgw-realm=test_realm --default
      [root@host01 ~]# radosgw-admin zonegroup create --rgw-zonegroup=test_zonegroup --default
      [root@host01 ~]# radosgw-admin zone create --rgw-zonegroup=test_zonegroup --rgw-zone=test_zone --default
      [root@host01 ~]# radosgw-admin period update --rgw-realm=test_realm --commit

   2. Create the radosgw.yml file with the following details:

      Example

      service_type: rgw
      service_id: test_realm.test_zone
      placement:
        hosts:
        - host01
        - host02
        - host03
        count_per_host: 2
      spec:
        rgw_realm: test_realm
        rgw_zone: test_zone
        rgw_frontend_port: 1234
      networks:
        - 192.169.142.0/24

4. Mount the radosgw.yml file under a directory in the container:

   Example

   [root@host01 ~]# cephadm shell --mount radosgw.yml:/var/lib/ceph/radosgw/radosgw.yml

   Note

   Every time you exit the shell, you have to mount the file in the container before deploying the daemon.

5. Deploy the Ceph Object Gateway using the service specification:

   Syntax

   ceph orch apply -i FILE_NAME.yml

   Example

   [ceph: root@host01 /]# ceph orch apply -i radosgw.yml

Procedure

1. In the cephadm shell, configure the primary zone:

   1. Create a realm:

      Syntax

      radosgw-admin realm create --rgw-realm=REALM_NAME --default

      Example

      [ceph: root@host01 /]# radosgw-admin realm create --rgw-realm=test_realm --default

      If the storage cluster has a single realm, then specify the --default flag.

   2. Create a primary zone group:

      Syntax

      radosgw-admin zonegroup create --rgw-zonegroup=ZONE_GROUP_NAME --endpoints=http://RGW_PRIMARY_HOSTNAME:RGW_PRIMARY_PORT_NUMBER_1 --master --default

      Example

      [ceph: root@host01 /]# radosgw-admin zonegroup create --rgw-zonegroup=us --endpoints=http://rgw1:80 --master --default

   3. Create a primary zone:

      Syntax

      radosgw-admin zone create --rgw-zonegroup=PRIMARY_ZONE_GROUP_NAME --rgw-zone=PRIMARY_ZONE_NAME --endpoints=http://RGW_PRIMARY_HOSTNAME:RGW_PRIMARY_PORT_NUMBER_1 --access-key=SYSTEM_ACCESS_KEY --secret=SYSTEM_SECRET_KEY

      Example

      [ceph: root@host01 /]# radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-1 --endpoints=http://rgw1:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ

   4. Optional: Delete the default zone, zone group, and the associated pools.

      Important

      Do not delete the default zone and its pools if you are using the default zone and zone group to store data. Also, removing the default zone group deletes the system user.

      To access old data in the default zone and zonegroup, use --rgw-zone default and --rgw-zonegroup default in radosgw-admin commands.

      Example

      [ceph: root@host01 /]# radosgw-admin zonegroup delete --rgw-zonegroup=default
      [ceph: root@host01 /]# ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it
      [ceph: root@host01 /]# ceph osd pool rm default.rgw.meta default.rgw.meta --yes-i-really-really-mean-it
      [ceph: root@host01 /]# ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it
      [ceph: root@host01 /]# ceph osd pool rm default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
      [ceph: root@host01 /]# ceph osd pool rm default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it

   5. Create a system user:

      Syntax

      radosgw-admin user create --uid=USER_NAME --display-name="USER_NAME" --access-key=SYSTEM_ACCESS_KEY --secret=SYSTEM_SECRET_KEY --system

      Example

      [ceph: root@host01 /]# radosgw-admin user create --uid=zone.user --display-name="Zone user" --system

      Make a note of the access_key and secret_key.

   6. Add the access key and system key to the primary zone:

      Syntax

      radosgw-admin zone modify --rgw-zone=PRIMARY_ZONE_NAME --access-key=ACCESS_KEY --secret=SECRET_KEY

      Example

      [ceph: root@host01 /]# radosgw-admin zone modify --rgw-zone=us-east-1 --access-key=NE48APYCAODEPLKBCZVQ --secret=u24GHQWRE3yxxNBnFBzjM4jn14mFIckQ4EKL6LoW

   7. Commit the changes:

      Syntax

      radosgw-admin period update --commit

      Example

      [ceph: root@host01 /]# radosgw-admin period update --commit

   8. Outside the cephadm shell, fetch the FSID of the storage cluster and the processes:

      Example

      [root@host01 ~]#  systemctl list-units | grep ceph

   9. Start the Ceph Object Gateway daemon:

      Syntax

      systemctl start ceph-FSID@DAEMON_NAME
      systemctl enable ceph-FSID@DAEMON_NAME

      Example

      [root@host01 ~]# systemctl start ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-1.host01.ahdtsw.service
      [root@host01 ~]# systemctl enable ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-1.host01.ahdtsw.service

2. In the Cephadm shell, configure the secondary zone.

   1. Pull the primary realm configuration from the host:

      Syntax

      radosgw-admin realm pull --rgw-realm=PRIMARY_REALM --url=URL_TO_PRIMARY_ZONE_GATEWAY --access-key=ACCESS_KEY --secret-key=SECRET_KEY --default

      Example

      [ceph: root@host04 /]# radosgw-admin realm pull --rgw-realm=test_realm --url=http://10.74.249.26:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ --default

   2. Pull the primary period configuration from the host:

      Syntax

      radosgw-admin period pull --url=URL_TO_PRIMARY_ZONE_GATEWAY --access-key=ACCESS_KEY --secret-key=SECRET_KEY

      Example

      [ceph: root@host04 /]# radosgw-admin period pull --url=http://10.74.249.26:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ

   3. Configure a secondary zone:

      Syntax

      radosgw-admin zone create --rgw-zonegroup=ZONE_GROUP_NAME \
                   --rgw-zone=SECONDARY_ZONE_NAME --endpoints=http://RGW_SECONDARY_HOSTNAME:RGW_PRIMARY_PORT_NUMBER_1 \
                   --access-key=SYSTEM_ACCESS_KEY --secret=SYSTEM_SECRET_KEY \
                   --endpoints=http://FQDN:80 \
                   [--read-only]

      Example

      [ceph: root@host04 /]# radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-2 --endpoints=http://rgw2:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ

   4. Optional: Delete the default zone.

      Important

      Do not delete the default zone and its pools if you are using the default zone and zone group to store data.

      To access old data in the default zone and zonegroup, use --rgw-zone default and --rgw-zonegroup default in radosgw-admin commands.

      Example

      [ceph: root@host04 /]# radosgw-admin zone rm --rgw-zone=default
      [ceph: root@host04 /]# ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it
      [ceph: root@host04 /]# ceph osd pool rm default.rgw.meta default.rgw.meta --yes-i-really-really-mean-it
      [ceph: root@host04 /]# ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it
      [ceph: root@host04 /]# ceph osd pool rm default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
      [ceph: root@host04 /]# ceph osd pool rm default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it

   5. Update the Ceph configuration database:

      Syntax

      ceph config set SERVICE_NAME rgw_zone SECONDARY_ZONE_NAME

      Example

      [ceph: root@host04 /]# ceph config set rgw rgw_zone us-east-2

   6. Commit the changes:

      Syntax

      radosgw-admin period update --commit

      Example

      [ceph: root@host04 /]# radosgw-admin period update --commit

   7. Outside the Cephadm shell, fetch the FSID of the storage cluster and the processes:

      Example

      [root@host04 ~]#  systemctl list-units | grep ceph

   8. Start the Ceph Object Gateway daemon:

      Syntax

      systemctl start ceph-FSID@DAEMON_NAME
      systemctl enable ceph-FSID@DAEMON_NAME

      Example

      [root@host04 ~]# systemctl start ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-2.host04.ahdtsw.service
      [root@host04 ~]# systemctl enable ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-2.host04.ahdtsw.service

3. Optional: Deploy multi-site Ceph Object Gateways using the placement specification:

   Syntax

   ceph orch apply rgw NAME --realm=REALM_NAME --zone=PRIMARY_ZONE_NAME --placement="NUMBER_OF_DAEMONS HOST_NAME_1 HOST_NAME_2"

   Example

   [ceph: root@host04 /]# ceph orch apply rgw east --realm=test_realm --zone=us-east-1 --placement="2 host01 host02"

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. List the service:

   Example

   [ceph: root@host01 /]# ceph orch ls

3. Remove the service:

   Syntax

   ceph orch rm SERVICE_NAME

   Example

   [ceph: root@host01 /]# ceph orch rm rgw.test_realm.test_zone_bb

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Enable the mgr/nfs module:

   Example

   [ceph: root@host01 /]# ceph mgr module enable nfs

3. Create the cluster:

   Syntax

   ceph nfs cluster create CLUSTER_NAME ["HOST_NAME_1 HOST_NAME_2 HOST_NAME_3"]

   The CLUSTER_NAME is an arbitrary string and HOST_NAME_1 is an optional string signifying the hosts to deploy NFS-Ganesha daemons.

   Example

   [ceph: root@host01 /]# ceph nfs cluster create nfsganesha "host01 host02"
   NFS Cluster Created Successful

   This creates an NFS_Ganesha cluster nfsganesha with one daemon on host01 and host02.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Create the RADOS pool namespace, and enable the application. For RBD pools, enable RBD.

   Syntax

   ceph osd pool create POOL_NAME
   ceph osd pool application enable POOL_NAME freeform/rgw/rbd/cephfs/nfs
   rbd pool init -p POOL_NAME

   Example

   [ceph: root@host01 /]# ceph osd pool create nfs-ganesha
   [ceph: root@host01 /]# ceph osd pool application enable nfs-ganesha nfs
   [ceph: root@host01 /]# rbd pool init -p nfs-ganesha

3. Deploy NFS-Ganesha gateway using placement specification in the command line interface:

   Syntax

   ceph orch apply nfs SERVICE_ID --placement="NUMBER_OF_DAEMONS HOST_NAME_1 HOST_NAME_2 HOST_NAME_3"

   Example

   [ceph: root@host01 /]# ceph orch apply nfs foo --placement="2 host01 host02"

   This deploys an NFS-Ganesha cluster nfsganesha with one daemon on host01 and host02.

Procedure

1. Create the nfs.yaml file:

   Example

   [root@host01 ~]# touch nfs.yaml

2. Edit the nfs.yaml file to include the following details:

   Syntax

   service_type: nfs
   service_id: SERVICE_ID
   placement:
     hosts:
       - HOST_NAME_1
       - HOST_NAME_2

   Example

   service_type: nfs
   service_id: foo
   placement:
     hosts:
       - host01
       - host02

3. Mount the YAML file under a directory in the container:

   Example

   [root@host01 ~]# cephadm shell --mount nfs.yaml:/var/lib/ceph/nfs.yaml

4. Create the RADOS pool, namespace, and enable RBD:

   Syntax

   ceph osd pool create POOL_NAME
   ceph osd pool application enable POOL_NAME rbd
   rbd pool init -p POOL_NAME

   Example

   [ceph: root@host01 /]# ceph osd pool create nfs-ganesha
   [ceph: root@host01 /]# ceph osd pool application enable nfs-ganesha rbd
   [ceph: root@host01 /]# rbd pool init -p nfs-ganesha

5. Navigate to the directory:

   Example

   [ceph: root@host01 /]# cd /var/lib/ceph/

6. Deploy NFS-Ganesha gateway using service specification:

   Syntax

   ceph orch apply -i FILE_NAME.yaml

   Example

   [ceph: root@host01 ceph]# ceph orch apply -i nfs.yaml

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Create the NFS cluster with the --ingress flag:

   Syntax

   ceph nfs cluster create CLUSTER_ID [PLACEMENT] [--port PORT_NUMBER] [--ingress --virtual-ip IP_ADDRESS/CIDR_PREFIX]

   • Replace CLUSTER_ID with a unique string to name the NFS Ganesha cluster.
   • Replace PLACEMENT with the number of NFS servers to deploy and the host or hosts that you want to deploy the NFS Ganesha daemon containers on.
   • Use the --port PORT_NUMBER flag to deploy NFS on a port other than the default port of 2049.
   • The --ingress flag combined with the --virtual-ip flag, deploys NFS with a high-availability front-end (virtual IP and load balancer).

   • Replace --virtual-ip IP_ADDRESS with an IP address to provide a known, stable NFS endpoint that all clients can use to mount NFS exports. The --virtual-ip must include a CIDR prefix length. The virtual IP will normally be configured on the first identified network interface that has an existing IP in the same subnet.

     Note

     The number of hosts you allocate for the NFS service must be greater than the number of active NFS servers you request to deploy, specified by the placement: count parameter. In the below example, one active NFS server is requested and two hosts are allocated.

     Example

     [ceph: root@host01 /]# ceph nfs cluster create mycephnfs "1 host02 host03" --ingress --virtual-ip 10.10.128.75/22

     Note

     Deployment of NFS daemons and the ingress service is asynchronous and the command might return before the services have completely started.

3. Check that the services have successfully started:

   Syntax

   ceph orch ls --service_name=nfs.CLUSTER_ID
   ceph orch ls --service_name=ingress.nfs.CLUSTER_ID

   Example

   [ceph: root@host01 /]# ceph orch ls --service_name=nfs.mycephnfs
   
   NAME           PORTS    RUNNING  REFRESHED  AGE  PLACEMENT
   nfs.mycephnfs  ?:12049      1/2  0s ago     20s  host02;host03
   
   [ceph: root@host01 /]# ceph orch ls --service_name=ingress.nfs.mycephnfs
   
   NAME                   PORTS                  RUNNING  REFRESHED  AGE  PLACEMENT
   ingress.nfs.mycephnfs  10.10.128.75:2049,9049      4/4   46s ago    73s  count:2

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. List existing NFS clusters:

   Example

   [ceph: root@host01 /]# ceph nfs cluster ls
   
   mynfs

   Note

   If a standalone NFS cluster is created on one node, you need to increase it to two or more nodes for HA. To increase the NFS service, edit the nfs.yaml file and increase the placements with the same port number.

   The number of hosts you allocate for the NFS service must be greater than the number of active NFS servers you request to deploy, specified by the placement: count parameter.

   Syntax

   service_type: nfs
   service_id: SERVICE_ID
   placement:
     hosts:
       - HOST_NAME_1
       - HOST_NAME_2
     count: COUNT
   spec:
     port: PORT_NUMBER

   Example

   service_type: nfs
   service_id: mynfs
   placement:
     hosts:
       - host02
       - host03
     count: 1
   spec:
     port: 12345

   In this example, the existing NFS service is running on port 12345 and an additional node is added to the NFS cluster with the same port.

   Apply the nfs.yaml service specification changes to upgrade to a two node NFS service:

   Example

   [ceph: root@host01 ceph]# ceph orch apply -i nfs.yaml

3. Edit the ingress.yaml specification file with the existing NFS cluster ID:

   Syntax

   service_type: SERVICE_TYPE
   service_id: SERVICE_ID
   placement:
     count: PLACEMENT
   spec:
     backend_service: SERVICE_ID_BACKEND 1
     frontend_port: FRONTEND_PORT
     monitor_port: MONITOR_PORT 2
     virtual_ip: VIRTUAL_IP_WITH_CIDR

   1

   SERVICE_ID_BACKEND should include a PORT property that is not 2049 to avoid conflicting with the ingress service, which could be placed on the same host.

   2

   MONITOR_PORT is used to access the haproxy load status page.

   Example

   service_type: ingress
   service_id: nfs.mynfs
   placement:
     count: 2
   spec:
     backend_service: nfs.mynfs
     frontend_port: 2049
     monitor_port: 9000
     virtual_ip: 10.10.128.75/22

4. Deploy the ingress service:

   Example

   [ceph: root@host01 /]# ceph orch apply -i ingress.yaml

   Note

   Deployment of NFS daemons and the ingress service is asynchronous and the command might return before the services have completely started.

5. Check that the ingress services have successfully started:

   Syntax

   ceph orch ls --service_name=ingress.nfs.CLUSTER_ID

   Example

   [ceph: root@host01 /]# ceph orch ls --service_name=ingress.nfs.mynfs
   
   NAME                   PORTS                  RUNNING  REFRESHED  AGE  PLACEMENT
   ingress.nfs.mynfs  10.10.128.75:2049,9000      4/4   4m ago     22m  count:2

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Ensure the NFS module is enabled:

   Example

   [ceph: root@host01 /]# ceph mgr module ls | more

3. Exit out of the Cephadm shell and create the nfs.yaml file:

   Example

   [root@host01 ~]# touch nfs.yaml

4. Edit the nfs.yaml file to include the following details:

   Syntax

   service_type: nfs
   service_id: SERVICE_ID
   placement:
     hosts:
       - HOST_NAME_1
       - HOST_NAME_2
     count: COUNT
   spec:
     port: PORT_NUMBER

   Note

   The number of hosts you allocate for the NFS service must be greater than the number of active NFS servers you request to deploy, specified by the placement: count parameter.

   Example

   service_type: nfs
   service_id: cephfsnfs
   placement:
     hosts:
       - host02
       - host03
     count: 1
   spec:
     port: 12345

   In this example, the server is run on the non-default port of 12345, instead of the default port of 2049, on host02 and host03.

5. Mount the YAML file under a directory in the container:

   Example

   [root@host01 ~]# cephadm shell --mount nfs.yaml:/var/lib/ceph/nfs.yaml

6. Log into the Cephadm shell and navigate to the directory:

   Example

   [ceph: root@host01 /]# cd /var/lib/ceph/

7. Deploy the NFS service using service specification:

   Syntax

   ceph orch apply -i FILE_NAME.yaml

   Example

   [ceph: root@host01 ceph]# ceph orch apply -i nfs.yaml

   Note

   Deployment of the NFS service is asynchronous and the command might return before the services have completely started.

8. Check that the NFS services have successfully started:

   Syntax

   ceph orch ls --service_name=nfs.CLUSTER_ID

   Example

   [ceph: root@host01 /]# ceph orch ls --service_name=nfs.cephfsnfs
   
   NAME           PORTS    RUNNING  REFRESHED  AGE  PLACEMENT
   nfs.cephfsnfs  ?:12345      2/2  3m ago     13m  host02;host03

9. Exit out of the Cephadm shell and create the ingress.yaml file:

   Example

   [root@host01 ~]# touch ingress.yaml

10. Edit the ingress.yaml file to include the following details:

    Syntax

    service_type: SERVICE_TYPE
    service_id: SERVICE_ID
    placement:
      count: PLACEMENT
    spec:
      backend_service: SERVICE_ID_BACKEND
      frontend_port: FRONTEND_PORT
      monitor_port: MONITOR_PORT
      virtual_ip: VIRTUAL_IP_WITH_CIDR

    Example

    service_type: ingress
    service_id: nfs.cephfsnfs
    placement:
      count: 2
    spec:
      backend_service: nfs.cephfsnfs
      frontend_port: 2049
      monitor_port: 9000
      virtual_ip: 10.10.128.75/22

    Note

    In this example, placement: count: 2 deploys the keepalived and haproxy service on random nodes. To specify the nodes to deploy keepalived and haproxy on, use the placement: hosts option:

    Example

    service_type: ingress
    service_id: nfs.cephfsnfs
    placement:
      hosts:
        - host02
        - host03

11. Mount the YAML file under a directory in the container:

    Example

    [root@host01 ~]# cephadm shell --mount ingress.yaml:/var/lib/ceph/ingress.yaml

12. Log into the Cephadm shell and navigate to the directory:

    Example

    [ceph: root@host01 /]# cd /var/lib/ceph/

13. Deploy the ingress service using service specification:

    Syntax

    ceph orch apply -i FILE_NAME.yaml

    Example

    [ceph: root@host01 ceph]# ceph orch apply -i ingress.yaml

14. Check that the ingress services have successfully started:

    Syntax

    ceph orch ls --service_name=ingress.nfs.CLUSTER_ID

    Example

    [ceph: root@host01 /]# ceph orch ls --service_name=ingress.nfs.cephfsnfs
    
    NAME                   PORTS                  RUNNING  REFRESHED  AGE  PLACEMENT
    ingress.nfs.cephfsnfs  10.10.128.75:2049,9000      4/4   4m ago     22m  count:2

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Update the cluster:

   Syntax

   ceph orch apply nfs CLUSTER_NAME ["HOST_NAME_1,HOST_NAME_2,HOST_NAME_3"]

   The CLUSTER_NAME is an arbitrary string, HOST_NAME_1 is an optional string signifying the hosts to update the deployed NFS-Ganesha daemons.

   Example

   [ceph: root@host01 /]# ceph orch apply nfs nfsganesha "host02"

   This updates the nfsganesha cluster on host02.