Chapter 6. Managing Cluster Size

Procedure: Adding a Monitor by Using ceph-deploy

1. Edit the Ceph configuration file on the administration node:

   1. Add the monitor address:

      [mon.<hostname>]
      public_addr = <ip-address>

      Replace <hostname> with the host name of the new monitor node, <ip-address> with the node IP address, for example:

      [mon.node5]
      public_addr = 127.0.0.1

   2. Optionally, add the new monitor host to the initial quorum:

      [global]
      mon_initial_members = node0, node1, <hostname>

      Replace <hostname> with the monitor host name, for example:

      [global]
      mon_initial_members = node0, node1, node5

      Important

      If you add a monitor to a cluster that has only one monitor, you must add the next two monitors to the mon_initial_members and mon_host options. Production clusters require at least three monitors set in mon_initial_members and mon_host to ensure high availability.

      If you add two more monitors to a cluster with only one initial monitor, but do not list the new monitors under mon_initial_members and mon_host, the failure of the initial monitor will cause the cluster to lock up.

      If the monitor you are adding is replacing a monitor that is part of mon_initial_members and mon_host, the new monitor must be added to mon_initial_members and mon_host too.

2. Redistribute the updated Ceph configuration file to the nodes in the Ceph cluster:

   ceph-deploy --overwrite-conf config push <ceph-node0 ceph-node1 ...>

3. Add the monitor to the cluster:

   ceph-deploy mon add <hostname>

   Replace <hostname> with the monitor host name, for example:

   $ ceph-deploy mon add node5

4. Ensure the monitor joined the quorum:

   # ceph quorum_status --format json-pretty

5. Connect the monitor to the Calamari server:

   ceph-deploy calamari connect --master '<calamari-node-FQDN>' <hostname>

   Replace <hostname> with the monitor host name and <calamari-node-FQDN> with the fully qualified domain name of the Calamari server, for example:

   $ ceph-deploy calamari connect --master 'calamari.domain' node5

1. To ensure the cluster identifies the monitor on start/restart, add the monitor IP address to your Ceph configuration file.

   To add the monitor in the [mon] or [global] section of the Ceph configuration file, you may specify it in the mon_host setting, which is a list of DNS-resolvable hostnames or IP addresses (separated by "," or ";" or " "). You may also create a specific section in the Ceph configuration file for the monitor you are adding. For example:

   [mon]
   mon_host = {mon-ip:port} {mon-ip:port} {new-mon-ip:port}
   
   [mon.{mon-id}]
   host = {mon-id}

   If you want to make the monitor part of the initial quorum, you must also add the hostname to mon_initial_members in the [global] section of your Ceph configuration file.

   Important

   If you are adding a monitor to a cluster that has only one monitor, you MUST add the next two monitors to mon_initial_members and mon_host. Production clusters REQUIRE at least 3 monitors set in mon_initial_members and mon_host to ensure high availability. If a cluster with only one initial monitor adds two more monitors but does not add them to mon_initial_members and mon_host, the failure of the initial monitor will cause the cluster to lock up. If the monitor you are adding is replacing a monitor that is part of mon_initial_members and mon_host, the new monitor must be added to mon_initial_members and mon_host too.

   For example:

   mon_initial_members = mon-node1, mon-node2, new-mon-node
   mon_host = mon-node1, mon-node2, new-mon-node

   Also, ensure you have pid file = /var/run/ceph/$name.pid set in the [global] section of your Ceph configuration file.

   Finally, push a new copy of the Ceph configuration file to your Ceph nodes and Ceph clients.

   ceph-deploy --overwrite-conf config push <ceph-node0 ceph-node1 ...>

2. Create the default directory on the machine that will host your new monitor.

   ssh {new-mon-host}
   sudo mkdir /var/lib/ceph/mon/ceph-{mon-id}

3. Create a temporary directory {tmp} to keep the files needed during this process. This directory should be different from the monitor’s default directory created in the previous step, and can be removed after all the steps are executed.

   mkdir {tmp}

4. Copy the admin key from your admin node to the monitor node so that you can run ceph CLI commands.

   ceph-deploy --overwrite-conf admin <ceph-node>

5. Retrieve the keyring for your monitors, where {tmp} is the path to the retrieved keyring, and {key-filename} is the name of the file containing the retrieved monitor key.

   ceph auth get mon. -o {tmp}/{key-filename}

6. Retrieve the monitor map, where {tmp} is the path to the retrieved monitor map, and {map-filename} is the name of the file containing the retrieved monitor monitor map.

   ceph mon getmap -o {tmp}/{map-filename}

7. Prepare the monitor’s data directory created in the first step. You must specify the path to the monitor map so that you can retrieve the information about a quorum of monitors and their fsid. You must also specify a path to the monitor keyring:

   sudo ceph-mon -i {mon-id} --mkfs --monmap {tmp}/{map-filename} --keyring {tmp}/{key-filename}

8. Start the new monitor and it will automatically join the cluster. The daemon needs to know which address to bind to, either via --public-addr {ip:port} or by setting mon addr in the appropriate section of ceph.conf. For example:

   sudo ceph-mon -i {mon-id} --public-addr {ip:port} --pid-file /var/run/ceph/mon.{mon-id}.pid

9. Finally, from the admin node in the directory where you keep you cluster’s Ceph configuration, connect your monitor to Calamari.

   ceph-deploy calamari connect --master '<calamari-node-FQDN>' <ceph-node>[<ceph-node> ...]

1. Stop the monitor. :

   service ceph -a stop mon.{mon-id}

2. Remove the monitor from the cluster. :

   ceph mon remove {mon-id}

3. On your admin node, remove the monitor entry from your Ceph configuration file.

4. Redistribute the Ceph configuration file.

   ceph-deploy --overwrite-conf config push <ceph-node0 ceph-node1 ..>

5. Archive the monitor data (optional).

   mv /var/lib/ceph/mon/{cluster}-{daemon-id} /var/lib/ceph/mon/removed-{cluster}-{daemon-id}

6. Remove the monitor data (only if previous archive step not executed).

   sudo rm -r /var/lib/ceph/mon/{cluster}-{daemon-id}

1. Identify a surviving monitor and log in to that host. :

   ceph mon dump
   ssh {mon-host}

2. Stop the ceph-mon daemon and extract a copy of the monap file. :

   service ceph stop mon || stop ceph-mon-all
       ceph-mon -i {mon-id} --extract-monmap {map-path}
   # for example,
       ceph-mon -i a --extract-monmap /tmp/monmap

3. Remove the non-surviving monitors. For example, if you have three monitors, mon.a, mon.b, and mon.c, where only mon.a will survive, follow the example below:

   monmaptool {map-path} --rm {mon-id}
   # for example,
   monmaptool /tmp/monmap --rm b
   monmaptool /tmp/monmap --rm c

4. Inject the surviving map with the removed monitors into the surviving monitors. For example, to inject a map into monitor mon.a, follow the example below:

   ceph-mon -i {mon-id} --inject-monmap {map-path}
   # for example,
   ceph-mon -i a --inject-monmap /tmp/monmap

Procedure: Adding an OSD by Using ceph-deploy

1. From your administration node, prepare the OSDs:

   ceph-deploy osd prepare <hostname>:<disk_device> [<ceph_node>:<disk_device>]

   For example:

   $ ceph-deploy osd prepare node2:/dev/sdb

   The prepare command creates two partitions on a disk device; one partition is for OSD data, and the other one is for the OSD journal.

2. Activate the OSD:

   ceph-deploy osd activate <hostname>:<data_partition>

   For example:

   $ ceph-deploy osd activate node2:/dev/sdb1

   Note

   In the ceph-deploy osd activate command, specify a particular disk partition, for example /dev/sdb1.

   It is also possible to use a disk device that is wholly formatted without a partition table. In that case, a partition on an additional disk must be used to serve as the journal store:

   ceph-deploy osd activate <hostame>:<disk_device>:<journal_partition>

   In the following example, sdd is a spinning hard drive that Ceph uses entirely for OSD data. ssdb1 is a partition of an SSD drive, which Ceph uses to store the OSD journal:

   $ ceph-deploy osd activate node{2,3,4}:sdd:ssdb1

1. On the administration node, prepare the encrypted device by using the ceph-deploy utility with the --dmcrypt and --dymcrypt-key-dir options:

   ceph-deploy osd prepare --dmcrypt --dmcrypt-key-dir <directory> <hostname>:<device>

   Replace <directory> with the path to the directory to store the key for the encrypted OSD, <hostname> with the host name of the OSD node, and <device> with the device that the OSD will use:

   ceph-deploy osd prepare --dmcrypt --dmcrypt-key-dir /etc/ceph/dmcrypt-keys node2:/dev/sdd

   The --dmcrypt-key-dir option is optional.

2. On the OSD node, open the newly created partition by using the cryptsetup utility:

   # /sbin/cryptsetup --key-file <key_path> luksOpen <data_partition> <uuid/name>

   Replace <key_path> with the path to the key for the encrypted partition, <data_partition> with the OSD partition, and <uuid> with the UUID of the partition:

   $ /sbin/cryptsetup --key-file /etc/ceph/dmcrypt-keys/596e7f3b-b565-40c8-bf70-451a1dafddbc.luks.key luksOpen /dev/sdd1 596e7f3b-b565-40c8-bf70-451a1dafddbc

3. From the administration node, activate the OSD:

   ceph-deploy osd activate <hostname>:/dev/dm-<data_partition_id>

   Replace <hostname> with the host name of the OSD node, <id> with the ID of the encrypted partition:

   $ ceph-deploy osd activate node2:/dev/dm-1

   To view the ID, use the ll command on the OSD node:

   $ ll /dev/mapper/

1. Create an OSD. If no UUID is given, it will be set automatically when the OSD starts up. To output the OSD number that is needed for subsequent steps:

   $ ceph osd create [{uuid}]

2. Create the default directory on the new OSD node:

   # mkdir /var/lib/ceph/osd/ceph-<osd-number>

3. If the OSD is for a drive other than the OS drive, prepare it for use with Ceph, and mount it to the directory you just created:

   # mkfs -t <fstype> /dev/<drive>
   # mount -o user_xattr /dev/<hdd> /var/lib/ceph/osd/ceph-<osd-number>

4. Initialize the OSD data directory:

   $ ceph-osd -i <osd-num> --mkfs --mkkey

   The directory must be empty before executing ceph-osd.

5. Register the OSD authentication key. The value of ceph for ceph-<osd-num> in the path is $cluster-$id. If your cluster name differs from ceph, use your cluster name instead:

   $ ceph auth add osd.<osd-num> osd 'allow *' mon 'allow rwx' -i /var/lib/ceph/osd/ceph-<osd-num>/keyring

6. Add the OSD to the CRUSH map so that the OSD can start receiving data. If you specify at least one bucket, the command will place the OSD into the most specific bucket you specify, and it will move that bucket underneath any other buckets you specify.

   Important

   If you specify only the root bucket, the command will attach the OSD directly to the root, but CRUSH rules expect OSDs to be inside of the host bucket.

   $ ceph osd crush add <id-or-name> <weigh>  [<bucket-type>=<bucket-name> ...]

   Note

   You can also decompile the CRUSH map, add the OSD to the device list, add the host as a bucket if it is not already in the CRUSH map, add the device as an item in the host, assign a weight to the device, recompile the CRUSH map, and set the CRUSH map.

1. Remove the OSD from the CRUSH map so that it no longer receives data. You may also decompile the CRUSH map, remove the OSD from the device list, remove the device as an item in the host bucket or remove the host bucket (if it’s in the CRUSH map and you intend to remove the host), recompile the map and set it. See the Storage Strategies guide for details.

   ceph osd crush remove {name}

2. Remove the OSD authentication key.

   ceph auth del osd.{osd-num}

   The value of ceph for ceph-{osd-num} in the path is the $cluster-$id. If your cluster name differs from ceph, use your cluster name instead.

3. Remove the OSD.

   ceph osd rm {osd-num}
   #for example
   ceph osd rm 1

4. Navigate to the host where you keep the master copy of the cluster’s ceph.conf file.

   ssh {admin-host}
   cd /etc/ceph
   vim ceph.conf

5. Remove the OSD entry from your ceph.conf file (if it exists). :

   [osd.1]
   host = {hostname}

6. From the host where you keep the master copy of the cluster’s ceph.conf file, copy the updated ceph.conf file to the /etc/ceph directory of other hosts in your cluster.