Red Hat Training

A Red Hat training course is available for Red Hat Ceph Storage

Chapter 6. Managing Cluster Size

Managing cluster size generally involves adding or removing Ceph Monitors or OSDs. If you are bootstrapping a cluster for the first time, see the Red Hat Ceph Storage 2 Installation Guide for Red Hat Enterprise Linux or Ubuntu.

6.1. Adding a Monitor

Ceph Monitors are light-weight processes that maintain a master copy of the cluster map. All Ceph clients contact a Ceph Monitor and retrieve the current copy of the cluster map, enabling clients to bind to a pool and read and write data.

Ceph Monitors use a variation of the Paxos protocol to establish consensus about maps and other critical information across the cluster. Due to the nature of Paxos, Ceph requires a majority of Monitors running to establish a quorum thus establishing consensus.

Note

Red Hat requires at least three Monitors on separate hosts to receive support for a production cluster.

Red Hat recommends deploying an odd number of Monitors, but it is not mandatory. An odd number of Monitors has a higher resiliency to failures than an even number of Monitors. For example, to maintain a quorum on a two Monitor deployment, Ceph cannot tolerate any failures; with three Monitors, one failure; with four Monitors, one failure; with five Monitors, two failures. This is why an odd number is advisable. Summarizing, Ceph needs a majority of Monitors to be running and to be able to communicate with each other, two out of three, three out of four, and so on.

For an initial deployment of a multi-node Ceph cluster, Red Hat requires three Monitors, increasing the number two at a time if a valid need for more than three Monitors exists.

Since Monitors are light-weight, it is possible to run them on the same host as OpenStack nodes. However, Red Hat recommends running Monitors on separate hosts. Co-locating Monitors and OSDs on the same node can impair performance and is not supported.

Note

A majority of Monitors in the storage cluster must be able to reach each other in order to establish a quorum.

6.1.1. Configuring a Host

When adding Ceph Monitors to a cluster, deploy them on separate nodes. Running Ceph Monitors on the same node does not provide any additional high availability assurance if a node fails. Ideally, the node hardware should be uniform for all Monitor nodes in the storage cluster.

For details on the minimum recommendations for Ceph Monitor hardware, see the Red Hat Ceph Storage Hardware Guide.

Before doing a new installation, review the requirements listed in the Prerequisites section in the Red Hat Ceph Storage Installation Guide for Red Hat Enterprise Linux or Ubuntu.

Add a new Monitor node to a rack in the cluster, connect it to the network and ensure that the Monitor has network connectivity.

Important

You must install NTP and open port 6789.

6.1.2. Adding a Monitor with Ansible

Before You Start…

Review the prerequisite procedures for using Red Hat Enterprise Linux or Ubuntu nodes.

Red Hat recommends adding two Monitors at a time to maintain an odd number of Monitors. For example, if you have a three Monitor cluster, Red Hat recommends expanding it to a five Monitor cluster.

To add a new Ceph Monitor to an existing Ansible cluster, simply open the /etc/ansible/hosts file and add another Monitor node under the [mons] section. Next, run the Ansible playbook again.

As root, add the Ceph Monitor hosts to the /etc/ansible/hosts file. Add Monitors under a [mons] section:

[mons]
<monitor-host-name>
<monitor-host-name>
<monitor-host-name>
<new-monitor-host-name>
<new-monitor-host-name>

Ensure that Ansible can reach the Ceph nodes:
```
# ansible all -m ping
```
Navigate to the Ansible configuration directory:
```
# cd /usr/share/ceph-ansible
```
Run the Ansible playbook:
```
$ ansible-playbook site.yml
```
If adding a monitor in Ceph deployed in a container:
```
$ ansible-playbook site-docker.yml
```
++ NOTE: If adding a node after upgrading ceph-ansible from a previous version, be sure to use the corresponding version of the site-docker.yml and site.yml files with respect to ceph-ansible. Updating ceph-ansible replaces site-docker.yml sample and site.yml sample, but not site-docker.yml and site.yml.

Ansible will add the new Monitors to the Ceph storage cluster.

6.1.3. Adding a Monitor with the Command Line Interface

This procedure creates a ceph-mon data directory, retrieves the Monitor map and Monitor keyring, and adds a ceph-mon daemon to the cluster. Add more Ceph Monitors by repeating this procedure until you have a sufficient number of ceph-mon daemons to achieve a quorum.

By convention, Ceph Monitors use the host name of the node where the ceph-mon daemon is running, and Red Hat recommends only running one Monitor daemon per node. The Ceph Monitor identifier in the Ceph configuration file can be defined as you see fit. For the purpose of this document, the Monitor identifier, <mon_id>, is the host name of the Monitor node.

Add the Red Hat Ceph Storage 2 Monitor repository.
1. Red Hat Enterprise Linux
2. Ubuntu
As root, install the ceph-mon package on the new Ceph Monitor node:
Red Hat Enterprise Linux
```
# yum install ceph-mon
```
Ubuntu
```
$ sudo apt-get install ceph-mon
```
To ensure the cluster identifies the Monitor on start or restart, add the Monitor IP address to the Ceph configuration file.
To add the new Monitor in the [mon] or [global] section of the Ceph configuration file on an existing Monitor node in the storage cluster. The mon_host setting, which is a list of DNS-resolvable host names or IP addresses, separated by "," or ";" or " ". Optionally, you can create a specific section in the Ceph configuration file for the new Monitor node:
Syntax
```
[mon]
mon host = <mon_ip:port> <mon_ip:port> ... <new_mon_ip:port>

[mon.<mon_id>]
host = <mon_id>
mon addr = <mon_ip>
```
If you want to make the Monitor part of the initial quorum, you must also add the host name to the mon_initial_members parameter in the [global] section of the Ceph configuration file.
Important
If you are adding a Monitor to a cluster that has only one Monitor, you MUST add the next two Monitors to mon_initial_members and mon_host. Production clusters REQUIRE at least three Monitors set in mon_initial_members and mon_host to ensure high availability. If a cluster with only one initial Monitor adds two more Monitors but does not add them to mon_initial_members and mon_host, the failure of the initial Monitor will cause the cluster to lock up. If the Monitor you are adding is replacing a Monitor that is part of mon_initial_members and mon_host, the new Monitor must be added to mon_initial_members and mon_host too.
Example
```
[global]
mon initial members = node1 node2 node3 node4
...
[mon]
mon host = 192.168.0.1:6789 192.168.0.2:6789 192.168.0.3:6789 192.168.0.4:6789
...
[mon.node4]
host = node4
mon addr = 192.168.0.4
```
Finally, send a new copy of the Ceph configuration file to the Ceph nodes and Ceph clients.
Syntax
```
# scp /etc/ceph/<cluster_name>.conf <target_host_name>:/etc/ceph
```
Example
```
# scp /etc/ceph/ceph.conf node4:/etc/ceph
```

Create the default Monitor directory on the new node:

Syntax

# mkdir /var/lib/ceph/mon/<cluster_name>-<mon_id>

Example

# mkdir /var/lib/ceph/mon/ceph-node4

Create a temporary directory <temp_directory> to keep the files needed during this process. This directory should be different from the Monitor’s default directory created in the previous step, and can be removed after all the steps are executed:
Syntax
```
# mkdir <temp_directory>
```
Exmaple
```
# mkdir /tmp/ceph
```

Copy the admin key from a running Monitor node to the new Monitor node so that you can run ceph commands:

Syntax

# scp /etc/ceph/<cluster_name>.client.admin.keyring <target_host_name>:/etc/ceph

Example

# scp /etc/ceph/ceph.client.admin.keyring node4:/etc/ceph

Retrieve the Monitor keyring, where <temp_directory> is the path to the retrieved keyring, and <key_file_name> is the name of the file containing the retrieved Monitor keyring:
Syntax
```
# ceph auth get mon. -o /<temp_directory>/<key_file_name>
```
Example
```
# ceph auth get mon. -o /tmp/ceph/ceph_keyring.out
```
Retrieve the Monitor map, where <tmp> is the path to the retrieved Monitor map, and <mon_map_file_name> is the name of the file containing the retrieved Monitor map:
Syntax
```
# ceph mon getmap -o /<temp_directory>/<mon_map_filename>
```
Example
```
# ceph mon getmap -o /tmp/ceph/ceph_mon_map.out
```
Prepare the Monitor’s data directory created in the fourth step. You must specify the path to the Monitor map so that you can retrieve the information about a quorum of Monitors and their fsid. You must also specify a path to the Monitor keyring:
Syntax
```
# ceph-mon -i <mon_id> --mkfs --monmap /<temp_directory>/<mon_map_filename> --keyring /<temp_directory>/<key_file_name>
```
Example
```
# ceph-mon -i node4 --mkfs --monmap /tmp/ceph/ceph_mon_map.out --keyring /tmp/ceph/ceph_keyring.out
```
For storage clusters with custom names, as root, add the the following line to the appropriate file:
Red Hat Enterprise Linux
```
# echo "CLUSTER=<custom_cluster_name>" >> /etc/sysconfig/ceph
```
Ubuntu
```
$ sudo echo "CLUSTER=<custom_cluster_name>" >> /etc/default/ceph
```

As root, update the owner and group permissions:

Syntax

# chown -R <owner>:<group> <path_to_directory>

Example

# chown -R ceph:ceph /var/lib/ceph/mon
# chown -R ceph:ceph /var/log/ceph
# chown -R ceph:ceph /var/run/ceph
# chown -R ceph:ceph /etc/ceph

As root, enable and start the ceph-mon process on the new Monitor node:

Syntax

# systemctl enable ceph-mon.target
# systemctl enable ceph-mon@<monitor_host_name>
# systemctl start ceph-mon@<monitor_host_name>

Example

# systemctl enable ceph-mon.target
# systemctl enable ceph-mon@node4
# systemctl start ceph-mon@node4

6.2. Removing a Monitor

When you remove monitors from a cluster, consider that Ceph monitors use PAXOS to establish consensus about the master cluster map. You must have a sufficient number of monitors to establish a quorum for consensus about the cluster map.

6.2.1. Removing a Monitor with the Command Line Interface

This procedure removes a ceph-mon daemon from the storage cluster. If this procedure results in only two monitor daemons, you may add or remove another monitor until you have a number of ceph-mon daemons that can achieve a quorum.

As root, stop the monitor:

Syntax

# systemctl stop ceph-mon@<monitor_hostname>

Example

# systemctl stop ceph-mon@node1

Remove the monitor from the cluster:
```
# ceph mon remove <mon_id>
```
On the Ceph Monitor node, remove the Monitor entry from the Ceph configuration file.

Redistribute the Ceph configuration file:

# scp /etc/ceph/ceph.conf <user>@<target_hostname>:/etc/ceph/

Optionally, archive the monitor data:

# mv /var/lib/ceph/mon/<cluster>-<daemon_id> /var/lib/ceph/mon/removed-<cluster>-<daemon_id>

Remove the monitor data, only if the previous archive step is executed:
```
# rm -r /var/lib/ceph/mon/<cluster>-<daemon_id>
```

6.2.2. Removing Monitors from an Unhealthy Cluster

This procedure removes a ceph-mon daemon from an unhealhty cluster. For example, a cluster that has placement groups persistently not active + clean.

Identify a surviving monitor and log in to that host. :
```
# ceph mon dump
# ssh <monitor_hostname>
```

As root, stop the ceph-mon daemon and extract a copy of the monap file. :

Syntax

# systemctl stop ceph-mon@<monitor_hostname>
# ceph-mon -i <monitor_id> --extract-monmap <map_path>

Example

# systemctl stop ceph-mon@node1
# ceph-mon -i a --extract-monmap /tmp/monmap

Remove the non-surviving monitors. For example, if you have three monitors, mon.a, mon.b, and mon.c, where only mon.a will survive, follow the example below:
Syntax
```
# monmaptool <map_path> --rm <monitor_id>
```
Example
```
# monmaptool /tmp/monmap --rm b
# monmaptool /tmp/monmap --rm c
```
Inject the surviving map with the removed monitors into the surviving monitors. For example, to inject a map into monitor mon.a, follow the example below:
Syntax
```
# ceph-mon -i <monitor_id> --inject-monmap <map_path>
```
Example
```
# ceph-mon -i a --inject-monmap /tmp/monmap
```

6.3. Adding an OSD

When a cluster is up and running, you can add OSDs or remove OSDs from the cluster at runtime.

A Ceph OSD generally consists of one ceph-osd daemon for one storage drive and its associated journal within a host machine. If a host has multiple storage drives, map one ceph-osd daemon for each drive.

Red Hat recommends checking the capacity of a cluster regularly to see if it is reaching the upper end of its storage capacity. As a cluster reaches its near full ratio, add one or more OSDs to expand the cluster’s capacity.

Important

Do not let a cluster reach the full ratio before adding an OSD. OSD failures that occur after the cluster reaches the near full ratio can cause the cluster to exceed the full ratio. Ceph blocks write access to protect the data until you resolve the storage capacity issues. Do not remove OSDs without considering the impact on the full ratio first.

6.3.1. Configuring a Host

OSDs and their supporting hardware should be similarly configured as a storage strategy for the pool(s) that will use the OSDs. Ceph prefers uniform hardware across pools for a consistent performance profile. For best performance, consider a CRUSH hierarchy with drives of the same type or size. View the Storage Strategies guide for details.

If you add drives of dissimilar size, adjust their weights. When you add the OSD to the CRUSH map, consider the weight for the new OSD. Hard drive capacity grows approximately 40% per year, so newer OSD hosts might have larger hard drives than older hosts in the cluster, that is, they might have greater weight.

Before doing a new OSD installation, review the requirements listed in the Prerequisites section in the Red Hat Ceph Storage Installation Guide for Red Hat Enterprise Linux or Ubuntu.

6.3.2. Adding an OSD with Ansible

Before You Start…

Review the prerequisite procedures for using Red Hat Enterprise Linux or Ubuntu nodes.

Using the Same Disk Topology

To add a new Ceph OSD node(s) to an existing storage cluster, under Ansible’s control, open the /etc/ansible/hosts file and add the new OSD node under the [osds] section.

An OSD node(s) should have the same number of OSD data drives and the same drive paths as other OSD nodes in the storage cluster. Ansible will add the same number of OSDs as other OSD nodes using the same paths specified in the devices: section of the /usr/share/ceph-ansible/group_vars/osds file.

As root, add the Ceph OSD host(s) to the /etc/ansible/hosts file, under the [osds] section:
```
[osds]
...
<existing_osd-host-name>
<new_osd-host-name>
```
Ensure that Ansible can reach the Ceph hosts:
```
$ ansible all -m ping
```
Navigate to the Ansible configuration directory:
```
$ cd /usr/share/ceph-ansible
```
Run the Ansible playbook:
```
$ ansible-playbook site.yml
```
If adding an OSD in Ceph deployed in a container:
```
$ ansible-playbook site-docker.yml
```
++ NOTE: If adding a node after upgrading ceph-ansbile from a previous version, be sure to use the corresponding version of the site-docker.yml and site.yml files with respect to ceph-ansible. Updating ceph-ansible replaces site-docker.yml sample and site.yml sample, but not site-docker.yml and site.yml.

Ansible will add the new OSD node(s) to the Ceph storage cluster.

Note

The new OSD node(s) will have the same configuration as the rest of the OSDs.

Using Different Disk Topologies

If the new Ceph OSD node(s) is(are) using a different disk topology, then there are two approaches for adding the new OSD node(s) to an existing storage cluster.

First Approach
1. As root, add the new Ceph OSD node(s) to the /etc/ansible/hosts file, under the [osds] section. Append the new OSD node name to the end of the OSD list. For example, adding the OSD node new_osdnode10:
```
[osds]
...
existing_osdnode09
new_osdnode10
```
2. As root, create a new file for each new Ceph OSD node added to the storage cluster. Name the new file the same as the new Ceph OSD node name. Create this new file under the /etc/ansible/host_vars/ directory, for example:
```
# touch /etc/ansible/host_vars/new_osdnode10
```
  1. As root, edit the new file, and add the devices: and raw_journal_devices: sections to the file. Under each of these sections add a -, space, then the full path to the block device names for this OSD node. For example:
    devices: - /dev/sdc - /dev/sdd - /dev/sde - /dev/sdf raw_journal_devices: - /dev/sda - /dev/sda - /dev/sdb - /dev/sdb
3. Ensure that Ansible can reach the Ceph hosts:
```
$ ansible all -m ping
```
4. Navigate to the Ansible configuration directory:
```
$ cd /usr/share/ceph-ansible
```
5. Run the Ansible playbook:
```
$ ansible-playbook site.yml
```

Ansible will add the new OSD node(s) to the Ceph storage cluster.

Second Approach
1. As root, modify the existing /etc/ansible/hosts file. Under the [osds] section, add the new OSD node name, and using the devices and raw_journal_devices settings, specify the different disk topology. For example:
```
[osds]
...
osdnode10 devices="['/dev/sdc', '/dev/sdd', '/dev/sde', '/dev/sdf']" raw_journal_devices="['/dev/sda', '/dev/sda', '/dev/sdb', '/dev/sdb']"
```
2. Ensure that Ansible can reach the Ceph hosts:
```
$ ansible all -m ping
```
3. Navigate to the Ansible configuration directory:
```
$ cd /usr/share/ceph-ansible
```
4. Run the Ansible playbook:
```
$ ansible-playbook site.yml
```

Ansible will add the new OSD node(s) to the Ceph storage cluster.

See Also..

For more details on the different OSD settings, see the Configuring Ceph OSD Settings table in the Red Hat Ceph Storage 2 Installation Guide.

6.3.3. Adding an OSD with the Command Line Interface

To add a new OSD node manually:

Note

Enter the commands that start with the number sign (#) as root or with the sudo utility.

In addition, if you use a custom cluster name, use the --cluster <cluster-name> option with the ceph and ceph-osd commands.

Before you Start

Add the Red Hat Ceph Storage 2 OSD repository:
1. Red Hat Enterprise Linux
2. Ubuntu
Create the /etc/ceph/ directory:
```
# mkdir /etc/ceph
```

On the new OSD node, copy the Ceph administration keyring and configuration file from the initial Monitor node:

scp <user_name>@<monitor_host_name>:/etc/ceph/<cluster_name>.client.admin.keyring /etc/ceph
scp <user_name>@<monitor_host_name>:/etc/ceph/<cluster_name>.conf /etc/ceph

For example:

# scp root@node1:/etc/ceph/ceph.client.admin.keyring /etc/ceph/
# scp root@node1:/etc/ceph/ceph.conf /etc/ceph/

Installing ceph-osd and Creating a New OSD Instance

Install the ceph-osd package on the new Ceph OSD node:
1. For Red Hat Enterprise Linux nodes:
```
# yum install ceph-osd
```
2. For Ubuntu nodes:
```
$ sudo apt-get install ceph-osd
```
Generate the Universally Unique Identifier (UUID) for the new OSD:
```
$ uuidgen
ebd24968-0f3a-499c-a5b7-929c71ac8c1a
```
Create the OSD instance. Note the OSD number identifier because you will need it for subsequent procedures.
```
ceph osd create <uuid> [<osd_id>]
```
For example:
```
# ceph osd create ebd24968-0f3a-499c-a5b7-929c71ac8c1a
4
```

Preparing the OSD Data and Journal Drives

Create the default directory for the new OSD:

mkdir /var/lib/ceph/osd/<cluster_name>-<osd_id>

For example:

# mkdir /var/lib/ceph/osd/ceph-4

Create a new GUID partition table on the drive to be used as an OSD:
```
parted <path_to_disk> mklabel gpt
```
For example:
```
# parted /dev/sdd mklabel gpt
```
If you want to use a separate drive for the journal, create a new GUID partition table on it, too, for example:
```
# parted /dev/sde mklabel gpt
```
Create the data and journal partitions:
```
parted <path_to_disk> mkpart primary <start> <end>
```
For example:
```
# parted /dev/sdd mkpart primary 1 10000
# parted /dev/sdd mkpart primary 10001 16000
```
If you use a separate drive for the journal, create the journal partition on it, for example:
```
# parted /dev/sde mkpart primary 1 6000
```
Note
The minimum size of a journal partition is 5 GB. Journal partitions bigger than 10 GB are not usually needed.

Create a file system on the data partition:

mkfs -t <fstype> <path_to_partition>

For example:

# mkfs -t xfs /dev/sdd1

Set the type code of the partitions using the proper GUID value:
```
# sgdisk --typecode=<partnum:{GUID}> -- <path to the device>
```
The GUID for Ceph OSD is 4fbd7e29-9d25-41b8-afd0-062c0ceff05d, and the GUID for Ceph Journal is 45B0969E-9B03-4F30-B4C6-B4B80CEFF106. For example:
```
# sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sdd
# sgdisk --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 -- /dev/sdd
```

Mount the data partition to the directory created in the first step:

mount -o noatime <path_to_partition> /var/lib/ceph/osd/<cluster_name>-<osd_id>

For example:

# mount -o noatime /dev/sdd1 /var/lib/ceph/osd/ceph-4

Mount the data partition permanently by editing the /etc/fstab file:

echo "<path_to_partition>  /var/lib/ceph/osd/<cluster_name>-<osd_id>   xfs defaults,noatime 1 2" >> /etc/fstab

For example:

# echo "/dev/sdd1 /var/lib/ceph/osd/ceph-4  xfs defaults,noatime 1 2" >> /etc/fstab

Initializing the OSD Data and Journal Directory and Registering the OSD Authentication Key

Initialize the OSD data directory:

ceph-osd -i <osd_id> --mkfs --mkkey --osd-uuid <uuid>

For example:

# ceph-osd -i 4 --mkfs --mkkey --osd-uuid ebd24968-0f3a-499c-a5b7-929c71ac8c1a
... auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory
... created new key in keyring /var/lib/ceph/osd/ceph-4/keyring

If you use a separate drive for the journal, use the --osd-journal option to specify the journal drive:

ceph-osd -i <osd_id> --mkfs --mkkey --osd-uuid <uuid> --osd-journal <path_to_journal>

For example:

# ceph-osd -i 4 --mkfs --mkkey --osd-uuid ebd24968-0f3a-499c-a5b7-929c71ac8c1a --osd-journal /dev/sde1

Important

The directory must be empty before you run ceph-osd with the --mkkey option.

If you use a separate drive for the journal, perform the following steps:

Determine the UUID of the journal drive:
```
$ ls -l /dev/disk/by-partuuid/*
```

Create a symbolic link between the journal drive and the journal file:

ln -s /dev/disk/by-partuuid/<uuid> /var/lib/ceph/osd/<cluster_name>-<osd_id>/journal

For example:

# ln -s /dev/disk/by-partuuid/62bc7076-0533-4440-9024-ddaeea4f7a7b
/var/lib/ceph/osd/ceph-4/journal

Create the journal on the journal disk:

ceph-osd -i <osd-id> --mkjournal

For example:

# ceph-osd -i 4 --mkjournal

ceph auth add osd.<osd_id> osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/<cluster_name>-<osd_id>/keyring

For example:

# ceph auth add osd.4 osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-4/keyring
added key for osd.4

Adding the New OSD Node to the CRUSH map

Add the OSD to the CRUSH map so that the OSD can start receiving data:

ceph osd crush add <osd_id> <weight> [<bucket_type>=<bucket_name> ...]

For example:

# ceph osd crush add 4 1 host=node4

If you specify more than one bucket, the command places the OSD into the most specific bucket out of those you specified, and it moves the bucket underneath any other buckets you specified.

Important

If you specify only the root bucket, the command attaches the OSD directly to the root, but CRUSH rules expect OSDs to be inside of the host bucket.

Note

You can also edit the CRUSH map manually. See the Editing a CRUSH map section in the Storage Strategies Guide for Red Hat Ceph Storage 2.

Updating the Owner and Group Permissions

Update the owner and group permissions for the newly created directories:

chown -R <owner>:<group> <path_to_directory>

For example:

# chown -R ceph:ceph /var/lib/ceph/osd
# chown -R ceph:ceph /var/log/ceph
# chown -R ceph:ceph /var/run/ceph
# chown -R ceph:ceph /etc/ceph

Enabling and Starting the `ceph-osd` Daemon

If you use clusters with custom names, add the following line to the appropriate file:
1. For Red Hat Enterprise Linux nodes:
```
# echo "CLUSTER=<custom_cluster_name>" >> /etc/sysconfig/ceph
```
2. For Ubuntu nodes:
```
$ sudo echo "CLUSTER=<custom_cluster_name>" >> /etc/default/ceph
```

To ensure that the new OSD is up and ready to receive data, enable and start the OSD process:

systemctl enable ceph-osd@<osd_id>
systemctl start ceph-osd@<osd_id>

For example:

# systemctl enable ceph-osd@4
# systemctl start ceph-osd@4

6.3.4. Observing Data Migration

When you add an OSD to the CRUSH map, Ceph begins rebalacing the server by migrating placement groups to the new OSD. To observe this process, use the ceph command line utility:

# ceph -w

The placement group states change from active+clean to active, some degraded objects, and finally active+clean when migration completes. To exit the utility, press Ctrl + C.

6.4. Removing an OSD

When you want to reduce the size of a cluster or replace hardware, you may remove an OSD at runtime. With Ceph, an OSD is generally one Ceph ceph-osd daemon for one storage drive within a host machine. If the node has multiple storage drives, you may need to remove one ceph-osd daemon for each drive. Generally, it’s a good idea to check the capacity of the storage cluster to see if you are reaching the upper end of its capacity. Ensure that when you remove an OSD that the storage cluster is not at its near full ratio.

Warning

Do not let the storage cluster reach its full ratio when removing an OSD. Removing OSDs could cause the cluster to reach or exceed its full ratio.

6.4.1. Removing an OSD with the Command Line Interface

This procedure takes out an OSD, removes an OSD from a cluster map, removes its authentication key, removes the OSD from the OSD map, and removes the OSD from the ceph.conf file. If the node has multiple drives, you may need to remove an OSD for each drive by repeating this procedure.

After you take an OSD out of the cluster, it may still be running. That is, the OSD may be up and out. You must stop the OSD before you remove it from the configuration. As root, disable and stop the OSD process:
Syntax
```
# systemctl disable ceph-osd@<osd_id>
# systemctl stop ceph-osd@<osd_id>
```
Example
```
# systemctl disable ceph-osd@4
# systemctl stop ceph-osd@4
```
Once the OSD is stopped, it is down.
Before removing the OSD, the OSD needs to be taken out of the storage cluster:
Syntax
```
# ceph osd out <osd_id>
```
Example
```
# ceph osd out 4
```
Important
Once the OSD is out, Ceph will start rebalancing and copying data to other OSDs in the storage cluster. Red Hat recommends waiting until the storage cluster becomes active+clean before proceeding to the next step. To observe the data migration, run the following command:
```
# ceph -w
```
Remove the OSD from the CRUSH map so that it no longer receives data. You may also decompile the CRUSH map, remove the OSD from the device list, remove the device as an item in the host bucket or remove the host bucket, if it’s in the CRUSH map and you intend to remove the host, recompile the map and set it. See the Storage Strategies Guide for details.
Syntax
```
# ceph osd crush remove <osd_name>
```
Example
```
# ceph osd crush remove osd.4
```

Remove the OSD authentication key:

Syntax

# ceph auth del osd.<osd_id>

Example

# ceph auth del osd.4

Remove the OSD:
Syntax
```
# ceph osd rm <osd_id>
```
Example
```
# ceph osd rm 4
```
Navigate to the host where you keep the master copy of the storage cluster’s ceph.conf file:
```
# cd /etc/ceph
# vim ceph.conf
```
Remove the OSD entry from the ceph.conf file, if it exists:
```
[osd.4]
host = <hostname>
```
Remove the reference to the OSD in the /etc/fstab file, if the OSD was added manually.
From the node where you keep the master copy of the storage cluster’s ceph.conf file, copy the updated ceph.conf file to the /etc/ceph/ directory of other nodes in the storage cluster.

Select Your Language

Language:

Language:

Red Hat Training

Chapter 6. Managing Cluster Size

6.1. Adding a Monitor

6.1.1. Configuring a Host

6.1.2. Adding a Monitor with Ansible

6.1.3. Adding a Monitor with the Command Line Interface

6.2. Removing a Monitor

6.2.1. Removing a Monitor with the Command Line Interface

6.2.2. Removing Monitors from an Unhealthy Cluster

6.3. Adding an OSD

6.3.1. Configuring a Host

6.3.2. Adding an OSD with Ansible

6.3.3. Adding an OSD with the Command Line Interface

Before you Start

Installing ceph-osd and Creating a New OSD Instance

Preparing the OSD Data and Journal Drives

Initializing the OSD Data and Journal Directory and Registering the OSD Authentication Key

Adding the New OSD Node to the CRUSH map

Updating the Owner and Group Permissions

Enabling and Starting the `ceph-osd` Daemon

6.3.4. Observing Data Migration

6.4. Removing an OSD

6.4.1. Removing an OSD with the Command Line Interface

Quick Links

Help

Site Info

Related Sites

About

Red Hat legal and privacy links

Red Hat legal and privacy links

Language:

Language and Page Formatting Options

Language:

Red Hat Training

Chapter 6. Managing Cluster Size

6.1. Adding a Monitor

6.1.1. Configuring a Host

6.1.2. Adding a Monitor with Ansible

6.1.3. Adding a Monitor with the Command Line Interface

6.2. Removing a Monitor

6.2.1. Removing a Monitor with the Command Line Interface

6.2.2. Removing Monitors from an Unhealthy Cluster

6.3. Adding an OSD

6.3.1. Configuring a Host

6.3.2. Adding an OSD with Ansible

6.3.3. Adding an OSD with the Command Line Interface

Before you Start

Installing ceph-osd and Creating a New OSD Instance

Preparing the OSD Data and Journal Drives

Initializing the OSD Data and Journal Directory and Registering the OSD Authentication Key

Adding the New OSD Node to the CRUSH map

Updating the Owner and Group Permissions

Enabling and Starting the ceph-osd Daemon

6.3.4. Observing Data Migration

6.4. Removing an OSD

6.4.1. Removing an OSD with the Command Line Interface

Quick Links

Help

Site Info

Related Sites

Systems Status

About

Red Hat legal and privacy links

Red Hat legal and privacy links

Enabling and Starting the `ceph-osd` Daemon