Installation Guide for Red Hat Enterprise Linux
Installing Red Hat Ceph Storage on Red Hat Enterprise Linux
Abstract
Chapter 1. What is Red Hat Ceph Storage?
Red Hat Ceph Storage is a scalable, open, software-defined storage platform that combines the most stable version of the Ceph storage system with a Ceph management platform, deployment utilities, and support services.
Red Hat Ceph Storage is designed for cloud infrastructure and web-scale object storage. Red Hat Ceph Storage clusters consist of the following types of nodes:
- Red Hat Ceph Storage Ansible administration node
This type of node acts as the traditional Ceph Administration node did for previous versions of Red Hat Ceph Storage. This type of node provides the following functions:
- Centralized storage cluster management
- The Ceph configuration files and keys
- Optionally, local repositories for installing Ceph on nodes that cannot access the Internet for security reasons
- Monitor nodes
-
Each monitor node runs the monitor daemon (
ceph-mon), which maintains a master copy of the cluster map. The cluster map includes the cluster topology. A client connecting to the Ceph cluster retrieves the current copy of the cluster map from the monitor which enables the client to read from and write data to the cluster.
Ceph can run with one monitor; however, to ensure high availability in a production cluster, Red Hat will only support deployments with at least three monitor nodes.
- OSD nodes
Each Object Storage Device (OSD) node runs the Ceph OSD daemon (
ceph-osd), which interacts with logical disks attached to the node. Ceph stores data on these OSD nodes.Ceph can run with very few OSD nodes, which the default is three, but production clusters realize better performance beginning at modest scales, for example 50 OSDs in a storage cluster. Ideally, a Ceph cluster has multiple OSD nodes, allowing isolated failure domains by creating the CRUSH map.
- MDS nodes
-
Each Metadata Server (MDS) node runs the MDS daemon (
ceph-mds), which manages metadata related to files stored on the Ceph File System (CephFS). The MDS daemon also coordinates access to the shared cluster. - Object Gateway node
Ceph Object Gateway node runs the Ceph RADOS Gateway daemon (
ceph-radosgw), and is an object storage interface built on top oflibradosto provide applications with a RESTful gateway to Ceph Storage Clusters. The Ceph Object Gateway supports two interfaces:S3
Provides object storage functionality with an interface that is compatible with a large subset of the Amazon S3 RESTful API.
Swift
Provides object storage functionality with an interface that is compatible with a large subset of the OpenStack Swift API.
For details on the Ceph architecture, see the Architecture Guide for Red Hat Ceph Storage 3.
For minimum recommended hardware, see the Red Hat Ceph Storage Hardware Selection Guide 3.
Chapter 2. Requirements for Installing Red Hat Ceph Storage
Figure 2.1. Prerequisite Workflow

Before installing Red Hat Ceph Storage (RHCS), review the following requirements and prepare each Monitor, OSD, Metadata Server, and client nodes accordingly.
2.1. Prerequisites
- Verify the hardware meets the minimum requirements. For details, see the Hardware Guide for Red Hat Ceph Storage 3.
2.2. Requirements Checklist for Installing Red Hat Ceph Storage
| Task | Required | Section | Recommendation |
|---|---|---|---|
| Verifying the operating system version | Yes | Section 2.3, “Operating system requirements for Red Hat Ceph Storage” | |
| Registering Ceph nodes | Yes | Section 2.4, “Registering Red Hat Ceph Storage Nodes to the CDN and Attaching Subscriptions” | |
| Enabling Ceph software repositories | Yes | Section 2.5, “Enabling the Red Hat Ceph Storage Repositories” | |
| Using a RAID controller with OSD nodes | No | Section 2.6, “Considerations for Using a RAID Controller with OSD Nodes (optional)” | Enabling write-back caches on a RAID controller might result in increased small I/O write throughput for OSD nodes. |
| Configuring the network | Yes | Section 2.8, “Verifying the Network Configuration for Red Hat Ceph Storage” | At minimum, a public network is required. However, a private network for cluster communication is recommended. |
| Configuring a firewall | No | Section 2.9, “Configuring a firewall for Red Hat Ceph Storage” | A firewall can increase the level of trust for a network. |
| Creating an Ansible user | Yes | Creating the Ansible user is required on all Ceph nodes. | |
| Enabling password-less SSH | Yes | Required for Ansible. |
2.3. Operating system requirements for Red Hat Ceph Storage
Red Hat Ceph Storage 3 requires Red Hat Enterprise Linux 7.5 Server with a homogeneous version, such as the AMD64 or Intel 64 architectures, across all Ceph nodes in the storage cluster.
Red Hat does not support clusters with heterogeneous operating systems and versions.
Additional Resources
- The Installation Guide for Red Hat Enterprise Linux 7.
- The System Administrator’s Guide for Red Hat Enterprise Linux 7.
2.4. Registering Red Hat Ceph Storage Nodes to the CDN and Attaching Subscriptions
Register each Red Hat Ceph Storage (RHCS) node to the Content Delivery Network (CDN) and attach the appropriate subscription so that the node has access to software repositories. Each RHCS node must be able to access the full Red Hat Enterprise Linux 7 base content and the extras repository content.
For RHCS nodes that cannot access the Internet during the installation, provide the software content by using the Red Hat Satellite server. Alternatively, mount a local Red Hat Enterprise Linux 7 Server ISO image and point the RHCS nodes to the ISO image. For additional details, contact Red Hat Support.
For more information on registering Ceph nodes with the Red Hat Satellite server, see the How to Register Ceph with Satellite 6 and How to Register Ceph with Satellite 5 articles on the Red Hat Customer Portal.
Prerequisites
- A valid Red Hat subscription
- RHCS nodes must be able to connect to the Internet.
Procedure
Perform the following steps on all nodes in the storage cluster as the root user.
Register the node. When prompted, enter your Red Hat Customer Portal credentials:
# subscription-manager register
Pull the latest subscription data from the CDN:
# subscription-manager refresh
List all available subscriptions for Red Hat Ceph Storage:
# subscription-manager list --available --all --matches="*Ceph*"
Identify the appropriate subscription and retrieve its Pool ID.
Attach the subscription:
# subscription-manager attach --pool=$POOL_ID
- Replace
-
$POOL_IDwith the Pool ID identified in the previous step.
-
Disable the default software repositories. Then, enable the Red Hat Enterprise Linux 7 Server and Red Hat Enterprise Linux 7 Server Extras repositories:
# subscription-manager repos --disable=* # subscription-manager repos --enable=rhel-7-server-rpms # subscription-manager repos --enable=rhel-7-server-extras-rpms
Update the system to receive the latest packages:
# yum update
Additional Resources
- See the Registering a System and Managing Subscriptions chapter in the System Administrator’s Guide for Red Hat Enterprise Linux 7.
- Section 2.5, “Enabling the Red Hat Ceph Storage Repositories”
2.5. Enabling the Red Hat Ceph Storage Repositories
Before installing Red Hat Ceph Storage (RHCS), enable the appropriate software repositories on each node in the storage cluster. The Red Hat Content Delivery Network (CDN) contains all the RHCS software.
Prerequisites
- Valid customer subscription.
- RHCS nodes can connect to the Internet.
- Register the cluster nodes with CDN.
Disable the EPEL software repository:
[root@monitor ~]# yum install yum-utils vim -y [root@monitor ~]# yum-config-manager --disable epel
Procedure
Enable the software repository for the appropriate RHCS node type as the root user.
On Monitor nodes enable the Red Hat Ceph Storage 3 Monitor repository:
[root@monitor ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-mon-rpms
On OSD nodes enable the Red Hat Ceph Storage 3 OSD repository:
[root@osd ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-osd-rpms
On the Ansible administration node, enable the Red Hat Ceph Storage 3 Tools repository and Ansible repository:
[root@admin ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms --enable=rhel-7-server-ansible-2.4-rpms
Optional. If you want to deploy the Ceph File System, enable the Red Hat Ceph Storage 3 Tools repository on the Metadata Server (MDS) nodes:
[root@mds ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms
Optional. If you want to deploy the Ceph Object Gateway, enable the Red Hat Ceph Storage 3 Tools repository on the Ceph Object Gateway nodes:
[root@gateway ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms
Optional. If you want to deploy a Ceph client, on the client node, enable the Red Hat Ceph Storage 3 Tools repository:
[root@client ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms
Additional Resources
- The Registering and Managing Subscriptions chapter in the System Administrator’s Guide for Red Hat Enterprise Linux.
2.6. Considerations for Using a RAID Controller with OSD Nodes (optional)
If an OSD node has a RAID controller with 1-2GB of cache installed, enabling the write-back cache might result in increased small I/O write throughput. However, the cache must be non-volatile.
Modern RAID controllers usually have super capacitors that provide enough power to drain volatile memory to non-volatile NAND memory during a power loss event. It is important to understand how a particular controller and its firmware behave after power is restored.
Some RAID controllers require manual intervention. Hard drives typically advertise to the operating system whether their disk caches should be enabled or disabled by default. However, certain RAID controllers and some firmware do not provide such information. Verify that disk level caches are disabled to avoid file system corruption.
Create a single RAID 0 volume with write-back for each Ceph OSD data drive with write-back cache enabled.
If Serial Attached SCSI (SAS) or SATA connected Solid-state Drive (SSD) disks are also present on the RAID controller, then investigate whether the controller and firmware support pass-through mode. Enabling pass-through mode helps avoid caching logic, and generally results in much lower latency for fast media.
2.7. Considerations for Using NVMe with Object Gateway (optional)
If you plan to use the Object Gateway feature of Red Hat Ceph Storage and your OSD nodes have NVMe based SSDs or SATA SSDs, consider following the procedures in Ceph Object Gateway for Production to use NVMe with LVM optimally. These procedures explain how to use specially designed Ansible playbooks which will place journals and bucket indexes together on SSDs, which can increase performance compared to having all journals on one device. The information on using NVMe with LVM optimally should be referenced in combination with this Installation Guide.
2.8. Verifying the Network Configuration for Red Hat Ceph Storage
All Red Hat Ceph Storage (RHCS) nodes require a public network. You must have a network interface card configured to a public network where Ceph clients can reach Ceph monitors and Ceph OSD nodes.
You might have a network interface card for a cluster network so that Ceph can conduct heart-beating, peering, replication, and recovery on a network separate from the public network.
Configure the network interface settings and ensure to make the changes persistent.
Red Hat does not recommend using a single network interface card for both a public and private network.
Prerequisites
- Network interface card connected to the network.
Procedure
Do the following steps on all RHCS nodes in the storage cluster, as the root user.
Verify the following settings are in the
/etc/sysconfig/network-scripts/ifcfg-*file corresponding the public-facing network interface card:-
The
BOOTPROTOparameter is set tononefor static IP addresses. The
ONBOOTparameter must be set toyes.If it is set to
no, the Ceph storage cluster might fail to peer on reboot.If you intend to use IPv6 addressing, you must set the IPv6 parameters such as
IPV6INITtoyes, except theIPV6_FAILURE_FATALparameter.Also, edit the Ceph configuration file,
/etc/ceph/ceph.conf, to instruct Ceph to use IPv6, otherwise, Ceph will use IPv4.
-
The
Additional Resources
- For details on configuring network interface scripts for Red Hat Enterprise Linux 7, see the Configuring a Network Interface Using ifcfg Files chapter in the Networking Guide for Red Hat Enterprise Linux 7.
- For more information on network configuration see the Network Configuration Reference chapter in the Configuration Guide for Red Hat Ceph Storage 3.
2.9. Configuring a firewall for Red Hat Ceph Storage
Red Hat Ceph Storage (RHCS) uses the firewalld service.
The Monitor daemons use port 6789 for communication within the Ceph storage cluster.
On each Ceph OSD node, the OSD daemons use several ports in the range 6800-7300:
- One for communicating with clients and monitors over the public network
- One for sending data to other OSDs over a cluster network, if available; otherwise, over the public network
- One for exchanging heartbeat packets over a cluster network, if available; otherwise, over the public network
The Ceph Manager (ceph-mgr) daemons use ports in range 6800-7300. Consider colocating the ceph-mgr daemons with Ceph Monitors on same nodes.
The Ceph Metadata Server nodes (ceph-mds) use port 6800.
The Ceph Object Gateway nodes use port 7480 by default. However, you can change the default port, for example to port 80.
To use the SSL/TLS service, open port 443.
Prerequisite
- Network hardware is connected.
Procedure
On all RHCS nodes, start the
firewalldservice. Enable it to run on boot, and ensure that it is running:# systemctl enable firewalld # systemctl start firewalld # systemctl status firewalld
On all Monitor nodes, open port
6789on the public network:[root@monitor ~]# firewall-cmd --zone=public --add-port=6789/tcp [root@monitor ~]# firewall-cmd --zone=public --add-port=6789/tcp --permanent
To limit access based on the source address:
firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \ port="6789" accept"
firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \ port="6789" accept" --permanent
- Replace
-
$IP_ADDRwith the network address of the Monitor node. -
$NETMASK_PREFIXwith the netmask in CIDR notation.
-
Example
[root@monitor ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="192.168.0.11/24" port protocol="tcp" \ port="6789" accept"
[root@monitor ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="192.168.0.11/24" port protocol="tcp" \ port="6789" accept" --permanent
On all OSD nodes, open ports
6800-7300on the public network:[root@osd ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp [root@osd ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent
If you have a separate cluster network, repeat the commands with the appropriate zone.
On all Ceph Manager (
ceph-mgr) nodes (usually the same nodes as Monitor ones), open ports6800-7300on the public network:[root@monitor ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp [root@monitor ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent
If you have a separate cluster network, repeat the commands with the appropriate zone.
On all Ceph Metadata Server (
ceph-mds) nodes, open port6800on the public network:[root@monitor ~]# firewall-cmd --zone=public --add-port=6800/tcp [root@monitor ~]# firewall-cmd --zone=public --add-port=6800/tcp --permanent
If you have a separate cluster network, repeat the commands with the appropriate zone.
On all Ceph Object Gateway nodes, open the relevant port or ports on the public network.
To open the default port
7480:[root@gateway ~]# firewall-cmd --zone=public --add-port=7480/tcp [root@gateway ~]# firewall-cmd --zone=public --add-port=7480/tcp --permanent
To limit access based on the source address:
firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \ port="7480" accept"
firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \ port="7480" accept" --permanent
- Replace
-
$IP_ADDRwith the network address of the object gateway node. -
$NETMASK_PREFIXwith the netmask in CIDR notation.
-
Example
[root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="192.168.0.31/24" port protocol="tcp" \ port="7480" accept"
[root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="192.168.0.31/24" port protocol="tcp" \ port="7480" accept" --permanent
Optional. If you changed the default Ceph Object Gateway port, for example, to port
80, open this port:[root@gateway ~]# firewall-cmd --zone=public --add-port=80/tcp [root@gateway ~]# firewall-cmd --zone=public --add-port=80/tcp --permanent
To limit access based on the source address, run the following commands:
firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \ port="80" accept"
firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \ port="80" accept" --permanent
- Replace
-
$IP_ADDRwith the network address of the object gateway node. -
$NETMASK_PREFIXwith the netmask in CIDR notation.
-
Example
[root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="192.168.0.31/24" port protocol="tcp" \ port="80" accept"
[root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="192.168.0.31/24" port protocol="tcp" \ port="80" accept" --permanent
Optional. To use SSL/TLS, open port
443:[root@gateway ~]# firewall-cmd --zone=public --add-port=443/tcp [root@gateway ~]# firewall-cmd --zone=public --add-port=443/tcp --permanent
To limit access based on the source address, run the following commands:
firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \ port="443" accept"
firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \ port="443" accept" --permanent
- Replace
-
$IP_ADDRwith the network address of the object gateway node. -
$NETMASK_PREFIXwith the netmask in CIDR notation.
-
Example
[root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="192.168.0.31/24" port protocol="tcp" \ port="443" accept" [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \ source address="192.168.0.31/24" port protocol="tcp" \ port="443" accept" --permanent
Additional Resources
- For more information about public and cluster network, see Verifying the Network Configuration for Red Hat Ceph Storage.
-
For additional details on
firewalld, see the Using Firewalls chapter in the Security Guide for Red Hat Enterprise Linux 7.
2.10. Creating an Ansible user with sudo access
Ansible must be able to log into all the Red Hat Ceph Storage (RHCS) nodes as a user that has root privileges to install software and create configuration files without prompting for a password. You must create an Ansible user with password-less root access on all nodes in the storage cluster when deploying and configuring a Red Hat Ceph Storage cluster with Ansible.
Prerequisite
-
Having
rootorsudoaccess to all nodes in the storage cluster.
Procedure
Log in to a Ceph node as the
rootuser:ssh root@$HOST_NAME
- Replace
-
$HOST_NAMEwith the host name of the Ceph node.
-
Example
# ssh root@mon01
Enter the
rootpassword when prompted.Create a new Ansible user:
adduser $USER_NAME
- Replace
-
$USER_NAMEwith the new user name for the Ansible user.
-
Example
# adduser admin
ImportantDo not use
cephas the user name. Thecephuser name is reserved for the Ceph daemons. A uniform user name across the cluster can improve ease of use, but avoid using obvious user names, because intruders typically use them for brute-force attacks.Set a new password for this user:
# passwd $USER_NAME
- Replace
-
$USER_NAMEwith the new user name for the Ansible user.
-
# passwd admin
Enter the new password twice when prompted.
Configure
sudoaccess for the newly created user:cat << EOF >/etc/sudoers.d/$USER_NAME $USER_NAME ALL = (root) NOPASSWD:ALL EOF
- Replace
-
$USER_NAMEwith the new user name for the Ansible user.
-
Example
# cat << EOF >/etc/sudoers.d/admin admin ALL = (root) NOPASSWD:ALL EOF
Assign the correct file permissions to the new file:
chmod 0440 /etc/sudoers.d/$USER_NAME
- Replace
-
$USER_NAMEwith the new user name for the Ansible user.
-
Example
# chmod 0440 /etc/sudoers.d/admin
Additional Resources
- The Adding a New User section in the System Administrator’s Guide for Red Hat Enterprise Linux 7.
2.11. Enabling Password-less SSH for Ansible
Generate an SSH key pair on the Ansible administration node and distribute the public key to each node in the storage cluster so that Ansible can access the nodes without being prompted for a password.
Prerequisites
Procedure
Do the following steps from the Ansible administration node, and as the Ansible user.
Generate the SSH key pair, accept the default file name and leave the passphrase empty:
[user@admin ~]$ ssh-keygen
Copy the public key to all nodes in the storage cluster:
ssh-copy-id $USER_NAME@$HOST_NAME
- Replace
-
$USER_NAMEwith the new user name for the Ansible user. -
$HOST_NAMEwith the host name of the Ceph node.
-
Example
[user@admin ~]$ ssh-copy-id ceph-admin@ceph-mon01
Create and edit the
~/.ssh/configfile.ImportantBy creating and editing the
~/.ssh/configfile you do not have to specify the-u $USER_NAMEoption each time you execute theansible-playbookcommand.Create the SSH
configfile:[user@admin ~]$ touch ~/.ssh/config
Open the
configfile for editing. Set theHostnameandUseroptions for each node in the storage cluster:Host node1 Hostname $HOST_NAME User $USER_NAME Host node2 Hostname $HOST_NAME User $USER_NAME ...
- Replace
-
$HOST_NAMEwith the host name of the Ceph node. -
$USER_NAMEwith the new user name for the Ansible user.
-
Example
Host node1 Hostname monitor User admin Host node2 Hostname osd User admin Host node3 Hostname gateway User admin
Set the correct file permissions for the
~/.ssh/configfile:[admin@admin ~]$ chmod 600 ~/.ssh/config
Additional Resources
-
The
ssh_config(5)manual page - The OpenSSH chapter in the System Administrator’s Guide for Red Hat Enterprise Linux 7
Chapter 3. Deploying Red Hat Ceph Storage
This chapter describes how to use the Ansible application to deploy a Red Hat Ceph Storage cluster and other components, such as Metadata Servers or the Ceph Object Gateway.
- To install a Red Hat Ceph Storage cluster, see Section 3.2, “Installing a Red Hat Ceph Storage Cluster”.
- To install Metadata Servers, see Section 3.3, “Installing Metadata Servers”.
-
To install the
ceph-clientrole, see Section 3.4, “Installing the Ceph Client Role”. - To install the Ceph Object Gateway, see Section 3.5, “Installing the Ceph Object Gateway”.
-
To learn about the Ansible
--limitoption, see Section 3.7, “Understanding thelimitoption”.
3.1. Prerequisites
- Obtain a valid customer subscription.
Prepare the cluster nodes. On each node:
3.2. Installing a Red Hat Ceph Storage Cluster
Use the Ansible application with the ceph-ansible playbook to install Red Hat Ceph Storage 3.
Production Ceph storage clusters start with a minimum of three monitor hosts and three OSD nodes containing multiple OSD daemons.

Prerequisites
On the Ansible administration node, install the
ceph-ansiblepackage:[root@admin ~]# yum install ceph-ansible
Procedure
Use the following commands from the Ansible administration node if not instructed otherwise.
In the user’s home directory, create the
ceph-ansible-keysdirectory where Ansible stores temporary values generated by theceph-ansibleplaybook.[user@admin ~]$ mkdir ~/ceph-ansible-keys
Create a symbolic link to the
/usr/share/ceph-ansible/group_varsdirectory in the/etc/ansible/directory:[root@admin ~]# ln -s /usr/share/ceph-ansible/group_vars /etc/ansible/group_vars
Navigate to the
/usr/share/ceph-ansible/directory:[user@admin ~]$ cd /usr/share/ceph-ansible
Create new copies of the
yml.samplefiles:[root@admin ceph-ansible]# cp group_vars/all.yml.sample group_vars/all.yml [root@admin ceph-ansible]# cp group_vars/osds.yml.sample group_vars/osds.yml [root@admin ceph-ansible]# cp site.yml.sample site.yml
Edit the copied files.
Edit the
group_vars/all.ymlfile. See the table below for the most common required and optional parameters to uncomment. Note that the table does not include all parameters.Table 3.1. General Ansible Settings
Option Value Required Notes ceph_repository_typecdnorisoYes
ceph_rhcs_version3Yes
ceph_rhcs_iso_pathThe path to the ISO image
Yes if using an ISO image
monitor_interfaceThe interface that the Monitor nodes listen to
monitor_interface,monitor_address, ormonitor_address_blockis requiredmonitor_addressThe address that the Monitor nodes listen to
monitor_address_blockThe subnet of the Ceph public network
Use when the IP addresses of the nodes are unknown, but the subnet is known
ip_versionipv6Yes if using IPv6 addressing
public_networkThe IP address and netmask of the Ceph public network
Yes
Section 2.8, “Verifying the Network Configuration for Red Hat Ceph Storage”
cluster_networkThe IP address and netmask of the Ceph cluster network
No, defaults to
public_networkAn example of the
all.ymlfile can look like:ceph_origin: repository ceph_repository: rhcs ceph_repository_type: cdn ceph_rhcs_version: 3 monitor_interface: eth0 public_network: 192.168.0.0/24
NoteHaving the
ceph_rhcs_versionoption set to3will pull in the latest version of Red Hat Ceph Storage 3.
For additional details, see the
all.ymlfile.Edit the
group_vars/osds.ymlfile. See the table below for the most common required and optional parameters to uncomment. Note that the table does not include all parameters.Table 3.2. OSD Ansible Settings
Option Value Required Notes osd_scenariocollocatedto use the same device for journal and OSD datanon-collocatedto use a dedicated device to store journal datalvmto use the Logical Volume Manager to store OSD dataYes
When using
osd_scenario: non-collocated,ceph-ansibleexpects the variablesdevicesanddedicated_devicesto match. For example, if you specify 10 disks indevices, you must specify 10 entries indedicated_devices. Currently, Red Hat only supports dedicated journals when usingosd_scenario: lvm, not collocated journals.osd_auto_discoverytrueto automatically discover OSDsYes if using
osd_scenario: collocatedCannot be used when
devicessetting is useddevicesList of devices where
ceph datais storedYes to specify the list of devices
Cannot be used when
osd_auto_discoverysetting is useddedicated_devicesList of dedicated devices for non-collocated OSDs where
ceph journalis storedYes if
osd_scenario: non-collocatedShould be nonpartitioned devices
dmcrypttrueto encrypt OSDsNo
Defaults to
falselvm_volumesa list of dictionaries
Yes if using
osd_scenario: lvmEach dictionary must contain a
data,journalanddata_vgkeys. Thedatakey must be a logical volume. Thejournalkey can be a logical volume (LV), device or partition, but do not use one journal for multipledataLVs. Thedata_vgkey must be the volume group containing thedataLV. Optionally, thejournal_vgkey can be used to specify the volume group containing the journal LV, if applicable.The following are examples of the
osds.ymlfile using these threeosd_scenario::collocated,non-collocated, andlvm.osd_scenario: non-collocated devices: - /dev/sda - /dev/sdb - /dev/sdc - /dev/sdd dedicated_devices: - /dev/nvme0n1 - /dev/nvme0n1 - /dev/nvme0n1 - /dev/nvme0n1
osd_scenario: non-collocated devices: - /dev/sda - /dev/sdb - /dev/sdc - /dev/sdd dedicated_devices: - /dev/nvme0n1 - /dev/nvme0n1 - /dev/nvme0n1 - /dev/nvme0n1
osd_scenario: lvm lvm_volumes: - data: data-lv1 data_vg: vg1 journal: journal-lv1 journal_vg: vg2 - data: data-lv2 journal: /dev/sda data_vg: vg1For additional details, see the comments in the
osds.ymlfile.NoteCurrently,
ceph-ansibledoes not create the volume groups or the logical volumes. This must be done before running the Anisble playbook.
Edit the Ansible inventory file located by default at
/etc/ansible/hosts. Remember to comment out example hosts.Add the Monitor nodes under the
[mons]section:[mons] <monitor-host-name> <monitor-host-name> <monitor-host-name>
Add OSD nodes under the
[osds]section. If the nodes have sequential naming, consider using a range:[osds] <osd-host-name[1:10]>
Optionally, use the
devicesparameter to specify devices that the OSD nodes will use. Use a comma-separated list to list multiple devices.[osds] <ceph-host-name> devices="[ '<device_1>', '<device_2>' ]"
For example:
[osds] ceph-osd-01 devices="[ '/dev/sdb', '/dev/sdc' ]" ceph-osd-02 devices="[ '/dev/sdb', '/dev/sdc', '/dev/sdd' ]"
When specifying no devices, set the
osd_auto_discoveryoption totruein theosds.ymlfile.NoteUsing the
devicesparameter is useful when OSDs use devices with different names or when one of the devices failed on one of the OSDs.Add the Ceph Manager (
ceph-mgr) nodes under the[mgrs]section. Colocate the Ceph Manager daemon with Monitor nodes.[mgrs] <monitor-host-name> <monitor-host-name> <monitor-host-name>
As the Ansible user, ensure that Ansible can reach the Ceph hosts:
[user@admin ~]$ ansible all -m ping
Add the following line to the
/etc/ansible/ansible.cfgfile:retry_files_save_path = ~/
As
root, create the/var/log/ansible/directory and assign the appropriate permissions for theansibleuser:[root@admin ceph-ansible]# mkdir /var/log/ansible [root@admin ceph-ansible]# chown ansible:ansible /var/log/ansible [root@admin ceph-ansible]# chmod 755 /var/log/ansible
Edit the
/usr/share/ceph-ansible/ansible.cfgfile, updating thelog_pathvalue as follows:log_path = /var/log/ansible/ansible.log
As the Ansible user, run the
ceph-ansibleplaybook.[user@admin ceph-ansible]$ ansible-playbook site.yml
From a Monitor node, verify the status of the Ceph cluster.
[root@monitor ~]# ceph health HEALTH_OK
Verify the cluster is functioning using
rados.From a monitor node, create a test pool with eight placement groups:
Syntax
[root@monitor ~]# ceph osd pool create <pool-name> <pg-number>
Example
[root@monitor ~]# ceph osd pool create test 8
Create a file called
hello-world.txt:Syntax
[root@monitor ~]# vim <file-name>
Example
[root@monitor ~]# vim hello-world.txt
Upload
hello-world.txtto the test pool using the object namehello-world:Syntax
[root@monitor ~]# rados --pool <pool-name> put <object-name> <object-file>
Example
[root@monitor ~]# rados --pool test put hello-world hello-world.txt
Download
hello-worldfrom the test pool as file namefetch.txt:Syntax
[root@monitor ~]# rados --pool <pool-name> get <object-name> <object-file>
Example
[root@monitor ~]# rados --pool test get hello-world fetch.txt
Check the contents of
fetch.txt:[root@monitor ~]# cat fetch.txt
The output should be:
"Hello World!"
NoteIn addition to verifying the cluster status, you can use the
ceph-medicutility to overall diagnose the Ceph Storage Cluster. See the Installing and Usingceph-medicto Diagnose a Ceph Storage Cluster chapter in the Red Hat Ceph Storage 3 Troubleshooting Guide.
3.3. Installing Metadata Servers
Use the Ansible automation application to install a Ceph Metadata Server (MDS). Metadata Server daemons are necessary for deploying a Ceph File System.
Procedure
Perform the following steps on the Ansible administration node.
Add a new section
[mdss]to the/etc/ansible/hostsfile:[mdss] <hostname> <hostname> <hostname>
Replace
<hostname>with the host names of the nodes where you want to install the Ceph Metadata Servers.Navigate to the
/usr/share/ceph-ansibledirectory:[root@admin ~]# cd /usr/share/ceph-ansible
Create a copy of the
group_vars/mdss.yml.samplefile namedmdss.yml:[root@admin ceph-ansible]# cp group_vars/mdss.yml.sample group_vars/mdss.yml
-
Optionally, edit parameters in
mdss.yml. Seemdss.ymlfor details. Run the Ansible playbook:
[user@admin ceph-ansible]$ ansible-playbook site-docker.yml --limit mdss
- After installing Metadata Servers, configure them. For details, see the Configuring Metadata Server Daemons chapter in the Ceph File System Guide for Red Hat Ceph Storage 3.
Additional Resources
- The Ceph File System Guide for Red Hat Ceph Storage 3
-
Section 3.7, “Understanding the
limitoption”
3.4. Installing the Ceph Client Role
The ceph-ansible utility provides the ceph-client role that copies the Ceph configuration file and the administration keyring to nodes. In addition, you can use this role to create custom pools and clients.
Prerequisites
-
A running Ceph storage cluster, preferably in the
active + cleanstate. - Perform the tasks listed in Chapter 2, Requirements for Installing Red Hat Ceph Storage.
Procedure
Perform the following tasks on the Ansible administration node.
Add a new section
[clients]to the/etc/ansible/hostsfile:[clients] <client-hostname>
Replace
<client-hostname>with the host name of the node where you want to install theceph-clientrole.Navigate to the
/usr/share/ceph-ansibledirectory:[root@admin ~]# cd /usr/share/ceph-ansible
Create a new copy of the
clients.yml.samplefile namedclients.yml:[root@admin ceph-ansible ~]# cp group_vars/clients.yml.sample group_vars/clients.yml
Optionally, instruct
ceph-clientto create pools and clients.Update
clients.yml.-
Uncomment the
user_configsetting and set it totrue. -
Uncomment the
poolsandkeyssections and update them as required. You can define custom pools and client names altogether with thecephxcapabilities.
-
Uncomment the
Add the
osd_pool_default_pg_numsetting to theceph_conf_overridessection in theall.ymlfile:ceph_conf_overrides: global: osd_pool_default_pg_num: <number>Replace
<number>with the default number of placement groups.
Run the Ansible playbook:
[user@admin ceph-ansible]$ ansible-playbook site.yml --limit clients
Additional Resources
3.5. Installing the Ceph Object Gateway
The Ceph Object Gateway, also know as the RADOS gateway, is an object storage interface built on top of the librados API to provide applications with a RESTful gateway to Ceph storage clusters.
Prerequisites
-
A running Ceph storage cluster, preferably in the
active + cleanstate. - On the Ceph Object Gateway node, perform the tasks listed in Chapter 2, Requirements for Installing Red Hat Ceph Storage.
Procedure
Perform the following tasks on the Ansible administration node.
Create the
rgwsfile from the sample file:[root@ansible ~]# cd /etc/ansible/group_vars [root@ansible ~]# cp rgws.yml.sample rgws.yml
Add gateway hosts to the
/etc/ansible/hostsfile under the[rgws]section to identify their roles to Ansible. If the hosts have sequential naming, use a range. For example:[rgws] <rgw_host_name_1> <rgw_host_name_2> <rgw_host_name[3..10]>
Navigate to the Ansible configuration directory,
/etc/ansible/:[root@ansible ~]# cd /usr/share/ceph-ansible
To copy the administrator key to the Ceph Object Gateway node, uncomment the
copy_admin_keysetting in the/usr/share/ceph-ansible/group_vars/rgws.ymlfile:copy_admin_key: true
The
rgws.ymlfile may specify a different default port than the default port7480. For example:ceph_rgw_civetweb_port: 80
The
all.ymlfile MUST specify aradosgw_interface. For example:radosgw_interface: eth0
Specifying the interface prevents Civetweb from binding to the same IP address as another Civetweb instance when running multiple instances on the same host.
Generally, to change default settings, uncomment the settings in the
rgw.ymlfile, and make changes accordingly. To make additional changes to settings that aren’t in thergw.ymlfile, useceph_conf_overrides:in theall.ymlfile. For example, set thergw_dns_name:with the host of the DNS server and ensure the cluster’s DNS server to configure it for wild cards to enable S3 subdomains.ceph_conf_overrides: client.rgw.rgw1: rgw_dns_name: <host_name> rgw_override_bucket_index_max_shards: 16 rgw_bucket_default_quota_max_objects: 1638400For advanced configuration details, see the Red Hat Ceph Storage 3 Ceph Object Gateway for Production guide. Advanced topics include:
- Configuring Ansible Groups
Developing Storage Strategies. See the Creating the Root Pool, Creating System Pools, and Creating Data Placement Strategies sections for additional details on how create and configure the pools.
See Bucket Sharding for configuration details on bucket sharding.
Uncomment the
radosgw_interfaceparameter in thegroup_vars/all.ymlfile.radosgw_interface: <interface>
Replace:
-
<interface>with the interface that the Ceph Object Gateway nodes listen to
For additional details, see the
all.ymlfile.-
Run the Ansible playbook:
[user@admin ceph-ansible]$ ansible-playbook site.yml --limit rgws
Ansible ensures that each Ceph Object Gateway is running.
For a single site configuration, add Ceph Object Gateways to the Ansible configuration.
For multi-site deployments, you should have an Ansible configuration for each zone. That is, Ansible will create a Ceph storage cluster and gateway instances for that zone.
After installation for a multi-site cluster is complete, proceed to the Multi-site chapter in the Object Gateway Guide for Red Hat Enterprise Linux for details on configuring a cluster for multi-site.
Additional Resources
3.6. Installing the NFS-Ganesha Gateway
The Ceph NFS Ganesha Gateway is an NFS interface built on top of the Ceph Object Gateway to provide applications with a POSIX filesystem interface to the Ceph Object Gateway for migrating files within filesystems to Ceph Object Storage.
Prerequisites
-
A running Ceph storage cluster, preferably in the
active + cleanstate. - At least one node running a Ceph Object Gateway.
- At least one S3 user with an access key and secret.
- Perform the Before You Start procedure.
Procedure
Perform the following tasks on the Ansible administration node.
Create the
nfssfile from the sample file:[root@ansible ~]# cd /etc/ansible/group_vars [root@ansible ~]# cp nfss.yml.sample nfss.yml
Add gateway hosts to the
/etc/ansible/hostsfile under an[nfss]group to identify their group membership to Ansible. If the hosts have sequential naming, use a range. For example:[nfss] <nfs_host_name_1> <nfs_host_name_2> <nfs_host_name[3..10]>
Navigate to the Ansible configuration directory,
/etc/ansible/:[root@ansible ~]# cd /usr/share/ceph-ansible
To copy the administrator key to the Ceph Object Gateway node, uncomment the
copy_admin_keysetting in the/usr/share/ceph-ansible/group_vars/nfss.ymlfile:copy_admin_key: true
Configure the FSAL (File System Abstraction Layer) sections of the
/usr/share/ceph-ansible/group_vars/nfss.ymlfile. Provide an ID, S3 user ID, S3 access key and secret. For NFSv4, it should look something like this:################### # FSAL RGW Config # ################### #ceph_nfs_rgw_export_id: <replace-w-numeric-export-id> #ceph_nfs_rgw_pseudo_path: "/" #ceph_nfs_rgw_protocols: "3,4" #ceph_nfs_rgw_access_type: "RW" #ceph_nfs_rgw_user: "cephnfs" # Note: keys are optional and can be generated, but not on containerized, where # they must be configered. #ceph_nfs_rgw_access_key: "<replace-w-access-key>" #ceph_nfs_rgw_secret_key: "<replace-w-secret-key>"
Run the Ansible playbook:
[user@admin ceph-ansible]$ ansible-playbook site.yml --limit nfss
Additional Resources
3.7. Understanding the limit option
This section contains information about the Ansible --limit option.
Ansible supports the --limit option that enables you to use the site, site-docker, and rolling_upgrade Ansible playbooks for a particular section of the inventory file.
$ ansible-playbook site.yml|rolling_upgrade.yml|site-docker.yml --limit osds|rgws|clients|mdss|nfss
For example, to redeploy only OSDs:
$ ansible-playbook /usr/share/ceph-ansible/site.yml --limit osds
If you colocate Ceph components on one node, Ansible applies a playbook to all components on the node despite that only one component type was specified with the limit option. For example, if you run the rolling_update playbook with the --limit osds option on a node that contains OSDs and Metadata Servers (MDS), Ansible will upgrade both components, OSDs and MDSs.
3.8. Additional Resources
Chapter 4. Upgrading a Red Hat Ceph Storage Cluster
This section describes how to upgrade to a new major or minor version of Red Hat Ceph Storage.
Please contact Red Hat support prior to upgrading, if you have a large Ceph Object Gateway storage cluster with millions of objects present in buckets.
For more details refer to the Red Hat Ceph Storage 3.0 Release Notes, under the Slow OSD startup after upgrading to Red Hat Ceph Storage 3.0 heading.
Use the Ansible rolling_update.yml playbook located in the /usr/share/ceph-ansible/infrastructure-playbooks/ directory from the administration node to upgrade between two major or minor versions of Red Hat Ceph Storage, or to apply asynchronous updates.
Ansible upgrades the Ceph nodes in the following order:
- Monitor nodes
- MGR nodes
- OSD nodes
- MDS nodes
- Ceph Object Gateway nodes
- All other Ceph client nodes
Red Hat Ceph Storage 3 introduces several changes in Ansible configuration files located in the /usr/share/ceph-ansible/group_vars/ directory; certain parameters were renamed or removed. Therefore, make backup copies of the all.yml and osds.yml files before creating new copies from the all.yml.sample and osds.yml.sample files after upgrading to version 3. For more details about the changes, see Appendix H, Changes in Ansible Variables Between Version 2 and 3.
Red Hat Ceph Storage 3.1 introduces new Ansible playbooks to optimize storage for performance when using Object Gateway and high speed NVMe based SSDs (and SATA SSDs). The playbooks do this by placing journals and bucket indexes together on SSDs, which can increase performance compared to having all journals on one device. These playbooks are designed to be used when installing Ceph. Existing OSDs continue to work and need no extra steps during an upgrade. There is no way to upgrade a Ceph cluster while simultaneously reconfiguring OSDs to optimize storage in this way. To use different devices for journals or bucket indexes requires reprovisioning OSDs. For more information see Using NVMe with LVM optimally in Ceph Object Gateway for Production.
The rolling_update.yml playbook includes the serial variable that adjusts the number of nodes to be updated simultaneously. Red Hat strongly recommends to use the default value (1), which ensures that Ansible will upgrade cluster nodes one by one.
When using the rolling_update.yml playbook to upgrade to Red Hat Ceph Storage 3.0 and from version 3.0 to other zStream releases of 3.0, users who use the Ceph File System (CephFS) must manually update the Metadata Server (MDS) cluster. This is due to a known issue.
Comment the MDS hosts in /etc/ansible/hosts before upgrading the entire cluster using ceph-ansible rolling-upgrade.yml, and then upgrade MDS manually. In the /etc/ansible/hosts file:
#[mdss] #host-abc
For more details about this known issue, including how to update the MDS cluster, refer to the Red Hat Ceph Storage 3.0 Release Notes.
Prerequisites
On all nodes in the cluster, enable the
rhel-7-server-extras-rpmsrepository.# subscription-manager repos --enable=rhel-7-server-extras-rpms
- If the Ceph nodes are not connected to the Red Hat Content Delivery Network (CDN) and you used an ISO image to install Red Hat Ceph Storage, update the local repository with the latest version of Red Hat Ceph Storage. See Section 2.5, “Enabling the Red Hat Ceph Storage Repositories” for details.
If upgrading from Red Hat Ceph Storage 2.x to 3.x, on the Ansible administration node and the RBD mirroring node, enable the Red Hat Ceph Storage 3 Tools repository and Ansible repository:
[root@admin ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms --enable=rhel-7-server-ansible-2.4-rpms
If upgrading from Red Hat Ceph Storage 3.0 to 3.1 and using Red Hat Ceph Storage Dashboard, before upgrading the cluster, purge the old cephmetrics installation from the cluster. This avoids an issue where the dashboard won’t display data after upgrade.
If the
cephmetrics-ansiblepackage isn’t already updated, update it:[root@admin ~]# yum update cephmetrics-ansible
Change to the /usr/share/cephmetrics-ansible/ directory.
[root@admin ~]# cd /usr/share/cephmetrics-ansible
Purge the existing cephmetrics installation.
[root@admin cephmetrics-ansible]# ansible-playbook -v purge.yml
Install the updated Red Hat Ceph Storage Dashboard
[root@admin cephmetrics-ansible]# ansible-playbook -v playbook.yml
- If the Ceph nodes are not connected to the Red Hat Content Delivery Network (CDN) and you used an ISO image to install Red Hat Ceph Storage, update the local repository with the latest version of Red Hat Ceph Storage. See Section 2.5, “Enabling the Red Hat Ceph Storage Repositories” for details.
On the Ansible administration node, ensure the latest version of the
ansibleandceph-ansiblepackages are installed.[root@admin ~]# yum update ansible ceph-ansible
In the
rolling_update.ymlplaybook, change thehealth_osd_check_retriesandhealth_osd_check_delayvalues to40and30respectively.health_osd_check_retries: 40 health_osd_check_delay: 30
For each OSD node, Ansible will wait up to 20 minutes. Also, Ansible will check the cluster health every 30 seconds, waiting before continuing the upgrade process.
If the cluster you want to upgrade contains Ceph Block Device images that use the
exclusive-lockfeature, ensure that all Ceph Block Device users have permissions to blacklist clients:ceph auth caps client.<ID> mon 'allow r, allow command "osd blacklist"' osd '<existing-OSD-user-capabilities>'
Procedure
Use the following commands from the Ansible administration node.
Navigate to the
/usr/share/ceph-ansible/directory:[user@admin ~]$ cd /usr/share/ceph-ansible/
Back up the
group_vars/all.ymlandgroup_vars/osds.ymlfiles. Skip this step when upgrading from version 3.x to the latest version.[root@admin ceph-ansible]# cp group_vars/all.yml group_vars/all_old.yml [root@admin ceph-ansible]# cp group_vars/osds.yml group_vars/osds_old.yml
Create new copies of the
group_vars/all.yml.sampleandgroup_vars/osds.yml.samplenamedgroup_vars/all.ymlandgroup_vars/osds.ymlrespectively and edit them according to you deployment. Skip this step when upgrading from version 3.x to the latest version. For details, see Appendix H, Changes in Ansible Variables Between Version 2 and 3 and Section 3.2, “Installing a Red Hat Ceph Storage Cluster” .[root@admin ceph-ansible]# cp group_vars/all.yml.sample group_vars/all.yml [root@admin ceph-ansible]# cp group_vars/osds.yml.sample group_vars/osds.yml
In the
group_vars/all.ymlfile, uncomment theupgrade_ceph_packagesoption and set it toTrue.upgrade_ceph_packages: True
In the
group_vars/all.ymlfile, setceph_rhcs_versionto3.ceph_rhcs_version: 3
NoteHaving the
ceph_rhcs_versionoption set to3will pull in the latest version of Red Hat Ceph Storage 3.Add the
fetch_directoryparameter to thegroup_vars/all.ymlfile.fetch_directory: <full_directory_path>
Replace:
-
<full_directory_path>with a writable location, such as the Ansible user’s home directory.
-
If the cluster you want to upgrade contains any Ceph Object Gateway nodes, add the
radosgw_interfaceparameter to thegroup_vars/all.ymlfile.radosgw_interface: <interface>
Replace:
-
<interface>with the interface that the Ceph Object Gateway nodes listen to.
-
In the Ansible inventory file located at
/etc/ansible/hosts, add the Ceph Manager (ceph-mgr) nodes under the[mgrs]section. Colocate the Ceph Manager daemon with Monitor nodes. Skip this step when upgrading from version 3.x to the latest version.[mgrs] <monitor-host-name> <monitor-host-name> <monitor-host-name>
Copy
rolling_update.ymlfrom theinfrastructure-playbooksdirectory to the current directory.[root@admin ceph-ansible]# cp infrastructure-playbooks/rolling_update.yml .
Create the
/var/log/ansible/directory and assign the appropriate permissions for theansibleuser:[root@admin ceph-ansible]# mkdir /var/log/ansible [root@admin ceph-ansible]# chown ansible:ansible /var/log/ansible [root@admin ceph-ansible]# chmod 755 /var/log/ansible
Edit the
/usr/share/ceph-ansible/ansible.cfgfile, updating thelog_pathvalue as follows:log_path = /var/log/ansible/ansible.log
Run the playbook:
[user@admin ceph-ansible]$ ansible-playbook rolling_update.yml
To use the playbook only for a particular group of nodes on the Ansible inventory file, use the
--limitoption. For details, see Section 3.7, “Understanding thelimitoption”.From the RBD mirroring daemon node, upgrade
rbd-mirrormanually:# yum upgrade rbd-mirror
Restart the daemon:
# systemctl restart ceph-rbd-mirror@<client-id>
Verify that the cluster health is OK.
[root@monitor ~]# ceph -s
Chapter 5. What to Do Next?
This is only the beginning of what Red Hat Ceph Storage can do to help you meet the challenging storage demands of the modern data center. Here are links to more information on a variety of topics:
- Benchmarking performance and accessing performance counters, see the Benchmarking Performance chapter in the Administration Guide for Red Hat Ceph Storage 3.
- Creating and managing snapshots, see the Snapshots chapter in the Block Device Guide for Red Hat Ceph Storage 3.
- Expanding the Red Hat Ceph Storage cluster, see the Managing Cluster Size chapter in the Administration Guide for Red Hat Ceph Storage 3.
- Mirroring Ceph Block Devices, see the Block Device Mirroring chapter in the Block Device Guide for Red Hat Ceph Storage 3.
- Process management, see the Process Management chapter in the Administration Guide for Red Hat Ceph Storage 3.
- Tunable parameters, see the Configuration Guide for Red Hat Ceph Storage 3.
- Using Ceph as the back end storage for OpenStack, see the Back-ends section in the Storage Guide for Red Hat OpenStack Platform.
Appendix A. Troubleshooting
A.1. Ansible Stops Installation Because It Detects Less Devices Than It Expected
The Ansible automation application stops the installation process and returns the following error:
- name: fix partitions gpt header or labels of the osd disks (autodiscover disks)
shell: "sgdisk --zap-all --clear --mbrtogpt -- '/dev/{{ item.0.item.key }}' || sgdisk --zap-all --clear --mbrtogpt -- '/dev/{{ item.0.item.key }}'"
with_together:
- "{{ osd_partition_status_results.results }}"
- "{{ ansible_devices }}"
changed_when: false
when:
- ansible_devices is defined
- item.0.item.value.removable == "0"
- item.0.item.value.partitions|count == 0
- item.0.rc != 0What this means:
When the osd_auto_discovery parameter is set to true in the /etc/ansible/group_vars/osds.yml file, Ansible automatically detects and configures all the available devices. During this process, Ansible expects that all OSDs use the same devices. The devices get their names in the same order in which Ansible detects them. If one of the devices fails on one of the OSDs, Ansible fails to detect the failed device and stops the whole installation process.
Example situation:
-
Three OSD nodes (
host1,host2,host3) use the/dev/sdb,/dev/sdc, anddev/sdddisks. -
On
host2, the/dev/sdcdisk fails and is removed. -
Upon the next reboot, Ansible fails to detect the removed
/dev/sdcdisk and expects that only two disks will be used forhost2,/dev/sdband/dev/sdc(formerly/dev/sdd). - Ansible stops the installation process and returns the above error message.
To fix the problem:
In the /etc/ansible/hosts file, specify the devices used by the OSD node with the failed disk (host2 in the Example situation above):
[osds] host1 host2 devices="[ '/dev/sdb', '/dev/sdc' ]" host3
See Chapter 3, Deploying Red Hat Ceph Storage for details.
Appendix B. Manually Installing Red Hat Ceph Storage
Red Hat does not support or test upgrading manually deployed clusters. Therefore, Red Hat recommends to use Ansible to deploy a new cluster with Red Hat Ceph Storage 3. See Chapter 3, Deploying Red Hat Ceph Storage for details.
You can use command-line utilities, such as Yum, to upgrade manually deployed clusters, but Red Hat does not support or test this.
All Ceph clusters require at least one monitor, and at least as many OSDs as copies of an object stored on the cluster. Red Hat recommends using three monitors for production environments and a minimum of three Object Storage Devices (OSD).
Bootstrapping the initial monitor is the first step in deploying a Ceph storage cluster. Ceph monitor deployment also sets important criteria for the entire cluster, such as:
- The number of replicas for pools
- The number of placement groups per OSD
- The heartbeat intervals
- Any authentication requirement
Most of these values are set by default, so it is useful to know about them when setting up the cluster for production.
Installing a Ceph storage cluster by using the command line interface involves these steps:
- Bootstrapping the initial Monitor node
- Adding an Object Storage Device (OSD) node
Configuring the Network Time Protocol for Red Hat Ceph Storage
All Ceph Monitor and OSD nodes requires configuring the Network Time Protocol (NTP). Ensure that Ceph nodes are NTP peers. NTP helps preempt issues that arise from clock drift.
When using Ansible to deploy a Red Hat Ceph Storage cluster, Ansible automatically installs, configures, and enables NTP.
Prerequisites
- Network access to a valid time source.
Procedure: Configuring the Network Time Protocol for RHCS
Do the following steps on the all RHCS nodes in the storage cluster, as the root user.
Install the
ntppackage:# yum install ntp
Start and enable the NTP service to be persistent across a reboot:
# systemctl start ntpd # systemctl enable ntpd
Ensure that NTP is synchronizing clocks properly:
$ ntpq -p
Additional Resources
- The Configuring NTP Using ntpd chapter in the System Administrator’s Guide for Red Hat Enterprise Linux 7.
Monitor Bootstrapping
Bootstrapping a Monitor and by extension a Ceph storage cluster, requires the following data:
- Unique Identifier
-
The File System Identifier (
fsid) is a unique identifier for the cluster. Thefsidwas originally used when the Ceph storage cluster was principally used for the Ceph file system. Ceph now supports native interfaces, block devices, and object storage gateway interfaces too, sofsidis a bit of a misnomer. - Cluster Name
Ceph clusters have a cluster name, which is a simple string without spaces. The default cluster name is
ceph, but you can specify a different cluster name. Overriding the default cluster name is especially useful when you work with multiple clusters.When you run multiple clusters in a multi-site architecture, the cluster name for example,
us-west,us-eastidentifies the cluster for the current command-line session.NoteTo identify the cluster name on the command-line interface, specify the Ceph configuration file with the cluster name, for example,
ceph.conf,us-west.conf,us-east.conf, and so on.Example:
# ceph --cluster us-west.conf ...
- Monitor Name
-
Each Monitor instance within a cluster has a unique name. In common practice, the Ceph Monitor name is the node name. Red Hat recommend one Ceph Monitor per node, and no co-locating the Ceph OSD daemons with the Ceph Monitor daemon. To retrieve the short node name, use the
hostname -scommand. - Monitor Map
Bootstrapping the initial Monitor requires you to generate a Monitor map. The Monitor map requires:
-
The File System Identifier (
fsid) -
The cluster name, or the default cluster name of
cephis used - At least one host name and its IP address.
-
The File System Identifier (
- Monitor Keyring
- Monitors communicate with each other by using a secret key. You must generate a keyring with a Monitor secret key and provide it when bootstrapping the initial Monitor.
- Administrator Keyring
-
To use the
cephcommand-line interface utilities, create theclient.adminuser and generate its keyring. Also, you must add theclient.adminuser to the Monitor keyring.
The foregoing requirements do not imply the creation of a Ceph configuration file. However, as a best practice, Red Hat recommends creating a Ceph configuration file and populating it with the fsid, the mon initial members and the mon host settings at a minimum.
You can get and set all of the Monitor settings at runtime as well. However, the Ceph configuration file might contain only those settings which overrides the default values. When you add settings to a Ceph configuration file, these settings override the default settings. Maintaining those settings in a Ceph configuration file makes it easier to maintain the cluster.
To bootstrap the initial Monitor, perform the following steps:
Enable the Red Hat Ceph Storage 3 Monitor repository:
[root@monitor ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-mon-rpms
On your initial Monitor node, install the
ceph-monpackage asroot:# yum install ceph-mon
As
root, create a Ceph configuration file in the/etc/ceph/directory. By default, Ceph usesceph.conf, wherecephreflects the cluster name:Syntax
# touch /etc/ceph/<cluster_name>.conf
Example
# touch /etc/ceph/ceph.conf
As
root, generate the unique identifier for your cluster and add the unique identifier to the[global]section of the Ceph configuration file:Syntax
# echo "[global]" > /etc/ceph/<cluster_name>.conf # echo "fsid = `uuidgen`" >> /etc/ceph/<cluster_name>.conf
Example
# echo "[global]" > /etc/ceph/ceph.conf # echo "fsid = `uuidgen`" >> /etc/ceph/ceph.conf
View the current Ceph configuration file:
$ cat /etc/ceph/ceph.conf [global] fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993
As
root, add the initial Monitor to the Ceph configuration file:Syntax
# echo "mon initial members = <monitor_host_name>[,<monitor_host_name>]" >> /etc/ceph/<cluster_name>.conf
Example
# echo "mon initial members = node1" >> /etc/ceph/ceph.conf
As
root, add the IP address of the initial Monitor to the Ceph configuration file:Syntax
# echo "mon host = <ip-address>[,<ip-address>]" >> /etc/ceph/<cluster_name>.conf
Example
# echo "mon host = 192.168.0.120" >> /etc/ceph/ceph.conf
NoteTo use IPv6 addresses, you set the
ms bind ipv6option totrue. For details, see the Bind section in the Configuration Guide for Red Hat Ceph Storage 3.As
root, create the keyring for the cluster and generate the Monitor secret key:Syntax
# ceph-authtool --create-keyring /tmp/<cluster_name>.mon.keyring --gen-key -n mon. --cap mon '<capabilites>'
Example
# ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *' creating /tmp/ceph.mon.keyring
As
root, generate an administrator keyring, generate a<cluster_name>.client.admin.keyringuser and add the user to the keyring:Syntax
# ceph-authtool --create-keyring /etc/ceph/<cluster_name>.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon '<capabilites>' --cap osd '<capabilites>' --cap mds '<capabilites>'
Example
# ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow' creating /etc/ceph/ceph.client.admin.keyring
As
root, add the<cluster_name>.client.admin.keyringkey to the<cluster_name>.mon.keyring:Syntax
# ceph-authtool /tmp/<cluster_name>.mon.keyring --import-keyring /etc/ceph/<cluster_name>.client.admin.keyring
Example
# ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring importing contents of /etc/ceph/ceph.client.admin.keyring into /tmp/ceph.mon.keyring
Generate the Monitor map. Specify using the node name, IP address and the
fsid, of the initial Monitor and save it as/tmp/monmap:Syntax
$ monmaptool --create --add <monitor_host_name> <ip-address> --fsid <uuid> /tmp/monmap
Example
$ monmaptool --create --add node1 192.168.0.120 --fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 /tmp/monmap monmaptool: monmap file /tmp/monmap monmaptool: set fsid to a7f64266-0894-4f1e-a635-d0aeaca0e993 monmaptool: writing epoch 0 to /tmp/monmap (1 monitors)
As
rooton the initial Monitor node, create a default data directory:Syntax
# mkdir /var/lib/ceph/mon/<cluster_name>-<monitor_host_name>
Example
# mkdir /var/lib/ceph/mon/ceph-node1
As
root, populate the initial Monitor daemon with the Monitor map and keyring:Syntax
# ceph-mon [--cluster <cluster_name>] --mkfs -i <monitor_host_name> --monmap /tmp/monmap --keyring /tmp/<cluster_name>.mon.keyring
Example
# ceph-mon --mkfs -i node1 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring ceph-mon: set fsid to a7f64266-0894-4f1e-a635-d0aeaca0e993 ceph-mon: created monfs at /var/lib/ceph/mon/ceph-node1 for mon.node1
View the current Ceph configuration file:
# cat /etc/ceph/ceph.conf [global] fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993 mon_initial_members = node1 mon_host = 192.168.0.120
For more details on the various Ceph configuration settings, see the Configuration Guide for Red Hat Ceph Storage 3. The following example of a Ceph configuration file lists some of the most common configuration settings:
Example
[global] fsid = <cluster-id> mon initial members = <monitor_host_name>[, <monitor_host_name>] mon host = <ip-address>[, <ip-address>] public network = <network>[, <network>] cluster network = <network>[, <network>] auth cluster required = cephx auth service required = cephx auth client required = cephx osd journal size = <n> osd pool default size = <n> # Write an object n times. osd pool default min size = <n> # Allow writing n copy in a degraded state. osd pool default pg num = <n> osd pool default pgp num = <n> osd crush chooseleaf type = <n>
As
root, create thedonefile:Syntax
# touch /var/lib/ceph/mon/<cluster_name>-<monitor_host_name>/done
Example
# touch /var/lib/ceph/mon/ceph-node1/done
As
root, update the owner and group permissions on the newly created directory and files:Syntax
# chown -R <owner>:<group> <path_to_directory>
Example
# chown -R ceph:ceph /var/lib/ceph/mon # chown -R ceph:ceph /var/log/ceph # chown -R ceph:ceph /var/run/ceph # chown ceph:ceph /etc/ceph/ceph.client.admin.keyring # chown ceph:ceph /etc/ceph/ceph.conf # chown ceph:ceph /etc/ceph/rbdmap
NoteIf the Ceph Monitor node is co-located with an OpenStack Controller node, then the Glance and Cinder keyring files must be owned by
glanceandcinderrespectively. For example:# ls -l /etc/ceph/ ... -rw-------. 1 glance glance 64 <date> ceph.client.glance.keyring -rw-------. 1 cinder cinder 64 <date> ceph.client.cinder.keyring ...
For storage clusters with custom names, as
root, add the the following line:Syntax
# echo "CLUSTER=<custom_cluster_name>" >> /etc/sysconfig/ceph
Example
# echo "CLUSTER=test123" >> /etc/sysconfig/ceph
As
root, start and enable theceph-monprocess on the initial Monitor node:Syntax
# systemctl enable ceph-mon.target # systemctl enable ceph-mon@<monitor_host_name> # systemctl start ceph-mon@<monitor_host_name>
Example
# systemctl enable ceph-mon.target # systemctl enable ceph-mon@node1 # systemctl start ceph-mon@node1
As
root, verify the monitor daemon is running:Syntax
# systemctl status ceph-mon@<monitor_host_name>
Example
# systemctl status ceph-mon@node1 ● ceph-mon@node1.service - Ceph cluster monitor daemon Loaded: loaded (/usr/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2018-06-27 11:31:30 PDT; 5min ago Main PID: 1017 (ceph-mon) CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@node1.service └─1017 /usr/bin/ceph-mon -f --cluster ceph --id node1 --setuser ceph --setgroup ceph Jun 27 11:31:30 node1 systemd[1]: Started Ceph cluster monitor daemon. Jun 27 11:31:30 node1 systemd[1]: Starting Ceph cluster monitor daemon...
To add more Red Hat Ceph Storage Monitors to the storage cluster, see the Adding a Monitor section in the Administration Guide for Red Hat Ceph Storage 3.
OSD Bootstrapping
Once you have your initial monitor running, you can start adding the Object Storage Devices (OSDs). Your cluster cannot reach an active + clean state until you have enough OSDs to handle the number of copies of an object.
The default number of copies for an object is three. You will need three OSD nodes at minimum. However, if you only want two copies of an object, therefore only adding two OSD nodes, then update the osd pool default size and osd pool default min size settings in the Ceph configuration file.
For more details, see the OSD Configuration Reference section in the Configuration Guide for Red Hat Ceph Storage 3.
After bootstrapping the initial monitor, the cluster has a default CRUSH map. However, the CRUSH map does not have any Ceph OSD daemons mapped to a Ceph node.
To add an OSD to the cluster and updating the default CRUSH map, execute the following on each OSD node:
Enable the Red Hat Ceph Storage 3 OSD repository:
[root@osd ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-osd-rpms
As
root, install theceph-osdpackage on the Ceph OSD node:# yum install ceph-osd
Copy the Ceph configuration file and administration keyring file from the initial Monitor node to the OSD node:
Syntax
# scp <user_name>@<monitor_host_name>:<path_on_remote_system> <path_to_local_file>
Example
# scp root@node1:/etc/ceph/ceph.conf /etc/ceph # scp root@node1:/etc/ceph/ceph.client.admin.keyring /etc/ceph
Generate the Universally Unique Identifier (UUID) for the OSD:
$ uuidgen b367c360-b364-4b1d-8fc6-09408a9cda7a
As
root, create the OSD instance:Syntax
# ceph osd create <uuid> [<osd_id>]
Example
# ceph osd create b367c360-b364-4b1d-8fc6-09408a9cda7a 0
NoteThis command outputs the OSD number identifier needed for subsequent steps.
As
root, create the default directory for the new OSD:Syntax
# mkdir /var/lib/ceph/osd/<cluster_name>-<osd_id>
Example
# mkdir /var/lib/ceph/osd/ceph-0
As
root, prepare the drive for use as an OSD, and mount it to the directory you just created. Create a partition for the Ceph data and journal. The journal and the data partitions can be located on the same disk. This example is using a 15 GB disk:Syntax
# parted <path_to_disk> mklabel gpt # parted <path_to_disk> mkpart primary 1 10000 # mkfs -t <fstype> <path_to_partition> # mount -o noatime <path_to_partition> /var/lib/ceph/osd/<cluster_name>-<osd_id> # echo "<path_to_partition> /var/lib/ceph/osd/<cluster_name>-<osd_id> xfs defaults,noatime 1 2" >> /etc/fstab
Example
# parted /dev/sdb mklabel gpt # parted /dev/sdb mkpart primary 1 10000 # parted /dev/sdb mkpart primary 10001 15000 # mkfs -t xfs /dev/sdb1 # mount -o noatime /dev/sdb1 /var/lib/ceph/osd/ceph-0 # echo "/dev/sdb1 /var/lib/ceph/osd/ceph-0 xfs defaults,noatime 1 2" >> /etc/fstab
As
root, initialize the OSD data directory:Syntax
# ceph-osd -i <osd_id> --mkfs --mkkey --osd-uuid <uuid>
Example
# ceph-osd -i 0 --mkfs --mkkey --osd-uuid b367c360-b364-4b1d-8fc6-09408a9cda7a ... auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory ... created new key in keyring /var/lib/ceph/osd/ceph-0/keyring
NoteThe directory must be empty before you run
ceph-osdwith the--mkkeyoption. If you have a custom cluster name, theceph-osdutility requires the--clusteroption.As
root, register the OSD authentication key. If your cluster name differs fromceph, insert your cluster name instead:Syntax
# ceph auth add osd.<osd_id> osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/<cluster_name>-<osd_id>/keyring
Example
# ceph auth add osd.0 osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-0/keyring added key for osd.0
As
root, add the OSD node to the CRUSH map:Syntax
# ceph [--cluster <cluster_name>] osd crush add-bucket <host_name> host
Example
# ceph osd crush add-bucket node2 host
As
root, place the OSD node under thedefaultCRUSH tree:Syntax
# ceph [--cluster <cluster_name>] osd crush move <host_name> root=default
Example
# ceph osd crush move node2 root=default
As
root, add the OSD disk to the CRUSH mapSyntax
# ceph [--cluster <cluster_name>] osd crush add osd.<osd_id> <weight> [<bucket_type>=<bucket-name> ...]
Example
# ceph osd crush add osd.0 1.0 host=node2 add item id 0 name 'osd.0' weight 1 at location {host=node2} to crush mapNoteYou can also decompile the CRUSH map, and add the OSD to the device list. Add the OSD node as a bucket, then add the device as an item in the OSD node, assign the OSD a weight, recompile the CRUSH map and set the CRUSH map. For more details, see the Editing a CRUSH map section in the Storage Strategies Guide for Red Hat Ceph Storage 3. for more details.
As
root, update the owner and group permissions on the newly created directory and files:Syntax
# chown -R <owner>:<group> <path_to_directory>
Example
# chown -R ceph:ceph /var/lib/ceph/osd # chown -R ceph:ceph /var/log/ceph # chown -R ceph:ceph /var/run/ceph # chown -R ceph:ceph /etc/ceph
For storage clusters with custom names, as
root, add the following line to the/etc/sysconfig/cephfile:Syntax
# echo "CLUSTER=<custom_cluster_name>" >> /etc/sysconfig/ceph
Example
# echo "CLUSTER=test123" >> /etc/sysconfig/ceph
The OSD node is in your Ceph storage cluster configuration. However, the OSD daemon is
downandin. The new OSD must beupbefore it can begin receiving data. Asroot, enable and start the OSD process:Syntax
# systemctl enable ceph-osd.target # systemctl enable ceph-osd@<osd_id> # systemctl start ceph-osd@<osd_id>
Example
# systemctl enable ceph-osd.target # systemctl enable ceph-osd@0 # systemctl start ceph-osd@0
Once you start the OSD daemon, it is
upandin.
Now you have the monitors and some OSDs up and running. You can watch the placement groups peer by executing the following command:
$ ceph -w
To view the OSD tree, execute the following command:
$ ceph osd tree
Example
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 2 root default -2 2 host node2 0 1 osd.0 up 1 1 -3 1 host node3 1 1 osd.1 up 1 1
To expand the storage capacity by adding new OSDs to the storage cluster, see the Adding an OSD section in the Administration Guide for Red Hat Ceph Storage 3.
Appendix C. Installing the Ceph Command Line Interface
The Ceph command-line interface (CLI) enables administrators to execute Ceph administrative commands. The CLI is provided by the ceph-common package and includes the following utilities:
-
ceph -
ceph-authtool -
ceph-dencoder -
rados
Prerequisites
-
A running Ceph storage cluster, preferably in the
active + cleanstate.
Procedure
On the client node, enable the Red Hat Ceph Storage 3 Tools repository:
[root@gateway ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms
On the client node, install the
ceph-commonpackage:# yum install ceph-common
From the initial monitor node, copy the Ceph configuration file, in this case
ceph.conf, and the administration keyring to the client node:Syntax
# scp /etc/ceph/<cluster_name>.conf <user_name>@<client_host_name>:/etc/ceph/ # scp /etc/ceph/<cluster_name>.client.admin.keyring <user_name>@<client_host_name:/etc/ceph/
Example
# scp /etc/ceph/ceph.conf root@node1:/etc/ceph/ # scp /etc/ceph/ceph.client.admin.keyring root@node1:/etc/ceph/
Replace
<client_host_name>with the host name of the client node.
Appendix D. Manually Installing Ceph Block Device
The following procedure shows how to install and mount a thin-provisioned, resizable Ceph Block Device.
Ceph Block Devices must be deployed on separate nodes from the Ceph Monitor and OSD nodes. Running kernel clients and kernel server daemons on the same node can lead to kernel deadlocks.
Prerequisites
- Ensure to perform the tasks listed in the Appendix C, Installing the Ceph Command Line Interface section.
- If you use Ceph Block Devices as a back end for virtual machines (VMs) that use QEMU, increase the default file descriptor. See the Ceph - VM hangs when transferring large amounts of data to RBD disk Knowledgebase article for details.
Procedure
Create a Ceph Block Device user named
client.rbdwith full permissions to files on OSD nodes (osd 'allow rwx') and output the result to a keyring file:ceph auth get-or-create client.rbd mon 'profile rbd' osd 'profile rbd pool=<pool_name>' \ -o /etc/ceph/rbd.keyring
Replace
<pool_name>with the name of the pool that you want to allowclient.rbdto have access to, for examplerbd:# ceph auth get-or-create \ client.rbd mon 'allow r' osd 'allow rwx pool=rbd' \ -o /etc/ceph/rbd.keyring
See the User Management section in the Red Hat Ceph Storage 3 Administration Guide for more information about creating users.
Create a block device image:
rbd create <image_name> --size <image_size> --pool <pool_name> \ --name client.rbd --keyring /etc/ceph/rbd.keyring
Specify
<image_name>,<image_size>, and<pool_name>, for example:$ rbd create image1 --size 4096 --pool rbd \ --name client.rbd --keyring /etc/ceph/rbd.keyring
WarningThe default Ceph configuration includes the following Ceph Block Device features:
-
layering -
exclusive-lock -
object-map -
deep-flatten -
fast-diff
If you use the kernel RBD (
krbd) client, you will not be able to map the block device image because the current kernel version included in Red Hat Enterprise Linux 7.3 does not supportobject-map,deep-flatten, andfast-diff.To work around this problem, disable the unsupported features. Use one of the following options to do so:
Disable the unsupported features dynamically:
rbd feature disable <image_name> <feature_name>
For example:
# rbd feature disable image1 object-map deep-flatten fast-diff
-
Use the
--image-feature layeringoption with therbd createcommand to enable onlylayeringon newly created block device images. Disable the features be default in the Ceph configuration file:
rbd_default_features = 1
This is a known issue, for details see the Known Issues chapter in the Release Notes for Red Hat Ceph Storage 3.
All these features work for users that use the user-space RBD client to access the block device images.
-
Map the newly created image to the block device:
rbd map <image_name> --pool <pool_name>\ --name client.rbd --keyring /etc/ceph/rbd.keyring
For example:
# rbd map image1 --pool rbd --name client.rbd \ --keyring /etc/ceph/rbd.keyring
ImportantKernel block devices currently only support the legacy straw bucket algorithm in the CRUSH map. If you have set the CRUSH tunables to optimal, you must set them to legacy or an earlier major release, otherwise, you will not be able to map the image.
Alternatively, replace
straw2withstrawin the CRUSH map. For details, see the Editing a CRUSH Map chapter in the Storage Strategies guide for Red Hat Ceph Storage 3.Use the block device by creating a file system:
mkfs.ext4 -m5 /dev/rbd/<pool_name>/<image_name>
Specify the pool name and the image name, for example:
# mkfs.ext4 -m5 /dev/rbd/rbd/image1
This can take a few moments.
Mount the newly created file system:
mkdir <mount_directory> mount /dev/rbd/<pool_name>/<image_name> <mount_directory>
For example:
# mkdir /mnt/ceph-block-device # mount /dev/rbd/rbd/image1 /mnt/ceph-block-device
For additional details, see the Block Device Guide for Red Hat Ceph Storage 3.
Appendix E. Manually Installing Ceph Object Gateway
The Ceph object gateway, also know as the RADOS gateway, is an object storage interface built on top of the librados API to provide applications with a RESTful gateway to Ceph storage clusters.
Prerequisites
-
A running Ceph storage cluster, preferably in the
active + cleanstate. - Perform the tasks listed in Chapter 2, Requirements for Installing Red Hat Ceph Storage.
Procedure
Enable the Red Hat Ceph Storage 3 Tools repository:
[root@gateway ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms
On the Object Gateway node, install the
ceph-radosgwpackage:# yum install ceph-radosgw
On the initial Monitor node, do the following steps.
Update the Ceph configuration file as follows:
[client.rgw.<obj_gw_hostname>] host = <obj_gw_hostname> rgw frontends = "civetweb port=80" rgw dns name = <obj_gw_hostname>.example.com
Where
<obj_gw_hostname>is a short host name of the gateway node. To view the short host name, use thehostname -scommand.Copy the updated configuration file to the new Object Gateway node and all other nodes in the Ceph storage cluster:
Syntax
# scp /etc/ceph/<cluster_name>.conf <user_name>@<target_host_name>:/etc/ceph
Example
# scp /etc/ceph/ceph.conf root@node1:/etc/ceph/
Copy the
<cluster_name>.client.admin.keyringfile to the new Object Gateway node:Syntax
# scp /etc/ceph/<cluster_name>.client.admin.keyring <user_name>@<target_host_name>:/etc/ceph/
Example
# scp /etc/ceph/ceph.client.admin.keyring root@node1:/etc/ceph/
On the Object Gateway node, create the data directory:
Syntax
# mkdir -p /var/lib/ceph/radosgw/<cluster_name>-rgw.`hostname -s`
Example
# mkdir -p /var/lib/ceph/radosgw/ceph-rgw.`hostname -s`
On the Object Gateway node, add a user and keyring to bootstrap the object gateway:
Syntax
# ceph auth get-or-create client.rgw.`hostname -s` osd 'allow rwx' mon 'allow rw' -o /var/lib/ceph/radosgw/<cluster_name>-rgw.`hostname -s`/keyring
Example
# ceph auth get-or-create client.rgw.`hostname -s` osd 'allow rwx' mon 'allow rw' -o /var/lib/ceph/radosgw/ceph-rgw.`hostname -s`/keyring
ImportantWhen you provide capabilities to the gateway key you must provide the read capability. However, providing the Monitor write capability is optional; if you provide it, the Ceph Object Gateway will be able to create pools automatically.
In such a case, ensure to specify a reasonable number of placement groups in a pool. Otherwise, the gateway uses the default number, which might not be suitable for your needs. See Ceph Placement Groups (PGs) per Pool Calculator for details.
On the Object Gateway node, create the
donefile:Syntax
# touch /var/lib/ceph/radosgw/<cluster_name>-rgw.`hostname -s`/done
Example
# touch /var/lib/ceph/radosgw/ceph-rgw.`hostname -s`/done
On the Object Gateway node, change the owner and group permissions:
# chown -R ceph:ceph /var/lib/ceph/radosgw # chown -R ceph:ceph /var/log/ceph # chown -R ceph:ceph /var/run/ceph # chown -R ceph:ceph /etc/ceph
For storage clusters with custom names, as
root, add the following line:Syntax
# echo "CLUSTER=<custom_cluster_name>" >> /etc/sysconfig/ceph
Example
# echo "CLUSTER=test123" >> /etc/sysconfig/ceph
On the Object Gateway node, open TCP port 80:
# firewall-cmd --zone=public --add-port=80/tcp # firewall-cmd --zone=public --add-port=80/tcp --permanent
On the Object Gateway node, start and enable the
ceph-radosgwprocess:Syntax
# systemctl enable ceph-radosgw.target # systemctl enable ceph-radosgw@rgw.<rgw_hostname> # systemctl start ceph-radosgw@rgw.<rgw_hostname>
Example
# systemctl enable ceph-radosgw.target # systemctl enable ceph-radosgw@rgw.node1 # systemctl start ceph-radosgw@rgw.node1
Once installed, the Ceph Object Gateway automatically creates pools if the write capability is set on the Monitor. See the Pools chapter in the Storage Strategies Guide for information on creating pools manually.
Additional Details
- The Red Hat Ceph Storage 3 the Object Gateway Guide for Red Hat Enterprise Linux
Appendix F. Overriding Ceph Default Settings
Unless otherwise specified in the Ansible configuration files, Ceph uses its default settings.
Because Ansible manages the Ceph configuration file, edit the /etc/ansible/group_vars/all.yml file to change the Ceph configuration. Use the ceph_conf_overrides setting to override the default Ceph configuration.
Ansible supports the same sections as the Ceph configuration file; [global], [mon], [osd], [mds], [rgw], and so on. You can also override particular instances, such as a particular Ceph Object Gateway instance. For example:
###################
# CONFIG OVERRIDE #
###################
ceph_conf_overrides:
client.rgw.rgw1:
log_file: /var/log/ceph/ceph-rgw-rgw1.logAnsible does not include braces when referring to a particular section of the Ceph configuration file. Sections and settings names are terminated with a colon.
Do not set the cluster network with the cluster_network parameter in the CONFIG OVERRIDE section because this can cause two conflicting cluster networks being set in the Ceph configuration file.
To set the cluster network, use the cluster_network parameter in the CEPH CONFIGURATION section. For details, see Section 3.2, “Installing a Red Hat Ceph Storage Cluster”.
Appendix G. Manually Upgrading from Red Hat Ceph Storage 2 to 3
This chapter provides information on how to manually upgrade from Red Hat Ceph storage version 2.4 to 3.0.
You can upgrade the Ceph Storage Cluster in a rolling fashion and while the cluster is running. Upgrade each node in the cluster sequentially, only proceeding to the next node after the previous node is done.
Red Hat recommends upgrading the Ceph components in the following order:
- Monitor nodes
- OSD nodes
- Ceph Object Gateway nodes
- All other Ceph client nodes
Two methods are available to upgrade a Red Hat Ceph Storage 2.3 to 3:
- Using Red Hat’s Content Delivery Network (CDN)
- Using a Red Hat provided ISO image file
After upgrading the storage cluster you might have a health warning regarding the CRUSH map using legacy tunables. For details, see the CRUSH Tunables section in the Storage Strategies guide for Red Hat Ceph Storage 3.
Example
$ ceph -s
cluster 848135d7-cdb9-4084-8df2-fb5e41ae60bd
health HEALTH_WARN
crush map has legacy tunables (require bobtail, min is firefly)
monmap e1: 1 mons at {ceph1=192.168.0.121:6789/0}
election epoch 2, quorum 0 ceph1
osdmap e83: 2 osds: 2 up, 2 in
pgmap v1864: 64 pgs, 1 pools, 38192 kB data, 17 objects
10376 MB used, 10083 MB / 20460 MB avail
64 active+clean
Red Hat recommends all Ceph clients to be running the same version as the Ceph storage cluster.
Prerequisites
If the cluster you want to upgrade contains Ceph Block Device images that use the
exclusive-lockfeature, ensure that all Ceph Block Device users have permissions to blacklist clients:ceph auth caps client.<ID> mon 'allow r, allow command "osd blacklist"' osd '<existing-OSD-user-capabilities>'
Upgrading Monitor Nodes
This section describes steps to upgrade a Ceph Monitor node to a later version. There must be an odd number of Monitors. While you are upgrading one Monitor, the storage cluster will still have quorum.
Procedure
Do the following steps on each Monitor node in the storage cluster. Upgrade only one Monitor node at a time.
If you installed Red Hat Ceph Storage 2 by using software repositories, disable the repositories:
# subscription-manager repos --disable=rhel-7-server-rhceph-2-mon-rpms --disable=rhel-7-server-rhceph-2-installer-rpms
Enable the Red Hat Ceph Storage 3 Monitor repository:
[root@monitor ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-mon-rpms
As
root, stop the Monitor process:Syntax
# service ceph stop <daemon_type>.<monitor_host_name>
Example
# service ceph stop mon.node1
As
root, update theceph-monpackage:# yum update ceph-mon
As
root, update the owner and group permissions:Syntax
# chown -R <owner>:<group> <path_to_directory>
Example
# chown -R ceph:ceph /var/lib/ceph/mon # chown -R ceph:ceph /var/log/ceph # chown -R ceph:ceph /var/run/ceph # chown ceph:ceph /etc/ceph/ceph.client.admin.keyring # chown ceph:ceph /etc/ceph/ceph.conf # chown ceph:ceph /etc/ceph/rbdmap
NoteIf the Ceph Monitor node is colocated with an OpenStack Controller node, then the Glance and Cinder keyring files must be owned by
glanceandcinderrespectively. For example:# ls -l /etc/ceph/ ... -rw-------. 1 glance glance 64 <date> ceph.client.glance.keyring -rw-------. 1 cinder cinder 64 <date> ceph.client.cinder.keyring ...
If SELinux is in enforcing or permissive mode, relabel the SELinux context on the next reboot.
# touch /.autorelabel
WarningRelabeling can take a long time to complete because SELinux must traverse every file system and fix any mislabeled files. To exclude directories from being relabeled, add the directories to the
/etc/selinux/fixfiles_exclude_dirsfile before rebooting.As
root, enable theceph-monprocess:# systemctl enable ceph-mon.target # systemctl enable ceph-mon@<monitor_host_name>
As
root, reboot the Monitor node:# shutdown -r now
Once the Monitor node is up, check the health of the Ceph storage cluster before moving to the next Monitor node:
# ceph -s
Upgrading OSD Nodes
This section describes steps to upgrade a Ceph OSD node to a later version.
Prerequisites
When upgrading an OSD node, some placement groups will become degraded because the OSD might be down or restarting. To prevent Ceph from starting the recovery process, on a Monitor node, set the noout and norebalance OSD flags:
[root@monitor ~]# ceph osd set noout [root@monitor ~]# ceph osd set norebalance
Procedure
Do the following steps on each OSD node in the storage cluster. Upgrade only one OSD node at a time. If an ISO-based installation was performed for Red Hat Ceph Storage 2.3, then skip this first step.
As
root, disable the Red Hat Ceph Storage 2 repositories:# subscription-manager repos --disable=rhel-7-server-rhceph-2-osd-rpms --disable=rhel-7-server-rhceph-2-installer-rpms
Enable the Red Hat Ceph Storage 3 OSD repository:
[root@osd ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-osd-rpms
As
root, stop any running OSD process:Syntax
# service ceph stop <daemon_type>.<osd_id>
Example
# service ceph stop osd.0
As
root, update theceph-osdpackage:# yum update ceph-osd
As
root, update the owner and group permissions on the newly created directory and files:Syntax
# chown -R <owner>:<group> <path_to_directory>
Example
# chown -R ceph:ceph /var/lib/ceph/osd # chown -R ceph:ceph /var/log/ceph # chown -R ceph:ceph /var/run/ceph # chown -R ceph:ceph /etc/ceph
NoteUsing the following
findcommand might quicken the process of changing ownership by using thechowncommand in parallel on a Ceph storage cluster with a large number of disks:# find /var/lib/ceph/osd -maxdepth 1 -mindepth 1 -print | xargs -P12 -n1 chown -R ceph:ceph
If SELinux is set to enforcing or permissive mode, then set a relabelling of the SELinux context on files for the next reboot:
# touch /.autorelabel
WarningRelabeling will take a long time to complete, because SELinux must traverse every file system and fix any mislabeled files. To exclude directories from being relabelled, add the directory to the
/etc/selinux/fixfiles_exclude_dirsfile before rebooting.NoteIn environments with large number of objects per placement group (PG), the directory enumeration speed will decrease, causing a negative impact to performance. This is caused by the addition of xattr queries which verifies the SELinux context. Setting the context at mount time removes the xattr queries for context and helps overall disk performance, especially on slower disks.
Add the following line to the
[osd]section in the/etc/ceph/ceph.conffile:+
osd_mount_options_xfs=rw,noatime,inode64,context="system_u:object_r:ceph_var_lib_t:s0"
As
root, replay device events from the kernel:# udevadm trigger
As
root, enable theceph-osdprocess:# systemctl enable ceph-osd.target # systemctl enable ceph-osd@<osd_id>
As
root, reboot the OSD node:# shutdown -r now
Move to the next OSD node.
NoteIf the
nooutandnorebalanceflags are set, the storage cluster is inHEALTH_WARNstate$ ceph health HEALTH_WARN noout,norebalance flag(s) set
Once you are done upgrading the Ceph Storage Cluster, unset the previously set OSD flags and verify the storage cluster status.
On a Monitor node, and after all OSD nodes have been upgraded, unset the noout and norebalance flags:
# ceph osd unset noout # ceph osd unset norebalance
In addition, execute the ceph osd require-osd-release <release> command. This command ensures that no more OSDs with Red Hat Ceph Storage 2.3 can be added to the storage cluster. If you do not run this command, the storage status will be HEALTH_WARN.
# ceph osd require-osd-release luminous
Additional Resources
- To expand the storage capacity by adding new OSDs to the storage cluster, see the Add an OSD section in the Administration Guide for Red Hat Ceph Storage 3
Upgrading the Ceph Object Gateway Nodes
This section describes steps to upgrade a Ceph Object Gateway node to a later version.
Prerequisites
- Red Hat recommends putting a Ceph Object Gateway behind a load balancer, such as HAProxy. If you use a load balancer, remove the Ceph Object Gateway from the load balancer once no requests are being served.
If you use a custom name for the region pool, specified in the
rgw_region_root_poolparameter, add thergw_zonegroup_root_poolparameter to the[global]section of the Ceph configuration file. Set the value ofrgw_zonegroup_root_poolto be the same asrgw_region_root_pool, for example:[global] rgw_zonegroup_root_pool = .us.rgw.root
Procedure
Do the following steps on each Ceph Object Gateway node in the storage cluster. Upgrade only one node at a time.
If you used online repositories to install Red Hat Ceph Storage, disable the 2 repositories.
# subscription-manager repos --disable=rhel-7-server-rhceph-2.3-tools-rpms --disable=rhel-7-server-rhceph-2-installer-rpms
Enable the Red Hat Ceph Storage 3 Tools repository:
[root@gateway ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms
Stop the Ceph Object Gateway process (
ceph-radosgw):# service ceph-radosgw stop
Update the
ceph-radosgwpackage:# yum update ceph-radosgw
Change the owner and group permissions on the newly created
/var/lib/ceph/radosgw/and/var/log/ceph/directories and their content toceph.# chown -R ceph:ceph /var/lib/ceph/radosgw # chown -R ceph:ceph /var/log/ceph
If SELinux is set to run in enforcing or permissive mode, instruct it to relabel SELinux context on the next boot.
# touch /.autorelabel
ImportantRelabeling takes a long time to complete, because SELinux must traverse every file system and fix any mislabeled files. To exclude directories from being relabeled, add them to the
/etc/selinux/fixfiles_exclude_dirsfile before rebooting.Enable the
ceph-radosgwprocess.# systemctl enable ceph-radosgw.target # systemctl enable ceph-radosgw@rgw.<hostname>
Replace
<hostname>with the name of the Ceph Object Gateway host, for examplegateway-node.# systemctl enable ceph-radosgw.target # systemctl enable ceph-radosgw@rgw.gateway-node
Reboot the Ceph Object Gateway node.
# shutdown -r now
- If you use a load balancer, add the Ceph Object Gateway node back to the load balancer.
See Also
Upgrading a Ceph Client Node
Ceph clients are:
- Ceph Block Devices
- OpenStack Nova compute nodes
- QEMU/KVM hypervisors
- Any custom application that uses the Ceph client-side libraries
Red Hat recommends all Ceph clients to be running the same version as the Ceph storage cluster.
Prerequisites
- Stop all I/O requests against a Ceph client node while upgrading the packages to prevent unexpected errors to occur
Procedure
If you installed Red Hat Ceph Storage 2 clients by using software repositories, disable the repositories:
# subscription-manager repos --disable=rhel-7-server-rhceph-2-tools-rpms --disable=rhel-7-server-rhceph-2-installer-rpms
NoteIf an ISO-based installation was performed for Red Hat Ceph Storage 2 clients, skip this first step.
On the client node, enable the Red Hat Ceph Storage Tools 3 repository:
[root@gateway ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms
On the client node, update the
ceph-commonpackage:# yum update ceph-common
Restart any application that depends on the Ceph client-side libraries after upgrading the ceph-common package.
If you are upgrading OpenStack Nova compute nodes that have running QEMU/KVM instances or use a dedicated QEMU/KVM client, stop and start the QEMU/KVM instance because restarting the instance does not work in this case.
Appendix H. Changes in Ansible Variables Between Version 2 and 3
With Red Hat Ceph Storage 3, certain variables in the configuration files located in the /usr/share/ceph-ansible/group_vars/ directory have changed or have been removed. The following table lists all the changes. After upgrading to version 3, copy the all.yml.sample and osds.yml.sample files again to reflect these changes. See Chapter 4, Upgrading a Red Hat Ceph Storage Cluster for details.
| Old Option | New Option | File |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Appendix I. Importing an Existing Ceph Cluster to Ansible
You can configure Ansible to use a cluster deployed without Ansible. For example, if you upgraded Red Hat Ceph Storage 1.3 clusters to version 2 manually, configure them to use Ansible by following this procedure:
- After manually upgrading from version 1.3 to version 2, install and configure Ansible on the administration node.
-
Ensure that the Ansible administration node has passwordless
sshaccess to all Ceph nodes in the cluster. See Section 2.11, “Enabling Password-less SSH for Ansible” for more details. As
root, create a symbolic link to the Ansiblegroup_varsdirectory in the/etc/ansible/directory:# ln -s /usr/share/ceph-ansible/group_vars /etc/ansible/group_vars
As
root, create anall.ymlfile from theall.yml.samplefile and open it for editing:# cd /etc/ansible/group_vars # cp all.yml.sample all.yml # vim all.yml
-
Set the
generate_fsidsetting tofalseingroup_vars/all.yml. -
Get the current cluster
fsidby executingceph fsid. -
Set the retrieved
fsidingroup_vars/all.yml. -
Modify the Ansible inventory in
/etc/ansible/hoststo include Ceph hosts. Add monitors under a[mons]section, OSDs under an[osds]section and gateways under an[rgws]section to identify their roles to Ansible. Make sure
ceph_conf_overridesis updated with the originalceph.confoptions used for[global],[osd],[mon], and[client]sections in theall.ymlfile.Options like
osd journal,public_networkandcluster_networkshould not be added inceph_conf_overridesbecause they are already part ofall.yml. Only the options that are not part ofall.ymland are in the originalceph.confshould be added toceph_conf_overrides.From the
/usr/share/ceph-ansible/directory run the playbook.# cd /usr/share/ceph-ansible/ # cp infrastructure-playbooks/take-over-existing-cluster.yml . $ ansible-playbook take-over-existing-cluster.yml -u <username>
Appendix J. Purging a Ceph Cluster by Using Ansible
If you deployed a Ceph cluster using Ansible and you want to purge the cluster, then use the purge-cluster.yml Ansible playbook located in the infrastructure-playbooks directory.
Purging a Ceph cluster will lose data stored on the cluster’s OSDs.
Before purging the Ceph cluster…
Check the osd_auto_discovery option in the osds.yml file. Having this option set to true will cause the purge to fail. To prevent the failure, do the following steps before running the purge:
-
Declare the OSD devices in the
osds.ymlfile. See Section 3.2, “Installing a Red Hat Ceph Storage Cluster” for more details. -
Comment out the
osd_auto_discoveryoption in theosds.ymlfile.
To purge the Ceph cluster…
As
root, navigate to the/usr/share/ceph-ansible/directory:# cd /usr/share/ceph-ansible
As
root, copy thepurge-cluster.ymlAnsible playbook to the current directory:# cp infrastructure-playbooks/purge-cluster.yml .
Run the
purge-cluster.ymlAnsible playbook:$ ansible-playbook purge-cluster.yml
