Chapter 1. Overview

Red Hat Ceph Storage 1.3 and later requires Red Hat Enterprise Linux 7 Server with a homogeneous version, for example, Red Hat Enterprise Linux 7.2 running on AMD64 and Intel 64 architectures for all Ceph nodes, including the Red Hat Storage Console node.

Ceph relies on packages in the Red Hat Enterprise Linux 7 Base content set. Each Ceph node must be able to access the full Red Hat Enterprise Linux 7 Base content.

To do so, register Ceph nodes that can connect to the Internet to the Red Hat Content Delivery Network (CDN) and attach appropriate Ceph subscriptions to the nodes:

Once you register the nodes, enable repositories that provide the Red Hat Ceph Storage packages.

Red Hat Ceph Storage supports two installation methods:

For CDN-based installations, enable the appropriate repository(ies) for each node.

For ISO-based installations, enable ALL repositories on the administration node ONLY.

# subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-rhceph-1.3-calamari-rpms --enable=rhel-7-server-rhceph-1.3-installer-rpms --enable=rhel-7-server-rhceph-1.3-mon-rpms --enable=rhel-7-server-rhceph-1.3-osd-rpms --enable=rhel-7-server-rhceph-1.3-tools-rpms

# yum update

Finally, disconnect your admin node from the internet.

Ceph nodes must be able to resolve short host names, not just fully qualified domain names. Set up a default search domain to resolve short host names. To retrieve a Ceph node’s short host name, execute:

hostname -s

Each Ceph node MUST be able to ping every other Ceph node in the cluster by its short host name.

All Ceph clusters require a public network. You MUST have a network interface card configured to a public network where Ceph clients can reach Ceph Monitors and Ceph OSDs. You SHOULD have a network interface card for a cluster network so that Ceph can conduct heart-beating, peering, replication and recovery on a network separate from the public network.

We DO NOT RECOMMEND using a single NIC for both a public and private network.

Ensure that you configure your network interfaces and make them persistent so that the settings are identical on reboot. For example:

Navigate to /etc/sysconfig/network-scripts and ensure that the ifcfg-<iface> settings for your public and cluster interfaces (assuming you will use a cluster network too [RECOMMENDED]) are properly configured.

For details on configuring network interface scripts for RHEL 7, see Configuring a Network Interface Using ifcfg Files.

Red Hat® Ceph Storage v1.3 uses firewalld. Start the firewall and ensure that you enable it to run on boot.

# systemctl start firewalld
# systemctl enable firewalld

Ensure firewalld is running:

# systemctl status firewalld.service

The default firewall configuration for RHEL is fairly strict. You MUST adjust your firewall settings on the Calamari node to allow inbound requests on port 80 so that clients in your network can access the Calamari web user interface.

Calamari also communicates with Ceph nodes via ports 2003, 4505 and 4506. For firewalld, add port 80, 4505, 4506 and 2003 to the public zone of your Calamari administration node and ensure that you make the setting permanent so that it is enabled on reboot.

You MUST open ports 80, 2003, and 4505-4506 on your Calamari node. First, open the port to ensure it opens immediately at runtime. Then, rerun the command with --permanent to ensure that the port opens on reboot.

# firewall-cmd --zone=public --add-port=80/tcp
# firewall-cmd --zone=public --add-port=80/tcp --permanent
# firewall-cmd --zone=public --add-port=2003/tcp
# firewall-cmd --zone=public --add-port=2003/tcp --permanent
# firewall-cmd --zone=public --add-port=4505-4506/tcp
# firewall-cmd --zone=public --add-port=4505-4506/tcp --permanent

You MUST open port 6789 on your public network on ALL Ceph monitor nodes.

# firewall-cmd --zone=public --add-port=6789/tcp
# firewall-cmd --zone=public --add-port=6789/tcp --permanent

Finally, you MUST also open ports for OSD traffic (6800-7300). Each OSD on each Ceph node needs a few ports: one for talking to clients and monitors (public network); one for sending data to other OSDs (cluster network, if available; otherwise, public network); and, one for heartbeating (cluster network, if available; otherwise, public network). To get started quickly, open up the default port range. For example:

# firewall-cmd --zone=public --add-port=6800-7300/tcp
# firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent

For additional details on firewalld, see Using Firewalls.

You MUST install Network Time Protocol (NTP) on all Ceph monitor nodes and admin nodes. Ensure that ceph nodes are NTP peers. You SHOULD consider installing NTP on Ceph OSD nodes, but it is not required. NTP helps preempt issues that arise from clock drift.

For additional details on NTP for RHEL 7, see Configuring NTP Using ntpd.

For ALL Ceph Nodes perform the following steps:

The ceph-deploy utility must login to a Ceph node as a user that has passwordless sudo privileges, because it needs to install software and configuration files without prompting for passwords.

ceph-deploy supports a --username option so you can specify any user that has password-less sudo (including root, although this is NOT recommended). To use ceph-deploy --username <username>, the user you specify must have password-less SSH access to the Ceph node, because ceph-deploy will not prompt you for a password.

We recommend creating a Ceph Deploy user on ALL Ceph nodes in the cluster. Please do NOT use "ceph" as the user name. A uniform user name across the cluster may improve ease of use (not required), but you should avoid obvious user names, because hackers typically use them with brute force hacks (for example, root, admin, or <productname>). The following procedure, substituting <username> for the user name you define, describes how to create a Ceph Deploy user with passwordless sudo on a Ceph node.

Since ceph-deploy will not prompt for a password, you must generate SSH keys on the admin node and distribute the public key to each Ceph node. ceph-deploy will attempt to generate the SSH keys for initial monitors.

For users that will run Ceph administrator commands on large clusters (for example, 1024 OSDs or more), create an /etc/security/limits.d/50-ceph.conf file on your admin node with the following contents:

<username>       soft    nproc     unlimited

Replace <username> with the name of the non-root account that you will use to run Ceph administrator commands.

If a RAID controller with 1-2 GB of cache is installed on a host, then enabling write-back caches might result in increased small I/O write throughput. In order for this to be done safely, the cache must be non-volatile.

Modern RAID controllers usually have super capacitors that provide enough power to drain volatile memory to non-volatile NAND memory during a power loss event. It is important to understand how a particular controller and firmware behave after power is restored.

Some of them might require manual intervention. Hard drives typically advertise to the operating system whether their disk caches should be enabled or disabled by default. However, certain RAID controllers or some firmware might not provide such information, so verify that disk level caches are disabled to avoid file system corruption.

Create a single RAID 0 volume with write-back for each OSD data drive with write-back cache enabled.

If SAS or SATA connected SSDs are also present on the controller, it is worth investigating whether your controller and firmware support passthrough mode. This will avoid the caching logic, and generally result in much lower latencies for fast media.

Hosts with high numbers of OSDs (more than 12) may spawn a lot of threads, especially during recovery and re-balancing. The standard RHEL 7 kernel defaults to a relatively small maximum number of threads (32768). Check your default settings to see if they are suitable.

cat /proc/sys/kernel/pid_max

Consider setting kernel.pid_max to a higher number of threads. The theoretical maximum is 4,194,303 threads. For example, you could add the following to the /etc/sysctl.conf file to set it to the maximum:

kernel.pid_max = 4194303

To see the changes you made without a reboot, execute:

# sysctl -p

To verify the changes, execute:

# sysctl -a | grep kernel.pid_max

When using a firewall and running several OSDs on a single host, busy clusters might create a lot of network connections and overflow the kernel nf_conntrack table on the OSD and monitor hosts. To find the current values, execute the following commands:

cat /proc/sys/net/netfilter/nf_conntrack_buckets
cat /proc/sys/net/netfilter/nf_conntrack_max

The nf_conntrack_max value defaults to the nf_conntrack_buckets value multiplied by 8. Consider setting nf_conntrack_buckets to a higher number on the OSD and monitor hosts. To do so, create a new /etc/modprobe.d/ceph.conf file with the following content:

options nf_conntrack hashsize=<size>

Where <size> specifies the new size of the nf_conntrack_buckets value. For example:

options nf_conntrack hashsize=128000

Having this option specified loads the nf_conntrack module with a maximum table size of 1024000 (128000 * 8).

To see the changes you made without a reboot, execute the following commands as root:

# systemctl stop firewalld
# modprobe -rv nf_conntrack
# systemctl start firewalld

To verify the changes, execute the following commands as root:

# sysctl -a | grep conntrack_buckets
# sysctl -a | grep conntrack_max

SELinux is set to enforcing mode by default. For Red Hat Ceph Storage 1.3 and 1.3.1, set SELinux to permissive mode, or disable it entirely and ensure that your installation and cluster are working properly. To set SELinux to permissive, execute the following command:

# setenforce 0

To configure SELinux persistently, modify the /etc/selinux/config configuration file.

With Red Hat Ceph Storage 1.3.2 or later, a new ceph-selinux package can be installed on Ceph nodes. This package provides SELinux support for Ceph, and SELinux therefore no longer needs to be in permissive or disabled mode. See the Install ceph-selinux section for detailed information on installing ceph-selinux and enabling SELinux.

Some Ceph package dependencies require versions that differ from the package versions from EPEL. Disable EPEL to ensure that you install the packages required for use with Ceph.

To disable epel, execute:

# yum-config-manager --disable epel

The above command will disable epel.repo in /etc/yum.repos.d/.

Create a working directory for the Ceph cluster configuration files and keys. Then, navigate to that directory. For example:

mkdir ~/ceph-config
cd ~/ceph-config

The ice-setup and ceph-deploy utilities must be executed within this working directory. See the Installation by ISO and Executing ceph-deploy sections for details.

To install the ceph-deploy utility and the Calamari server by using the Red Hat Content Delivery Network (CDN), execute the following command as root:

# yum install ceph-deploy calamari-server calamari-clients

To install the ceph-deploy utility and the Calamari server by using the ISO image, perform the following steps:

Once you have installed the Calamari package by using either the Content Delivery Network or the ISO image, initialize the Calamari monitoring and administration server:

# calamari-ctl initialize

As root, update existing cluster nodes that report to Calamari.

# salt '*' state.highstate

At this point, you should be able to access the Calamari web server using a web browser. Proceed to the Storage Cluster Quick Start.

Upgrading from Red Hat Enterprise Linux 6 to 7 requires either removing the PostgreSQL data or migrating it. Consider migrating this data if you have services other than Calamari using PostgreSQL. See the How to Migrate PostgreSQL Databases from RHEL6 to RHEL7 article for details.

If you do not have services other than Calamari using the PostgreSQL database, proceed as follows. In Red Hat Ceph Storage 1.2, Calamari stores only crash-recovery data in PostgreSQL. All that data will be rebuilt when first connected to the Ceph cluster. It is harmless to delete the data during this transition.

All the commands in the following steps must be run as root.

On the Calamari node perform these steps:

On nodes that were previously connected to Calamari and that you identified in step 7, perform the following steps:

On the Calamari node, perform the following steps: