Red Hat Ceph Storage v1.2.3 and beyond requires a homogeneous operating system distribution and version (e.g., RHEL 6, RHEL7) on x86_64 architecture for all Ceph nodes, including the Calamari cluster. We do not support clusters with heterogeneous operating systems and versions.

Ceph nodes must be able to resolve short host names, not just fully qualified domain names. Set up a default search domain to resolve short host names. To retrieve a Ceph node’s short host name, execute:

hostname -s

Each Ceph node MUST be able to ping every other Ceph node in the cluster by its short host name.

All Ceph clusters require a public network. You MUST have a network interface card configured to a public network where Ceph clients can reach Ceph Monitors and Ceph OSDs. You SHOULD have a network interface card for a cluster network so that Ceph can conduct heart-beating, peering, replication and recovery on a network separate from the public network.

We DO NOT RECOMMEND using a single NIC for both a public and private network.

Ensure that you configure your network interfaces and make them persistent so that the settings are identical on reboot. For example:

Navigate to /etc/sysconfig/network-scripts and ensure that the ifcfg-<iface> settings for your public and cluster interfaces (assuming you will use a cluster network too [RECOMMENDED]) are properly configured.

For details on configuring network interface scripts for RHEL 6, see Ethernet Interfaces.

For details on configuring network interface scripts for RHEL 7, see Configuring a Network Interface Using ifcfg Files.

The default firewall configuration for RHEL is fairly strict. You MUST adjust your firewall settings on the Calamari node to allow inbound requests on port 80 so that clients in your network can access the Calamari web user interface.

Calamari also communicates with Ceph nodes via ports 2003, 4505 and 4506. You MUST open ports 80, 2003, and 4505-4506 on your Calamari node.

sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 80 -j ACCEPT
sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 2003 -j ACCEPT
sudo iptables -I INPUT 1 -i <iface> -m multiport -p tcp -s <ip-address>/<netmask> --dports 4505:4506 -j ACCEPT

You MUST open port 6789 on your public network on ALL Ceph monitor nodes.

sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 6789 -j ACCEPT

Finally, you MUST also open ports for OSD traffic (e.g., 6800-7100). Each OSD on each Ceph node needs three ports: one for talking to clients and monitors (public network); one for sending data to other OSDs (cluster network, if available; otherwise, public network); and, one for heartbeating (cluster network, if available; otherwise, public network). For example, if you have 4 OSDs, open 4 x 3 ports (12).

sudo iptables -I INPUT 1 -i <iface> -m multiport -p tcp -s <ip-address>/<netmask> --dports 6800:6811 -j ACCEPT

Once you have finished configuring iptables, ensure that you make the changes persistent on each node so that they will be in effect when your nodes reboot. For example:

/sbin/service iptables save

The default firewall configuration for RHEL is fairly strict. You MUST adjust your firewall settings on the Calamari node to allow inbound requests on port 80 so that clients in your network can access the Calamari web user interface.

Calamari also communicates with Ceph nodes via ports 2003, 4505 and 4506. For firewalld, add port 80, 4505, 4506 and 2003 to the public zone and ensure that you make the setting permanent so that it is enabled on reboot.

You MUST open ports 80, 2003, and 4505-4506 on your Calamari node.

sudo firewall-cmd --zone=public --add-port=80/tcp --permanent
sudo firewall-cmd --zone=public --add-port=2003/tcp --permanent
sudo firewall-cmd --zone=public --add-port=4505-4506/tcp --permanent

You MUST open port 6789 on your public network on ALL Ceph monitor nodes.

sudo firewall-cmd --zone=public --add-port=6789/tcp --permanent

Finally, you MUST also open ports for OSD traffic (e.g., 6800-7100). Each OSD on each Ceph node needs three ports: one for talking to clients and monitors (public network); one for sending data to other OSDs (cluster network, if available; otherwise, public network); and, one for heartbeating (cluster network, if available; otherwise, public network). For example, if you have 4 OSDs, open 4 x 3 ports (12).

sudo firewall-cmd --zone=public --add-port=6800-6811/tcp --permanent

Once the foregoing procedures are complete, reload the firewall configuration to ensure that the changes take effect.

sudo firewall-cmd --reload

For additional details on firewalld, see Using Firewalls.

You MUST install Network Time Protocol (NTP) on all Ceph monitor hosts and ensure that monitor hosts are NTP peers. You SHOULD consider installing NTP on Ceph OSD nodes, but it is not required. NTP helps preempt issues that arise from clock drift.

For additional details on NTP for RHEL 6, see Network Time Protocol Setup.

For additional details on NTP for RHEL 7, see Configuring NTP Using ntpd.

For ALL Ceph Nodes perform the following steps:

For additional details on OpenSSH for RHEL 6, see OpenSSH.

For additional details on OpenSSH for RHEL 7, see OpenSSH.

The ceph-deploy utility must login to a Ceph node as a user that has passwordless sudo privileges, because it needs to install software and configuration files without prompting for passwords.

ceph-deploy supports a --username option so you can specify any user that has password-less sudo (including root, although this is NOT recommended). To use ceph-deploy --username <username>, the user you specify must have password-less SSH access to the Ceph node, because ceph-deploy will not prompt you for a password.

We recommend creating a Ceph user on ALL Ceph nodes in the cluster. A uniform user name across the cluster may improve ease of use (not required), but you should avoid obvious user names, because hackers typically use them with brute force hacks (e.g., root, admin, <productname>). The following procedure, substituting <username> for the user name you define, describes how to create a user with passwordless sudo on a node called ceph-server.

Since ceph-deploy will not prompt for a password, you must generate SSH keys on the admin node and distribute the public key to each Ceph node. ceph-deploy will attempt to generate the SSH keys for initial monitors.

For users that will run Ceph administrator commands on large clusters (e.g., 1024 OSDs or more), create an /etc/security/limits.d/50-ceph.conf file on your admin node with the following contents:

<username>       soft    nproc     unlimited

Replace <username> with the name of the non-root account that you will use to run Ceph administrator commands.

If you have RAID (not recommended), configure your RAID controllers to RAID 0 (JBOD).

Hosts with high numbers of OSDs (e.g., > 20) may spawn a lot of threads, especially during recovery and re-balancing. Many Linux kernels default to a relatively small maximum number of threads (e.g., 32768). Check your default settings to see if they are suitable.

cat /proc/sys/kernel/pid_max

Consider setting kernel.pid_max to a higher number of threads. The theoretical maximum is 4,194,303 threads. For example, you could add the following to the /etc/sysctl.conf file to set it to the maximum:

kernel.pid_max = 4194303

To see the changes you made without a reboot, execute:

sudo sysctl -p

To verify the changes, execute:

sudo sysctl -a | grep kernel.pid_max

Ceph aims for data safety, which means that when the Ceph Client receives notice that data was written to a storage drive, that data was actually written to the storage drive (i.e., it’s not in a journal or drive cache, but yet to be written). On RHEL 6, disable the write cache if the journal is on a raw drive.

Use hdparm to disable write caching on OSD storage drives:

sudo hdparm -W 0 /<path-to>/<disk> 0

RHEL 7 has a newer kernel that handles this automatically.

SELinux is set to Enforcing by default. For Ceph Storage v1.2.3, set SELinux to Permissive or disable it entirely and ensure that your installation and cluster is working properly. To set SELinux to Permissive, execute the following:

sudo setenforce 0

To configure SELinux persistently, modify the configuration file at /etc/selinux/config.

Some Ceph package dependencies require versions that differ from the package versions from EPEL. Disable EPEL to ensure that you install the packages required for use with Ceph.

Red Hat Ceph Storage for RHEL 6 requires xfsprogs for OSD nodes.

As part of the Red Hat Ceph Storage product, Red Hat includes an entitlement to the Scalable File System set of packages for RHEL6, which includes xfsprogs. On each Ceph Node, using sudo, enable the Scalable File System repo and install xfsprogs:

sudo subscription-manager repos --enable=rhel-scalefs-for-rhel-6-server-rpms
sudo yum install xfsprogs