Red Hat Training

A Red Hat training course is available for Red Hat Ceph Storage

Chapter 1. Overview

Designed for cloud infrastructures and web-scale object storage, Red Hat® Ceph Storage is a massively scalable, open, software-defined storage platform that combines the most stable version of Ceph with a Ceph management platform, deployment tools and support services. Providing the tools to flexibly and cost-effectively manage petabyte-scale data deployments in the enterprise, Red Hat Ceph Storage manages cloud data so enterprises can focus on managing their businesses.

This document provides procedures for installing Red Hat Ceph Storage v1.3 for x86_64 architecture on Ubuntu Trusty 14.04.

Red Hat® Ceph Storage clusters consist of the following types of nodes:

  • Administration node: We expect that you will have a dedicated administration node that will host the Calamari monitoring and administration server, your cluster’s configuration files and keys, and optionally local repositories for installing Ceph on nodes that cannot access the internet for security reasons.
  • Monitor nodes: Ceph can run with one monitor; however, for high availability, we expect that you will have at least three monitor nodes to ensure high availability in a production cluster.
  • OSD nodes: Ceph can run with very few OSDs (3, by default), but production clusters realize better performance beginning at modest scales (e.g., 50 OSDs). Ideally, a Ceph cluster will have multiple OSD nodes, allowing you to create a CRUSH map to isolate failure domains.

For minimum recommended hardware, see the Hardware Guide.

1.1. Prerequisites

Before installing Red Hat Ceph Storage, review the following prerequisites first and prepare the cluster nodes.

1.1.1. Operating System

Red Hat Ceph Storage 1.3 and later requires Ubuntu 16.04 with a homogeneous version running on AMD64 and Intel 64 architectures for all Ceph nodes, including the Red Hat Ceph Storage node.

Important

Red Hat does not support clusters with heterogeneous operating systems and versions.

1.1.2. Kernel

Red Hat Ceph Storage v1.3 is supported on Ubuntu 14.04 configurations including the original 14.04 LTS kernel (3.13). Configurations including the latest HWE kernel (3.19) are also supported. We recommend customers default their choice to the LTS kernel where the HWE is not required by their hardware in light of its longer support lifecycle.

1.1.3. DNS Name Resolution

Ceph nodes must be able to resolve short host names, not just fully qualified domain names. Set up a default search domain to resolve short host names. To retrieve a Ceph node’s short host name, execute: 

hostname -s

Each Ceph node MUST be able to ping every other Ceph node in the cluster by its short host name.

1.1.4. Network Interface Cards

All Ceph clusters require a public network. You MUST have a network interface card configured to a public network where Ceph clients can reach Ceph Monitors and Ceph OSDs. You SHOULD have a network interface card for a cluster network so that Ceph can conduct heart-beating, peering, replication and recovery on a network separate from the public network.

We DO NOT RECOMMEND using a single NIC for both a public and private network.

1.1.5. Firewall

You MUST adjust your firewall settings on the Calamari node to allow inbound requests on port 80 so that clients in your network can access the Calamari web user interface.

Calamari also communicates with Ceph nodes via ports 2003, 4505 and 4506. You MUST open ports 80, 2003, and 4505-4506 on your Calamari node. 

sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 80 -j ACCEPT
sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 2003 -j ACCEPT
sudo iptables -I INPUT 1 -i <iface> -m multiport -p tcp -s <ip-address>/<netmask> --dports 4505:4506 -j ACCEPT

You MUST open port 6789 on your public network on ALL Ceph monitor nodes. 

sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 6789 -j ACCEPT

Finally, you MUST also open ports for OSD traffic (e.g., 6800-7300). Each OSD on each Ceph node needs three ports: one for talking to clients and monitors (public network); one for sending data to other OSDs (cluster network, if available; otherwise, public network); and, one for heartbeating (cluster network, if available; otherwise, public network). The OSDs will bind to the next available port if they are restarted in certain scenarios, so open the entire port range 6800 through 7300 for completeness. 

sudo iptables -I INPUT 1 -i <iface> -m multiport -p tcp -s <ip-address>/<netmask> --dports 6800:7300 -j ACCEPT

Once you have finished configuring iptables, ensure that you make the changes persistent on each node so that they will be in effect when your nodes reboot.

Execute: 

sudo apt-get install iptables-persistent

A terminal UI will open up. Select yes for the prompts to save current IPv4 iptables rules to /etc/iptables/rules.v4 and current IPv6 iptables rules to /etc/iptables/rules.v6.

The IPv4 iptables rules that you set in the earlier steps will be loaded in /etc/iptables/rules.v4 and will be persistent across reboots.

If you add a new IPv4 iptables rule after installing iptables-persistent you will have to add it to the rule file. In such case, execute the following as a root user: 

iptables-save > /etc/iptables/rules.v4

1.1.6. Network Time Protocol

You MUST install Network Time Protocol (NTP) on all Ceph monitor hosts and ensure that monitor hosts are NTP peers. You SHOULD consider installing NTP on Ceph OSD nodes, but it is not required. NTP helps preempt issues that arise from clock drift.

  1. Install NTP 

    sudo apt-get install ntp

  2. Start the NTP service and ensure it’s running. 

    sudo service ntp start
sudo service ntp status

  3. Ensure that NTP is synchronizing Ceph monitor node clocks properly. 

    ntpq -p

1.1.7. Install SSH Server

For ALL Ceph Nodes perform the following steps:

  1. Install an SSH server (if necessary) on each Ceph Node: 

    sudo apt-get install openssh-server
  2. Ensure the SSH server is running on ALL Ceph Nodes.

1.1.8. Create a Ceph Deploy User

The ceph-deploy utility must login to a Ceph node as a user that has passwordless sudo privileges, because it needs to install software and configuration files without prompting for passwords.

ceph-deploy supports a --username option so you can specify any user that has password-less sudo (including root, although this is NOT recommended). To use ceph-deploy --username <username>, the user you specify must have password-less SSH access to the Ceph node, because ceph-deploy will not prompt you for a password.

We recommend creating a Ceph Deploy user on ALL Ceph nodes in the cluster. Please do NOT use "ceph" as the user name. A uniform user name across the cluster may improve ease of use (not required), but you should avoid obvious user names, because hackers typically use them with brute force hacks (for example, root, admin, or <productname>). The following procedure, substituting <username> for the user name you define, describes how to create a Ceph Deploy user with passwordless sudo on a Ceph node.

Note

In a future Ceph release, the "ceph" user name will be reserved for the Ceph daemons. If the "ceph" user already exists on the Ceph nodes, removing this user must be done before attempting an upgrade to future Ceph releases.

  1. Create a Ceph Deploy user on each Ceph Node. 

    ssh root@<hostname>
adduser <username>

    Replace <hostname> with the hostname of your Ceph node.

  2. For the Ceph Deploy user you added to each Ceph node, ensure that the user has sudo privileges and disable requiretty. 

    cat << EOF >/etc/sudoers.d/<username>
<username> ALL = (root) NOPASSWD:ALL
Defaults:<username> !requiretty
EOF

    Ensure the correct file permissions. 

    # chmod 0440 /etc/sudoers.d/<username>

1.1.9. Enable Password-less SSH

Since ceph-deploy will not prompt for a password, you must generate SSH keys on the admin node and distribute the public key to each Ceph node. ceph-deploy will attempt to generate the SSH keys for initial monitors.

  1. Generate the SSH keys, but do not use sudo or the root user. Leave the passphrase empty: 

    ssh-keygen

Generating public/private key pair.
Enter file in which to save the key (/ceph-admin/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /ceph-admin/.ssh/id_rsa.
Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub.

  2. Copy the key to each Ceph Node, replacing <username> with the user name you created with Create a Ceph Deploy User. 

    ssh-copy-id <username>@node1
ssh-copy-id <username>@node2
ssh-copy-id <username>@node3

  3. (Recommended) Modify the ~/.ssh/config file of your ceph-deploy admin node so that ceph-deploy can log in to Ceph nodes as the user you created without requiring you to specify --username <username> each time you execute ceph-deploy. This has the added benefit of streamlining ssh and scp usage. Replace <username> with the user name you created: 

    Host node1
   Hostname node1
   User <username>
Host node2
   Hostname node2
   User <username>
Host node3
   Hostname node3
   User <username>

    After editing the ~/.ssh/config file on the admin node, execute the following to ensure the permissions are correct: 

    chmod 600 ~/.ssh/config

1.1.10. Configuring RAID Controllers

If a RAID controller with 1-2 GB of cache is installed on a host, then enabling write-back caches might result in increased small I/O write throughput. In order for this to be done safely, the cache must be non-volatile.

Modern RAID controllers usually have super capacitors that provide enough power to drain volatile memory to non-volatile NAND memory during a power loss event. It is important to understand how a particular controller and firmware behave after power is restored.

Some of them might require manual intervention. Hard drives typically advertise to the operating system whether their disk caches should be enabled or disabled by default. However, certain RAID controllers or some firmware might not provide such information, so verify that disk level caches are disabled to avoid file system corruption.

Create a single RAID 0 volume with write-back for each OSD data drive with write-back cache enabled.

If SAS or SATA connected SSDs are also present on the controller, it is worth investigating whether your controller and firmware support passthrough mode. This will avoid the caching logic, and generally result in much lower latencies for fast media.

1.1.11. Adjust PID Count

Hosts with high numbers of OSDs (e.g., > 20) may spawn a lot of threads, especially during recovery and re-balancing. Ubuntu 14.04 kernels (3.13 and 3.16) default to a relatively small maximum number of threads (e.g., 32768). Check your default settings to see if they are suitable. 

cat /proc/sys/kernel/pid_max

Consider setting kernel.pid_max to a higher number of threads. The theoretical maximum is 4,194,303 threads. For example, you could add the following to the /etc/sysctl.conf file to set it to the maximum: 

kernel.pid_max = 4194303

To see the changes you made without a reboot, execute: 

# sysctl -p

To verify the changes, execute: 

# sysctl -a | grep kernel.pid_max

1.1.12. Enable Ceph Repositories

Starting from v1.3.1, Red Hat Ceph Storage supports two installation methods for Ubuntu Trusty:

  • Online Repositories: For Ceph Storage clusters with Ceph nodes that can connect directly to the internet, you can use online repositories for Calamari, Installer, Monitors, OSDs and RGW from https://rhcs.download.redhat.com/ubuntu. You will need your Customer Name and Customer Password received from https://rhcs.download.redhat.com to be able to use the repos.

    Important

    Please contact your account manager to obtain credentials for https://rhcs.download.redhat.com.

  • Local Repository: For Ceph Storage clusters where security measures preclude nodes from accessing the internet, you may install Red Hat Ceph Storage v1.3 from a single software build delivered as an ISO with the ice_setup package, which installs the ice_setup program. When you execute the ice_setup program, it will install local repositories, the Calamari monitoring and administration server and the Ceph installation scripts, including a hidden .cephdeploy.conf file pointing ceph-deploy to the local repositories.

For installation via online repositories, configure the Installer repository on your administration node in order to install ceph-deploy, then use ceph-deploy to configure all other RHCS repositories.

  1. For your administration node, set the Installer (ceph-deploy) repository then use ceph-deploy to enable the Calamari and Tools repositories. 

    sudo bash -c 'umask 0077; echo deb https://customername:customerpasswd@rhcs.download.redhat.com/ubuntu/1.3-updates/Installer $(lsb_release -sc) main | tee /etc/apt/sources.list.d/Installer.list'
sudo bash -c 'wget -O - https://www.redhat.com/security/fd431d51.txt | apt-key add -'
sudo apt-get update
sudo apt-get install ceph-deploy
ceph-deploy repo --repo-url 'https://customername:customerpasswd@rhcs.download.redhat.com/ubuntu/1.3-updates/Calamari' Calamari `hostname -f`
ceph-deploy repo --repo-url 'https://customername:customerpasswd@rhcs.download.redhat.com/ubuntu/1.3-updates/Tools' Tools `hostname -f`
sudo apt-get update

  2. For Ceph monitor nodes, install the MON repository by running the following from the Admin node: 

    ceph-deploy repo --repo-url 'https://customername:customerpasswd@rhcs.download.redhat.com/ubuntu/1.3-updates/MON' --gpg-url https://www.redhat.com/security/fd431d51.txt ceph-mon <MONHOST1> [<MONHOST2> [<MONHOST3> ...]]

  3. Update monitor nodes: 

    sudo apt-get update

  4. For Ceph OSD nodes, install the OSD repository by running the following from the Admin node: 

    ceph-deploy repo --repo-url 'https://customername:customerpasswd@rhcs.download.redhat.com/ubuntu/1.3-updates/OSD' --gpg-url https://www.redhat.com/security/fd431d51.txt ceph-osd <OSDHOST1> [<OSDHOST2> [<OSDHOST3> ...]]

  5. Update OSD nodes: 

    sudo apt-get update

  6. For gateway nodes, install the Tools repository by running the following from the Admin node: 

    ceph-deploy repo --repo-url 'https://customername:customerpasswd@rhcs.download.redhat.com/ubuntu/1.3-updates/Tools' --gpg-url https://www.redhat.com/security/fd431d51.txt Tools <RGWNODE1> [RGWNODE2 [RGWNODE3 ...]]

  7. Update gateway nodes: 

    sudo apt-get update

1.2. Setting Up the Administration Server

You are supposed to have a dedicated administration node that hosts the Calamari monitoring and administration server.

The administration server hardware requirements vary with the size of the cluster. A minimum recommended hardware configuration for a Calamari server includes:

  • at least 4 GB of RAM,
  • a dual core CPU on AMD64 or Intel 64 architecture,
  • enough network throughput to handle communication with Ceph hosts.

The hardware requirements scale linearly with the number of Ceph servers, so if you intend to run a fairly large cluster, ensure that you have enough RAM, processing power, and network throughput for the administration node.

Red Hat Ceph Storage uses an administration server for the Calamari monitoring and administration server, and the cluster’s Ceph configuration file and authentication keys. If you install Red Hat Ceph Storage from an ISO image and require local repositories for the Ceph packages, the administration server will also contain the Red Hat Ceph Storage repositories.

Note

To use the HTTPS protocol with Calamari, set up the Apache web server first. See the Setting Up an SSL Server chapter in the System Administrator’s Guide for Red Hat Enterprise Linux 7.

diag 2915299f42b3a889c9796f99ded88611

1.2.1. Create a Working Directory

Create a working directory for the Ceph cluster configuration files and keys. Then, navigate to that directory. For example: 

mkdir ~/ceph-config
cd ~/ceph-config

The ice-setup and ceph-deploy utilities must be executed within this working directory. See the Installation by ISO and Executing ceph-deploy sections for details.

1.2.2. Installation by CDN

If you have correctly set the online repositories for Calamari and Installer in section Set Online Ceph Repositories, execute the following to install Calamari: 

sudo apt-get install calamari-server calamari-clients

1.2.3. Installation by ISO

To install the ceph-deploy utility and the Calamari server by using the ISO image, execute following steps:

  1. Visit the Red Hat Ceph Storage for Ubuntu page on the Customer Portal to obtain the Red Hat Ceph Storage installation ISO image files.

  2. Using sudo, mount the downloaded ISO image to the /mnt/ directory, for example: 

    $ sudo mount <path_to_iso>/rhceph-1.3.2-ubuntu-x86_64-dvd.iso /mnt

  3. Using sudo, install the ice_setup program: 

    $ sudo dpkg -i /mnt/ice-setup_*.deb
    Note

    If you receive an error that the python-pkg-resources package is missing, run the sudo apt-get -f install command to install the missing python-pkg-resources package.

  4. Navigate to the working directory that you created in the Create a Working Directory section: 

    $ cd ~/ceph-config

  5. Within this directory, run ice_setup using sudo: 

    $ sudo ice_setup -d /mnt

    Follow the instructions in the interactive shell.

    The ice_setup program performs the following operations:

    • creates a local repository for the ceph-deploy and calamari packages
    • installs the Calamari server packages on the administration node
    • installs the ceph-deploy package on the administration node
    • creates the /opt/ICE/ and /opt/calamari/ directories
    • writes the .cephdeploy.conf file to the /root/ directory and to the current working directory, for example, ~/ceph-config

1.2.4. Initialize Calamari

Once you have installed the Calamari package by using either the Content Delivery Network or the ISO image, initialize the Calamari monitoring and administration server: 

# calamari-ctl initialize

As root, update existing cluster nodes that report to Calamari. 

# salt '*' state.highstate

At this point, you should be able to access the Calamari web server using a web browser. Proceed to the Storage Cluster Quick Start.

Note

The initialization program implies that you can only execute ceph-deploy when pointing to a remote site. You may also direct ceph-deploy to your Calamari administration node for example, ceph-deploy admin <admin-hostname>. You can also use the Calamari administration node to run a Ceph daemon, although this is not recommended.