Chapter 2. Requirements for Installing Red Hat Ceph Storage

Figure 2.1. Prerequisite Workflow

Ceph Installation Guide 459707 JCS

Before installing Red Hat Ceph Storage (RHCS), review the following requirements and prepare each Monitor, OSD, Metadata Server, and client nodes accordingly.

2.1. Prerequisites

  • Verify the hardware requirements are met. For details, see the Hardware Guide for Red Hat Ceph Storage 3.

2.2. Requirements Checklist for Installing Red Hat Ceph Storage

TaskRequiredSectionRecommendation

Verifying the operating system version

Yes

Section 2.3, “Operating System Requirements for Red Hat Ceph Storage”

If you have a node containing more than 12 OSDs, review the Increasing the PID count section for guidance to prevent potential daemon terminations.

Registering Ceph nodes

Yes

Section 2.5, “Registering Red Hat Ceph Storage Nodes to CDN and Attaching Subscriptions”

 

Enabling Ceph software repositories

Yes

Section 2.6, “Enabling the Red Hat Ceph Storage Repositories”

 

Using a RAID controller with OSD nodes

No

Section 2.7, “Considerations for Using a RAID Controller with OSD Nodes (optional)”

Enabling write-back caches on a RAID controller might result in increased small I/O write throughput for OSD nodes.

Configuring the network

Yes

Section 2.8, “Verifying the Network Configuration for Red Hat Ceph Storage”

At minimum, a public network is required. However, a private network for cluster communication is recommended.

Resolving short host names

Yes

Section 2.9, “Verifying Host Name Resolution for Red Hat Ceph Storage Nodes”

 

Configuring a firewall

No

Section 2.10, “Configuring a Firewall for Red Hat Ceph Storage (optional)”

A firewall can increase the level of trust for a network.

Creating an Ansible user

Yes

Section 2.11, “Creating an Ansible User with sudo Access”

Creating the Ansible user is required on all Ceph nodes.

Enabling password-less SSH

Yes

Section 2.12, “Enabling Password-less SSH for Ansible”

Required for Ansible.

2.3. Operating System Requirements for Red Hat Ceph Storage

Red Hat Ceph Storage 3 requires Red Hat Enterprise Linux 7 Server with a homogeneous version across the storage cluster. For example, Red Hat Enterprise Linux 7.4 running on AMD64 and Intel 64 architectures for all Ceph nodes.

Important

Red Hat does not support clusters with heterogeneous operating systems and versions.

Additional Resources

Return to requirements checklist

2.4. Increasing the PID count

If you have a node containing more than 12 OSDs, the default maximum number of threads can be insufficient, especially during recovery operations. As a consequence, some OSD daemons (ceph-osd) can terminate and fail to start again.

Procedure

Do the following steps on the OSD nodes in the storage cluster, as the root user.

  1. Verify the current pid_max settings:

    [root@osd ~]# cat /proc/sys/kernel/pid_max
  2. Increase the maximum PID number online:

    [root@osd ~]# sysctl -w kernel.pid_max=4194303
  3. Permanently set the new number, so it is persistent across a reboot. Open the /etc/sysctl.conf file for editing, and add the following line:

    kernel.pid_max = 4194303

2.5. Registering Red Hat Ceph Storage Nodes to CDN and Attaching Subscriptions

Register each Red Hat Ceph Storage (RHCS) node to the Content Delivery Network (CDN) and attach the appropriate subscription so that the node has access to software repositories. Each RHCS node must be able to access the full Red Hat Enterprise Linux 7 base content and content from the extras repository.

Note

For RHCS nodes that cannot access the Internet during the installation, provide the software content by using the Red Hat Satellite server or mounting a local Red Hat Enterprise Linux 7 Server ISO image and pointing the RHCS nodes to it. For additional details, contact the Red Hat Support.

For more information on registering Ceph nodes with the Red Hat Satellite server, see the How to Register Ceph with Satellite 6 and How to Register Ceph with Satellite 5 articles on the Red Hat Customer Portal.

Prerequisites

  • Valid Red Hat subscription
  • RHCS nodes can connect to the Internet.

Procedure

Do the following steps on all nodes in the storage cluster, and as the root user.

  1. Register the node and when prompted, enter your Red Hat Customer Portal credentials:

    # subscription-manager register
  2. Pull the latest subscription data from CDN:

    # subscription-manager refresh
  3. List all available subscriptions for Red Hat Ceph Storage and get its Pool ID:

    # subscription-manager list --available --all --matches="*Ceph*"
  4. Attach the subscriptions:

    subscription-manager attach --pool=$POOL_ID
    Replace
    • $POOL_ID with the Pool ID determined in the previous step.
  5. Disable the default software repositories, and enable the Red Hat Enterprise Linux 7 Server base and Red Hat Enterprise Linux 7 Server extras repositories:

    # subscription-manager repos --disable=*
    # subscription-manager repos --enable=rhel-7-server-rpms
    # subscription-manager repos --enable=rhel-7-server-extras-rpms
  6. Update to the latest packages:

    # yum update

Additional Resources

Return to requirements checklist

2.6. Enabling the Red Hat Ceph Storage Repositories

Before installing Red Hat Ceph Storage (RHCS), enable the appropriate software repositories on each node in the storage cluster. The Red Hat Content Delivery Network (CDN) contains all the RHCS software.

Prerequisites

  • Valid customer subscription.
  • RHCS nodes can connect to the Internet.
  • Register the cluster nodes with CDN.
  • Disable the EPEL software repository:

    [root@monitor ~]# yum-config-manager --disable epel

Procedure

Enable the software repository for the appropriate RHCS node type, as the root user.

  1. On Monitor nodes enable the Red Hat Ceph Storage 3 Monitor repository:

    [root@monitor ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-mon-rpms
  2. On OSD nodes enable the Red Hat Ceph Storage 3 OSD repository:

    [root@osd ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-osd-rpms
  3. On Ansible administration node, enable the Red Hat Ceph Storage 3 Tools repository:

    [root@admin ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms
  4. Optional. If you want to deploy Ceph File System, on the Metadata Server (MDS) nodes, enable the Red Hat Ceph Storage 3 Tools repository:

    [root@mds ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms
  5. Optional. If you want to deploy the Ceph Object Gateway, on the Ceph Object Gateway node, enable the Red Hat Ceph Storage 3 Tools repository:

    [root@gateway ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms
  6. Optional. If you want to deploy a Ceph client, on the client node, enable the Red Hat Ceph Storage 3 Tools repository:

    [root@client ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms

Additional Resources

Return to the requirements checklist

2.7. Considerations for Using a RAID Controller with OSD Nodes (optional)

If a RAID controller with 1-2 GB of cache is installed on a node, enabling write-back caches might result in increased small I/O write throughput, but the cache must be non-volatile.

Modern RAID controllers usually have super capacitors that provide enough power to drain volatile memory to non-volatile NAND memory during a power loss event. It is important to understand how a particular controller and firmware behave after power is restored.

Some RAID controllers require manual intervention. Hard drives typically advertise to the operating system whether their disk caches should be enabled or disabled by default. However, certain RAID controllers or some firmware do not provide such information, so verify that disk level caches are disabled to avoid file system corruption.

Create a single RAID 0 volume with write-back for each Ceph OSD data drive with write-back cache enabled.

If Serial Attached SCSI (SAS) or SATA connected Solid-state Drive (SSD) disks are also present on the RAID controller, then investigate whether the controller and firmware support pass-through mode. Enabling pass-through mode helps avoid caching logic, and generally results in much lower latency for fast media.

Return to requirements checklist

2.8. Verifying the Network Configuration for Red Hat Ceph Storage

All Red Hat Ceph Storage (RHCS) nodes require a public network. You must have a network interface card configured to a public network where Ceph clients can reach Ceph monitors and Ceph OSD nodes.

You might have a network interface card for a cluster network so that Ceph can conduct heart-beating, peering, replication, and recovery on a network separate from the public network.

Configure the network interface settings and ensure to make the changes persistent across a reboot.

Important

Red Hat does not recommend using a single network interface card for both a public and private network.

Prerequisites

  • Network interface card connected to the network.

Procedure

Do the following steps on all RHCS nodes in the storage cluster, as the root user.

  1. Verify the following settings are in the /etc/sysconfig/network-scripts/ifcfg-* file corresponding the public-facing network interface card:

    1. The BOOTPROTO parameter is set to none for static IP addresses.
    2. The ONBOOT parameter must be set to yes.

      If it is set to no, the Ceph storage cluster might fail to peer on reboot.

    3. If you intend to use IPv6 addressing, the IPv6 parameters, for example IPV6INIT must be set to yes except for the IPV6_FAILURE_FATAL parameter.

      Also, edit the Ceph configuration file, /etc/ceph/ceph.conf, to instruct Ceph to use IPv6, otherwise, Ceph will use IPv4.

Additional Resources

Return to requirements checklist

2.9. Verifying Host Name Resolution for Red Hat Ceph Storage Nodes

Red Hat Ceph Storage (RHCS) nodes must be able to resolve short host names, and fully qualified domain names (FQDN). Each RHCS node must be able to ping every other RHCS node in the storage cluster by its short host name.

Prerequisites

  • A search domain

Procedure

Do the following step on all nodes in the storage cluster.

  • Verify that the short host name of a node can be resolved:

    $ hostname -s

Return to requirements checklist

2.10. Configuring a Firewall for Red Hat Ceph Storage (optional)

Red Hat Ceph Storage (RHCS) uses the firewalld service.

The Monitor daemons use port 6789 for communication within the Ceph storage cluster.

On each Ceph OSD node, the OSD daemons use several ports in the range 6800-7300:

  • One for communicating with clients and monitors over the public network
  • One for sending data to other OSDs over a cluster network, if available; otherwise, over the public network
  • One for exchanging heartbeat packets over a cluster network, if available; otherwise, over the public network

The Ceph Manager (ceph-mgr) daemons use ports in range 6800-7300. Consider to colocate ceph-mgr daemons with Monitors on same nodes.

The Ceph Metadata Server nodes (ceph-mds) use port 6800.

The Ceph Object Gateway nodes use port 7480 by default. However, you can change the default port, for example to port 80.

To use the SSL/TLS service, open port 443.

Prerequisites

  • Network hardware is connected.

Procedure

Do the following steps on each node as specified, and as the root user.

  1. On all RHCS nodes, start the firewalld service. Enable it to run on boot, and ensure that it is running:

    # systemctl enable firewalld
    # systemctl start firewalld
    # systemctl status firewalld
  2. On all Monitor nodes, open port 6789 on the public network:

    [root@monitor ~]# firewall-cmd --zone=public --add-port=6789/tcp
    [root@monitor ~]# firewall-cmd --zone=public --add-port=6789/tcp --permanent

    To limit access based on the source address:

    firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
    source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \
    port="6789" accept"
    firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
    source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \
    port="6789" accept" --permanent
    Replace
    • $IP_ADDR with the network address of the Monitor node.
    • $NETMASK_PREFIX with the netmask in CIDR notation.

    Example

    [root@monitor ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
    source address="192.168.0.11/24" port protocol="tcp" \
    port="6789" accept"

    [root@monitor ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
    source address="192.168.0.11/24" port protocol="tcp" \
    port="6789" accept" --permanent
  3. On all OSD nodes, open ports 6800-7300 on the public network:

    [root@osd ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp
    [root@osd ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent

    If you have a separate cluster network, repeat the commands with the appropriate zone.

  4. On all Ceph Manager (ceph-mgr) nodes (usually the same nodes as Monitor ones), open ports 6800-7300 on the public network:

    [root@monitor ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp
    [root@monitor ~]# firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent

    If you have a separate cluster network, repeat the commands with the appropriate zone.

  5. On all Ceph Metadata Server (ceph-mds) nodes, open port 6800 on the public network:

    [root@monitor ~]# firewall-cmd --zone=public --add-port=6800/tcp
    [root@monitor ~]# firewall-cmd --zone=public --add-port=6800/tcp --permanent

    If you have a separate cluster network, repeat the commands with the appropriate zone.

  6. On all Ceph Object Gateway nodes, open the relevant port or ports on the public network.

    1. To open the default port 7480:

      [root@gateway ~]# firewall-cmd --zone=public --add-port=7480/tcp
      [root@gateway ~]# firewall-cmd --zone=public --add-port=7480/tcp --permanent

      To limit access based on the source address:

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \
      port="7480" accept"
      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \
      port="7480" accept" --permanent
      Replace
      • $IP_ADDR with the network address of the object gateway node.
      • $NETMASK_PREFIX with the netmask in CIDR notation.

      Example

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="7480" accept"

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="7480" accept" --permanent
    2. Optional. If you changed the default Ceph Object Gateway port, for example, to port 80, open this port:

      [root@gateway ~]# firewall-cmd --zone=public --add-port=80/tcp
      [root@gateway ~]# firewall-cmd --zone=public --add-port=80/tcp --permanent

      To limit access based on the source address, run the following commands:

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \
      port="80" accept"
      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \
      port="80" accept" --permanent
      Replace
      • $IP_ADDR with the network address of the object gateway node.
      • $NETMASK_PREFIX with the netmask in CIDR notation.

      Example

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="80" accept"

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="80" accept" --permanent
    3. Optional. To use SSL/TLS, open port 443:

      [root@gateway ~]# firewall-cmd --zone=public --add-port=443/tcp
      [root@gateway ~]# firewall-cmd --zone=public --add-port=443/tcp --permanent

      To limit access based on the source address, run the following commands:

      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \
      port="443" accept"
      firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="$IP_ADDR/$NETMASK_PREFIX" port protocol="tcp" \
      port="443" accept" --permanent
      Replace
      • $IP_ADDR with the network address of the object gateway node.
      • $NETMASK_PREFIX with the netmask in CIDR notation.

      Example

      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="443" accept"
      [root@gateway ~]# firewall-cmd --zone=public --add-rich-rule="rule family="ipv4" \
      source address="192.168.0.31/24" port protocol="tcp" \
      port="443" accept" --permanent

Additional Resources

Return to requirements checklist

2.11. Creating an Ansible User with sudo Access

Ansible must be able to log in to the Red Hat Ceph Storage (RHCS) nodes as a user that has root privileges to install software and create configuration files without prompting for a password.

Prerequisites

  • Having root or sudo access to all nodes in the storage cluster.

Procedure

Do the following steps on all nodes in the storage cluster, as the root user.

  1. Log in to a Ceph node as the root user:

    ssh root@$HOST_NAME
    Replace
    • $HOST_NAME with the host name of the Ceph node.

    Example

    # ssh root@mon01

    Enter the root password when prompted.

  2. Create a new Ansible user:

    useradd $USER_NAME
    Replace
    • $USER_NAME with the new user name for the Ansible user.

    Example

    # useradd admin

    Enter the new password twice when prompted.

    Important

    Do not use ceph as the user name. The ceph user name is reserved for the Ceph daemons. A uniform user name across the cluster can improve ease of use, but avoid using obvious user names, because intruders typically use them for brute-force attacks.

  3. Set a new password for this user:

    passwd $USER_NAME
    Replace
    • $USER_NAME with the new user name for the Ansible user.

    Example

    # passwd admin

    Enter the new password twice when prompted.

  4. Configure sudo access for the newly created user:

    cat << EOF >/etc/sudoers.d/$USER_NAME
    $USER_NAME ALL = (root) NOPASSWD:ALL
    EOF
    Replace
    • $USER_NAME with the new user name for the Ansible user.

    Example

    # cat << EOF >/etc/sudoers.d/admin
    admin ALL = (root) NOPASSWD:ALL
    EOF

  5. Assign the correct file permissions to the new file:

    chmod 0440 /etc/sudoers.d/$USER_NAME
    Replace
    • $USER_NAME with the new user name for the Ansible user.

    Example

    # chmod 0440 /etc/sudoers.d/admin

Additional Resources

  • The {customerportal}/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/[Adding a New User] section in the System Administrator’s Guide for Red Hat Enterprise Linux 7.

Return to the requirements checklist

2.12. Enabling Password-less SSH for Ansible

Generate an SSH key pair on the Ansible administration node and distribute the public key to each node in the storage cluster so that Ansible can access the nodes without being prompted for a password.

Prerequisites

Procedure

Do the following steps from the Ansible administration node, and as the Ansible user.

  1. Generate the SSH key pair, accept the default file name and leave the passphrase empty:

    [user@admin ~]$ ssh-keygen
  2. Copy the public key to all nodes in the storage cluster:

    ssh-copy-id $USER_NAME@$HOST_NAME
    Replace
    • $USER_NAME with the new user name for the Ansible user.
    • $HOST_NAME with the host name of the Ceph node.

    Example

    [user@admin ~]$ ssh-copy-id ceph-admin@ceph-mon01

  3. Create and edit the ~/.ssh/config file.

    Important

    By creating and editing the ~/.ssh/config file you do not have to specify the -u $USER_NAME option each time you execute the ansible-playbook command.

    1. Create the SSH config file:

      [user@admin ~]$ touch ~/.ssh/config
    2. Open the config file for editing. Set the Hostname and User options for each node in the storage cluster:

      Host node1
         Hostname $HOST_NAME
         User $USER_NAME
      Host node2
         Hostname $HOST_NAME
         User $USER_NAME
      ...
      Replace
      • $HOST_NAME with the host name of the Ceph node.
      • $USER_NAME with the new user name for the Ansible user.

      Example

      Host node1
         Hostname monitor
         User admin
      Host node2
         Hostname osd
         User admin
      Host node3
         Hostname gateway
         User admin

  4. Set the correct file permissions for the ~/.ssh/config file:

    [admin@admin ~]$ chmod 600 ~/.ssh/config

Additional Resources

  • The ssh_config(5) manual page
  • The {customerportal}/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/#ch-OpenSSH[OpenSSH] chapter in the System Administrator’s Guide for Red Hat Enterprise Linux 7

Return to requirements checklist