Red Hat Training

A Red Hat training course is available for Red Hat Ceph Storage

Chapter 6. HAProxy/keepalived Configuration

The Ceph Object Gateway allows you to assign many instances of the object gateway to a single zone so that you can scale out as load increases (i.e., the same region and zone; however, you do not need a federated architecture to use HAProxy/keepalived). Since each object gateway instance has its own IP address, you can use HAProxy and keepalived to balance the load across Ceph Object Gateway servers.

Another use case for HAProxy and keepalived is to terminate HTTPS at the HAProxy server. Red Hat Ceph Storage (RHCS) 1.3.x uses Civetweb, and the implementation in RHCS 1.3.x doesn’t support HTTPS. You can use an HAProxy server to terminate HTTPS at the HAProxy server and use HTTP between the HAProxy server and the Civetweb gateway instances.

6.1. Prerequisites

To set up an HA Proxy with the Ceph Object Gateway, you must have:

  • A running Ceph cluster
  • At least two Ceph Object Gateway servers within the same zone configured to run on port 80. If you follow the simple installation procedure, the gateway instances are in the same region and zone by default. If you are using a federated architecture, ensure that the instances are in the same region and zone; and,
  • At least two servers for HAProxy and keepalived.
Note

This document assumes that you have at least two Ceph Object Gateway servers running, and that you get a valid response from each of them when running test scripts over port 80.

For a detailed discussion of HAProxy and keepalived, see Load Balancer Administration.

6.2. Preparing HAProxy Nodes

The following setup assumes two HAProxy nodes named haproxy and haproxy2, and two Ceph Object Gateway servers named rgw1 and rgw2. You may use any naming convention you prefer. Perform the following procedure on your at least two HAProxy nodes:

  1. Install RHEL 7.x.
  2. Register the nodes.

    sudo subscription-manager register
  3. Enable the RHEL server repository.

    sudo subscription-manager repos --enable=rhel-7-server-rpms
  4. Update the server.

    sudo yum update -y
  5. Install admin tools (e.g., wget, vim, etc.) as needed.
  6. Open port 80.

    sudo firewall-cmd --zone=public --add-port 80/tcp --permanent
    sudo firewall-cmd --reload
  7. For HTTPS, open port 443.

    sudo firewall-cmd --zone=public --add-port 443/tcp --permanent
    sudo firewall-cmd --reload

6.3. Install and Configure keepalived

Perform the following procedure on your at least two HAProxy nodes:

  1. Install keepalived.

    sudo yum install -y keepalived
  2. Configure keepalived.

    sudo vim /etc/keepalived/keepalived.conf

    In the following configuration, there is a script to check the haproxy processes. The instance uses eth0 as the network interface and configures haproxy as the master server and haproxy2 as the backup server. It also assigns a virtual IP address (i.e., 192.168.0.100).

    vrrp_script chk_haproxy {
      script "killall -0 haproxy" # check the haproxy process
      interval 2 # every 2 seconds
      weight 2 # add 2 points if OK
    }
    
    vrrp_instance VI_1 {
      interface eth0 # interface to monitor
      state MASTER # MASTER on haproxy, BACKUP on haproxy2
      virtual_router_id 51
      priority 101 # 101 on haproxy, 100 on haproxy2
      virtual_ipaddress {
        192.168.0.100 # virtual ip address
      }
      track_script {
        chk_haproxy
      }
    }

    For a detailed discussion of configuring keepalived, refer to Initial Load Balancer Configuration with Keepalived.

  3. Enable/start keepalived.

    sudo systemctl enable keepalived
    sudo systemctl start keepalived

6.4. Install and Configure HAProxy

Perform the following procedure on your at least two HAProxy nodes:

  1. Install haproxy.

    sudo yum install haproxy
  2. Configure haproxy for SELinux and HTTP.

    sudo vim /etc/firewalld/services/haproxy-http.xml

    Add the following lines:

    <?xml version="1.0" encoding="utf-8"?>
    <service>
    <short>HAProxy-HTTP</short>
    <description>HAProxy load-balancer</description>
    <port protocol="tcp" port="80"/>
    </service>

    As root, assign the correct SELinux context and file permissions to the haproxy-http.xml file.

    # cd /etc/firewalld/services
    # restorecon haproxy-http.xml
    # chmod 640 haproxy-http.xml
  3. If you intend to use HTTPS, configure haproxy for SELinux and HTTPS.

    sudo vim /etc/firewalld/services/haproxy-https.xml

    Add the following lines:

    <?xml version="1.0" encoding="utf-8"?>
    <service>
    <short>HAProxy-HTTPS</short>
    <description>HAProxy load-balancer</description>
    <port protocol="tcp" port="443"/>
    </service>

    As root, assign the correct SELinux context and file permissions to the haproxy-https.xml file.

    # cd /etc/firewalld/services
    # restorecon haproxy-https.xml
    # chmod 640 haproxy-https.xml
  4. If you intend to use HTTPS, generate keys for SSL. If you do not have a certificate, you may use a self-signed certificate. To generate a key, refer to generating a key.

    Finally, put the certificate and key into a PEM file.

    cat example.com.crt example.com.key > example.com.pem
    sudo cp example.com.pem /etc/ssl/private/
  5. Configure haproxy.

    sudo vim /etc/haproxy/haproxy.cfg

    The global and defaults may remain unchanged. After the defaults section, you will need to configure frontend and backend sections. For example:

    frontend http_web *:80
        mode http
        default_backend rgw
    
    frontend rgw­-https
      bind *:443 ssl crt /etc/ssl/private/example.com.pem
      default_backend rgw
    
    backend rgw
        balance roundrobin
        mode http
        server  rgw1 10.0.0.71:80 check
        server  rgw2 10.0.0.80:80 check

    For a detailed discussion of HAProxy configuration, refer to HAProxy Configuration.

  6. Enable/start haproxy

    sudo systemctl enable haproxy
    sudo systemctl start haproxy

6.5. Test Your HAProxy Configuration

On your HAProxy nodes, check to ensure the virtual IP address from your keepalived configuration appears.

ip addr show

On your calamari node, see if you can reach the gateway nodes via the load balancer configuration. For example:

wget haproxy

This should return the same result as:

wget rgw1

If it returns an index.html file with the following contents:

<?xml version="1.0" encoding="UTF-8"?>
	<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
		<Owner>
			<ID>anonymous</ID>
			<DisplayName></DisplayName>
		</Owner>
		<Buckets>
		</Buckets>
	</ListAllMyBucketsResult>

Then, your configuration is working properly.