14.3. Preparing to Deploy Geo-replication

This section provides an overview of geo-replication deployment scenarios, lists prerequisites, and describes how to setup the environment for geo-replication session.

14.3.1. Exploring Geo-replication Deployment Scenarios

Geo-replication provides an incremental replication service over Local Area Networks (LANs), Wide Area Network (WANs), and the Internet. This section illustrates the most common deployment scenarios for geo-replication, including the following:
  • Geo-replication over LAN
  • Geo-replication over WAN
  • Geo-replication over the Internet
  • Multi-site cascading geo-replication
Geo-replication over LAN
Geo-replication over LAN
Geo-replication over WAN
Geo-replication over WAN
Geo-replication over Internet
Geo-replication over Internet
Multi-site cascading Geo-replication
Multi-site cascading geo-replication

14.3.2. Geo-replication Deployment Overview

Deploying geo-replication involves the following steps:
  1. Verify that your environment matches the minimum system requirements. See Section 14.3.3, “Prerequisites”.
  2. Determine the appropriate deployment scenario. See Section 14.3.1, “Exploring Geo-replication Deployment Scenarios”.
  3. Start geo-replication on the master and slave systems. See Section 14.4, “Starting Geo-replication”.

14.3.3. Prerequisites

The following are prerequisites for deploying geo-replication:
  • The master and slave volumes must be of same version of Red Hat Gluster Storage instances.
  • Slave node must not be a peer of the any of the nodes of the Master trusted storage pool.
  • Passwordless SSH access is required between one node of the master volume (the node from which the geo-replication create command will be executed), and one node of the slave volume (the node whose IP/hostname will be mentioned in the slave name when running the geo-replication create command).
    Create the public and private keys using ssh-keygen (without passphrase) on the master node:
    # ssh-keygen
    Copy the public key to the slave node using the following command:
    # ssh-copy-id -i identity_file root@slave_node_IPaddress/Hostname
    If you are setting up a non-root geo-replicaton session, then copy the public key to the respective user location.

    Note

    - Passwordless SSH access is required from the master node to slave node, whereas passwordless SSH access is not required from the slave node to master node.
    - ssh-copy-id command does not work if ssh authorized_keys file is configured in the custom location. You must copy the contents of .ssh/id_rsa.pub file from the Master and paste it to authorized_keys file in the custom location on the Slave node.
    A passwordless SSH connection is also required for gsyncd between every node in the master to every node in the slave. The gluster system:: execute gsec_create command creates secret-pem files on all the nodes in the master, and is used to implement the passwordless SSH connection. The push-pem option in the geo-replication create command pushes these keys to all the nodes in the slave.
    For more information on the gluster system::execute gsec_create and push-pem commands, see Section 14.3.4.1, “Setting Up your Environment for Geo-replication Session”.

14.3.4. Setting Up your Environment

You can set up your environment for a geo-replication session in the following ways:
Time Synchronization
Before configuring the geo-replication environment, ensure that the time on all the servers are synchronized.
  • All the servers' time must be uniform on bricks of a geo-replicated master volume. It is recommended to set up a NTP (Network Time Protocol) service to keep the bricks' time synchronized, and avoid out-of-time sync effects.
    For example: In a replicated volume where brick1 of the master has the time 12:20, and brick2 of the master has the time 12:10 with a 10 minute time lag, all the changes on brick2 between in this period may go unnoticed during synchronization of files with a Slave.

14.3.4.1. Setting Up your Environment for Geo-replication Session

Creating Geo-replication Sessions

  1. To create a common pem pub file, run the following command on the master node where the passwordless SSH connection is configured:
    # gluster system:: execute gsec_create
  2. Create the geo-replication session using the following command. The push-pem option is needed to perform the necessary pem-file setup on the slave nodes.
    # gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL create push-pem [force]
    For example:
    # gluster volume geo-replication Volume1 example.com::slave-vol create push-pem

    Note

    There must be passwordless SSH access between the node from which this command is run, and the slave host specified in the above command. This command performs the slave verification, which includes checking for a valid slave URL, valid slave volume, and available space on the slave. If the verification fails, you can use the force option which will ignore the failed verification and create a geo-replication session.
  3. Configure the meta-volume for geo-replication:
    # gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL config use_meta_volume true
    For example:
    # gluster volume geo-replication Volume1 example.com::slave-vol config use_meta_volume true
    For more information on configuring meta-volume, see Section 14.3.5, “Configuring a Meta-Volume”.
  4. Start the geo-replication by running the following command on the master node:
    For example,
    # gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL start [force]
  5. Verify the status of the created session by running the following command:
    # gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL status

14.3.4.2. Setting Up your Environment for a Secure Geo-replication Slave

Geo-replication supports access to Red Hat Gluster Storage slaves through SSH using an unprivileged account (user account with non-zero UID). This method is more secure and it reduces the master's capabilities over slave to the minimum. This feature relies on mountbroker, an internal service of glusterd which manages the mounts for unprivileged slave accounts. You must perform additional steps to configure glusterd with the appropriate mountbroker's access control directives. The following example demonstrates this process:
Perform the following steps on all the Slave nodes to setup an auxiliary glusterFS mount for the unprivileged account:
  1. Create a new group. For example, geogroup.
  2. Create a unprivileged account. For example, geoaccount. Add geoaccount as a member of geogroup group.
  3. As a root, create a new directory with permissions 0711 and with correct SELinux context. Ensure that the location where this directory is created is writeable only by root but geoaccount is able to access it.
    For example,
    # mkdir /var/mountbroker-root
    # chmod 0711 /var/mountbroker-root
    # semanage fcontext -a -e /home /var/mountbroker-root
    # restorecon -Rv /var/mountbroker-root
  4. Run the following commands in any one of the Slave node:
    # gluster system:: execute mountbroker opt mountbroker-root /var/mountbroker-root
    # gluster system:: execute mountbroker user geoaccount slavevol
    # gluster system:: execute mountbroker opt geo-replication-log-group geogroup
    # gluster system:: execute mountbroker opt rpc-auth-allow-insecure on
    See Section 2.4, “Storage Concepts” for information on glusterd.vol volume file of a Red Hat Gluster Storage volume.
    If the above commands fails, check if the glusterd.vol file is available at /etc/glusterfs/ directory. If not found, create a glusterd.vol file containing the default configuration and save it at /etc/glusterfs/ directory. Now re-run the above commands listed above to get all the required geo-replication options.
    The following is the sample glusterd.vol file along with default options:
    volume management
        type mgmt/glusterd
        option working-directory /var/lib/glusterd
        option transport-type socket,rdma
        option transport.socket.keepalive-time 10
        option transport.socket.keepalive-interval 2
        option transport.socket.read-fail-log off
        option rpc-auth-allow-insecure on
        
        option mountbroker-root /var/mountbroker-root 
        option mountbroker-geo-replication.geoaccount slavevol
        option geo-replication-log-group geogroup
    end-volume
    • If you have multiple slave volumes on Slave, repeat Step 2 for each of them and run the following commands to update the vol file:
      # gluster system:: execute mountbroker user geoaccount2 slavevol2
      # gluster system:: execute mountbroker user geoaccount3 slavevol3
      You can use gluster system:: execute mountbroker info command to view the configured mountbroker options.
    • You can add multiple slave volumes within the same account (geoaccount) by providing comma-separated list (without spaces) as the argument of mountbroker-geo-replication.geogroup. You can also have multiple options of the form mountbroker-geo-replication.*. It is recommended to use one service account per Master machine. For example, if there are multiple slave volumes on Slave for the master machines Master1, Master2, and Master3, then create a dedicated service user on Slave for them by repeating Step 2. for each (like geogroup1, geogroup2, and geogroup3), and then run the following commands to add the corresponding options to the volfile:
      # gluster system:: execute mountbroker user geoaccount1 slavevol11,slavevol12,slavevol13
      # gluster system:: execute mountbroker user geoaccount2 slavevol21,slavevol22
      # gluster system:: execute mountbroker user geoaccount3 slavevol31
  5. Restart glusterd service on all the Slave nodes.
    After you setup an auxiliary glusterFS mount for the unprivileged account on all the Slave nodes, perform the following steps to setup a non-root geo-replication session.:
  6. Setup a passwordless SSH from one of the master node to the user on one of the slave node.
    For example, to setup a passwordless SSH to the user geoaccount.
    # ssh-keygen
    # ssh-copy-id -i identity_file geoaccount@slave_node_IPaddress/Hostname
  7. Create a common pem pub file by running the following command on the master node where the passwordless SSH connection is configured to the user on the slave node:
    # gluster system:: execute gsec_create
  8. Create a geo-replication relationship between master and slave to the user by running the following command on the master node:
    For example,
    # gluster volume geo-replication MASTERVOL geoaccount@SLAVENODE::slavevol create push-pem
    If you have multiple slave volumes and/or multiple accounts, create a geo-replication session with that particular user and volume.
    For example,
    # gluster volume geo-replication MASTERVOL geoaccount2@SLAVENODE::slavevol2 create push-pem
  9. In the slavenode, which is used to create relationship, run /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh as a root with user name, master volume name, and slave volume names as the arguments.
    For example,
     # /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh geoaccount MASTERVOL SLAVEVOL_NAME 
  10. Configure the meta-volume for geo-replication:
    # gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL config use_meta_volume true
    For example:
    # gluster volume geo-replication Volume1 example.com::slave-vol config use_meta_volume true
    For more information on configuring meta-volume, see Section 14.3.5, “Configuring a Meta-Volume”.
  11. Start the geo-replication with slave user by running the following command on the master node:
    For example,
    # gluster volume geo-replication MASTERVOL geoaccount@SLAVENODE::slavevol start
  12. Verify the status of geo-replication session by running the following command on the master node:
    # gluster volume geo-replication MASTERVOL geoaccount@SLAVENODE::slavevol status
Deleting a mountbroker geo-replication options after deleting session

When mountbroker geo-replicaton session is deleted, use the following command to remove volumes per mountbroker user. If the volume to be removed is the last one for the mountbroker user, the user is also removed.

  • To delete a volumes per mountbroker user:
    # gluster system:: execute mountbroker volumedel geoaccount2 slavevol2
    You can delete multiple volumes per mountbroker user by providing comma-separated list (without spaces) as the argument of this command.
    # gluster system:: execute mountbroker volumedel geoaccount2 slavevol2,slavevol3

Important

If you have a secured geo-replication setup, you must ensure to prefix the unprivileged user account to the slave volume in the command. For example, to execute a geo-replication status command, run the following:
# gluster volume geo-replication MASTERVOL geoaccount@SLAVENODE::slavevol status
In this command, geoaccount is the name of the unprivileged user account.

14.3.5. Configuring a Meta-Volume

For effective handling of node fail-overs in Master volume, geo-replication requires a shared storage to be available across all nodes of the cluster. Hence, you must ensure that a gluster volume named gluster_shared_storage is created in the cluster, and is mounted at /var/run/gluster/shared_storage on all the nodes in the cluster. For more information on setting up shared storage volume, see Section 10.8, “Setting up Shared Storage Volume”.
  • Configure the meta-volume for geo-replication:
    # gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL config use_meta_volume true
    For example:
    # gluster volume geo-replication Volume1 example.com::slave-vol config use_meta_volume true