Chapter 6. Configuring Red Hat High Availability Cluster on Google Cloud Platform

This chapter includes information and procedures for configuring a Red Hat High Availability (HA) cluster on Google Cloud Platform (GCP) using Google Compute Engine (GCE) virtual machine (VM) instances as cluster nodes.

The chapter includes prerequisite procedures for setting up your environment for GCP. Once you have set up your environment, you can create and configure VM instances.

The chapter also includes procedures specific to the creation of HA clusters, which transform individual nodes into a cluster of HA nodes on GCP. These include procedures for installing the High Availability packages and agents on each cluster node, configuring fencing, and installing network resource agents.

Prerequisites

  • You must be enrolled in the Red Hat Cloud Access program and have unused RHEL subscriptions. The attached subscription must include access to the following repositories for each GCP instance.

    • Red Hat Enterprise Linux 8 Server: rhel-8-server-rpms/8Server/x86_64
    • Red Hat Enterprise Linux 8 Server (High Availability): rhel-8-server-ha-rpms/8Server/x86_64
  • You must belong to an active GCP project and have sufficient permissions to create resources in the project.
  • Your project should have a service account that belongs to a VM instance and not an individual user. See Using the Compute Engine Default Service Account for information about using the default service account instead of creating a separate service account.

If you or your project administrator create a custom service account, the service account should be configured for the following roles.

  • Cloud Trace Agent
  • Compute Admin
  • Compute Network Admin
  • Cloud Datastore User
  • Logging Admin
  • Monitoring Editor
  • Monitoring Metric Writer
  • Service Account Administrator
  • Storage Admin

6.1. Required system packages

The procedures in this chapter assume you are using a host system running Red Hat Enterprise Linux. To successfully complete the procedures, your host system must have the following packages installed.

Table 6.1. System packages

PackageRepositoryDescription

libvirt

rhel-8-for-x86_64-appstream-rpms

Open source API, daemon, and management tool for managing platform virtualization

virt-install

rhel-8-for-x86_64-appstream-rpms

A command-line utility for building VMs

libguestfs

rhel-8-for-x86_64-appstream-rpms

A library for accessing and modifying VM file systems

libguestfs-tools

rhel-8-for-x86_64-appstream-rpms

System administration tools for VMs; includes the guestfish utility

6.2. Red Hat Enterprise Linux image options on GCP

The following table lists image choices and the differences in the image options.

Table 6.2. Image options

Image optionSubscriptionsSample scenarioConsiderations

Choose to deploy a custom image that you move to GCP.

Leverage your existing Red Hat subscriptions.

Enable subscriptions through the Red Hat Cloud Access program, upload your custom image, and attach your subscriptions.

The subscription includes the Red Hat product cost; you pay all other instance costs.

Custom images that you move to GCP are called "Cloud Access" images because you leverage your existing Red Hat subscriptions. Red Hat provides support directly for Cloud Access images.

Choose to deploy an existing GCP image that includes RHEL.

The GCP images include a Red Hat product.

Choose a RHEL image when you launch an instance on the GCP Compute Engine, or choose an image from the Google Cloud Platform Marketplace.

You pay GCP hourly on a pay-as-you-go model. Such images are called "on-demand" images. GCP offers support for on-demand images through a support agreement.

Note

You can create a custom image for GCP using Red Hat Image Builder. See Composing a Customized RHEL System Image for more information.

Important

You cannot convert an on-demand instance to a Red Hat Cloud Access instance. To change from an on-demand image to a Red Hat Cloud Access (BYOS) image, create a new Red Hat Cloud Access instance and migrate data from your on-demand instance. Cancel your on-demand instance after you migrate your data to avoid double billing.

The remainder of this chapter includes information and procedures pertaining to custom images.

6.3. Installing the Google Cloud SDK

Complete the following steps to install the Google Cloud SDK.

Prerequisites

Procedure

  1. Follow the GCP instructions for downloading and extracting the Google Cloud SDK archive. See the GCP document Quickstart for Linux for details.
  2. Follow the same instructions for initializing the Google Cloud SDK.

    Note

    Once you have initialized the Google Cloud SDK, you can use the gcloud CLI commands to perform tasks and obtain information about your project and instances. For example, you can display project information with the gcloud compute project-info describe --project <project-name> command.

6.4. Creating a GCP image bucket

The following document includes the minimum requirements for creating a multi-regional bucket in your default location.

Prerequisites

GCP storage utility (gsutil)

Procedure

  1. If you are not already logged in to Google Cloud Platform, log in with the following command.

    # gcloud auth login
  2. Create a storage bucket.

    $ gsutil mb gs://BucketName

    Example:

    $ gsutil mb gs://rhel-ha-bucket

Additional resources

Make buckets

6.5. Creating a custom virtual private cloud network and subnet

Complete the following steps to create a custom virtual private cloud (VPC) network and subnet.

Procedure

  1. Launch the GCP Console.
  2. Select VPC networks under Networking in the left navigation pane.
  3. Click Create VPC Network.
  4. Enter a name for the VPC network.
  5. Under the New subnet, create a Custom subnet in the region where you want to create the cluster.
  6. Click Create.

6.6. Preparing and importing a base GCP image

Complete the following steps to prepare the image for GCP. The following procedures assume you have created an image from a KVM Guest Image.

See Create a VM from a KVM Guest image for more information.

Procedure

  1. Enter the following command to convert the file. Images uploaded to GCP must be in raw format and named disk.raw.

    $ qemu-img convert -f qcow2 ImageName.qcow2 -O raw disk.raw
  2. Enter the following command to compress the raw file. Images uploaded to GCP must be compressed.

    $ tar -Sczf ImageName.tar.gz disk.raw
  3. Import the compressed image to the bucket created earlier.

    $ gsutil cp ImageName.tar.gz gs://BucketName

6.7. Creating and configuring a base GCP instance

Complete the following steps to create and configure a GCP instance that complies with GCP operating and security requirements.

Procedure

  1. Enter the following command to create an image from the compressed file in the bucket.

    $ gcloud compute images create BaseImageName --source-uri gs://BucketName/BaseImageName.tar.gz

    Example:

    [admin@localhost ~] $ gcloud compute images create rhel-76-server --source-uri gs://user-rhelha/rhel-server-76.tar.gz
    Created [https://www.googleapis.com/compute/v1/projects/MyProject/global/images/rhel-server-76].
    NAME            PROJECT                 FAMILY  DEPRECATED  STATUS
    rhel-76-server  rhel-ha-testing-on-gcp                      READY
  2. Enter the following command to create a template instance from the image. The minimum size required for a base RHEL instance is n1-standard-2. See gcloud compute instances create for additional configuration options.

    $ gcloud compute instances create BaseInstanceName --can-ip-forward --machine-type n1-standard-2 --image BaseImageName --service-account ServiceAccountEmail

    Example:

    [admin@localhost ~] $ gcloud compute instances create rhel-76-server-base-instance --can-ip-forward --machine-type n1-standard-2 --image rhel-76-server --service-account account@project-name-on-gcp.iam.gserviceaccount.com
    Created [https://www.googleapis.com/compute/v1/projects/rhel-ha-testing-on-gcp/zones/us-east1-b/instances/rhel-76-server-base-instance].
    NAME   ZONE   MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
    rhel-76-server-base-instance  us-east1-bn1-standard-2          10.10.10.3   192.227.54.211  RUNNING
  3. Connect to the instance with an SSH terminal session.

    $ ssh root@PublicIPaddress
  4. Update the RHEL software.

    1. Register with Red Hat Subscription Manager (RHSM).
    2. Enable a Subscription pool ID (or use the --auto-attach command).
    3. Disable all repositories.

      # subscription-manager repos --disable=*
    4. Enable the following repository.

      # subscription-manager repos --enable=rhel-8-server-rpms
    5. Run yum update.

      # yum update -y
  5. Install the GCP Linux Guest Environment on the running instance (in-place installation).

    See Install the guest environment in-place for instructions.

  6. Select the CentOS/RHEL option.
  7. Copy the command script and paste it at the command prompt to run the script immediately.
  8. Make the following configuration changes to the instance. These changes are based on GCP recommendations for custom images. See gcloudcompute images list for more information.

    1. Edit the /etc/chrony.conf file and remove all NTP servers.
    2. Add the following NTP server.

      metadata.google.internal iburst Google NTP server
    3. Remove any persistent network device rules.

      # rm -f /etc/udev/rules.d/70-persistent-net.rules
      
      # rm -f /etc/udev/rules.d/75-persistent-net-generator.rules
    4. Set the network service to start automatically.

      # chkconfig network on
    5. Set the ssh service to start automatically.

      # systemctl enable sshd
      # systemctl is-enabled sshd
    6. Enter the following command to set the time zone to UTC.

      # ln -sf /usr/share/zoneinfo/UTC /etc/localtime
    7. (Optional) Edit the /etc/ssh/ssh_config file and add the following lines to the end of the file. This keeps your SSH session alive during longer periods of inactivity.

      # Server times out connections after several minutes of inactivity.
      # Keep alive ssh connections by sending a packet every 7 minutes.
      ServerAliveInterval 420
    8. Edit the /etc/ssh/sshd_config file and make the following changes, if necessary. The ClientAliveInterval 420 setting is optional; this keeps your SSH session alive during longer periods of inactivity.

      PermitRootLogin no
      PasswordAuthentication no
      AllowTcpForwarding yes
      X11Forwarding no
      PermitTunnel no
      # Compute times out connections after 10 minutes of inactivity.
      # Keep ssh connections alive by sending a packet every 7 minutes.
      ClientAliveInterval 420
  9. Enter the following command to disable password access. Edit the /etc/cloud/cloud.cfg file.

    ssh_pwauth from 1 to 0.
    ssh_pwauth: 0
    Important

    Previously, you enabled password access to allow SSH session access to configure the instance. You must disable password access. All SSH session access must be passwordless.

  10. Enter the following command to unregister the instance from the subscription manager.

    # subscription-manager unregister
  11. Enter the following command to clean the shell history. Keep the instance running for the next procedure.

    # export HISTSIZE=0

6.8. Creating a snapshot image

Complete the following steps to preserve the instance configuration settings and create a snapshot.

Procedure

  1. On the running instance, enter the following command to synchronize data to disk.

    # sync
  2. On your host system, enter the following command to create the snapshot.

    $ gcloud compute disks snapshot InstanceName --snapshot-names SnapshotName
  3. On your host system, enter the following command to create the configured image from the snapshot.

    $ gcloud compute images create ConfiguredImageFromSnapshot --source-snapshot SnapshotName

6.9. Creating an HA node template instance and HA nodes

Once you have configured an image from the snapshot, you can create a node template. Use this template to create all HA nodes. Complete the following steps to create the template and HA nodes.

Procedure

  1. Enter the following command to create an instance template.

    $ gcloud compute instance-templates create InstanceTemplateName --can-ip-forward --machine-type n1-standard-2  --image ConfiguredImageFromSnapshot --service-account ServiceAccountEmailAddress

    Example:

    [admin@localhost ~] $ gcloud compute instance-templates create rhel-81-instance-template --can-ip-forward --machine-type n1-standard-2 --image rhel-81-gcp-image --service-account account@project-name-on-gcp.iam.gserviceaccount.com
    Created [https://www.googleapis.com/compute/v1/projects/project-name-on-gcp/global/instanceTemplates/rhel-81-instance-template].
    NAME  MACHINE_TYPE   PREEMPTIBLE  CREATION_TIMESTAMP
    rhel-81-instance-template   n1-standard-2          2018-07-25T11:09:30.506-07:00
  2. Enter the following command to create multiple nodes in one zone.

    # gcloud compute instances create NodeName01 NodeName02 --source-instance-template InstanceTemplateName --zone RegionZone --network=NetworkName --subnet=SubnetName

    Example:

    [admin@localhost ~] $ gcloud compute instances create rhel81-node-01 rhel81-node-02 rhel81-node-03 --source-instance-template rhel-81-instance-template --zone us-west1-b --network=projectVPC --subnet=range0
    Created [https://www.googleapis.com/compute/v1/projects/project-name-on-gcp/zones/us-west1-b/instances/rhel81-node-01].
    Created [https://www.googleapis.com/compute/v1/projects/project-name-on-gcp/zones/us-west1-b/instances/rhel81-node-02].
    Created [https://www.googleapis.com/compute/v1/projects/project-name-on-gcp/zones/us-west1-b/instances/rhel81-node-03].
    NAME            ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
    rhel81-node-01  us-west1-b  n1-standard-2               10.10.10.4   192.230.25.81   RUNNING
    rhel81-node-02  us-west1-b  n1-standard-2               10.10.10.5   192.230.81.253  RUNNING
    rhel81-node-03  us-east1-b  n1-standard-2               10.10.10.6   192.230.102.15  RUNNING

6.10. Installing HA packages and agents

Complete the following steps on all nodes.

Procedure

  1. In the Google Cloud Console, select Compute Engine and then select VM instances.
  2. Select the instance, click the arrow next to SSH, and select the View gcloud command option.
  3. Paste this command at a command prompt for passwordless access to the instance.
  4. Enable sudo account access and register with Red Hat Subscription Manager.
  5. Enable a Subscription pool ID (or use the --auto-attach command).
  6. Disable all repositories.

    # subscription-manager repos --disable=*
  7. Enable the following repositories.

    # subscription-manager repos --enable=rhel-8-server-rpms
    # subscription-manager repos --enable=rhel-ha-for-rhel-8-server-rpms
  8. Install pcs pacemaker, the fence agents, and the resource agents.

    # yum install -y pcs pacemaker fence-agents-gce resource-agents-gcp
  9. Update all packages.

    # yum update -y

6.11. Configuring HA services

Complete the following steps on all nodes to configure HA services.

Procedure

  1. The user hacluster was created during the pcs and pacemaker installation in the previous step. Create a password for the user hacluster on all cluster nodes. Use the same password for all nodes.

    # passwd hacluster
  2. If the firewalld service is installed, enter the following command to add the HA service.

    # firewall-cmd --permanent --add-service=high-availability
    
    # firewall-cmd --reload
  3. Enter the following command to start the pcs service and enable it to start on boot.

    # systemctl start pcsd.service
    
    # systemctl enable pcsd.service
    
    Created symlink from /etc/systemd/system/multi-user.target.wants/pcsd.service to /usr/lib/systemd/system/pcsd.service.

Verification steps

  1. Ensure the pcs service is running.

    # systemctl status pcsd.service
    
    pcsd.service - PCS GUI and remote configuration interface
    Loaded: loaded (/usr/lib/systemd/system/pcsd.service; enabled; vendor preset: disabled)
    Active: active (running) since Mon 2018-06-25 19:21:42 UTC; 15s ago
    Docs: man:pcsd(8)
    man:pcs(8)
    Main PID: 5901 (pcsd)
    CGroup: /system.slice/pcsd.service
    └─5901 /usr/bin/ruby /usr/lib/pcsd/pcsd > /dev/null &
  2. Edit the /etc/hosts file. Add RHEL host names and internal IP addresses for all nodes.

6.12. Creating a cluster

Complete the following steps to create the cluster of nodes.

Procedure

  1. On one of the nodes, enter the following command to authenticate the pcs user. Specify the name of each node in the cluster in the command.

    # pcs host auth  hostname1 hostname2 hostname3
    Username: hacluster
    Password:
    hostname1: Authorized
    hostname2: Authorized
    hostname3: Authorized
  2. Enter the following command to create the cluster.

    # pcs cluster setup cluster-name hostname1 hostname2 _hostname3-

Verification steps

  1. Run the following command to enable nodes to join the cluster automatically when started.

    # pcs cluster enable --all
  2. Enter the following command to start the cluster.

    # pcs cluster start --all

6.13. Creating a fencing device

For most default configurations, the GCP instance names and the RHEL host names are identical.

Complete the following steps to create a fencing device.

Procedure

  1. Enter the following command to get GCP instance names. Note that the output also shows the internal ID for the instance.

    # fence_gce --zone us-west1-b --project=rhel-ha-on-gcp -o list

    Example:

    Example:
    [root@rhel81-node-01 ~]# fence_gce --zone us-west1-b --project=rhel-ha-testing-on-gcp -o list
    44358**********3181,InstanceName-3
    40819**********6811,InstanceName-1
    71736**********3341,InstanceName-2
  2. Enter the following command to create a fence device.

    # pcs stonith create _FenceDeviceName_ fence_gce zone=_Region-Zone_ project=_MyProject_

Verification step

Verify that the fence devices started.

# pcs status

Example:

[root@rhel81-node-01 ~]# pcs status
Cluster name: gcp-cluster
Stack: corosync
Current DC: rhel81-node-02 (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
Last updated: Fri Jul 27 12:53:25 2018
Last change: Fri Jul 27 12:51:43 2018 by root via cibadmin on rhel81-node-01

3 nodes configured
3 resources configured

Online: [ rhel81-node-01 rhel81-node-02 rhel81-node-03 ]

Full list of resources:

us-west1-b-fence    (stonith:fence_gce):    Started rhel81-node-01

Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled

6.14. Configuring GCP node authorization

Configure cloud SDK tools to use your account credentials to access GCP.

Procedure

Enter the following command on each node to initialize each node with your project ID and account credentials.

# gcloud-ra init

6.15. Configuring the gcp-vcp-move-vip resource agent

The gcp-vpc-move-vip resource agent attaches a secondary IP address (alias IP) to a running instance. This is a floating IP address that can be passed between different nodes in the cluster.

Enter the following command to show more information about this resource.

# pcs resource describe gcp-vpc-move-vip

You can configure the resource agent to use a primary subnet address range or a secondary subnet address range. This section includes procedures for both.

Primary subnet address range

Complete the following steps to configure the resource for the primary VPC subnet.

Procedure

  1. Enter the following command to create the aliasip resource. Include an unused internal IP address. Include the CIDR block in the command.

    # pcs resource create aliasip gcp-vpc-move-vip  alias_ip=UnusedIPaddress/CIDRblock

    Example:

    [root@rhel81-node-01 ~]# pcs resource create aliasip gcp-vpc-move-vip alias_ip=10.10.10.200/32
  2. Enter the following command to create an IPaddr2 resource for managing the IP on the node.

    # pcs resource create vip IPaddr2 nic=interface ip=AliasIPaddress cidr_netmask=32

    Example:

    [root@rhel81-node-01 ~]# pcs resource create vip IPaddr2 nic=eth0 ip=10.10.10.200 cidr_netmask=32
  3. Enter the following command to group the network resources under vipgrp.

    # pcs resource group add vipgrp aliasip vip

Verification steps

  1. Enter the following command to verify that the resources have started and are grouped under vipgrp.

    [root@rhel81-node-01 ~]# pcs status
  2. Enter the following command to verify that the resource can move to a different node.

    # pcs resource move vip _Node_

    Example:

    [root@rhel81-node-01 ~]# pcs resource move vip rhel81-node-03
  3. Enter the following command to verify that the vip successfully started on a different node.

    [root@rhel81-node-01 ~]# pcs status

Secondary subnet address range

Complete the following steps to configure the resource for a secondary subnet address range.

Procedure

  1. Enter the following command to create a secondary subnet address range.

    # gcloud-ra compute networks subnets update SubnetName --region RegionName --add-secondary-ranges SecondarySubnetName=SecondarySubnetRange

    Example:

    # gcloud-ra compute networks subnets update range0 --region us-west1 --add-secondary-ranges range1=10.10.20.0/24
  2. Enter the following command to create the aliasip resource. Create an unused internal IP address in the secondary subnet address range. Include the CIDR block in the command.

    # pcs resource create aliasip gcp-vpc-move-vip alias_ip=UnusedIPaddress/CIDRblock

    Example:

    [root@rhel81-node-01 ~]# pcs resource create aliasip gcp-vpc-move-vip alias_ip=10.10.20.200/32
  3. Enter the following command to create an IPaddr2 resource for managing the IP on the node.

    # pcs resource create vip IPaddr2 nic=interface ip=AliasIPaddress cidr_netmask=32

    Example:

    [root@rhel81-node-01 ~]# pcs resource create vip IPaddr2 nic=eth0 ip=10.10.20.200 cidr_netmask=32
  4. Group the network resources under vipgrp.

    # pcs resource group add vipgrp aliasip vip

Verification steps

  1. Enter the following command to verify that the resources have started and are grouped under vipgrp.

    [root@rhel81-node-01 ~]# pcs status
  2. Enter the following command to verify that the resource can move to a different node.

    # pcs resource move vip _Node_

    Example:

    [root@rhel81-node-01 ~]# pcs resource move vip rhel81-node-03
  3. Enter the following command to verify that the vip successfully started on a different node.

    [root@rhel81-node-01 ~]# pcs status