Procedure

1. Convert the image from qcow2 to raw format.

   $ qemu-img convert -f qcow2 -O raw <image-name>.qcow2 <image-name>.raw

2. Create a shell script using the contents below.

   #!/bin/bash
   MB=$((1024 * 1024))
   size=$(qemu-img info -f raw --output json "$1" | gawk 'match($0, /"virtual-size": ([0-9]+),/, val) {print val[1]}')
   rounded_size=$((($size/$MB + 1) * $MB))
   if [ $(($size % $MB)) -eq  0 ]
   then
    echo "Your image is already aligned. You do not need to resize."
    exit 1
   fi
   echo "rounded size = $rounded_size"
   export rounded_size

3. Run the script. This example uses the name align.sh.

   $ sh align.sh <image-xxx>.raw

   • If the message "Your image is already aligned. You do not need to resize." displays, proceed to the following step.
   • If a value displays, your image is not aligned.

4. Use the following command to convert the file to a fixed VHD format.

   The sample uses qemu-img version 2.12.0.

   $ qemu-img convert -f raw -o subformat=fixed,force_size -O vpc <image-xxx>.raw <image.xxx>.vhd

   Once converted, the VHD file is ready to upload to Azure.

5. If the raw image is not aligned, complete the following steps to align it.

   1. Resize the raw file using the rounded value displayed when you ran the verification script.

      $ qemu-img resize -f raw <image-xxx>.raw <rounded-value>

   2. Convert the raw image file to a VHD format.

      The sample uses qemu-img version 2.12.0.

      $ qemu-img convert -f raw -o subformat=fixed,force_size -O vpc <image-xxx>.raw <image.xxx>.vhd

      Once converted, the VHD file is ready to upload to Azure.

Procedure

1. Import the Microsoft repository key.

   $ sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc

2. Create a local Azure CLI repository entry.

   $ sudo sh -c 'echo -e "[azure-cli]\nname=Azure CLI\nbaseurl=https://packages.microsoft.com/yumrepos/azure-cli\nenabled=1\ngpgcheck=1\ngpgkey=https://packages.microsoft.com/keys/microsoft.asc" > /etc/yum.repos.d/azure-cli.repo'

3. Update the dnf package index.

   $ dnf check-update

4. Check your Python version (python --version) and install Python 3.x, if necessary.

   $ sudo dnf install python3

5. Install the Azure CLI.

   $ sudo dnf install -y azure-cli

6. Run the Azure CLI.

   $ az

Procedure

1. Authenticate your system with Azure and log in.

   $ az login

   Note

   If a browser is available in your environment, the CLI opens your browser to the Azure sign-in page. See Sign in with Azure CLI for more information and options.

2. Create a resource group in an Azure region.

   $ az group create --name <resource-group> --location <azure-region>

   Example:

   [clouduser@localhost]$ az group create --name azrhelclirsgrp --location southcentralus
   {
     "id": "/subscriptions//resourceGroups/azrhelclirsgrp",
     "location": "southcentralus",
     "managedBy": null,
     "name": "azrhelclirsgrp",
     "properties": {
       "provisioningState": "Succeeded"
     },
     "tags": null
   }

3. Create a storage account. See SKU Types for more information about valid SKU values.

   $ az storage account create -l <azure-region> -n <storage-account-name> -g <resource-group> --sku <sku_type>

   Example:

   [clouduser@localhost]$ az storage account create -l southcentralus -n azrhelclistact -g azrhelclirsgrp --sku Standard_LRS
   {
     "accessTier": null,
     "creationTime": "2017-04-05T19:10:29.855470+00:00",
     "customDomain": null,
     "encryption": null,
     "id": "/subscriptions//resourceGroups/azrhelclirsgrp/providers/Microsoft.Storage/storageAccounts/azrhelclistact",
     "kind": "StorageV2",
     "lastGeoFailoverTime": null,
     "location": "southcentralus",
     "name": "azrhelclistact",
     "primaryEndpoints": {
       "blob": "https://azrhelclistact.blob.core.windows.net/",
       "file": "https://azrhelclistact.file.core.windows.net/",
       "queue": "https://azrhelclistact.queue.core.windows.net/",
       "table": "https://azrhelclistact.table.core.windows.net/"
   },
   "primaryLocation": "southcentralus",
   "provisioningState": "Succeeded",
   "resourceGroup": "azrhelclirsgrp",
   "secondaryEndpoints": null,
   "secondaryLocation": null,
   "sku": {
     "name": "Standard_LRS",
     "tier": "Standard"
   },
   "statusOfPrimary": "available",
   "statusOfSecondary": null,
   "tags": {},
     "type": "Microsoft.Storage/storageAccounts"
   }

4. Get the storage account connection string.

   $ az storage account show-connection-string -n <storage-account-name> -g <resource-group>

   Example:

   [clouduser@localhost]$ az storage account show-connection-string -n azrhelclistact -g azrhelclirsgrp
   {
     "connectionString": "DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=azrhelclistact;AccountKey=NreGk...=="
   }

5. Export the connection string by copying the connection string and pasting it into the following command. This string connects your system to the storage account.

   $ export AZURE_STORAGE_CONNECTION_STRING="<storage-connection-string>"

   Example:

   [clouduser@localhost]$ export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=azrhelclistact;AccountKey=NreGk...=="

6. Create the storage container.

   $ az storage container create -n <container-name>

   Example:

   [clouduser@localhost]$ az storage container create -n azrhelclistcont
   {
     "created": true
   }

7. Create a virtual network.

   $ az network vnet create -g <resource group> --name <vnet-name> --subnet-name <subnet-name>

   Example:

   [clouduser@localhost]$ az network vnet create --resource-group azrhelclirsgrp --name azrhelclivnet1 --subnet-name azrhelclisubnet1
   {
     "newVNet": {
       "addressSpace": {
         "addressPrefixes": [
         "10.0.0.0/16"
         ]
     },
     "dhcpOptions": {
       "dnsServers": []
     },
     "etag": "W/\"\"",
     "id": "/subscriptions//resourceGroups/azrhelclirsgrp/providers/Microsoft.Network/virtualNetworks/azrhelclivnet1",
     "location": "southcentralus",
     "name": "azrhelclivnet1",
     "provisioningState": "Succeeded",
     "resourceGroup": "azrhelclirsgrp",
     "resourceGuid": "0f25efee-e2a6-4abe-a4e9-817061ee1e79",
     "subnets": [
       {
         "addressPrefix": "10.0.0.0/24",
         "etag": "W/\"\"",
         "id": "/subscriptions//resourceGroups/azrhelclirsgrp/providers/Microsoft.Network/virtualNetworks/azrhelclivnet1/subnets/azrhelclisubnet1",
         "ipConfigurations": null,
         "name": "azrhelclisubnet1",
         "networkSecurityGroup": null,
         "provisioningState": "Succeeded",
         "resourceGroup": "azrhelclirsgrp",
         "resourceNavigationLinks": null,
         "routeTable": null
       }
     ],
     "tags": {},
     "type": "Microsoft.Network/virtualNetworks",
     "virtualNetworkPeerings": null
     }
   }

Procedure

1. Upload the VHD file to the storage container. It may take several minutes. To get a list of storage containers, enter the az storage container list command.

   $ az storage blob upload \
       --account-name <storage-account-name> --container-name <container-name> \
       --type page --file <path-to-vhd> --name <image-name>.vhd

   Example:

   [clouduser@localhost]$ az storage blob upload \
   --account-name azrhelclistact --container-name azrhelclistcont \
   --type page --file rhel-image-9.vhd --name rhel-image-9.vhd
   
   Percent complete: %100.0

2. Get the URL for the uploaded VHD file to use in the following step.

   $ az storage blob url -c <container-name> -n <image-name>.vhd

   Example:

   $ az storage blob url -c azrhelclistcont -n rhel-image-9.vhd "https://azrhelclistact.blob.core.windows.net/azrhelclistcont/rhel-image-9.vhd"

3. Create the Azure custom image.

   $ az image create -n <image-name> -g <resource-group> -l <azure-region> --source <URL> --os-type linux

   Note

   The default hypervisor generation of the VM is V1. You can optionally specify a V2 hypervisor generation by including the option --hyper-v-generation V2. Generation 2 VMs use a UEFI-based boot architecture. See Support for generation 2 VMs on Azure for information on generation 2 VMs.

   The command may return the error "Only blobs formatted as VHDs can be imported." This error may mean that the image was not aligned to the nearest 1 MB boundary before it was converted to VHD.

   Example:

   $ az image create -n rhel9 -g azrhelclirsgrp2 -l southcentralus --source https://azrhelclistact.blob.core.windows.net/azrhelclistcont/rhel-image-9.vhd --os-type linux

Procedure

1. Enter the following command to create the VM.

   $ az vm create \
       -g <resource-group> -l <azure-region> -n <vm-name> \
       --vnet-name <vnet-name> --subnet <subnet-name> --size Standard_A2 \
       --os-disk-name <simple-name> --admin-username <administrator-name> \
       --generate-ssh-keys --image <path-to-image>

   Note

   The option --generate-ssh-keys creates a private/public key pair. Private and public key files are created in ~/.ssh on your system. The public key is added to the authorized_keys file on the VM for the user specified by the --admin-username option. See Other authentication methods for additional information.

   Example:

   [clouduser@localhost]$ az vm create \
   -g azrhelclirsgrp2 -l southcentralus -n rhel-azure-vm-1 \
   --vnet-name azrhelclivnet1 --subnet azrhelclisubnet1 --size Standard_A2 \
   --os-disk-name vm-1-osdisk --admin-username clouduser \
   --generate-ssh-keys --image rhel9
   
   {
     "fqdns": "",
     "id": "/subscriptions//resourceGroups/azrhelclirsgrp/providers/Microsoft.Compute/virtualMachines/rhel-azure-vm-1",
     "location": "southcentralus",
     "macAddress": "",
     "powerState": "VM running",
     "privateIpAddress": "10.0.0.4",
     "publicIpAddress": "<public-IP-address>",
     "resourceGroup": "azrhelclirsgrp2"

   Note the publicIpAddress. You need this address to log in to the VM in the following step.

2. Start an SSH session and log in to the VM.

   [clouduser@localhost]$ ssh -i /home/clouduser/.ssh/id_rsa clouduser@<public-IP-address>.
   The authenticity of host ',<public-IP-address>' can't be established.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added '<public-IP-address>' (ECDSA) to the list of known hosts.
   
   [clouduser@rhel-azure-vm-1 ~]$

Procedure

1. Register your system.

   # subscription-manager register --auto-attach

2. Attach your subscriptions.

   • You can use an activation key to attach subscriptions. See Creating Red Hat Customer Portal Activation Keys for more information.
   • Alternatively, you can manually attach a subscription using the ID of the subscription pool (Pool ID). See Attaching and Removing Subscriptions Through the Command Line.

Procedure

1. Use the Gold Image to create a RHEL 9 VM in your Azure instance. For instructions, see Creating and starting the VM in Azure.
2. Start the created VM.

3. In the RHEL 9 VM, enable automatic registration.

   # subscription-manager config --rhsmcertd.auto_registration=1

4. Enable the rhsmcertd service.

   # systemctl enable rhsmcertd.service

5. Disable the redhat.repo repository.

   # subscription-manager config --rhsm.manage_repos=0

6. Power off the VM, and save it as a managed image on Azure. For instructions, see How to create a managed image of a virtual machine or VHD.
7. Create VMs using the managed image. They will be automatically subscribed to RHSM.

Procedure

1. Authenticate your system with Azure and log in.

   $ az login

   Note

   If a browser is available in your environment, the CLI opens your browser to the Azure sign-in page.

   Example:

   [clouduser@localhost]$ az login
   
   To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code FDMSCMETZ to authenticate.
     [
       {
         "cloudName": "AzureCloud",
         "id": "Subscription ID",
         "isDefault": true,
         "name": "MySubscriptionName",
         "state": "Enabled",
         "tenantId": "Tenant ID",
         "user": {
           "name": "clouduser@company.com",
           "type": "user"
         }
       }
     ]

2. Create a resource group in an Azure region.

   $ az group create --name resource-group --location azure-region

   Example:

   [clouduser@localhost]$ az group create --name azrhelclirsgrp --location southcentralus
   
   {
     "id": "/subscriptions//resourceGroups/azrhelclirsgrp",
     "location": "southcentralus",
     "managedBy": null,
     "name": "azrhelclirsgrp",
     "properties": {
       "provisioningState": "Succeeded"
     },
     "tags": null
   }

3. Create a storage account.

   $ az storage account create -l azure-region -n storage-account-name -g resource-group --sku sku_type --kind StorageV2

   Example:

   [clouduser@localhost]$ az storage account create -l southcentralus -n azrhelclistact -g azrhelclirsgrp --sku Standard_LRS --kind StorageV2
   
   {
     "accessTier": null,
     "creationTime": "2017-04-05T19:10:29.855470+00:00",
     "customDomain": null,
     "encryption": null,
     "id": "/subscriptions//resourceGroups/azrhelclirsgrp/providers/Microsoft.Storage/storageAccounts/azrhelclistact",
     "kind": "StorageV2",
     "lastGeoFailoverTime": null,
     "location": "southcentralus",
     "name": "azrhelclistact",
     "primaryEndpoints": {
       "blob": "https://azrhelclistact.blob.core.windows.net/",
       "file": "https://azrhelclistact.file.core.windows.net/",
       "queue": "https://azrhelclistact.queue.core.windows.net/",
       "table": "https://azrhelclistact.table.core.windows.net/"
   },
   "primaryLocation": "southcentralus",
   "provisioningState": "Succeeded",
   "resourceGroup": "azrhelclirsgrp",
   "secondaryEndpoints": null,
   "secondaryLocation": null,
   "sku": {
     "name": "Standard_LRS",
     "tier": "Standard"
   },
   "statusOfPrimary": "available",
   "statusOfSecondary": null,
   "tags": {},
     "type": "Microsoft.Storage/storageAccounts"
   }

4. Get the storage account connection string.

   $ az storage account show-connection-string -n storage-account-name -g resource-group

   Example:

   [clouduser@localhost]$ az storage account show-connection-string -n azrhelclistact -g azrhelclirsgrp
   {
     "connectionString": "DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=azrhelclistact;AccountKey=NreGk...=="
   }

5. Export the connection string by copying the connection string and pasting it into the following command. This string connects your system to the storage account.

   $ export AZURE_STORAGE_CONNECTION_STRING="storage-connection-string"

   Example:

   [clouduser@localhost]$ export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=azrhelclistact;AccountKey=NreGk...=="

6. Create the storage container.

   $ az storage container create -n container-name

   Example:

   [clouduser@localhost]$ az storage container create -n azrhelclistcont
   
   {
     "created": true
   }

7. Create a virtual network. All cluster nodes must be in the same virtual network.

   $ az network vnet create -g resource group --name vnet-name --subnet-name subnet-name

   Example:

   [clouduser@localhost]$ az network vnet create --resource-group azrhelclirsgrp --name azrhelclivnet1 --subnet-name azrhelclisubnet1
   {
     "newVNet": {
       "addressSpace": {
         "addressPrefixes": [
         "10.0.0.0/16"
         ]
     },
     "dhcpOptions": {
       "dnsServers": []
     },
     "etag": "W/\"\"",
     "id": "/subscriptions//resourceGroups/azrhelclirsgrp/providers/Microsoft.Network/virtualNetworks/azrhelclivnet1",
     "location": "southcentralus",
     "name": "azrhelclivnet1",
     "provisioningState": "Succeeded",
     "resourceGroup": "azrhelclirsgrp",
     "resourceGuid": "0f25efee-e2a6-4abe-a4e9-817061ee1e79",
     "subnets": [
       {
         "addressPrefix": "10.0.0.0/24",
         "etag": "W/\"\"",
         "id": "/subscriptions//resourceGroups/azrhelclirsgrp/providers/Microsoft.Network/virtualNetworks/azrhelclivnet1/subnets/azrhelclisubnet1",
         "ipConfigurations": null,
         "name": "azrhelclisubnet1",
         "networkSecurityGroup": null,
         "provisioningState": "Succeeded",
         "resourceGroup": "azrhelclirsgrp",
         "resourceNavigationLinks": null,
         "routeTable": null
       }
     ],
     "tags": {},
     "type": "Microsoft.Network/virtualNetworks",
     "virtualNetworkPeerings": null
     }
   }

8. Create an availability set. All cluster nodes must be in the same availability set.

   $ az vm availability-set create --name MyAvailabilitySet --resource-group MyResourceGroup

   Example:

   [clouduser@localhost]$ az vm availability-set create --name rhelha-avset1 --resource-group azrhelclirsgrp
   {
     "additionalProperties": {},
       "id": "/subscriptions/.../resourceGroups/azrhelclirsgrp/providers/Microsoft.Compute/availabilitySets/rhelha-avset1",
       "location": "southcentralus",
       "name": “rhelha-avset1",
       "platformFaultDomainCount": 2,
       "platformUpdateDomainCount": 5,
   
   [omitted]

Procedure

1. Enter the following grep command to determine if the required Hyper-V device drivers are installed.

   # lsinitrd | grep hv

   In the example below, all required drivers are installed.

   # lsinitrd | grep hv
   drwxr-xr-x   2 root     root            0 Aug 12 14:21 usr/lib/modules/3.10.0-932.el9.x86_64/kernel/drivers/hv
   -rw-r--r--   1 root     root        31272 Aug 11 08:45 usr/lib/modules/3.10.0-932.el9.x86_64/kernel/drivers/hv/hv_vmbus.ko.xz
   -rw-r--r--   1 root     root        25132 Aug 11 08:46 usr/lib/modules/3.10.0-932.el9.x86_64/kernel/drivers/net/hyperv/hv_netvsc.ko.xz
   -rw-r--r--   1 root     root         9796 Aug 11 08:45 usr/lib/modules/3.10.0-932.el9.x86_64/kernel/drivers/scsi/hv_storvsc.ko.xz

   If all the drivers are not installed, complete the remaining steps.

   Note

   An hv_vmbus driver may exist in the environment. Even if this driver is present, complete the following steps.

2. Create a file named hv.conf in /etc/dracut.conf.d.

3. Add the following driver parameters to the hv.conf file.

   add_drivers+=" hv_vmbus "
   add_drivers+=" hv_netvsc "
   add_drivers+=" hv_storvsc "
   add_drivers+=" nvme "

   Note

   Note the spaces before and after the quotes, for example, add_drivers+=" hv_vmbus ". This ensures that unique drivers are loaded in the event that other Hyper-V drivers already exist in the environment.

4. Regenerate the initramfs image.

   # dracut -f -v --regenerate-all

Verification

1. Reboot the machine.
2. Run the lsinitrd | grep hv command to verify that the drivers are installed.

Procedure

1. Log in to the VM.

2. Register the VM, and enable the Red Hat Enterprise Linux 9 repository.

   # subscription-manager register --auto-attach
   Installed Product Current Status:
   Product Name: Red Hat Enterprise Linux for x86_64
   Status: Subscribed

3. Ensure that the cloud-init and hyperv-daemons packages are installed.

   # dnf install cloud-init hyperv-daemons -y

4. Create cloud-init configuration files that are needed for integration with Azure services:

   1. To enable logging to the Hyper-V Data Exchange Service (KVP), create the /etc/cloud/cloud.cfg.d/10-azure-kvp.cfg configuration file and add the following lines to that file.

      reporting:
          logging:
              type: log
          telemetry:
              type: hyperv

   2. To add Azure as a datasource, create the /etc/cloud/cloud.cfg.d/91-azure_datasource.cfg configuration file, and add the following lines to that file.

      datasource_list: [ Azure ]
      datasource:
          Azure:
              apply_network_config: False

5. To ensure that specific kernel modules are blocked from loading automatically, edit or create the /etc/modprobe.d/blocklist.conf file and add the following lines to that file.

   blacklist nouveau
   blacklist lbm-nouveau
   blacklist floppy
   blacklist amdgpu
   blacklist skx_edac
   blacklist intel_cstate

6. Modify udev network device rules:

   1. Remove the following persistent network device rules if present.

      # rm -f /etc/udev/rules.d/70-persistent-net.rules
      # rm -f /etc/udev/rules.d/75-persistent-net-generator.rules
      # rm -f /etc/udev/rules.d/80-net-name-slot-rules

   2. To ensure that Accelerated Networking on Azure works as intended, create a new network device rule /etc/udev/rules.d/68-azure-sriov-nm-unmanaged.rules and add the following line to it.

      SUBSYSTEM=="net", DRIVERS=="hv_pci", ACTION=="add", ENV{NM_UNMANAGED}="1"

7. Set the sshd service to start automatically.

   # systemctl enable sshd
   # systemctl is-enabled sshd

8. Modify kernel boot parameters:

   1. Open the /etc/default/grub file, and ensure the GRUB_TIMEOUT line has the following value.

      GRUB_TIMEOUT=10

   2. Remove the following options from the end of the GRUB_CMDLINE_LINUX line if present.

      rhgb quiet

   3. Ensure the /etc/default/grub file contains the following lines with all the specified options.

      GRUB_CMDLINE_LINUX="loglevel=3 crashkernel=auto console=tty1 console=ttyS0 earlyprintk=ttyS0 rootdelay=300"
      GRUB_TIMEOUT_STYLE=countdown
      GRUB_TERMINAL="serial console"
      GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

   4. Regenerate the grub.cfg file.

      On a BIOS-based machine:

      # grub2-mkconfig -o /boot/grub2/grub.cfg

      On a UEFI-based machine:

      # grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

      If your system uses a non-default location for grub.cfg, adjust the command accordingly.

9. Configure the Windows Azure Linux Agent (WALinuxAgent):

   1. Install and enable the WALinuxAgent package.

      # dnf install WALinuxAgent -y
      # systemctl enable waagent

   2. To ensure that a swap partition is not used in provisioned VMs, edit the following lines in the /etc/waagent.conf file.

      Provisioning.DeleteRootPassword=y
      ResourceDisk.Format=n
      ResourceDisk.EnableSwap=n

10. Prepare the VM for Azure provisioning:

    1. Unregister the VM from Red Hat Subscription Manager.

       # subscription-manager unregister

    2. Clean up the existing provisioning details.

       # waagent -force -deprovision

       Note

       This command generates warnings, which are expected because Azure handles the provisioning of VMs automatically.

    3. Clean the shell history and shut down the VM.

       # export HISTSIZE=0
       # poweroff

Procedure

1. Ensure you are an Administrator or Owner for the Microsoft Azure subscription. You need this authorization to create an Azure AD application.

2. Log in to your Azure account.

   $ az login

3. Enter the following command to create the Azure AD Application. To use your own password, add the --password option to the command. Ensure that you create a strong password.

   $ az ad sp create-for-rbac --name FencingApplicationName --role owner --scopes "/subscriptions/SubscriptionID/resourceGroups/MyResourseGroup"

   Example:

   [clouduser@localhost ~] $ az ad sp create-for-rbac --name FencingApp --role owner --scopes "/subscriptions/2586c64b-xxxxxx-xxxxxxx-xxxxxxx/resourceGroups/azrhelclirsgrp"
   Retrying role assignment creation: 1/36
   Retrying role assignment creation: 2/36
   Retrying role assignment creation: 3/36
   {
     "appId": "1a3dfe06-df55-42ad-937b-326d1c211739",
     "displayName": "FencingApp",
     "name": "http://FencingApp",
     "password": "43a603f0-64bb-482e-800d-402efe5f3d47",
     "tenant": "77ecefb6-xxxxxxxxxx-xxxxxxx-757a69cb9485"
   }

4. Save the following information before proceeding. You need this information to set up the fencing agent.

   • Azure AD Application ID
   • Azure AD Application Password
   • Tenant ID
   • Microsoft Azure Subscription ID

Procedure

1. Convert the image from qcow2 to raw format.

   $ qemu-img convert -f qcow2 -O raw <image-name>.qcow2 <image-name>.raw

2. Create a shell script using the contents below.

   #!/bin/bash
   MB=$((1024 * 1024))
   size=$(qemu-img info -f raw --output json "$1" | gawk 'match($0, /"virtual-size": ([0-9]+),/, val) {print val[1]}')
   rounded_size=$((($size/$MB + 1) * $MB))
   if [ $(($size % $MB)) -eq  0 ]
   then
    echo "Your image is already aligned. You do not need to resize."
    exit 1
   fi
   echo "rounded size = $rounded_size"
   export rounded_size

3. Run the script. This example uses the name align.sh.

   $ sh align.sh <image-xxx>.raw

   • If the message "Your image is already aligned. You do not need to resize." displays, proceed to the following step.
   • If a value displays, your image is not aligned.

4. Use the following command to convert the file to a fixed VHD format.

   The sample uses qemu-img version 2.12.0.

   $ qemu-img convert -f raw -o subformat=fixed,force_size -O vpc <image-xxx>.raw <image.xxx>.vhd

   Once converted, the VHD file is ready to upload to Azure.

5. If the raw image is not aligned, complete the following steps to align it.

   1. Resize the raw file using the rounded value displayed when you ran the verification script.

      $ qemu-img resize -f raw <image-xxx>.raw <rounded-value>

   2. Convert the raw image file to a VHD format.

      The sample uses qemu-img version 2.12.0.

      $ qemu-img convert -f raw -o subformat=fixed,force_size -O vpc <image-xxx>.raw <image.xxx>.vhd

      Once converted, the VHD file is ready to upload to Azure.

Procedure

1. Upload the VHD file to the storage container. It may take several minutes. To get a list of storage containers, enter the az storage container list command.

   $ az storage blob upload \
       --account-name <storage-account-name> --container-name <container-name> \
       --type page --file <path-to-vhd> --name <image-name>.vhd

   Example:

   [clouduser@localhost]$ az storage blob upload \
   --account-name azrhelclistact --container-name azrhelclistcont \
   --type page --file rhel-image-9.vhd --name rhel-image-9.vhd
   
   Percent complete: %100.0

2. Get the URL for the uploaded VHD file to use in the following step.

   $ az storage blob url -c <container-name> -n <image-name>.vhd

   Example:

   $ az storage blob url -c azrhelclistcont -n rhel-image-9.vhd "https://azrhelclistact.blob.core.windows.net/azrhelclistcont/rhel-image-9.vhd"

3. Create the Azure custom image.

   $ az image create -n <image-name> -g <resource-group> -l <azure-region> --source <URL> --os-type linux

   Note

   The default hypervisor generation of the VM is V1. You can optionally specify a V2 hypervisor generation by including the option --hyper-v-generation V2. Generation 2 VMs use a UEFI-based boot architecture. See Support for generation 2 VMs on Azure for information on generation 2 VMs.

   The command may return the error "Only blobs formatted as VHDs can be imported." This error may mean that the image was not aligned to the nearest 1 MB boundary before it was converted to VHD.

   Example:

   $ az image create -n rhel9 -g azrhelclirsgrp2 -l southcentralus --source https://azrhelclistact.blob.core.windows.net/azrhelclistcont/rhel-image-9.vhd --os-type linux

Procedure

1. Launch an SSH terminal session and connect to the VM using the administrator name and public IP address.

   $ ssh administrator@PublicIP

   To get the public IP address for an Azure VM, open the VM properties in the Azure Portal or enter the following Azure CLI command.

   $ az vm list -g <resource-group> -d --output table

   Example:

   [clouduser@localhost ~] $ az vm list -g azrhelclirsgrp -d --output table
   Name    ResourceGroup           PowerState      PublicIps        Location
   ------  ----------------------  --------------  -------------    --------------
   node01  azrhelclirsgrp          VM running      192.98.152.251    southcentralus

2. Register the VM with Red Hat.

   $ sudo -i
   # subscription-manager register --auto-attach

   Note

   If the --auto-attach command fails, manually register the VM to your subscription.

3. Disable all repositories.

   # subscription-manager repos --disable=*

4. Enable the RHEL 9 Server HA repositories.

   # subscription-manager repos --enable=rhel-9-for-x86_64-highavailability-rpms

5. Update all packages.

   # dnf update -y

6. Install the Red Hat High Availability Add-On software packages, along with all available fencing agents from the High Availability channel.

   # dnf install pcs pacemaker fence-agents-azure-arm

7. The user hacluster was created during the pcs and pacemaker installation in the previous step. Create a password for hacluster on all cluster nodes. Use the same password for all nodes.

   # passwd hacluster

8. Add the high availability service to the RHEL Firewall if firewalld.service is installed.

   # firewall-cmd --permanent --add-service=high-availability
   # firewall-cmd --reload

9. Start the pcs service and enable it to start on boot.

   # systemctl start pcsd.service
   # systemctl enable pcsd.service
   
   Created symlink from /etc/systemd/system/multi-user.target.wants/pcsd.service to /usr/lib/systemd/system/pcsd.service.

Procedure

1. On one of the nodes, enter the following command to authenticate the pcs user hacluster. In the command, specify the name of each node in the cluster.

   # pcs host auth <hostname1> <hostname2> <hostname3>

   Example:

   [root@node01 clouduser]# pcs host auth node01 node02 node03
   Username: hacluster
   Password:
   node01: Authorized
   node02: Authorized
   node03: Authorized

2. Create the cluster.

   # pcs cluster setup <cluster_name> <hostname1> <hostname2> <hostname3>

   Example:

   [root@node01 clouduser]# pcs cluster setup new_cluster node01 node02 node03
   
   [...]
   
   Synchronizing pcsd certificates on nodes node01, node02, node03...
   node02: Success
   node03: Success
   node01: Success
   Restarting pcsd on the nodes in order to reload the certificates...
   node02: Success
   node03: Success
   node01: Success

Verification

1. Enable the cluster.

   [root@node01 clouduser]# pcs cluster enable --all
   node02: Cluster Enabled
   node03: Cluster Enabled
   node01: Cluster Enabled

2. Start the cluster.

   [root@node01 clouduser]# pcs cluster start --all
   node02: Starting Cluster...
   node03: Starting Cluster...
   node01: Starting Cluster...

Procedure

1. Identify the Azure node name for each RHEL VM. You use the Azure node names to configure the fence device.

   # fence_azure_arm \
       -l <AD-Application-ID> -p <AD-Password> \
       --resourceGroup <MyResourceGroup> --tenantId <Tenant-ID> \
       --subscriptionId <Subscription-ID> -o list

   Example:

   [root@node01 clouduser]# fence_azure_arm \
   -l e04a6a49-9f00-xxxx-xxxx-a8bdda4af447 -p z/a05AwCN0IzAjVwXXXXXXXEWIoeVp0xg7QT//JE=
   --resourceGroup azrhelclirsgrp --tenantId 77ecefb6-cff0-XXXX-XXXX-757XXXX9485
   --subscriptionId XXXXXXXX-38b4-4527-XXXX-012d49dfc02c -o list
   
   node01,
   node02,
   node03,

2. View the options for the Azure ARM STONITH agent.

   # pcs stonith describe fence_azure_arm

   Example:

   # pcs stonith describe fence_apc
   Stonith options:
   password: Authentication key
   password_script: Script to run to retrieve password

   Warning

   For fence agents that provide a method option, do not specify a value of cycle as it is not supported and can cause data corruption.

   Some fence devices can fence only a single node, while other devices can fence multiple nodes. The parameters you specify when you create a fencing device depend on what your fencing device supports and requires.

   You can use the pcmk_host_list parameter when creating a fencing device to specify all of the machines that are controlled by that fencing device.

   You can use pcmk_host_map parameter when creating a fencing device to map host names to the specifications that comprehends the fence device.

3. Create a fence device.

   # pcs stonith create clusterfence fence_azure_arm

Verification

1. Test the fencing agent for one of the other nodes.

   # pcs stonith fence azurenodename

   Example:

   [root@node01 clouduser]# pcs status
   Cluster name: newcluster
   Stack: corosync
   Current DC: node01 (version 1.1.18-11.el7-2b07d5c5a9) - partition with quorum
   Last updated: Fri Feb 23 11:44:35 2018
   Last change: Fri Feb 23 11:21:01 2018 by root via cibadmin on node01
   
   3 nodes configured
   1 resource configured
   
   Online: [ node01 node03 ]
   OFFLINE: [ node02 ]
   
   Full list of resources:
   
     clusterfence  (stonith:fence_azure_arm):  Started node01
   
   Daemon Status:
     corosync: active/disabled
     pacemaker: active/disabled
     pcsd: active/enabled

2. Start the node that was fenced in the previous step.

   # pcs cluster start <hostname>

3. Check the status to verify the node started.

   # pcs status

   Example:

   [root@node01 clouduser]# pcs status
   Cluster name: newcluster
   Stack: corosync
   Current DC: node01 (version 1.1.18-11.el7-2b07d5c5a9) - partition with quorum
   Last updated: Fri Feb 23 11:34:59 2018
   Last change: Fri Feb 23 11:21:01 2018 by root via cibadmin on node01
   
   3 nodes configured
   1 resource configured
   
   Online: [ node01 node02 node03 ]
   
   Full list of resources:
   
   clusterfence    (stonith:fence_azure_arm):  Started node01
   
   Daemon Status:
     corosync: active/disabled
     pacemaker: active/disabled
     pcsd: active/enabled

Procedure

1. Create a Basic load balancer. Select Internal load balancer, the Basic SKU, and Dynamic for the type of IP address assignment.
2. Create a back-end address pool. Associate the backend pool to the availability set created while creating Azure resources in HA. Do not set any target network IP configurations.
3. Create a health probe. For the health probe, select TCP and enter port 61000. You can use TCP port number that does not interfere with another service. For certain HA product applications (for example, SAP HANA and SQL Server), you may need to work with Microsoft to identify the correct port to use.
4. Create a load balancer rule. To create the load balancing rule, the default values are prepopulated. Ensure to set Floating IP (direct server return) to Enabled.

Procedure

1. Install the nmap-ncat resource agents on all nodes.

   # dnf install nmap-ncat resource-agents

   Perform the following steps on a single node.

2. Create the pcs resources and group. Use your load balancer FrontendIP for the IPaddr2 address.

   # pcs resource create resource-name IPaddr2 ip="10.0.0.7" --group cluster-resources-group

3. Configure the load balancer resource agent.

   # pcs resource create resource-loadbalancer-name azure-lb port=port-number --group cluster-resources-group

Procedure

1. Create a shared block volume using the Azure command az disk create.

   $ az disk create -g <resource_group> -n <shared_block_volume_name> --size-gb <disk_size> --max-shares <number_vms> -l <location>

   For example, the following command creates a shared block volume named shared-block-volume.vhd in the resource group sharedblock within the Azure Availability Zone westcentralus.

   $ az disk create -g sharedblock-rg -n shared-block-volume.vhd --size-gb 1024 --max-shares 3 -l westcentralus
   
   {
     "creationData": {
       "createOption": "Empty",
       "galleryImageReference": null,
       "imageReference": null,
       "sourceResourceId": null,
       "sourceUniqueId": null,
       "sourceUri": null,
       "storageAccountId": null,
       "uploadSizeBytes": null
     },
     "diskAccessId": null,
     "diskIopsReadOnly": null,
     "diskIopsReadWrite": 5000,
     "diskMbpsReadOnly": null,
     "diskMbpsReadWrite": 200,
     "diskSizeBytes": 1099511627776,
     "diskSizeGb": 1024,
     "diskState": "Unattached",
     "encryption": {
       "diskEncryptionSetId": null,
       "type": "EncryptionAtRestWithPlatformKey"
     },
     "encryptionSettingsCollection": null,
     "hyperVgeneration": "V1",
     "id": "/subscriptions/12345678910-12345678910/resourceGroups/sharedblock-rg/providers/Microsoft.Compute/disks/shared-block-volume.vhd",
     "location": "westcentralus",
     "managedBy": null,
     "managedByExtended": null,
     "maxShares": 3,
     "name": "shared-block-volume.vhd",
     "networkAccessPolicy": "AllowAll",
     "osType": null,
     "provisioningState": "Succeeded",
     "resourceGroup": "sharedblock-rg",
     "shareInfo": null,
     "sku": {
       "name": "Premium_LRS",
       "tier": "Premium"
     },
     "tags": {},
     "timeCreated": "2020-08-27T15:36:56.263382+00:00",
     "type": "Microsoft.Compute/disks",
     "uniqueId": "cd8b0a25-6fbe-4779-9312-8d9cbb89b6f2",
     "zones": null
   }

2. Verify that you have created the shared block volume using the Azure command az disk show.

   $ az disk show -g <resource_group> -n <shared_block_volume_name>

   For example, the following command shows details for the shared block volume shared-block-volume.vhd within the resource group sharedblock-rg.

   $ az disk show -g sharedblock-rg -n shared-block-volume.vhd
   
   {
     "creationData": {
       "createOption": "Empty",
       "galleryImageReference": null,
       "imageReference": null,
       "sourceResourceId": null,
       "sourceUniqueId": null,
       "sourceUri": null,
       "storageAccountId": null,
       "uploadSizeBytes": null
     },
     "diskAccessId": null,
     "diskIopsReadOnly": null,
     "diskIopsReadWrite": 5000,
     "diskMbpsReadOnly": null,
     "diskMbpsReadWrite": 200,
     "diskSizeBytes": 1099511627776,
     "diskSizeGb": 1024,
     "diskState": "Unattached",
     "encryption": {
       "diskEncryptionSetId": null,
       "type": "EncryptionAtRestWithPlatformKey"
     },
     "encryptionSettingsCollection": null,
     "hyperVgeneration": "V1",
     "id": "/subscriptions/12345678910-12345678910/resourceGroups/sharedblock-rg/providers/Microsoft.Compute/disks/shared-block-volume.vhd",
     "location": "westcentralus",
     "managedBy": null,
     "managedByExtended": null,
     "maxShares": 3,
     "name": "shared-block-volume.vhd",
     "networkAccessPolicy": "AllowAll",
     "osType": null,
     "provisioningState": "Succeeded",
     "resourceGroup": "sharedblock-rg",
     "shareInfo": null,
     "sku": {
       "name": "Premium_LRS",
       "tier": "Premium"
     },
     "tags": {},
     "timeCreated": "2020-08-27T15:36:56.263382+00:00",
     "type": "Microsoft.Compute/disks",
     "uniqueId": "cd8b0a25-6fbe-4779-9312-8d9cbb89b6f2",
     "zones": null
   }

3. Create three network interfaces using the Azure command az network nic create. Run the following command three times using a different <nic_name> for each.

   $ az network nic create \
       -g <resource_group> -n <nic_name> --subnet <subnet_name> \
       --vnet-name <virtual_network> --location <location> \
       --network-security-group <network_security_group> --private-ip-address-version IPv4

   For example, the following command creates a network interface with the name shareblock-nodea-vm-nic-protected.

   $ az network nic create \
       -g sharedblock-rg -n sharedblock-nodea-vm-nic-protected --subnet sharedblock-subnet-protected \
       --vnet-name sharedblock-vn --location westcentralus \
       --network-security-group sharedblock-nsg --private-ip-address-version IPv4

4. Create three VMs and attach the shared block volume using the Azure command az vm create. Option values are the same for each VM except that each VM has its own <vm_name>, <new_vm_disk_name>, and <nic_name>.

   $ az vm create \
       -n <vm_name> -g <resource_group> --attach-data-disks <shared_block_volume_name> \
       --data-disk-caching None --os-disk-caching ReadWrite --os-disk-name <new-vm-disk-name> \
       --os-disk-size-gb <disk_size> --location <location> --size <virtual_machine_size> \
       --image <image_name> --admin-username <vm_username> --authentication-type ssh \
       --ssh-key-values <ssh_key> --nics <nic_name> --availability-set <availability_set> --ppg <proximity_placement_group>

   For example, the following command creates a VM named sharedblock-nodea-vm.

   $ az vm create \
   -n sharedblock-nodea-vm -g sharedblock-rg --attach-data-disks shared-block-volume.vhd \
   --data-disk-caching None --os-disk-caching ReadWrite --os-disk-name sharedblock-nodea-vm.vhd \
   --os-disk-size-gb 64 --location westcentralus --size Standard_D2s_v3 \
   --image /subscriptions/12345678910-12345678910/resourceGroups/sample-azureimagesgroupwestcentralus/providers/Microsoft.Compute/images/sample-azure-rhel-9.3.0-20200713.n.0.x86_64 --admin-username sharedblock-user --authentication-type ssh \
   --ssh-key-values @sharedblock-key.pub --nics sharedblock-nodea-vm-nic-protected --availability-set sharedblock-as --ppg sharedblock-ppg
   
   {
     "fqdns": "",
     "id": "/subscriptions/12345678910-12345678910/resourceGroups/sharedblock-rg/providers/Microsoft.Compute/virtualMachines/sharedblock-nodea-vm",
     "location": "westcentralus",
     "macAddress": "00-22-48-5D-EE-FB",
     "powerState": "VM running",
     "privateIpAddress": "198.51.100.3",
     "publicIpAddress": "",
     "resourceGroup": "sharedblock-rg",
     "zones": ""
   }

Verification

1. For each VM in your cluster, verify that the block device is available by using the ssh command with your VM’s IP address.

   # ssh <ip_address> "hostname ; lsblk -d | grep ' 1T '"

   For example, the following command lists details including the host name and block device for the VM IP 198.51.100.3.

   # ssh 198.51.100.3 "hostname ; lsblk -d | grep ' 1T '"
   
   nodea
   sdb    8:16   0    1T  0 disk

2. Use the ssh command to verify that each VM in your cluster uses the same shared disk.

   # ssh <ip_address> "hostname ; lsblk -d | grep ' 1T ' | awk '{print \$1}' | xargs -i udevadm info --query=all --name=/dev/{} | grep '^E: ID_SERIAL='"

   For example, the following command lists details including the host name and shared disk volume ID for the instance IP address 198.51.100.3.

   # ssh 198.51.100.3 "hostname ; lsblk -d | grep ' 1T ' | awk '{print \$1}' | xargs -i udevadm info --query=all --name=/dev/{} | grep '^E: ID_SERIAL='"
   
   nodea
   E: ID_SERIAL=3600224808dd8eb102f6ffc5822c41d89

Procedure

1. Launch the AWS Console.
2. Click on your AWS Account ID to display the drop-down menu and select My Security Credentials.
3. Click Users.
4. Select the user and open the Summary screen.
5. Click the Security credentials tab.
6. Click Create access key.
7. Download the .csv file (or save both keys). You need to enter these keys when creating the fencing device.

Procedure

1. Install the AWS command line tools using the dnf command.

   # dnf install awscli

2. Use the aws --version command to verify that you installed the AWS CLI.

   $ aws --version
   aws-cli/1.19.77 Python/3.6.15 Linux/5.14.16-201.fc34.x86_64 botocore/1.20.77

3. Configure the AWS command line client according to your AWS access details.

   $ aws configure
   AWS Access Key ID [None]:
   AWS Secret Access Key [None]:
   Default region name [None]:
   Default output format [None]:

Procedure

1. From the AWS EC2 Dashboard, select Images and then AMIs.
2. Right-click on your image and select Launch.

3. Choose an Instance Type that meets or exceeds the requirements of your workload. Depending on your HA application, each instance may need to have higher capacity.

   See Amazon EC2 Instance Types for information on instance types.

4. Click Next: Configure Instance Details.

   1. Enter the Number of instances you want to create for the cluster. This example procedure uses three cluster nodes.

      Note

      Do not launch into an Auto Scaling Group.

   2. For Network, select the VPC you created in Set up the AWS environment. Select the subnet for the instance to create a new subnet.

   3. Select Enable for Auto-assign Public IP. These are the minimum selections you need to make for Configure Instance Details. Depending on your specific HA application, you may need to make additional selections.

      Note

      These are the minimum configuration options necessary to create a basic instance. Review additional options based on your HA application requirements.

5. Click Next: Add Storage and verify that the default storage is sufficient. You do not need to modify these settings unless your HA application requires other storage options.

6. Click Next: Add Tags.

   Note

   Tags can help you manage your AWS resources. See Tagging Your Amazon EC2 Resources for information on tagging.

7. Click Next: Configure Security Group. Select the existing security group you created in Setting up the AWS environment.
8. Click Review and Launch and verify your selections.
9. Click Launch. You are prompted to select an existing key pair or create a new key pair. Select the key pair you created when Setting up the AWS environment.
10. Click Launch Instances.

11. Click View Instances. You can name the instance(s).

    Note

    Alternatively, you can launch instances using the AWS CLI. See Launching, Listing, and Terminating Amazon EC2 Instances in the Amazon documentation for more information.

Procedure

1. Move the key file from the Downloads directory to your Home directory or to your ~/.ssh directory.

2. Change the permissions of the key file so that only the root user can read it.

   # chmod 400 KeyName.pem

Procedure

1. Launch the AWS Console and select the EC2 instance.
2. Click Connect and select A standalone SSH client.
3. From your SSH terminal session, connect to the instance using the AWS example provided in the pop-up window. Add the correct path to your KeyName.pem file if the path is not shown in the example.

Procedure

1. Remove the AWS Red Hat Update Infrastructure (RHUI) client. Because you are going to use a Red Hat Cloud Access subscription, you should not use AWS RHUI in addition to your subscription.

   $ sudo -i
   # dnf -y remove rh-amazon-rhui-client*

2. Register the VM with Red Hat.

   # subscription-manager register --auto-attach

3. Disable all repositories.

   # subscription-manager repos --disable=*

4. Enable the RHEL 9 Server HA repositories.

   # subscription-manager repos --enable=rhel-9-for-x86_64-highavailability-rpms

5. Update the RHEL AWS instance.

   # dnf update -y

6. Install the Red Hat High Availability Add-On software packages, along with all available fencing agents from the High Availability channel.

   # dnf install pcs pacemaker fence-agents-aws

7. The user hacluster was created during the pcs and pacemaker installation in the previous step. Create a password for hacluster on all cluster nodes. Use the same password for all nodes.

   # passwd hacluster

8. Add the high availability service to the RHEL Firewall if firewalld.service is installed.

   # firewall-cmd --permanent --add-service=high-availability
   # firewall-cmd --reload

9. Start the pcs service and enable it to start on boot.

   # systemctl start pcsd.service
   # systemctl enable pcsd.service

10. Edit /etc/hosts and add RHEL host names and internal IP addresses. See How should the /etc/hosts file be set up on RHEL cluster nodes? for details.

Procedure

1. On one of the nodes, enter the following command to authenticate the pcs user hacluster. In the command, specify the name of each node in the cluster.

   # pcs host auth <hostname1> <hostname2> <hostname3>

   Example:

   [root@node01 clouduser]# pcs host auth node01 node02 node03
   Username: hacluster
   Password:
   node01: Authorized
   node02: Authorized
   node03: Authorized

2. Create the cluster.

   # pcs cluster setup <cluster_name> <hostname1> <hostname2> <hostname3>

   Example:

   [root@node01 clouduser]# pcs cluster setup new_cluster node01 node02 node03
   
   [...]
   
   Synchronizing pcsd certificates on nodes node01, node02, node03...
   node02: Success
   node03: Success
   node01: Success
   Restarting pcsd on the nodes in order to reload the certificates...
   node02: Success
   node03: Success
   node01: Success

Verification

1. Enable the cluster.

   [root@node01 clouduser]# pcs cluster enable --all
   node02: Cluster Enabled
   node03: Cluster Enabled
   node01: Cluster Enabled

2. Start the cluster.

   [root@node01 clouduser]# pcs cluster start --all
   node02: Starting Cluster...
   node03: Starting Cluster...
   node01: Starting Cluster...

Standard procedure

1. Enter the following AWS metadata query to get the Instance ID for each node. You need these IDs to configure the fence device. See Instance Metadata and User Data for additional information.

   # echo $(curl -s http://169.254.169.254/latest/meta-data/instance-id)

   Example:

   [root@ip-10-0-0-48 ~]# echo $(curl -s http://169.254.169.254/latest/meta-data/instance-id) i-07f1ac63af0ec0ac6

2. Enter the following command to configure the fence device. Use the pcmk_host_map command to map the RHEL host name to the Instance ID. Use the AWS Access Key and AWS Secret Access Key that you previously set up.

   # pcs stonith \
       create <name> fence_aws access_key=access-key secret_key=<secret-access-key> \
       region=<region> pcmk_host_map="rhel-hostname-1:Instance-ID-1;rhel-hostname-2:Instance-ID-2;rhel-hostname-3:Instance-ID-3" \
       power_timeout=240 pcmk_reboot_timeout=480 pcmk_reboot_retries=4

   Example:

   [root@ip-10-0-0-48 ~]# pcs stonith \
   create clusterfence fence_aws access_key=AKIAI123456MRMJA secret_key=a75EYIG4RVL3hdsdAslK7koQ8dzaDyn5yoIZ/ \
   region=us-east-1 pcmk_host_map="ip-10-0-0-48:i-07f1ac63af0ec0ac6;ip-10-0-0-46:i-063fc5fe93b4167b2;ip-10-0-0-58:i-08bd39eb03a6fd2c7" \
   power_timeout=240 pcmk_reboot_timeout=480 pcmk_reboot_retries=4

Alternate procedure

1. Obtain the VPC ID of the cluster.

   # aws ec2 describe-vpcs --output text --filters "Name=tag:Name,Values=clustername-vpc" --query 'Vpcs[*].VpcId'
   vpc-06bc10ac8f6006664

2. Using the VPC ID of the cluster, obtain the VPC instances.

   $ aws ec2 describe-instances --output text --filters "Name=vpc-id,Values=vpc-06bc10ac8f6006664" --query 'Reservations[*].Instances[*].{Name:Tags[? Key==Name]|[0].Value,Instance:InstanceId}' | grep "\-node[a-c]"
   i-0b02af8927a895137     clustername-nodea-vm
   i-0cceb4ba8ab743b69     clustername-nodeb-vm
   i-0502291ab38c762a5     clustername-nodec-vm

3. Use the obtained instance IDs to configure fencing on each node on the cluster. For example:

   [root@nodea ~]# CLUSTER=clustername && pcs stonith create fence${CLUSTER} fence_aws access_key=XXXXXXXXXXXXXXXXXXXX pcmk_host_map=$(for NODE \
   in node{a..c}; do ssh ${NODE} "echo -n \${HOSTNAME}:\$(curl -s http://169.254.169.254/latest/meta-data/instance-id)\;"; done) \
   pcmk_reboot_retries=4 pcmk_reboot_timeout=480 power_timeout=240 region=xx-xxxx-x secret_key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
   
   [root@nodea ~]# pcs stonith config fence${CLUSTER}
   
   Resource: clustername (class=stonith type=fence_aws)
   Attributes: access_key=XXXXXXXXXXXXXXXXXXXX pcmk_host_map=nodea:i-0b02af8927a895137;nodeb:i-0cceb4ba8ab743b69;nodec:i-0502291ab38c762a5;
   pcmk_reboot_retries=4 pcmk_reboot_timeout=480 power_timeout=240 region=xx-xxxx-x secret_key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
   Operations: monitor interval=60s (clustername-monitor-interval-60s)

Verification

1. Test the fencing agent for one of the cluster nodes.

   # pcs stonith fence awsnodename

   Note

   The command response may take several minutes to display. If you watch the active terminal session for the node being fenced, you see that the terminal connection is immediately terminated after you enter the fence command.

   Example:

   [root@ip-10-0-0-48 ~]# pcs stonith fence ip-10-0-0-58
   
   Node: ip-10-0-0-58 fenced

2. Check the status to verify that the node is fenced.

   # pcs status

   Example:

   [root@ip-10-0-0-48 ~]# pcs status
   
   Cluster name: newcluster
   Stack: corosync
   Current DC: ip-10-0-0-46 (version 1.1.18-11.el7-2b07d5c5a9) - partition with quorum
   Last updated: Fri Mar  2 19:55:41 2018
   Last change: Fri Mar  2 19:24:59 2018 by root via cibadmin on ip-10-0-0-46
   
   3 nodes configured
   1 resource configured
   
   Online: [ ip-10-0-0-46 ip-10-0-0-48 ]
   OFFLINE: [ ip-10-0-0-58 ]
   
   Full list of resources:
   clusterfence  (stonith:fence_aws):    Started ip-10-0-0-46
   
   Daemon Status:
   corosync: active/disabled
   pacemaker: active/disabled
   pcsd: active/enabled

3. Start the node that was fenced in the previous step.

   # pcs cluster start awshostname

4. Check the status to verify the node started.

   # pcs status

   Example:

   [root@ip-10-0-0-48 ~]# pcs status
   
   Cluster name: newcluster
   Stack: corosync
   Current DC: ip-10-0-0-46 (version 1.1.18-11.el7-2b07d5c5a9) - partition with quorum
   Last updated: Fri Mar  2 20:01:31 2018
   Last change: Fri Mar  2 19:24:59 2018 by root via cibadmin on ip-10-0-0-48
   
   3 nodes configured
   1 resource configured
   
   Online: [ ip-10-0-0-46 ip-10-0-0-48 ip-10-0-0-58 ]
   
   Full list of resources:
   
     clusterfence  (stonith:fence_aws):    Started ip-10-0-0-46
   
   Daemon Status:
     corosync: active/disabled
     pacemaker: active/disabled
     pcsd: active/enabled

Procedure

1. Install the AWS CLI. For instructions, see Installing the AWS CLI.

2. Verify that the AWS CLI is configured properly. The instance IDs and instance names should display.

   Example:

   [root@ip-10-0-0-48 ~]# aws ec2 describe-instances --output text --query 'Reservations[].Instances[].[InstanceId,Tags[?Key==Name].Value]'
   
   i-07f1ac63af0ec0ac6
   ip-10-0-0-48
   i-063fc5fe93b4167b2
   ip-10-0-0-46
   i-08bd39eb03a6fd2c7
   ip-10-0-0-58

Procedure

1. View the AWS Secondary Private IP Address resource agent (awsvip) description. This shows the options and default operations for this agent.

   # pcs resource describe awsvip

2. Create the Secondary Private IP address using an unused private IP address in the VPC CIDR block.

   # pcs resource create privip awsvip secondary_private_ip=Unused-IP-Address --group group-name

   Example:

   [root@ip-10-0-0-48 ~]# pcs resource create privip awsvip secondary_private_ip=10.0.0.68 --group networking-group

3. Create a virtual IP resource. This is a VPC IP address that can be rapidly remapped from the fenced node to the failover node, masking the failure of the fenced node within the subnet.

   # pcs resource create vip IPaddr2 ip=secondary-private-IP --group group-name

   Example:

   root@ip-10-0-0-48 ~]# pcs resource create vip IPaddr2 ip=10.0.0.68 --group networking-group

1. Add the two resources to the same group that was previously created to enforce order and colocation constraints.

2. Enter the following AWS CLI command to create an elastic IP address.

   [root@ip-10-0-0-48 ~]# aws ec2 allocate-address --domain vpc --output text
   eipalloc-4c4a2c45   vpc 35.169.153.122

3. View the AWS Secondary Elastic IP Address resource agent (awseip) description. The following command shows the options and default operations for this agent.

   # pcs resource describe awseip

4. Create the Secondary Elastic IP address resource using the allocated IP address created in Step 1.

   # pcs resource create elastic awseip elastic_ip=Elastic-IP-Address allocation_id=Elastic-IP-Association-ID --group networking-group

   Example:

   # pcs resource create elastic awseip elastic_ip=35.169.153.122 allocation_id=eipalloc-4c4a2c45 --group networking-group

Procedure

1. Launch an SSH session from your local workstation to the elastic IP address previously created.

   $ ssh -l ec2-user -i ~/.ssh/<KeyName>.pem elastic-IP

   Example:

   $ ssh -l ec2-user -i ~/.ssh/cluster-admin.pem 35.169.153.122

2. Verify that the host you connected to via SSH is the host associated with the elastic resource created.

Procedure

1. Create a shared block volume using the AWS command create-volume.

   $ aws ec2 create-volume --availability-zone <availability_zone> --no-encrypted --size 1024 --volume-type io1 --iops 51200 --multi-attach-enabled

   For example, the following command creates a volume in the us-east-1a availability zone.

   $ aws ec2 create-volume --availability-zone us-east-1a --no-encrypted --size 1024 --volume-type io1 --iops 51200 --multi-attach-enabled
   
   {
       "AvailabilityZone": "us-east-1a",
       "CreateTime": "2020-08-27T19:16:42.000Z",
       "Encrypted": false,
       "Size": 1024,
       "SnapshotId": "",
       "State": "creating",
       "VolumeId": "vol-042a5652867304f09",
       "Iops": 51200,
       "Tags": [ ],
       "VolumeType": "io1"
   }

   Note

   You need the VolumeId in the next step.

2. For each instance in your cluster, attach a shared block volume using the AWS command attach-volume. Use your <instance_id> and <volume_id>.

   $ aws ec2 attach-volume --device /dev/xvdd --instance-id <instance_id> --volume-id <volume_id>

   For example, the following command attaches a shared block volume vol-042a5652867304f09 to instance i-0eb803361c2c887f2.

   $ aws ec2 attach-volume --device /dev/xvdd --instance-id i-0eb803361c2c887f2 --volume-id vol-042a5652867304f09
   
   {
       "AttachTime": "2020-08-27T19:26:16.086Z",
       "Device": "/dev/xvdd",
       "InstanceId": "i-0eb803361c2c887f2",
       "State": "attaching",
       "VolumeId": "vol-042a5652867304f09"
   }

Verification

1. For each instance in your cluster, verify that the block device is available by using the ssh command with your instance <ip_address>.

   # ssh <ip_address> "hostname ; lsblk -d | grep ' 1T '"

   For example, the following command lists details including the host name and block device for the instance IP 198.51.100.3.

   # ssh 198.51.100.3 "hostname ; lsblk -d | grep ' 1T '"
   
   nodea
   nvme2n1 259:1    0   1T  0 disk

2. Use the ssh command to verify that each instance in your cluster uses the same shared disk.

   # ssh <ip_address> "hostname ; lsblk -d | grep ' 1T ' | awk '{print \$1}' | xargs -i udevadm info --query=all --name=/dev/{} | grep '^E: ID_SERIAL='"

   For example, the following command lists details including the host name and shared disk volume ID for the instance IP address 198.51.100.3.

   # ssh 198.51.100.3 "hostname ; lsblk -d | grep ' 1T ' | awk '{print \$1}' | xargs -i udevadm info --query=all --name=/dev/{} | grep '^E: ID_SERIAL='"
   
   nodea
   E: ID_SERIAL=Amazon Elastic Block Store_vol0fa5342e7aedf09f7

Procedure

1. Follow the GCP instructions for downloading and extracting the Google Cloud SDK archive. See the GCP document Quickstart for Linux for details.

2. Follow the same instructions for initializing the Google Cloud SDK.

   Note

   Once you have initialized the Google Cloud SDK, you can use the gcloud CLI commands to perform tasks and obtain information about your project and instances. For example, you can display project information with the gcloud compute project-info describe --project <project-name> command.

Procedure

1. If you are not already logged in to Google Cloud Platform, log in with the following command.

   # gcloud auth login

2. Create a storage bucket.

   $ gsutil mb gs://BucketName

   Example:

   $ gsutil mb gs://rhel-ha-bucket

Procedure

1. Launch the GCP Console.
2. Select VPC networks under Networking in the left navigation pane.
3. Click Create VPC Network.
4. Enter a name for the VPC network.
5. Under the New subnet, create a Custom subnet in the region where you want to create the cluster.
6. Click Create.

Procedure

1. Convert the file. Images uploaded to GCP must be in raw format and named disk.raw.

   $ qemu-img convert -f qcow2 ImageName.qcow2 -O raw disk.raw

2. Compress the raw file. Images uploaded to GCP must be compressed.

   $ tar -Sczf ImageName.tar.gz disk.raw

3. Import the compressed image to the bucket created earlier.

   $ gsutil cp ImageName.tar.gz gs://BucketName

Procedure

1. Create an image from the compressed file in the bucket.

   $ gcloud compute images create BaseImageName --source-uri gs://BucketName/BaseImageName.tar.gz

   Example:

   [admin@localhost ~] $ gcloud compute images create rhel-76-server --source-uri gs://user-rhelha/rhel-server-76.tar.gz
   Created [https://www.googleapis.com/compute/v1/projects/MyProject/global/images/rhel-server-76].
   NAME            PROJECT                 FAMILY  DEPRECATED  STATUS
   rhel-76-server  rhel-ha-testing-on-gcp                      READY

2. Create a template instance from the image. The minimum size required for a base RHEL instance is n1-standard-2. See gcloud compute instances create for additional configuration options.

   $ gcloud compute instances create BaseInstanceName --can-ip-forward --machine-type n1-standard-2 --image BaseImageName --service-account ServiceAccountEmail

   Example:

   [admin@localhost ~] $ gcloud compute instances create rhel-76-server-base-instance --can-ip-forward --machine-type n1-standard-2 --image rhel-76-server --service-account account@project-name-on-gcp.iam.gserviceaccount.com
   Created [https://www.googleapis.com/compute/v1/projects/rhel-ha-testing-on-gcp/zones/us-east1-b/instances/rhel-76-server-base-instance].
   NAME   ZONE   MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
   rhel-76-server-base-instance  us-east1-bn1-standard-2          10.10.10.3   192.227.54.211  RUNNING

3. Connect to the instance with an SSH terminal session.

   $ ssh root@PublicIPaddress

4. Update the RHEL software.

   1. Register with Red Hat Subscription Manager (RHSM).
   2. Enable a Subscription Pool ID (or use the --auto-attach command).

   3. Disable all repositories.

      # subscription-manager repos --disable=*

   4. Enable the following repository.

      # subscription-manager repos --enable=rhel-9-server-rpms

   5. Run the dnf update command.

      # dnf update -y

5. Install the GCP Linux Guest Environment on the running instance (in-place installation).

   See Install the guest environment in-place for instructions.

6. Select the CentOS/RHEL option.
7. Copy the command script and paste it at the command prompt to run the script immediately.

8. Make the following configuration changes to the instance. These changes are based on GCP recommendations for custom images. See gcloudcompute images list for more information.

   1. Edit the /etc/chrony.conf file and remove all NTP servers.

   2. Add the following NTP server.

      metadata.google.internal iburst Google NTP server

   3. Remove any persistent network device rules.

      # rm -f /etc/udev/rules.d/70-persistent-net.rules
      
      # rm -f /etc/udev/rules.d/75-persistent-net-generator.rules

   4. Set the network service to start automatically.

      # chkconfig network on

   5. Set the sshd service to start automatically.

      # systemctl enable sshd
      # systemctl is-enabled sshd

   6. Set the time zone to UTC.

      # ln -sf /usr/share/zoneinfo/UTC /etc/localtime

   7. (Optional) Edit the /etc/ssh/ssh_config file and add the following lines to the end of the file. This keeps your SSH session active during longer periods of inactivity.

      # Server times out connections after several minutes of inactivity.
      # Keep alive ssh connections by sending a packet every 7 minutes.
      ServerAliveInterval 420

   8. Edit the /etc/ssh/sshd_config file and make the following changes, if necessary. The ClientAliveInterval 420 setting is optional; this keeps your SSH session active during longer periods of inactivity.

      PermitRootLogin no
      PasswordAuthentication no
      AllowTcpForwarding yes
      X11Forwarding no
      PermitTunnel no
      # Compute times out connections after 10 minutes of inactivity.
      # Keep ssh connections alive by sending a packet every 7 minutes.
      ClientAliveInterval 420

9. Disable password access.

   ssh_pwauth from 1 to 0.
   ssh_pwauth: 0

   Important

   Previously, you enabled password access to allow SSH session access to configure the instance. You must disable password access. All SSH session access must be passwordless.

10. Unregister the instance from the subscription manager.

    # subscription-manager unregister

11. Clean the shell history. Keep the instance running for the next procedure.

    # export HISTSIZE=0

Procedure

1. On the running instance, synchronize data to disk.

   # sync

2. On your host system, create the snapshot.

   $ gcloud compute disks snapshot InstanceName --snapshot-names SnapshotName

3. On your host system, create the configured image from the snapshot.

   $ gcloud compute images create ConfiguredImageFromSnapshot --source-snapshot SnapshotName

Procedure

1. Create an instance template.

   $ gcloud compute instance-templates create InstanceTemplateName --can-ip-forward --machine-type n1-standard-2 --image ConfiguredImageFromSnapshot --service-account ServiceAccountEmailAddress

   Example:

   [admin@localhost ~] $ gcloud compute instance-templates create rhel-91-instance-template --can-ip-forward --machine-type n1-standard-2 --image rhel-91-gcp-image --service-account account@project-name-on-gcp.iam.gserviceaccount.com
   Created [https://www.googleapis.com/compute/v1/projects/project-name-on-gcp/global/instanceTemplates/rhel-91-instance-template].
   NAME  MACHINE_TYPE   PREEMPTIBLE  CREATION_TIMESTAMP
   rhel-91-instance-template   n1-standard-2          2018-07-25T11:09:30.506-07:00

2. Create multiple nodes in one zone.

   # gcloud compute instances create NodeName01 NodeName02 --source-instance-template InstanceTemplateName --zone RegionZone --network=NetworkName --subnet=SubnetName

   Example:

   [admin@localhost ~] $ gcloud compute instances create rhel81-node-01 rhel81-node-02 rhel81-node-03 --source-instance-template rhel-91-instance-template --zone us-west1-b --network=projectVPC --subnet=range0
   Created [https://www.googleapis.com/compute/v1/projects/project-name-on-gcp/zones/us-west1-b/instances/rhel81-node-01].
   Created [https://www.googleapis.com/compute/v1/projects/project-name-on-gcp/zones/us-west1-b/instances/rhel81-node-02].
   Created [https://www.googleapis.com/compute/v1/projects/project-name-on-gcp/zones/us-west1-b/instances/rhel81-node-03].
   NAME            ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
   rhel81-node-01  us-west1-b  n1-standard-2               10.10.10.4   192.230.25.81   RUNNING
   rhel81-node-02  us-west1-b  n1-standard-2               10.10.10.5   192.230.81.253  RUNNING
   rhel81-node-03  us-east1-b  n1-standard-2               10.10.10.6   192.230.102.15  RUNNING

Procedure

1. In the Google Cloud Console, select Compute Engine and then select VM instances.
2. Select the instance, click the arrow next to SSH, and select the View gcloud command option.
3. Paste this command at a command prompt for passwordless access to the instance.
4. Enable sudo account access and register with Red Hat Subscription Manager.
5. Enable a Subscription Pool ID (or use the --auto-attach command).

6. Disable all repositories.

   # subscription-manager repos --disable=*

7. Enable the following repositories.

   # subscription-manager repos --enable=rhel-9-server-rpms
   # subscription-manager repos --enable=rhel-9-for-x86_64-highavailability-rpms

8. Install pcs pacemaker, the fence agents, and the resource agents.

   # dnf install -y pcs pacemaker fence-agents-gce resource-agents-gcp

9. Update all packages.

   # dnf update -y

Procedure

1. The user hacluster was created during the pcs and pacemaker installation in the previous step. Create a password for the user hacluster on all cluster nodes. Use the same password for all nodes.

   # passwd hacluster

2. If the firewalld service is installed, add the HA service.

   # firewall-cmd --permanent --add-service=high-availability
   
   # firewall-cmd --reload

3. Start the pcs service and enable it to start on boot.

   # systemctl start pcsd.service
   
   # systemctl enable pcsd.service
   
   Created symlink from /etc/systemd/system/multi-user.target.wants/pcsd.service to /usr/lib/systemd/system/pcsd.service.

Verification

1. Ensure the pcsd service is running.

   # systemctl status pcsd.service
   
   pcsd.service - PCS GUI and remote configuration interface
   Loaded: loaded (/usr/lib/systemd/system/pcsd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-06-25 19:21:42 UTC; 15s ago
   Docs: man:pcsd(8)
   man:pcs(8)
   Main PID: 5901 (pcsd)
   CGroup: /system.slice/pcsd.service
   └─5901 /usr/bin/ruby /usr/lib/pcsd/pcsd > /dev/null &

2. Edit the /etc/hosts file. Add RHEL host names and internal IP addresses for all nodes.

Procedure

1. On one of the nodes, authenticate the pcs user. Specify the name of each node in the cluster in the command.

   # pcs host auth hostname1 hostname2 hostname3
   Username: hacluster
   Password:
   hostname1: Authorized
   hostname2: Authorized
   hostname3: Authorized

2. Create the cluster.

   # pcs cluster setup cluster-name hostname1 hostname2 hostname3

Verification

1. Run the following command to enable nodes to join the cluster automatically when started.

   # pcs cluster enable --all

2. Start the cluster.

   # pcs cluster start --all

Procedure

1. Obtain GCP instance names. Note that the output of the following command also shows the internal ID for the instance.

   # fence_gce --zone us-west1-b --project=rhel-ha-on-gcp -o list

   Example:

   [root@rhel81-node-01 ~]# fence_gce --zone us-west1-b --project=rhel-ha-testing-on-gcp -o list
   
   4435801234567893181,InstanceName-3
   4081901234567896811,InstanceName-1
   7173601234567893341,InstanceName-2

2. Create a fence device.

   # pcs stonith create FenceDeviceName fence_gce zone=Region-Zone project=MyProject

Procedure

1. Create the aliasip resource. Include an unused internal IP address. Include the CIDR block in the command.

   # pcs resource create aliasip gcp-vpc-move-vip alias_ip=UnusedIPaddress/CIDRblock

   Example:

   [root@rhel81-node-01 ~]# pcs resource create aliasip gcp-vpc-move-vip alias_ip=10.10.10.200/32

2. Create an IPaddr2 resource for managing the IP on the node.

   # pcs resource create vip IPaddr2 nic=interface ip=AliasIPaddress cidr_netmask=32

   Example:

   [root@rhel81-node-01 ~]# pcs resource create vip IPaddr2 nic=eth0 ip=10.10.10.200 cidr_netmask=32

3. Group the network resources under vipgrp.

   # pcs resource group add vipgrp aliasip vip

Verification

1. Verify that the resources have started and are grouped under vipgrp.

   # pcs status

2. Verify that the resource can move to a different node.

   # pcs resource move vip Node

   Example:

   [root@rhel81-node-01 ~]# pcs resource move vip rhel81-node-03

3. Verify that the vip successfully started on a different node.

   # pcs status

Procedure

1. Create a secondary subnet address range.

   # gcloud-ra compute networks subnets update SubnetName --region RegionName --add-secondary-ranges SecondarySubnetName=SecondarySubnetRange

   Example:

   # gcloud-ra compute networks subnets update range0 --region us-west1 --add-secondary-ranges range1=10.10.20.0/24

2. Create the aliasip resource. Create an unused internal IP address in the secondary subnet address range. Include the CIDR block in the command.

   # pcs resource create aliasip gcp-vpc-move-vip alias_ip=UnusedIPaddress/CIDRblock

   Example:

   [root@rhel81-node-01 ~]# pcs resource create aliasip gcp-vpc-move-vip alias_ip=10.10.20.200/32

3. Create an IPaddr2 resource for managing the IP on the node.

   # pcs resource create vip IPaddr2 nic=interface ip=AliasIPaddress cidr_netmask=32

   Example:

   [root@rhel81-node-01 ~]# pcs resource create vip IPaddr2 nic=eth0 ip=10.10.20.200 cidr_netmask=32

4. Group the network resources under vipgrp.

   # pcs resource group add vipgrp aliasip vip

Verification

1. Verify that the resources have started and are grouped under vipgrp.

   # pcs status

2. Verify that the resource can move to a different node.

   # pcs resource move vip Node

   Example:

   [root@rhel81-node-01 ~]# pcs resource move vip rhel81-node-03

3. Verify that the vip successfully started on a different node.

   # pcs status