Configure SAP HANA System Replication in Pacemaker on Amazon Web Services

Updated -

For feedback on this preview version feel free to use comment section at the end of this page.

1. Overview

This article describes how to configure SAP HANA System Replication in a scale-up scenario, in a pacemaker-based cluster on supported Red Hat Enterprise Linux (RHEL) virtual machines on Amazon Web Services. For more generic documentation check the SAP HANA system replication in pacemaker cluster.

1.1 Supported Scenarios

See: Support Policies for RHEL High Availability Clusters - Management of SAP HANA in a Cluster.

Note the parts specific to Amazon Web Services.

1.2. System registration and required repositories

Make sure that system has attached subscription providing EUS or E4S repositories and enable them as described in the articles below.

2. Required AWS Configurations

2.1. Initial AWS Setup

For instructions on the initial setup of the AWS environment, please refer to Installing and Configuring a Red Hat Enterprise Linux 7.4 (and later) High-Availability Cluster on Amazon Web Services.

As a summary, you should have configured the following components:
- An AWS account
- A Key Pair
- An IAM user with permissions to: modify routing tables and create security groups, create IAM policies and roles
- A VPC
- 3 subnets: one public subnet, and two private subnets spanning in two different availability zones (that's recommended to minimize the service disruption related to zone-wise failures. However single availability zone is also acceptable)
- NAT Gateway
- Security Group for the jump server
- Security Group for the HANA cluster nodes

2.2. Choose the Supported SAP HANA Instance Type and Storage

Instance Type of the two HANA nodes:

Please follow the guidelines in the SAP HANA Quick Start Guide, Planning the Deployment, on instance types, storage, memory sizing, and Multi-AZ deployments.

  • It's recommended to create the two nodes in different availability zones.
  • To enable access, copy your private key to the jump server, and each of the HANA nodes.
  • The hostname should meet the SAP HANA requirement. Optionally, change hostname following the instructions.

In Services -> EC2 -> IMAGES -> AMIs choose one of the following AMIs:

  • RHEL 7.4: RHEL-7.4_HVM-20180122-x86_64-1-Hourly2-GP2

Instance Type of Jump Server:

  • You can select an instance type from the free tier, e.g. t2.micro with 30GiB EBS storage volume.

After the HANA nodes are created, note the following information for use in the next section:

  • Region: e.g. us-east-1
  • Account ID of the AWS user account
  • Instance ID of the two HANA nodes

2.3. Create Policies

For the IAM user, you need to create three policies:

Services -> IAM -> Policies -> Create DataProvider Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "EC2:DescribeInstances",
                "EC2:DescribeVolumes"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "cloudwatch:GetMetricStatistics",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::aws-data-provider/config.properties"
        }
    ]
}

Services -> IAM -> Policies -> Create OverlayIPAgent Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1424870324000",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceAttribute",
                "ec2:DescribeTags",
                "ec2:DescribeRouteTables"

            ],
            "Resource": "*"
        },
        {
            "Sid": "Stmt1424860166260",
            "Action": [
                "ec2:CreateRoute",
                "ec2:DeleteRoute",
                "ec2:ReplaceRoute"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:ec2:<region>:<account-id>:route-table/<ClusterRouteTableID>"
        }
    ]
}

In the last Resource clause, replace the following parameters with the real values:

  • region: e.g. us-east-1
  • account-id: the account ID of the user account
  • ClusterRouteTableID: the route table ID for the existing cluster VPC route table, in format of rtb-XXXXX.

Services -> IAM -> Policies -> Create STONITH Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1424870324000",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceAttribute",
                "ec2:DescribeTags"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Stmt1424870324001",
            "Effect": "Allow",
            "Action": [
                "ec2:ModifyInstanceAttribute",
                "ec2:RebootInstances",
                "ec2:StartInstances",
                "ec2:StopInstances"
            ],
            "Resource": [
                "arn:aws:ec2:<region>:<account-id>:instance/<instance-id-node1>",
                "arn:aws:ec2:<region>:<account-id>:instance/<instance-id-node2>"
            ]
        }
    ]
}

In the last Resource clause, replace the following parameters with the real values:

  • region: e.g. us-east-1
  • account-id: the account ID of the user account
  • instance-id-node1, instance-id-node2: instance ID of the two SAP HANA instances

2.4. Create an IAM Role

Create an IAM Role, attached the 3 policies that are created in previous step, and assign the role to the two HANA instances.

  • In EC2 -> IAM -> Roles -> Create a new role, e.g. PacemakerRole, attach the 3 policies to it: DataProvider, OverlayIPAgent, and STONITH
  • Assign the role to the HANA instances: Perform for BOTH HANA nodes: in AWS EC2 console, right click the HANA node -> Instance Settings -> Attach/Replace IAM Role -> Select PacemakerRole, click Apply.

2.5. Install AWS CLI

Follow the Install the AWS CLI section to install and verify the AWS CLI configuration on both nodes.

3. Install SAP HANA and setup HANA SR

Installation of SAP HANA and setup of HANA SR is not specific to AWS environemnt and you can follow the usual procedures:

4. Configure SAP HANA in a pacemaker cluster on AWS

Install and setup pacemaker cluster in AWS and then configure and test fencing as described in all following sections of Installing and Configuring a Red Hat Enterprise Linux 7.4 (and later) High-Availability Cluster on Amazon Web Services article:

At this point this guide assumes that following is true:

4.1. Create `SAPHanaTopology` and `SAPHana` resources

Follow documentation below to create SAPHanaTopology and SAPHana resources:

4.2. Configure the OverlayIP Resource Agent

  1. Disable "Source/Destination Check" on BOTH HANA nodes:
    Perform for BOTH HANA nodes: in AWS EC2 console, right click the HANA node -> Networking -> Change Source/Des. Check -> In the pop up window, click Yes, Disable.

  2. On ONE node in the cluster, enter the following command:

    [root]# aws ec2 create-route --route-table-id <ClusterRouteTableID> --destination-cidr-block <NewCIDRblockIP/NetMask> --instance-id <ClusterNodeID>
    

    Notes:

    • ClusterRouteTableID is the route table ID for the existing cluster VPC route table.
    • NewCIDRblockIP/NetMask is a new IP address and netmask outside of the VPC CIDR block. For example, if the VPC CIDR block is 172.31.0.0/16, the new IP address/netmask could be 192.168.0.15/32.
    • ClusterNodeID is the instance ID for another node in the cluster.
  3. Create the aws-vpc-move-ip resource
    Choose a free IP for the virtual IP address, that the client can access HANA, e.g. 192.168.0.15. On BOTH nodes, modify the /etc/hosts file:

    [root]# vi /etc/hosts
    192.168.0.15 hahadb
    

    On ONE node in the cluster, create the aws-vpc-move-ip resource, e.g. vpcip:

    [root]# pcs resource create vpcip aws-vpc-move-ip ip=192.168.0.15 interface=eth0 routing_table=<ClusterRouteTableID> monapi=true
    
  4. Test failover of the resource
    Don't forget to clear the automatically created constraint after the succesful move of vpcip resource

    [root]# pcs resource move vpcip
    [root]# pcs resource clear vpcip 
    

4.3. Configure constraints

For correct operation we need to ensure that SAPHanaTopology resources are started before starting the SAPHana resources and also that the virtual IP address is present on the node where the Master resource of SAPHana is running. To achieve this, the following 2 constraints need to be created.

4.3.1. constraint - start `SAPHanaTopology` before `SAPHana`

Follow same steps as in 4.5.1. constraint - start SAPHanaTopology before SAPHana.

4.3.2. constraint - colocate `aws-vpc-move-ip` resource with Master of `SAPHana` resource

Below is an example command that will colocate the aws-vpc-move-ip resource with SAPHana resource that was promoted as Master.

[root]# pcs constraint colocation add vpcip with master SAPHana_RH1_00-master 2000

Note that the constraint is using a score of 2000 instead of the default INFINITY. This allows the aws-vpc-move-ip resource to be taken down by the cluster in case there is no Master promoted in the SAPHana resource so it is still possible to use this address with tools like SAP Management Console or SAP LVM that can use this address to query the status information about the SAP Instance.

The resulting constraint should look like one in the example below.

[root]# pcs constraint
...
Colocation Constraints:
  vpcip with SAPHana_RH1_00-master (score:2000) (rsc-role:Started) (with-rsc-role:Master)
...

4.4. Test failover of `SAPHana` resource

There are several ways how to simulate a failure to test the proper functionality of cluster in such case. You can for example Stop the master node in the AWS EC2 console, or simulate a crash and panic the master node with sysrq-trigger:

[root]# echo c > /proc/sysrq-trigger

Kill the HANA instance on the master node, by running the command:

[rh2adm]# HDB kill-9