Automated SAP HANA System Replication in Scale-Up in pacemaker cluster

Updated -

For feedback on this document feel free to use comment section at the end of this page.

1. Overview

This article describes how to configure Automated HANA System Replication in Scale-Up in a Pacemaker cluster on supported RHEL releases.

This article does NOT cover preparation of a RHEL system for SAP HANA installation nor the SAP HANA installation procedure. For more details on these topics refer to SAP Note 2009879 - SAP HANA Guidelines for RedHat Enterprise Linux (RHEL).

1.1. Supported scenarios

See: Support Policies for RHEL High Availability Clusters - Management of SAP HANA in a Cluster

1.2. Subscription and Repos

The following repos are required:

  • RHEL Server: provides the RHEL kernel packages
  • RHEL HA Add-On: provides the Pacemaker framework
  • RHEL for SAP HANA: provides the resource agents for the automation of HANA System Replication in Scale-Up

1.2.1. On-Premise or Bring Your Own Subscription through Cloud Access

For on-premise or Bring Your Own Subscription through Red Hat Cloud Access, the subscription to use is RHEL for SAP Solutions.

Below is the example of repos enabled with RHEL for SAP Solutions 7.6, on-premise or through Cloud Access:

# yum repolist
repo id                                                  repo name                                                                                                status
rhel-7-server-e4s-rpms/7Server/x86_64                    Red Hat Enterprise Linux 7 Server - Update Services for SAP Solutions (RPMs)                             18,929
rhel-ha-for-rhel-7-server-e4s-rpms/7Server/x86_64        Red Hat Enterprise Linux High Availability (for RHEL 7 Server) Update Services for SAP Solutions (RPMs)     437
rhel-sap-hana-for-rhel-7-server-e4s-rpms/7Server/x86_64  RHEL for SAP HANA (for RHEL 7 Server) Update Services for SAP Solutions (RPMs)                               38

1.2.2. On-Demand on Public Clouds through RHUI

For deployment in on-demand images on public cloud, the software packages are delivered in Red Hat Enterprise Linux for SAP with High Availability and Update Services, a variant of RHEL for SAP Solutions, customized for public clouds, available through RHUI.

Below is the example of repos enabled on a RHUI system with RHEL for SAP with High Availability and Update Services 7.5. For configuration of Automated HANA System Replication in Scale-Up, the following repos must present:

# yum repolist
repo id                                                        repo name                                                                  status
rhui-rhel-7-server-rhui-eus-rpms/7.5/x86_64                   Red Hat Enterprise Linux 7 Server - Extended Update Support (RPMs) from RH 21,199
rhui-rhel-ha-for-rhel-7-server-eus-rhui-rpms/7.5/x86_64       Red Hat Enterprise Linux High Availability from RHUI (for RHEL 7 Server) -    501
rhui-rhel-sap-hana-for-rhel-7-server-eus-rhui-rpms/7.5/x86_64 RHEL for SAP HANA (for RHEL 7 Server) Extended Update Support (RPMs) from      43

2. SAP HANA System Replication

The following example shows how to set up system replication between 2 nodes running SAP HANA.

Configuration used in the example:

SID:                   RH2
Instance Number:       02
node1 FQDN:            node1.example.com
node2 FQDN:            node2.example.com
node1 HANA site name:  DC1
node2 HANA site name:  DC2
SAP HANA 'SYSTEM' user password: <HANA_SYSTEM_PASSWORD>
SAP HANA administrative user:    rh2adm

Ensure that both systems can resolve the FQDN of both systems without issues. To ensure that FQDNs can be resolved even without DNS you can place them into /etc/hosts like in the example below.

# /etc/hosts
192.168.0.11 node1.example.com node1
192.168.0.12 node2.example.com node2

For the system replication to work, the SAP HANA log_mode variable must be set to normal. This can be verified as HANA system user using the command below on both nodes.

[rh2adm]# hdbsql -u system -p <HANA_SYSTEM_PASSWORD> -i 02 "select value from "SYS"."M_INIFILE_CONTENTS" where key='log_mode'"
VALUE "normal"
1 row selected

Note that later configuration of primary and secondary node is used only during setup. The roles (primary/secondary) may change during cluster operation based on cluster configuration.

A lot of the configuration steps are performed from the SAP HANA administrative user on the system whose name was selected during installation. In examples we will use rh2adm as we use SID RH2. To become the SAP HANA administrative user you can use the command below.

[root]# sudo -i -u rh2adm
[rh2adm]#

2.1. Configure HANA primary node

SAP HANA system replication will only work after initial backup has been performed. The following command will create an initial backup in /tmp/foo directory. Please note that the size of the backup depends on the database size and may take some time to complete. The directory to which the backup will be placed must by writeable by the SAP HANA administrative user.

a) On single container systems following command can be used for backup:

[rh2adm]# hdbsql -i 02 -u system -p <HANA_SYSTEM_PASSWORD> "BACKUP DATA USING FILE ('/tmp/foo')"
0 rows affected (overall time xx.xxx sec; server time xx.xxx sec)

b) On multiple container systems (MDC) SYSTEMDB and all tenant databases needs to be backed up:

[rh2adm]# hdbsql -i 02 -u system -p <HANA_SYSTEM_PASSWORD> -d SYSTEMDB "BACKUP DATA USING FILE ('/tmp/foo')"
0 rows affected (overall time xx.xxx sec; server time xx.xxx sec)
[rh2adm]# hdbsql -i 02 -u system -p <HANA_SYSTEM_PASSWORD> -d SYSTEMDB "BACKUP DATA FOR RH2 USING FILE ('/tmp/foo-RH2')"
0 rows affected (overall time xx.xxx sec; server time xx.xxx sec)

After the initial backup, initialize the replication using the command below.

[rh2adm]# hdbnsutil -sr_enable --name=DC1
checking for active nameserver ...
nameserver is active, proceeding ...
successfully enabled system as system replication source site
done.

Verify that initialization is showing current node as 'primary' and that SAP HANA is running on it.

[rh2adm]# hdbnsutil -sr_state
checking for active or inactive nameserver ...
System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~
mode: primary
site id: 1
site name: DC1
Host Mappings:

2.2. Configure HANA secondary node

Secondary node needs to be registered to, now running, primary node. SAP HANA on the secondary node must be shut down before using the command bellow.

[rh2adm]# HDB stop

(SAP HANA2.0 only) Copy the SAP HANA system PKI SSFS_RH2.KEY and SSFS_RH2.DAT files from primary node to secondary node.

[rh2adm]# scp root@node1:/usr/sap/RH2/SYS/global/security/rsecssfs/key/SSFS_RH2.KEY /usr/sap/RH2/SYS/global/security/rsecssfs/key/SSFS_RH2.KEY
[rh2adm]# scp root@node1:/usr/sap/RH2/SYS/global/security/rsecssfs/data/SSFS_RH2.DAT /usr/sap/RH2/SYS/global/security/rsecssfs/data/SSFS_RH2.DAT

To register secondary node use the command below.

[rh2adm]# hdbnsutil -sr_register --remoteHost=node1 --remoteInstance=02 --replicationMode=syncmem --name=DC2
adding site ...
checking for inactive nameserver ...
nameserver node2:30201 not responding.
collecting information ...
updating local ini files ...
done.

Start SAP HANA on the secondary node.

[rh2adm]# HDB start

Verify that the secondary node is running and that 'mode' is syncmem. Output should look similar to the output below.

[rh2adm]# hdbnsutil -sr_state
checking for active or inactive nameserver ...

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~
mode: syncmem
site id: 2
site name: DC2
active primary site: 1

Host Mappings:
~~~~~~~~~~~~~~
node2 -> [DC1] node1
node2 -> [DC2] node2

2.3. Testing SAP HANA System Replication

To manually test the SAP HANA System Replication setup you can follow the procedure described in following SAP documents:

2.4. Checking SAP HANA System Replication state

To check the current state of SAP HANA System Replication you can execute the following command as the SAP HANA administrative user on current primary SAP HANA node.

On single_container system:

[rh2adm]# python /usr/sap/RH2/HDB02/exe/python_support/systemReplicationStatus.py

| Host  | Port  | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary     | Replication | Replication | Replication    |
|       |       |              |           |         |           | Host      | Port      | Site ID   | Site Name | Active Status | Mode        | Status      | Status Details |
| ----- | ----- | ------------ | --------- | ------- | --------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| node1 | 30201 | nameserver   |         1 |       1 | DC1       | node2     |     30201 |         2 | DC2       | YES           | SYNCMEM     | ACTIVE      |                |
| node1 | 30207 | xsengine     |         2 |       1 | DC1       | node2     |     30207 |         2 | DC2       | YES           | SYNCMEM     | ACTIVE      |                |
| node1 | 30203 | indexserver  |         3 |       1 | DC1       | node2     |     30203 |         2 | DC2       | YES           | SYNCMEM     | ACTIVE      |                |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: DC1

On multiple_containers system (MDC):

[rh2adm]# python /usr/sap/RH2/HDB02/exe/python_support/systemReplicationStatus.py
| Database | Host  | Port  | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary     | Replication | Replication | Replication    |
|          |       |       |              |           |         |           | Host      | Port      | Site ID   | Site Name | Active Status | Mode        | Status      | Status Details |
| -------- | ----- | ----- | ------------ | --------- | ------- | --------- | ----------| --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- |
| SYSTEMDB | node1 | 30201 | nameserver   |         1 |       1 | DC1       | node2     |     30201 |         2 | DC2       | YES           | SYNCMEM     | ACTIVE      |                |
| RH2      | node1 | 30207 | xsengine     |         2 |       1 | DC1       | node2     |     30207 |         2 | DC2       | YES           | SYNCMEM     | ACTIVE      |                |
| RH2      | node1 | 30203 | indexserver  |         3 |       1 | DC1       | node2     |     30203 |         2 | DC2       | YES           | SYNCMEM     | ACTIVE      |                |

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: DC1

3. Configuring monitoring account in SAP HANA for cluster resource agents (SAP HANA 1.0 SPS12 and earlier)

Starting with SAP HANA 2.0 SPS0 monitoring account is no longer needed
A technical user with CATALOG READ and MONITOR ADMIN privileges must exist in SAP HANA for the resource agents to be able to run queries on the system replication status. The example below shows how to create such a user, assign him the correct permissions and disable password expiration for this user.

monitoring user username: rhelhasync
monitoring user password: <MONITORING_USER_PASSWORD>

3.1. Creating monitoring user

When SAP HANA System replication is active then only the primary system is able to access the database. Accessing the secondary system will fail.

On the primary system run the following commands to create the monitoring user.

[rh2adm]# hdbsql -i 02 -u system -p <HANA_SYSTEM_PASSWORD> "create user rhelhasync password \"<MONITORING_USER_PASSWORD>\""
[rh2adm]# hdbsql -i 02 -u system -p <HANA_SYSTEM_PASSWORD> "grant CATALOG READ to rhelhasync"
[rh2adm]# hdbsql -i 02 -u system -p <HANA_SYSTEM_PASSWORD> "grant MONITOR ADMIN to rhelhasync"
[rh2adm]# hdbsql -i 02 -u system -p <HANA_SYSTEM_PASSWORD> "ALTER USER rhelhasync DISABLE PASSWORD LIFETIME"

3.2. Store monitoring user credentials on all nodes

The SAP HANA userkey allows the "root" user on OS level to access SAP HANA via monitoring user without asking for password. This is needed by resource agents so they can run queries on HANA System Replication status.

[root]# /usr/sap/RH2/HDB02/exe/hdbuserstore SET SAPHANARH2SR localhost:30215 rhelhasync "<MONITORING_USER_PASSWORD>"

To verify that the userkey has been created correctly in root's userstore, you can run hdbuserstore list command on each node and check if the monitoring account is present in the output as shown below:

[root]# /usr/sap/RH2/HDB02/exe/hdbuserstore list

DATA FILE      :  /root/.hdb/node1/SSFS_HDB.DAT
KEY FILE       :  /root/.hdb/node1/SSFS_HDB.KEY

KEY SAPHANARH2SR
  ENV : localhost:30215
  USER: rhelhasync

Please also verify that it is possible to run hdbsql commands as root using the SAPHANASR userkey without being prompted for a password by running the following command on the primary node of the SAP HANA SR setup:

[root]# /usr/sap/RH2/HDB02/exe/hdbsql -U SAPHANARH2SR -i 02 "select distinct REPLICATION_STATUS from SYS.M_SERVICE_REPLICATION"
REPLICATION_STATUS
"ACTIVE"
1 row selected

If you get an error message about issues with the password or if you are prompted for a password please verify with hdbsql command or HANA Studio that the password for the user created with the hdbsql commands above is not configured 'to be changed on first login' or that the password has not expired. You can use the command below.
(Note: be sure to use the name of monitoring user in capital letters)

[root]# /usr/sap/RH2/HDB02/exe/hdbsql -i 02 -u system -p <HANA_SYSTEM_PASSWORD> "select * from sys.users where USER_NAME='RHELHASYNC'"

USER_NAME,USER_ID,USER_MODE,EXTERNAL_IDENTITY,CREATOR,CREATE_TIME,VALID_FROM,VALID_UNTIL,LAST_SUCCESSFUL_CONNECT,LAST_INVALID_CONNECT_ATTEMPT,INVALID_CONNECT_A
TTEMPTS,ADMIN_GIVEN_PASSWORD,LAST_PASSWORD_CHANGE_TIME,PASSWORD_CHANGE_NEEDED,IS_PASSWORD_LIFETIME_CHECK_ENABLED,USER_DEACTIVATED,DEACTIVATION_TIME,IS_PASSWORD
_ENABLED,IS_KERBEROS_ENABLED,IS_SAML_ENABLED,IS_X509_ENABLED,IS_SAP_LOGON_TICKET_ENABLED,IS_SAP_ASSERTION_TICKET_ENABLED,IS_RESTRICTED,IS_CLIENT_CONNECT_ENABLE
D,HAS_REMOTE_USERS,PASSWORD_CHANGE_TIME
"RHELHASYNC",156529,"LOCAL",?,"SYSTEM","2017-05-12 15:10:49.971000000","2017-05-12 15:10:49.971000000",?,"2017-05-12 15:21:12.117000000",?,0,"TRUE","2017-05-12
 15:10:49.971000000","FALSE","FALSE","FALSE",?,"TRUE","FALSE","FALSE","FALSE","FALSE","FALSE","FALSE","TRUE","FALSE",?
1 row selected

4. Configuring SAP HANA in a pacemaker cluster

Please refer to Reference Document for the High Availability Add-On for Red Hat Enterprise Linux 7 documentation to first set up a pacemaker cluster. Note that the cluster must conform to article Support Policies for RHEL High Availability Clusters - General Requirements for Fencing/STONITH.

This guide will assume that following things are working properly:

  • Pacemaker cluster is configured according to documentation and has proper and working fencing
  • SAP HANA startup on boot is disabled on all cluster nodes as the start and stop will be managed by the cluster
  • SAP HANA system replication and takeover using tools from SAP are working properly between cluster nodes
  • SAP HANA contains monitoring account that can be used by the cluster from both cluster nodes
  • Both nodes are subscribed to 'High-availability' and 'RHEL for SAP HANA' (RHEL 6,RHEL 7) channels

4.1. Install SAP HANA Resource Agents

[root]# yum install resource-agents-sap-hana

4.2. Configure general cluster properties

Before putting the environment into production, when testing SAP HANA you may wish to limit the number of failovers, because restart on the same node is much faster than failover to another. You can set up stickiness and migration threshold using commands below. These settings are optional and so are not required for proper setup of SAP HANA in pacemaker. Commands should be executed only on one node but they will take effect in the whole cluster.

[root]# pcs resource defaults resource-stickiness=1000
[root]# pcs resource defaults migration-threshold=5000

Note:
resource-stickiness=1000: control a service to strongly prefer to stay running where it is.
migration-threshold=5000: resource will migrate to a new node after 5000 failures. 5000 is generally high enough to prevent resource from failover to another node.

To remove above options after testing you can use commands below.

[root]# pcs resource defaults resource-stickiness=
[root]# pcs resource defaults migration-threshold=

In previous versions of this guide you might find the recommendation to set up no-quorum-policy to ignore which is currently NOT supported. In the default configuration there is no need to change the no-quorum-policy property of cluster. If you would like to achieve behavior provided by this option please check for more information in the article Can I configure pacemaker to continue to manage resources after a loss of quorum in RHEL 6 or 7?.

4.3. Create cloned SAPHanaTopology resource

SAPHanaTopology resource gathers status and configuration of SAP HANA System Replication on each node. In addition, it starts and monitors the local SAP HostAgent which is required for starting, stopping, and monitoring the SAP HANA instances. It has the following attributes:

Attribute Name Required? Default value Description
SID yes null The SAP System Identifier (SID) of the SAP HANA installation (must be identical for all nodes). Example: RH2
InstanceNumber yes null The Instance Number of the SAP HANA installation (must be identical for all nodes). Example: 02

Below is an example command to create the SAPHanaTopology cloned resource.

[root]# pcs resource create SAPHanaTopology_RH2_02 SAPHanaTopology SID=RH2 InstanceNumber=02 --clone clone-max=2 clone-node-max=1 interleave=true

Resulting resource should look like the following.

[root]# pcs resource show SAPHanaTopology_RH2_02-clone

 Clone: SAPHanaTopology_RH2_02-clone
  Meta Attrs: clone-max=2 clone-node-max=1 interleave=true
  Resource: SAPHanaTopology_RH2_02 (class=ocf provider=heartbeat type=SAPHanaTopology)
   Attributes: SID=RH2 InstanceNumber=02
   Operations: start interval=0s timeout=180 (SAPHanaTopology_RH2_02-start-interval-0s)
               stop interval=0s timeout=60 (SAPHanaTopology_RH2_02-stop-interval-0s)
               monitor interval=60 timeout=60 (SAPHanaTopology_RH2_02-monitor-interval-60)

Once the resource is started you will see the collected information stored in the form of node attributes that can be viewed with the command crm_mon -A1. Below is an example of what attributes can look like when only SAPHanaTopology is started.

[root]# crm_mon -A1
...
Node Attributes:
* Node node1:
    + hana_rh2_remoteHost               : node2
    + hana_rh2_roles                    : 1:P:master1::worker:
    + hana_rh2_site                     : DC1
    + hana_rh2_srmode                   : syncmem
    + hana_rh2_vhost                    : node1
* Node node2:
    + hana_rh2_remoteHost               : node1
    + hana_rh2_roles                    : 1:S:master1::worker:
    + hana_rh2_site                     : DC2
    + hana_rh2_srmode                   : syncmem
    + hana_rh2_vhost                    : node2
...

4.4. Create Master/Slave SAPHana resource

The SAPHana resource agent manages two SAP HANA instances (databases) that are configured in HANA System Replication.

Attribute Name Required? Default value Description
SID yes null The SAP System Identifier (SID) of the SAP HANA installation (must be identical for all nodes). Example: RH2
InstanceNumber yes null The Instance Number of the SAP HANA installation (must be identical for all nodes). Example: 02
PREFER_SITE_TAKEOVER no null Should resource agent prefer to switch over to the secondary instance instead of restarting primary locally? true: do prefer takeover to the secondary site; false: do prefer restart locally; never: under no circumstances do a takeover to the other node
AUTOMATED_REGISTER no false If a takeover event has occurred, and the DUPLICATE_PRIMARY_TIMEOUT has expired, should the former primary instance be registered as secondary? ("false": no, manual intervention will be needed; "true": yes, the former primary will be registered by resource agent as secondary) [1]
DUPLICATE_PRIMARY_TIMEOUT no 7200 The time difference (in seconds) needed between two primary time stamps, if a dual-primary situation occurs. If the time difference is less than the time gap, the cluster will hold one or both instances in a "WAITING" status. This is to give the system admin a chance to react to a takeover. After the time difference has passed, if AUTOMATED_REGISTER is set to true, the failed former primary will be registered as secondary. After the registration to the new primary, all data on the former primary will be overwritten by the system replication.

[1] - As a good practice for test and PoC, we recommend to leave AUTOMATED_REGISTER at its default value (AUTOMATED_REGISTER="false") to prevent that a failed primary instance automatically registers as a secondary instance. After testing, if the failover scenarios work as expected, especially for production environment, we recommend to set AUTOMATED_REGISTER="true", so that after a takeover, the system replication will resume in a timely manner, to avoid disruption. When AUTOMATED_REGISTER="false", in case of a failure on the primary node, after investigation, you will need to manually register it as the secondary HANA System Replication node.

Below is an example command to create the SAPHana Master/Slave resource.

[root]# pcs resource create SAPHana_RH2_02 SAPHana SID=RH2 InstanceNumber=02 PREFER_SITE_TAKEOVER=true DUPLICATE_PRIMARY_TIMEOUT=7200 AUTOMATED_REGISTER=false --master meta notify=true clone-max=2 clone-node-max=1 interleave=true

When running pcs-0.9.158-6.el7, or newer, use the command below to avoid deprecation warning. More information about the change is explained in What are differences between master and --master option in pcs resource create command?.

[root]# pcs resource create SAPHana_RH2_02 SAPHana SID=RH2 InstanceNumber=02 PREFER_SITE_TAKEOVER=true DUPLICATE_PRIMARY_TIMEOUT=7200 AUTOMATED_REGISTER=false master notify=true clone-max=2 clone-node-max=1 interleave=true

Resulting resource should look like the following.

[root]# pcs resource show SAPHana_RH2_02-master

 Master: SAPHana_RH2_02-master
  Meta Attrs: notify=true clone-max=2 clone-node-max=1 interleave=true
  Resource: SAPHana_RH2_02 (class=ocf provider=heartbeat type=SAPHana)
   Attributes: SID=RH2 InstanceNumber=02 PREFER_SITE_TAKEOVER=true DUPLICATE_PRIMARY_TIMEOUT=7200 AUTOMATED_REGISTER=false
   Operations: start interval=0s timeout=180 (SAPHana_RH2_02-start-interval-0s)
               stop interval=0s timeout=240 (SAPHana_RH2_02-stop-interval-0s)
               monitor interval=120 timeout=60 (SAPHana_RH2_02-monitor-interval-120)
               monitor interval=121 role=Slave timeout=60 (SAPHana_RH2_02-monitor-interval-121)
               monitor interval=119 role=Master timeout=60 (SAPHana_RH2_02-monitor-interval-119)
               promote interval=0s timeout=320 (SAPHana_RH2_02-promote-interval-0s)
               demote interval=0s timeout=320 (SAPHana_RH2_02-demote-interval-0s)

Once the resource is started it will add additional node attributes describing the current state of SAP HANA databases on nodes as seen below.

[root]# crm_mon -A1
...
Node Attributes:
* Node node1:
    + hana_rh2_clone_state              : PROMOTED
    + hana_rh2_op_mode                  : delta_datashipping
    + hana_rh2_remoteHost               : node2
    + hana_rh2_roles                    : 4:S:master1:master:worker:master
    + hana_rh2_site                     : DC1
    + hana_rh2_sync_state               : PRIM
    + hana_rh2_srmode                   : syncmem
    + hana_rh2_vhost                    : node1
    + lpa_rh2_lpt                       : 1495204085
    + master-hana                       : 150
* Node node2:
    + hana_rh2_clone_state              : DEMOTED
    + hana_rh2_remoteHost               : node1
    + hana_rh2_roles                    : 4:P:master1:master:worker:master
    + hana_rh2_site                     : DC2
    + hana_rh2_srmode                   : syncmem
    + hana_rh2_sync_state               : SOK
    + hana_rh2_vhost                    : node2
    + lpa_rh2_lpt                       : 30
    + master-hana                       : 100
...

4.5. Create Virtual IP address resource

Cluster will contain Virtual IP address in order to reach the Master instance of SAP HANA. Below is example command to create IPaddr2 resource with IP 192.168.0.15.

[root]# pcs resource create vip_RH2_02 IPaddr2 ip="192.168.0.15"

Resulting resource should look like one below.

[root]# pcs resource show vip_RH2_02

 Resource: vip_RH2_02 (class=ocf provider=heartbeat type=IPaddr2)
  Attributes: ip=192.168.0.15
  Operations: start interval=0s timeout=20s (vip_RH2_02-start-interval-0s)
              stop interval=0s timeout=20s (vip_RH2_02-stop-interval-0s)
              monitor interval=10s timeout=20s (vip_RH2_02-monitor-interval-10s)

4.6. Create constraints

For correct operation we need to ensure that SAPHanaTopology resources are started before starting the SAPHana resources and also that the virtual IP address is present on the node where the Master resource of SAPHana is running. To achieve this, the following 2 constraints need to be created.

4.6.1. constraint - start `SAPHanaTopology` before `SAPHana`

Example command below will create the constraint that mandates the start order of these resources. There are 2 things worth mentioning here:

  • symmetrical=false attribute defines that we care only about the start of resources and they don't need to be stopped in reverse order.
  • Both resources (SAPHana and SAPHanaTopology) have the attribute interleave=true that allows parallel start of these resources on nodes. This permits that despite of ordering we will not wait for all nodes to start SAPHanaTopology but we can start the SAPHana resource on any of nodes as soon as SAPHanaTopology is running there.

Command for creating the constraint:

[root]# pcs constraint order SAPHanaTopology_RH2_02-clone then SAPHana_RH2_02-master symmetrical=false

The resulting constraint should look like the one in the example below.

[root]# pcs constraint
...
Ordering Constraints:
  start SAPHanaTopology_RH2_02-clone then start SAPHana_RH2_02-master (kind:Mandatory) (non-symmetrical)
...

4.6.2. constraint - colocate the `IPaddr2` resource with Master of `SAPHana` resource

Below is an example command that will colocate the IPaddr2 resource with SAPHana resource that was promoted as Master.

[root]# pcs constraint colocation add vip_RH2_02 with master SAPHana_RH2_02-master 2000

Note that the constraint is using a score of 2000 instead of the default INFINITY. This allows the IPaddr2 resource to be taken down by the cluster in case there is no Master promoted in the SAPHana resource so it is still possible to use this address with tools like SAP Management Console or SAP LVM that can use this address to query the status information about the SAP Instance.

The resulting constraint should look like one in the example below.

[root]# pcs constraint
...
Colocation Constraints:
  vip_RH2_02 with SAPHana_RH2_02-master (score:2000) (rsc-role:Started) (with-rsc-role:Master)
...

4.6.3 Adding a secondary virtual IP address resource for an `Active/Read-Enabled` setup

Starting with SAP HANA 2.0 SPS1 SAP allows the so called 'Active/Read-Enabled' setups for SAP HANA System Replication, where the secondary systems of SAP HANA system replication can be used actively for read-intense workloads. To be able to support such setups a second virtual IP address is required which allows clients to access the secondary SAP HANA database. To ensure that the secondary replication site can still be accessed after a takeover has occured the cluster needs to move the virtual IP address around with the slave of the master/slave SAPHana resource. The second virtual IP and the appropriate colocation constraint can be configured with the following commands:

[root]#  pcs resource create vip2_SAPHana_RH2_02 IPaddr2 ip="192.168.1.11"
[root]#  pcs constraint colocation add vip2_SAPHana_RH2_02 with slave msl_rsc_SAPHana_RH2_02 2000

4.7. Testing the manual move of SAPHana resource to another node (SAP Hana takeover by cluster)

To test out the move of the SAPHana resource from one node to another, use the command below. Note that the option --master should NOT be used when running the below command due to the way how the SAPHana resource works internally.

[root]# pcs resource move SAPHana_RH2_02-master

IMPORTANT: After each pcs resource move command invocation the cluster creates location constraints to achieve the move of the resource. These constraints must be removed in order to allow automatic failover in the future. To remove them you can use the command pcs resource clear SAPHana_RH2_02-master.

Table of Contents

Automatically generate a table of contents

2 Comments

How are RHEL version requirements (a la RHEL7.3 ceiling) realized with RHUI repositories? Last I checked, a setting a release level was not permitted with RHUI.

Hello John, standard ccsp certificate contains for example EUS repos - those have RHEL versions. So customers need to sync those repos into their RHUIs and then create client package with repo paths pointing to concrete versions. It is possible to hand modify pregenerated client package to have paths containing $releasever instead of hard coded version - from RHUI 3.0.5 version (iirc) there is even script rhui-set-release present in every RHUI generated client package that enables clients to set different releasever. I've written KB article on how to "manually" create you client package: https://access.redhat.com/articles/4070201 that notes also $releasever.

Bottomline: It is possible to change versions, but you need to know what you are doing - eg. changing version to non-existing repo will break your client.