The SAPInstance resource agent will be used for managing the ASCS and ERS resources in this example. All operations of the SAPInstance resource agent are done by using the SAP start-up service framework sapstartsrv.

This resource agent is is gathering status and configuration of SAP HANA System Replication on each cluster node. It is essential to have the data from this agent present in cluster node attributes for SAPHana resource agent to work properly.

This resource is responsible for starting, stopping, and relocating (failover) of SAP HANA database. This resource agent takes information gathered by SAPHanaTopology and based on that it is interacting with SAP HANA database to do things. It also adds additional information to cluster node attributes about SAP HANA status on cluster nodes.

Pacemaker cluster resource-agent for Filesystem. It manages a Filesystem on a shared storage medium exported by NFS or iSCSI etc.

Manages virtual IPv4 and IPv6 addresses and aliases.

There must be a separate SAN LUN or NFS export for the ASCS and ERS instances that can be mounted by the cluster on each node.

For example, as shown below, 'ASCS' and 'ERS' instances, respectively, the instance specific directory must be present on the corresponding node.

Note: For the Application Servers, the following directory must be made available on the nodes where the Application Server instances will run:

When using SAN LUNs for the instance directories, customers must use HA-LVM to ensure that the instance directories can only be mounted on one node at a time.

When using NFS exports, if the directories are created on the same directory tree on an NFS file server, such as Azure NetApp Files or Amazon EFS, the option force_unmount=safe must be used when configuring the Filesystem resource. This option will ensure that the cluster only stops the processes running on the specific NFS export instead of stopping all processes running on the directory tree where the exports are created.

The following mount points must be available on ASCS, ERS, HANA, and Application Servers nodes.

/sapmnt
/usr/sap/trans
/usr/sap/SID/SYS

Shared storage can be achieved by:

These mount points must be either managed by the cluster or mounted before the cluster is started.

The following file systems should be mounted on s4node1, where ASCS will be installed:

/usr/sap/S4H/ASCS20
/usr/sap/S4H/SYS
/usr/sap/trans
/sapmnt

Virtual IP for s4ascs should be enabled on s4node1.

Run the installer:

[root@s4node1]# ./sapinst SAPINST_USE_HOSTNAME=s4ascs

Select High-Availability System option.

The following file systems should be mounted on s4node2, where ERS will be installed:

/usr/sap/S4H/ERS29 /usr/sap/S4H/SYS /usr/sap/trans /sapmnt

Virtual IP for s4ers should be enabled on s4node2.

Run the installer:

[root@s4node2]# ./sapinst SAPINST_USE_HOSTNAME=s4ers

Select High-Availability System option.

In this example, we will be using SAP HANA with the following configuration. You can also use other supported databases as per the support policies.

SAP HANA SID: S4D
SAP HANA Instance number: 00

In this example, SAP HANA Database server can be installed on both nodes using the hdblcm command line tool and then Automated HANA System Replication will be established in the same way as mentioned in the following document: SAP HANA system replication in pacemaker cluster.
It is obvious to note that in this setup, the ASCS and HANA primary instances can failover independently of each other; hence, there will come a situation where the ASCS and the primary SAP HANA instances start running on the same node. Therefore, it is also important to ensure that both nodes have enough resources and free memory available for the ASCS/ERS and the primary SAP HANA instances to run on one node in an event of the other node’s failure.

To achieve this, the SAP HANA instance can be set with certain memory restrictions/limits, and we highly recommend reaching out to SAP for exploring the available options as per your SAP HANA environment. Some related links are as follows:

The ERS instance requires the following modification to its profile, to prevent the automatic restart of the enqueue server, as it will be managed by cluster. To apply the change, run the following command at your ERS profile /sapmnt/S4H/profile/S4H_ERS29_s4ers.

[root]# sed -i -e 's/Restart_Program_00/Start_Program_00/'
/sapmnt/S4H/profile/S4H_ERS29_s4ers+

On both s4node1 and s4node2, make sure the following two lines are commented out in /usr/sap/sapservices file:

#LD_LIBRARY_PATH=/usr/sap/S4H/ERS29/exe:$LD_LIBRARY_PATH; export
LD_LIBRARY_PATH; /usr/sap/S4H/ERS29/exe/sapstartsrv
pf=/usr/sap/S4H/SYS/profile/S4H_ERS29_s4ers -D -u s4hadm
#LD_LIBRARY_PATH=/usr/sap/S4H/ASCS20/exe:$LD_LIBRARY_PATH; export
LD_LIBRARY_PATH; /usr/sap/S4H/ASCS20/exe/sapstartsrv
pf=/usr/sap/S4H/SYS/profile/S4H_ASCS20_s4ascs -D -u s4hadm

[root@s4node1 ~]# mkdir /usr/sap/S4H/ERS29/
[root@s4node1 ~]# chown s4hadm:sapsys /usr/sap/S4H/ERS29/

[root@s4node2 ~]# mkdir /usr/sap/S4H/ASCS20
[root@s4node2 ~]# chown s4hadm:sapsys /usr/sap/S4H/ASCS20

Stop ASCS and ERS instances. Move the instance specific directory to the other node:

[root@s4node1 ~]# umount /usr/sap/S4H/ASCS20
[root@s4node2 ~]# mount /usr/sap/S4H/ASCS20

[root@s4node2 ~]# umount /usr/sap/S4H/ERS29/
[root@s4node1 ~]# mount /usr/sap/S4H/ERS29/

Manually start the ASCS and ERS instances on the other cluster node, then manually stop them, respectively.

On all nodes check if SAP HostAgent has the same version and meets the minimum version requirement:

[root]# /usr/sap/hostctrl/exe/saphostexec -version

To upgrade/install SAP HostAgent. Please follow SAP OSS Note 1031096.

SAP hardware key determination in the high-availability scenario has been improved. It might be necessary to install several SAP license keys based on the hardware key of each cluster node. Please see SAP OSS Note 1178686 - Linux: Alternative method to generate a SAP hardware key for more information.

The cloned Filesystem cluster resource can be used to mount the shares from external NFS server on all cluster nodes as shown below.

[root]# pcs resource create s4h_fs_sapmnt Filesystem \
device='<NFS_Server>:<sapmnt_nfs_share>' directory='/sapmnt' \
fstype='nfs' --clone interleave=true
[root]# pcs resource create s4h_fs_sap_trans Filesystem \
device='<NFS_Server>:<sap_trans_nfs_share>' directory='/usr/sap/trans' \
fstype='nfs' --clone interleave=true
[root]# pcs resource create s4h_fs_sap_sys Filesystem \
device='<NFS_Server>:<s4h_sys_nfs_share>' directory='/usr/sap/S4H/SYS' \
fstype='nfs' --clone interleave=true

After creating the Filesystem resources verify that they have started properly on all nodes.

[root]# pcs status
...
Clone Set: s4h_fs_sapmnt-clone [s4h_fs_sapmnt]
Started: [ s4node1 s4node2 ]
Clone Set: s4h_fs_sap_trans-clone [s4h_fs_sap_trans]
Started: [ s4node1 s4node2 ]
Clone Set: s4h_fs_sys-clone [s4h_fs_sys]
Started: [ s4node1 s4node2 ]
...

In case that shared filesystems will NOT be managed by cluster, it must be ensured that they are available before the pacemaker service is started.
In RHEL 7 due to systemd parallelization you must ensure that shared filesystems are started in resource-agents-deps target. More details on this can be found in documentation section 9.6. Configuring Startup Order for Resource Dependencies not Managed by Pacemaker (Red Hat Enterprise Linux 7.4 and later).

Below is the example of creating resource for NFS filesystem

[root]# pcs resource create s4h_fs_ascs20 Filesystem \
device='<NFS_Server>:<s4h_ascs20_nfs_share>' \
directory=/usr/sap/S4H/ASCS20 fstype=nfs force_unmount=safe \
--group s4h_ASCS20_group op start interval=0 timeout=60 \
op stop interval=0 timeout=120 \
op monitor interval=200 timeout=40

Below is the example of creating resources for HA-LVM filesystem

[root]# pcs resource create s4h_fs_ascs20_lvm LVM \
volgrpname='<ascs_volume_group>' exclusive=true \
--group s4h_ASCS20_group

[root]# pcs resource create s4h_fs_ascs20 Filesystem \
device='/dev/mapper/<ascs_logical_volume>' \
directory=/usr/sap/S4H/ASCS20 fstype=ext4 \
--group s4h_ASCS20_group

[root]# pcs resource create s4h_ascs20 SAPInstance \
InstanceName="S4H_ASCS20_s4ascs" \
START_PROFILE=/sapmnt/S4H/profile/S4H_ASCS20_s4ascs \
AUTOMATIC_RECOVER=false \
meta resource-stickiness=5000 \
--group s4h_ASCS20_group \
op monitor interval=20 on-fail=restart timeout=60 \
op start interval=0 timeout=600 \
op stop interval=0 timeout=600

Note: meta resource-stickiness=5000 is here to balance out the failover constraint with ERS so the resource stays on the node where it started and doesn’t migrate around the cluster uncontrollably.
Add a resource stickiness to the group to ensure that the ASCS will stay on a node if possible:

[root]# pcs resource meta s4h_ASCS20_group resource-stickiness=3000

Below is the example of creating resource for NFS filesystem

[root]# pcs resource create s4h_fs_ers29 Filesystem \
device='<NFS_Server>:<s4h_ers29_nfs_share>' \
directory=/usr/sap/S4H/ERS29 fstype=nfs force_unmount=safe \
--group s4h_ERS29_group op start interval=0 timeout=60 \
op stop interval=0 timeout=120 op monitor interval=200 timeout=40

Below is the example of creating resources for HA-LVM filesystem

[root]# pcs resource create s4h_fs_ers29_lvm LVM \
volgrpname='<ers_volume_group>' exclusive=true --group s4h_ERS29_group

[root]# pcs resource create s4h_fs_ers29 Filesystem \
device='/dev/mapper/<ers_logical_volume>' directory=/usr/sap/S4H/ERS29 \
fstype=ext4 --group s4h_ERS29_group

Create an ERS instance cluster resource.
Note: In ENSA2 deployments the IS_ERS attribute is optional. To learn more about IS_ERS, additional information can be found in How does the IS_ERS attribute work on a SAP NetWeaver cluster with Standalone Enqueue Server (ENSA1 and ENSA2)?.

[root]# pcs resource create s4h_ers29 SAPInstance \
InstanceName="S4H_ERS29_s4ers" \
START_PROFILE=/sapmnt/S4H/profile/S4H_ERS29_s4ers \
AUTOMATIC_RECOVER=false \
--group s4h_ERS29_group \
op monitor interval=20 on-fail=restart timeout=60 \
op start interval=0 timeout=600 \
op stop interval=0 timeout=600

ASCS20 instance rh2_ascs20 prefers to run on node where ERS was running before failover.

# pcs constraint location rh2_ascs20 rule score=2000 runs_ers_RH2 eq 1

Prefer to start s4h_ASCS20_group before the s4h_ERS29_group

[root]# pcs constraint order start s4h_ASCS20_group then start \
s4h_ERS29_group symmetrical=false kind=Optional
[root]# pcs constraint order start s4h_ASCS20_group then stop \
s4h_ERS29_group symmetrical=false kind=Optional

If the shared filesystem /sapmnt is managed by the cluster, then the following constraints ensure that resource groups with ASCS and ERS SAPInstance resources are started only once the filesystem is available.

[root]# pcs constraint order s4h_fs_sapmnt-clone then s4h_ASCS20_group
[root]# pcs constraint order s4h_fs_sapmnt-clone then s4h_ERS29_group