Configuring SAP NetWeaver ASCS/ERS ENSA1 with Standalone Resources in RHEL 7.5+ and RHEL 8

Updated -


1. Overview

1.1. Introduction

SAP NetWeaver-based systems (like SAP ERP, SAP CRM, ...) play an important role in business processes, thus it's critical for SAP NetWeaver-based systems to be highly available. The underlying idea of Clustering is a fairly simple one: Not a single large machine bears all of the load and risk; but rather one or more machines automatically drop in as an instant full replacement for the service or the machine that has failed. In the best case, this replacement process causes no interruption to the systems' users.

1.2. Audience

This document is intended for SAP and Red Hat certified or trained administrators and consultants who already have experience setting up high available solutions using the RHEL HA add-on or other clustering solutions. Access to both SAP Service Marketplace and Red Hat Customer Portal is required to be able to download software and additional documentation.

Red Hat Consulting is highly recommend to set up the cluster and customize the solution to meet customers' data center requirements, that are normally more complex than the solution presented in this document.

1.3. Concepts

This document describes how to set up a two-node-cluster solution that conforms to the guidelines for high availability that have been established by both SAP and Red Hat. It is based on SAP NetWeaver on top of Red Hat Enterprise Linux 7.5 or newer, RHEL 8, with RHEL HA Add-on. The reference architecture described in this document is under the tests of SAP High Availability Interface certification NW-HA-CLU 7.50. This document will be updated after the certification is obtained.

At application tier, the lock table of the enqueue server is the most critical piece. To protect it, SAP has developed the “Enqueue Replication Server” (ERS) which maintains a backup copy of the lock table. While the (A)SCS is running on node1, the ERS always needs to maintain a copy of the current enqueue table on node2. When the system needs to failover the (A)SCS to node2, it first starts on node2, and then shuts down the ERS, taking over its' shared memory segment and thereby acquiring an up-to-date enqueue table, thus the replicated enqueue table becomes the new primary enqueue table. ERS will start on node1 when it becomes available.

Required by the SAP HA-Interface certification, this document focuses on the high availability of (A)SCS and ERS, both are controlled by the cluster software. In normal mode, cluster ensures that ERS and (A)SCS always run on different nodes. In the event of failover, the (A)SCS needs to “follow” the ERS. (A)SCS needs to switch to the node where ERS is running. Cluster has to ensure that the ERS is already running when (A)SCS starts up, because (A)SCS needs to take over the replicated enqueue table from ERS.

The concept described above is known as Standalone Enqueue Server 1 (ENSA1), available in ABAP 1709 or older. For Standalone Enqueue Server 2 (ENSA2) which is now the default installation in ABAP 1809 or newer, check Configure SAP S/4HANA ASCS/ERS with Standalone Enqueue Server 2 (ENSA2) in Pacemaker

1.4. Resources: Standalone vs. Master/Slave

There are two approaches to configure (A)SCS and ERS resources in Pacemaker: Master/Slave and Standalone. Master/Slave approach has already been supported in all RHEL 7 minor releases. Standalone approach is supported in RHEL 7.5 or newer, and RHEL 8.

In any new deployment, Standalone is recommended for the following reasons:
- it meets the requirements of the current SAP HA Interface Certification
- it is compatible with the new Standalone Enqueue Server 2 (ENSA2) configuration
- (A)SCS/ERS instances can be started and stopped independently
- (A)SCS/ERS instance directories can be managed as part of the cluster

This article outlines the configuration procedure of the SAPinstance Standalone approach. For instructions on SAPInstance Master/Slave configuration, please refer to kabse article SAP NetWeaver in Pacemaker cluster with Master/Slave SAPInstance resource

1.5. Support Policies

See: Support Policies for RHEL High Availability Clusters - Management of SAP NetWeaver in a Cluster

2. Requirements

2.1 Subscription

It’s important to keep the subscription, kernel, and patch level identical on both nodes.

Please follow this kbase article to subscribe your systems to the Update Service for RHEL for SAP Solutions.

2.2. Pacemaker Resource Agents

  • RHEL for SAP Solutions 7.5 or newer, and RHEL 8

2.3. SAP NetWeaver High-Availability Architecture

A typical setup for SAP NetWeaver High-Availability consists of 3 distinctive components:

This article focuses on the configuration of SAP NetWeaver ASCS and ERS in a pacemaker cluster. As the best practice, we recommend to install Application Servers and Database on separate nodes outside of the two-node cluster designated for (A)SCS and ERS.

Below is the architecture diagram of the example installation:

EFS Creation

2.4. SAPInstance resource agent

SAPInstance is a pacemaker resource agent used for both ASCS and ERS resources. All operations of the SAPInstance resource agent are done by using the SAP start service framework sapstartsrv.

2.5. Storage requirements

Directories created for NetWeaver installation should be put on shared storage, following the rules:

2.5.1. Instance Specific Directory

There must be a separate SAN LUN or NFS export for each instance directory that can be mounted by the cluster on the node where the instance is supposed to be running.

For 'ASCS' and 'ERS', respectively, the instance specific directory must be present on the corresponding node.

  • ASCS node: /usr/sap/SID/ASCS<Ins#>
  • ERS node: /usr/sap/SID/ERS<Ins#>

For Application Servers, the following directory should be made available on corresponding node designated for the Application Server instance:

  • App Server D<Ins#>: /usr/sap/SID/D<Ins#>

When using SAN LUNs for the instance directories customers must use HA-LVM to ensure that the instance directories can only be mounted on one node at a time.

When using NFS exports, if the directories are created on the same directory tree on an NFS file server, such as NetApp Files or Amazon EFS, the option force_unmount=safe must be used when configuring the Filesystem resource. This option will ensure that the cluster only stops the processes running on the specific NFS export instead of stopping all processes running on the directory tree where the exports are created.

2.5.2. Shared Directories

The following mount points must be available on ASCS, ERS, and Application Servers nodes.


2.5.3. Shared Directories on HANA

The following mount point(s) must be available on the HANA node.


Shared storage can be achieved by:

These mount points must be either managed by cluster or mounted before cluster is started.

3. Install SAP NetWeaver

3.1. Configuration options used in this document

Below are configuration options that will be used for instances in this document:

Two nodes will be running the ASCS/ERS instances in pacemaker:

1st node hostname:      node1
2nd node hostname:      node2

SID:                    RH2

ASCS Instance number:   20
ASCS virtual hostname:  rhascs

ERS Instance number:    29
ERS virtual hostname:   rhers

Outside the two-node cluster:

PAS Instance number:    21
AAS Instance number:    22

HANA database:

SID:                    RH0
HANA Instance number:   00
HANA virtual hostname:  rhdb

3.2. Prepare hosts

Before starting installation ensure that:

  • Install RHEL for SAP Solutions 7.x or RHEL 8.x (latest is recommended)
  • Register system to RHN or Satellite, enable RHEL for SAP Applications channel, or Update Services (E4S) channel
  • Enable High Availability Add-on channel
  • Shared storage and filesystems are present at correct mount points
  • Virtual IP addresses used by instances are present and reachable
  • Hostnames that will be used by instances can be resolved to IP addresses and back
  • Installation medias are available
  • System is configured according to the recommendation for running SAP NetWeaver

3.3. Install NetWeaver

Using software provisioning manager (SWPM) install instances in the following order:

  • ASCS instance
  • ERS instance
  • DB instance
  • PAS instance
  • AAS instances

3.3.1. Install ASCS on node1

The following file systems should be mounted on node1, where ASCS will be installed:


Virtual IP for rhascs should be enabled on node1.

Run the installer:

[root@node1]# ./sapinst SAPINST_USE_HOSTNAME=rhascs

Select High-Availability System option.

ASCS Installation

3.3.2. Install ERS on node2

The following file systems should be mounted on node2, where ERS will be installed:


Virtual IP for rhers should be enabled on node2.

Run the installer:

[root@node2]# ./sapinst SAPINST_USE_HOSTNAME=rhers

Select High-Availability System option.

ERS Installation

3.3.3. SAP HANA

In the example, SAP HANA will be using the following configuration. You can also use other supported database.

SAP HANA SID:                    RH0
SAP HANA Instance number:        00

SAP HANA should be installed on separate host. Optionally, Automated HANA System Replication can be installed in another pacemaker cluster by following document SAP HANA system replication in pacemaker cluster.

Run the installer on the HANA host:

[root]# ./sapinst SAPINST_USE_HOSTNAME=rhdb

3.3.4. Install Application Servers

The following file systems should be mounted on the host to run the Application Server instance. If you have multiple application servers, install each one on corresponding host:


Run the installer:

[root]# ./sapinst

Select High-Availability System option.

3.4. Post Installation

3.4.1. (A)SCS profile modification

(A)SCS instance requires following modification in profile to prevent automatic restart of enqueue server as it will be managed by cluster. To apply the change run the following command at your ASCS profile /sapmnt/RH2/profile/RH2_ASCS20_rhascs.

[root]# sed -i -e 's/Restart_Program_01/Start_Program_01/' /sapmnt/RH2/profile/RH2_ASCS20_rhascs

3.4.2. ERS profile modification

ERS instance requires following modification in profile to prevent automatic restart as it will be managed by cluster. To apply the change run the following command at your ERS profile /sapmnt/RH2/profile/RH2_ERS29_rhers.

[root]# sed -i -e 's/Restart_Program_00/Start_Program_00/' /sapmnt/RH2/profile/RH2_ERS29_rhers

3.4.3. Update the /usr/sap/sapservices file

On both node1 and node2, make sure following two lines are commented out in /usr/sap/sapservices file:

#LD_LIBRARY_PATH=/usr/sap/RH2/ASCS20/exe:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; /usr/sap/RH2/ASCS20/exe/sapstartsrv pf=/usr/sap/RH2/SYS/profile/RH2_ASCS20_rhascs -D -u rh2adm
#LD_LIBRARY_PATH=/usr/sap/RH2/ERS29/exe:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; /usr/sap/RH2/ERS29/exe/sapstartsrv pf=/usr/sap/RH2/ERS29/profile/RH2_ERS29_rhers -D -u rh2adm

3.4.4. Create mount points for ASCS and ERS on the failover node

[root@node1 ~]# mkdir /usr/sap/RH2/ERS29
[root@node1 ~]# chown rh2adm:sapsys /usr/sap/RH2/ERS29

[root@node2 ~]# mkdir /usr/sap/RH2/ASCS20
[root@node2 ~]# chown rh2adm:sapsys /usr/sap/RH2/ASCS20

3.4.5. Manual Testing Instance on Other Node

Stop ASCS and ERS instances. Move the instance specific directory to the other node:

[root@node1 ~]# umount /usr/sap/RH2/ASCS20
[root@node2 ~]# mount /usr/sap/RH2/ASCS20

[root@node2 ~]# umount /usr/sap/RH2/ERS29
[root@node1 ~]# mount /usr/sap/RH2/ERS29

Manually start ASCS and ERS instances on other cluster node, then manually stop them, respectively.

3.4.6. Check SAP HostAgent on all nodes

On all nodes check if SAP HostAgent has the same version and meets the minimum version requirement:

[root]# /usr/sap/hostctrl/exe/saphostexec -version

To upgrade/install SAP HostAgent, follow SAP note 1031096.

3.4.7. Install permanent SAP license keys

SAP hardware key determination in the high-availability scenario has been improved. It might be necessary to install several SAP license keys based on the hardware key of each cluster node. Please see SAP Note 1178686 - Linux: Alternative method to generate a SAP hardware key for more information.

4. Install Pacemaker

Follow Pacemaker documentation:
Reference Document for the High Availability Add-On for Red Hat Enterprise Linux 7
Configuring and Managing High Availability Clusters on RHEL 8
How can I configure power fencing for the IBM POWER platform using an HMC in a RHEL High Availability cluster?

Below is a sample procedure to install pacemaker. It's recommended to work with a Red Hat consultant to install and configure Pacemaker in your environment.

4.1. Install Pacemaker rpm's

# yum -y install pcs pacemaker
# passwd hacluster
[provide a password]
# systemctl enable pcsd.service; systemctl start pcsd.service

4.2. Create a Cluster

Create a cluster named nwha, consisting of node1 and node2, and start the cluster. Please note that at this point, cluster is not yet configured to auto-start after reboot.

# pcs cluster auth node1 node2
# pcs cluster setup --name nwha node1 node2
# pcs cluster start --all

4.2.1. Define General Cluster Properties

Set the resource stickiness:

# pcs resource defaults resource-stickiness=1
# pcs resource defaults migration-threshold=3

4.3. Configure STONITH

The fencing mechanism STONITH depends on the underneath platform. Please check the corresponding document to configure the STONITH. Support Policies for RHEL High Availability Clusters - General Requirements for Fencing/STONITH.

After configuring the STONITH, on node1 test fencing node2.

[root@node1]# pcs stonith fence node2

node2 should be properly fenced. After fencing, start cluster on node2 using the following command. This is because the cluster has not yet been enabled to auto-start. Auto-start will be enabled after initial testings showing the cluster is properly configured.

[root@node2 ~]# pcs cluster start

4.4. Install resource-agents-sap on all cluster nodes

[root]# yum install resource-agents-sap

4.5. Configure cluster resources for shared filesystems

Configure shared filesystem to provide following mount points on all of cluster nodes.


4.5.1. Configure shared filesystems managed by the cluster

The cloned Filesystem cluster resource can be used to mount the shares from external NFS server on all cluster nodes as shown below.

[root]# pcs resource create rh2_fs_sapmnt Filesystem device='<NFS_Server>:<sapmnt_nfs_share>' directory='/sapmnt' fstype='nfs' --clone interleave=true
[root]# pcs resource create rh2_fs_sap_trans Filesystem device='<NFS_Server>:<sap_trans_nfs_share>' directory='/usr/sap/trans' fstype='nfs' --clone interleave=true
[root]# pcs resource create rh2_fs_sap_sys Filesystem device='<NFS_Server>:<rh2_sys_nfs_share>' directory='/usr/sap/RH2/SYS' fstype='nfs' --clone interleave=true

After creating the Filesystem resources verify that they have started properly on all nodes.

[root]# pcs status
Clone Set: rh2_fs_sapmnt-clone [rh2_fs_sapmnt]
    Started: [ node1 node2 ]
Clone Set: rh2_fs_sap_trans-clone [rh2_fs_sap_trans]
    Started: [ node1 node2 ]
Clone Set: rh2_fs_sys-clone [rh2_fs_sys]
    Started: [ node1 node2 ]


4.5.2. Configure shared filesystems managed outside of cluster

In case that shared filesystems will NOT be managed by cluster, it must be ensured that they are available before the pacemaker service is started.

In RHEL 7 due to systemd parallelization you must ensure that shared filesystems are started in resource-agents-deps target. More details on this can be found in documentation section 9.6. Configuring Startup Order for Resource Dependencies not Managed by Pacemaker (Red Hat Enterprise Linux 7.4 and later).

4.6. Configure ASCS resource group

4.6.1. Create resource for virtual IP address

# pcs resource create rh2_vip_ascs20 IPaddr2 ip= --group rh2_ASCS20_group

4.6.2. Create resource for ASCS filesystem.

Below is the example of creating resource for NFS filesystem

# pcs resource create rh2_fs_ascs20 Filesystem device='<NFS_Server>:<rh2_ascs20_nfs_share>' directory=/usr/sap/RH2/ASCS20 fstype=nfs force_unmount=safe --group rh2_ASCS20_group \
  op start interval=0 timeout=60 \
  op stop interval=0 timeout=120 \
  op monitor interval=200 timeout=40

Below is the example of creating resources for HA-LVM filesystem

# pcs resource create rh2_fs_ascs20_lvm LVM volgrpname='<ascs_volume_group>' exclusive=true --group rh2_ASCS20_group
# pcs resource create rh2_fs_ascs20 Filesystem device='/dev/mapper/<ascs_logical_volume>' directory=/usr/sap/RH2/ASCS20 fstype=ext4 --group rh2_ASCS20_group

4.6.3. Create resource for ASCS instance

# pcs resource create rh2_ascs20 SAPInstance InstanceName="RH2_ASCS20_rhascs" START_PROFILE=/sapmnt/RH2/profile/RH2_ASCS20_rhascs AUTOMATIC_RECOVER=false meta resource-stickiness=5000 migration-threshold=1 failure-timeout=60 --group rh2_ASCS20_group \
  op monitor interval=20 on-fail=restart timeout=60 \
  op start interval=0 timeout=600 \
  op stop interval=0 timeout=600

Note: meta resource-stickiness=5000 is here to balance out the failover constraint with ERS so the resource stays on the node where it started and doesn't migrate around cluster uncontrollably. migration-threshold=1 ensures that ASCS failover to another node when issue is detected instead of restarting on same node.

Add a resource stickiness to the group to ensure that the ASCS will stay on a node if possible:

# pcs resource meta rh2_ASCS20_group resource-stickiness=3000

4.7. Configure ERS resource group

4.7.1. Create resource for virtual IP address

# pcs resource create rh2_vip_ers29 IPaddr2 ip= --group rh2_ERS29_group

4.7.2. Create resource for ERS filesystem

Below is the example of creating resource for NFS filesystem

# pcs resource create rh2_fs_ers29 Filesystem device='<NFS_Server>:<rh2_ers29_nfs_share>' directory=/usr/sap/RH2/ERS29 fstype=nfs force_unmount=safe --group rh2_ERS29_group \
  op start interval=0 timeout=60 \
  op stop interval=0 timeout=120 \
  op monitor interval=200 timeout=40

Below is the example of creating resources for HA-LVM filesystem

# pcs resource create rh2_fs_ers29_lvm LVM volgrpname='<ers_volume_group>' exclusive=true --group rh2_ERS29_group
# pcs resource create rh2_fs_ers29 Filesystem device='/dev/mapper/<ers_logical_volume>' directory=/usr/sap/RH2/ERS29 fstype=ext4 --group rh2_ERS29_group

4.7.3. Create resource for ERS instance

Create the ERS instance cluster resource. Note: IS_ERS=true attribute is mandatory for ENSA1 deployments. More information about IS_ERS can be found in How does the IS_ERS attribute work on a SAP NetWeaver cluster with Standalone Enqueue Server (ENSA1 and ENSA2)?.

# pcs resource create rh2_ers29 SAPInstance InstanceName="RH2_ERS29_rhers" START_PROFILE=/sapmnt/RH2/profile/RH2_ERS29_rhers AUTOMATIC_RECOVER=false IS_ERS=true --group rh2_ERS29_group \
  op monitor interval=20 on-fail=restart timeout=60 \
  op start interval=0 timeout=600 \
  op stop interval=0 timeout=600

4.8. Create constraints

4.8.1. Create colocation constraint for ASCS and ERS resource groups

Resource groups rh2_ASCS20_group and rh2_ERS29_group should try to avoid running on same node. Order of groups matters.

# pcs constraint colocation add rh2_ERS29_group with rh2_ASCS20_group -5000

4.8.2. Create location constraint for ASCS resource

ASCS20 instance rh2_ascs20 prefers to run on node where ERS was running before when failover is happening

# pcs constraint location rh2_ascs20 rule score=2000 runs_ers_RH2 eq 1

4.8.3. Create order constraint for ASCS and ERS resource groups

Prefer to start rh2_ASCS20_group before the rh2_ERS29_group (optionally)

# pcs constraint order start rh2_ASCS20_group then stop rh2_ERS29_group symmetrical=false kind=Optional

4.8.4. Create order constraint for /sapmnt resource managed by cluster

If the shared filesystem /sapmnt is managed by cluster, then following constraints ensures that resource groups with ASCS and ERS SAPInstance resource are started only once the filesystem is available.

# pcs constraint order rh2_fs_sapmnt-clone then rh2_ASCS20_group
# pcs constraint order rh2_fs_sapmnt-clone then rh2_ERS29_group

4.9. Configure Primary Application Server group (PAS)

Below is example of configuring resource group rh1_PAS_D01_group containing the Primary Application Server (PAS) instance rh1_pas_d01 managed by SAPInstance resource agent.

4.9.1. Create Virtual IP address for PAS instance

To create the virtual IP address that will be part of the rh1_PAS_D01_group use the command below.

[root]# pcs resource create rh1_vip_pas_d01 IPaddr2 ip= --group rh1_PAS_D01_group

To verify that resource got created in new group rh1_PAS_D01_group the output from pcs status command should look like below.

[root]# pcs status
 Resource Group: rh1_PAS_D01_group
  rh1_vip_pas_d01       (ocf::heartbeat:IPaddr2):   Started node1

4.9.2. Configure LVM and Filesystem cluster resources in PAS resource group

First LVM cluster resource is added then followed by Filesystem cluster resource. LVM shared by cluster is expected to be configured as described in the article What is a Highly Available LVM (HA-LVM) configuration and how do I implement it?.

[root]# pcs resource create rh1_lvm_pas_d01 LVM volgrpname=vg_d01 exclusive=true --group rh1_PAS_D01_group
[root]# pcs resource create rh1_fs_pas_d01 Filesystem device=/dev/vg_d01/lv_d01 directory=/usr/sap/RH1/D01 fstype=xfs --group rh1_PAS_D01_group

Verify that resources were added to rh1_PAS_D01_group resource group and started as shown below.

[root]# pcs status
 Resource Group: rh1_PAS_D01_group
  rh1_vip_pas_d01       (ocf::heartbeat:IPaddr2):       Started node1
  rh1_lvm_pas_d01       (ocf::heartbeat:LVM):       Started node1
  rh1_fs_pas_d01        (ocf::heartbeat:Filesystem):    Started node1

4.9.3. Configure constraints for PAS resource group

PAS requires the ASCS and database instance to be running before it can start properly. Below are example commands on how to setup constraints to achieve this for various databases that can be used by SAP Netweaver. Deplyoments with rh1_SAPDatabase_group group

For configurations that has one cluster resource group that will start all resources needed by database. In example here the SAPDatabase resource agent is used to manage the database and is part of the the database group rh1_SAPDatabase_group. Commands bellow will create constraints that will start the whole rh1_PAS_D01_group only once the ASCS instance was promoted and when the database group rh1_SAPDatabase_group is running.

[root]# pcs constraint order rh1_SAPDatabase_group then rh1_PAS_D01_group symmetrical=false
[root]# pcs constraint order promote rh1_ascs_ers-master then rh1_PAS_D01_group

After executing the commands verify that the constraints were added properly.

[root]# pcs constraint
Ordering Constraints:
  start rh1_SAPDatabase_group then start rh1_PAS_group (kind:Mandatory) (non-symmetrical)
  promote rh1_ascs_ers-master then start rh1_PAS_group (kind:Mandatory)
... Deplyoments with SAP HANA with SR as database

When using SAP HANA database that is configured for system replication (SR) that is managed by cluster, the following constraints will ensure that whole rh1_PAS_D01_group will start only once the ASCS instance was promoted and when the SAP HANA SAPHana_RH2_02-master was promoted.

[root]# pcs constraint order promote SAPHana_RH2_02-master then rh1_PAS_D01_group symmetrical=false
[root]# pcs constraint order promote rh1_ascs_ers-master then rh1_PAS_D01_group

After executing the commands verify that the constraints were added properly.

[root]# pcs constraint
Ordering Constraints:
  promote SAPHana_RH2_02-master then start rh1_PAS_group (kind:Mandatory) (non-symmetrical)
  promote rh1_ascs_ers-master then start rh1_PAS_group (kind:Mandatory)

4.9.4. Configure PAS SAPInstance cluster resource

To run the PAS instance the same SAPInstance resource agents as for (A)SCS/ERS instance is used. The PAS instance is compared to (A)SCS/ERS a simple instance and requires less attributes to be configured. Check the command below for example on how to create a PAS instance for 'D01' instance and place it at the end of the rh1_PAS_D01_group resource group.

[root]# pcs resource create rh1_pas_d01 SAPInstance InstanceName="RH1_D01_rh1-pas" DIR_PROFILE=/sapmnt/RH1/profile START_PROFILE=/sapmnt/RH1/profile/RH1_D01_rh1-pas --group rh1_PAS_D01_group

Verify the configuration of the PAS SAPInstance resource using command below.

[root]# pcs resource show rh1_pas_d01
 Resource: rh1_pas_d01 (class=ocf provider=heartbeat type=SAPInstance)
  Attributes: DIR_PROFILE=/sapmnt/RH1/profile InstanceName=RH1_D01_rh1-pas START_PROFILE=/sapmnt/RH1/profile/RH1_D01_rh1-pas
  Operations: demote interval=0s timeout=320 (rh1_pas_d01-demote-interval-0s)
              monitor interval=120 timeout=60 (rh1_pas_d01-monitor-interval-120)
              monitor interval=121 role=Slave timeout=60 (rh1_pas_d01-monitor-interval-121)
              monitor interval=119 role=Master timeout=60 (rh1_pas_d01-monitor-interval-119)
              promote interval=0s timeout=320 (rh1_pas_d01-promote-interval-0s)
              start interval=0s timeout=180 (rh1_pas_d01-start-interval-0s)
              stop interval=0s timeout=240 (rh1_pas_d01-stop-interval-0s)

5. Test the cluster configuration

5.1. Check the constraints

# pcs constraint
Location Constraints:
  Resource: rh2_ascs20
    Constraint: location-rh2_ascs20
      Rule: score=2000
Expression: runs_ers_RH2 eq 1
Ordering Constraints:
  start rh2_ASCS20_group then stop rh2_ERS29_group (kind:Optional) (non-symmetrical)
Colocation Constraints:
  rh2_ERS29_group with rh2_ASCS20_group (score:-5000)
Ticket Constraints:

5.2. Failover ASCS due to node crash

Before the crash, ASCS is running on node1 while ERS running on node2.

# pcs status
Resource Group: rh2_ASCS20_group
    rh2_vip_ascs20  (ocf::heartbeat:IPaddr2):   Started node1
    rh2_fs_ascs20   (ocf::heartbeat:Filesystem):    Started node1
    rh2_ascs20  (ocf::heartbeat:SAPInstance):   Started node1
Resource Group: rh2_ERS29_group
    rh2_vip_ers29   (ocf::heartbeat:IPaddr2):   Started node2
    rh2_fs_ers29    (ocf::heartbeat:Filesystem):    Started node2
    rh2_ers29   (ocf::heartbeat:SAPInstance):   Started node2

On node2, run the following command to monitor the status changes in the cluster:

[root@node2 ~]# crm_mon -Arf

Crash node1 by running the following command. Please note that connection to node1 will be lost after the command.

[root@node1 ~]# echo c > /proc/sysrq-trigger

On node2, monitor the failover process. After failover, cluster should be in such state, with ASCS and ERS both on node2.

[root@node2 ~]# pcs status
Resource Group: rh2_ASCS20_group
    rh2_fs_ascs20   (ocf::heartbeat:Filesystem):    Started node2
    rh2_ascs20  (ocf::heartbeat:SAPInstance):   Started node2
    rh2_vip_ascs20  (ocf::heartbeat:IPaddr2):   Started node2
Resource Group: rh2_ERS29_group
    rh2_fs_ers29    (ocf::heartbeat:Filesystem):    Started node2
    rh2_vip_ers29   (ocf::heartbeat:IPaddr2):   Started node2
    rh2_ers29   (ocf::heartbeat:SAPInstance):   Started node2

5.3. ERS moves to the previously failed node

Bring node1 back online, and start the cluster on node1:

[root@node1 ~]# pcs cluster start

ERS should move to node1, while ASCS remaining on node2. Wait for ERS to finish the migration, and at the end the cluster should be in such state:

[root@node1 ~]# pcs status
Resource Group: rh2_ASCS20_group
    rh2_fs_ascs20   (ocf::heartbeat:Filesystem):    Started node2
    rh2_ascs20  (ocf::heartbeat:SAPInstance):   Started node2
    rh2_vip_ascs20  (ocf::heartbeat:IPaddr2):   Started node2
Resource Group: rh2_ERS29_group
    rh2_fs_ers29    (ocf::heartbeat:Filesystem):    Started node1
    rh2_vip_ers29   (ocf::heartbeat:IPaddr2):   Started node1
    rh2_ers29   (ocf::heartbeat:SAPInstance):   Started node1

6. Enable cluster to auto-start after reboot

The cluster is not yet enabled to auto-start after reboot. System admin needs to manually start the cluster after the node is fenced and rebooted.

After testing the previous section, when everything works fine, enable the cluster to auto-start after reboot.

# pcs cluster enable --all

Note: in some situations it can be beneficial not to have the cluster auto-start after a node has been rebooted. For example, if there is an issue with a filesystem that is required by a cluster resource, and the filesystem needs to be repaired first before it can be used again, having the cluster auto-start but then fail because the filesystem doesn't work can cause even more trouble.

Now please rerun the tests in previous section to make sure that cluster still works fine. Please note that in section 5.3., there is no need to run command pcs cluster start after a node is rebooted. Cluster should automatically start after reboot.

7. Optional - Enable SAP HAlib for Management by Outside Tools

When a system admin controls a SAP instance that is running inside the pacemaker cluster, either manually or using tools such as SAP Management Console (MC/MMC), the change needs to be done through the HA interface that's provided by the HA cluster software. SAP Start Service sapstartsrv controls the SAP instances, and needs to be configured to communicate with the pacemaker cluster software through the HA interface.

Please follow the kbase article to configure the HAlib: How to configure SAP HAlib for SAPInstance resources?

8. Multi-SID Support

The setup described in this document can also be used to manage the (A)SCS/ERS instances for multiple SAP environments (Multi-SID) within the same cluster. Below are some of the considerations:

8.1. Prerequisites

8.1.1. Unique SID and Instance Number

To avoid conflict, each pair of (A)SCS/ERS instances must use a different SID and each instance must use a unique Instance Number, even if they belong to different SID.

8.1.2. Sizing

Each cluster node must meet the SAP requirements for sizing to support the multiple instances.

8.1.3. Shared File Systems

Since shared file systems can be managed either by the cluster or outside the cluster, if you want to simplify the cluster management, you can choose to configure shared file systems managed outside of cluster, as documented in section 4.5.2.

8.2. Installation

For each (A)SCS/ERS pair please repeat all the steps documented in sections 4.6, 4.7 and 4.8 above.

Each (A)SCS/ERS pair will failover independently following the configuration rules.


Is there a way to setup the SAP dual ABAP & Java stack with standalone SAPInstance resources using separate resource groups for the ABAP stack and separate resource groups for the Java stack? I have followed the procedures but hit a problem when using the location constraint to instruct the java scs instance to run on the same node where the java ers instance was running prior to a failover event:

pcs constraint location rule score=2000 runs_ers_$SID eq 1

Unfortunately because the ABAP and Java stack share the same $SID, the java scs instance always moves to the node which is running the ABAP ers instance resource group, not the Java ers instance resource group. I believe the limitation is due to the condition 'runs_ers_$SID' where $SID needs to be a unique SAP id not a shared SAP id.

The only way I can overcome this is to use a master/slave resource to manage the ABAP instance and a master/slave resource to manage the Java instance. This is my least preferred, but currently only, method of managing the dual stack under pacemaker.

Hello Jonathan,

with the current version of the SAPInstance resource agent shipped with RHEL this won't be possible. To make it work we would have to make changes to the code of the resource agent, and then test it to make sure the changes don't break anything else, etc.

But since SAP no longer recommends to use dual-stack setups where both ABAP and JAVA instances share the same SID it would be better to consider doing a "dual-stack split of the SAP environment (see for more information), so that the ABAP and JAVA stack will use separate SIDs.

With such a setup you then could use the 'standalone' approach described in this article for managing the ASCS/ERS instances for the ABAP stack and the SCS/ERS instances for the JAVA stack by configuring separate resource groups for each instance and set up appropriate constraints based on the separate SIDs for each part of the environment.

Frank Danapfel
Senior Software Engineer
Red Hat SAP Alliance Technology Team

Dear, this solution need external NFS server for shared filesystem,if external NFS server is down,it means this cluster will down also,is it right? would try to use gfs2 filesystem for share filesystem? I try to use gfs2,but face "node unclean" issue where one node crashed.

There seems to be an issue in this constraint:

pcs constraint order start rh2_ASCS20_group then stop rh2_ERS29_group symmetrical=false

It should be :

pcs constraint order start rh2_ASCS20_group then start rh2_ERS29_group symmetrical=false

otherwise the ERS never starts as per our tests in normal situations.

4.8.3. Create order constraint for ASCS and ERS resource groups

I echo this concern. The constraint listed does not match the preceding text ("Prefer to start rh2_ASCS20_group before the rh2_ERS29_group (optionally)"). We need to fix the preceding text, the recommended constraint, or both -- depending on what's warranted. If we need this "start then stop" constraint, then we need to explain our reasoning in the text.

Making more explanations is always good. However in case here I wonder if kind=Optional part of constraint was used as default is kind=Mandatory if nothing is specified which would fit the description that "ERS never starts".