Table of Contents
- 1. Overview
- 1.1. Introduction
- 1.2. Audience
- 1.3. Concepts
- 1.4. Resources: Standalone vs. Master/Slave
- 1.5. Support Policies
- 2. Requirements
- 2.1 Subscription
- 2.2. Pacemaker Resource Agents
- 2.3. SAP NetWeaver High-Availability Architecture
- 2.4. `SAPInstance` resource agent
- 2.5. Storage requirements
- 2.5.1. Instance Specific Directory
- 2.5.2. Shared Directories
- 2.5.3. Shared Directories on HANA
- 3. Install SAP Netweaver
- 3.1. Configuration options used in this document
- 3.2. Prepare hosts
- 3.3. Install Netweaver
- 3.3.1. Install ASCS on node1
- 3.3.2. Install ERS on node2
- 3.3.3. SAP HANA
- 3.3.4. Install Application Servers
- 3.4. Post Installation
- 3.4.1. (A)SCS profile modification
- 3.4.2. ERS profile modification
- 3.4.3. Update the `/usr/sap/sapservices` file
- 3.4.4. Create mount points for ASCS and ERS on the failover node, respectively:
- 3.4.5. Manual Testing Instance on Other Node
- 3.4.6. Check SAP HostAgent on all nodes
- 3.4.7. Install permanent SAP license keys
- 4. Install Pacemaker
- 4.1. Install Pacemaker rpm's
- 4.2. Create a Cluster
- 4.3. Configure STONITH
- 4.4. Install `resource-agents-sap` on all cluster nodes
- 4.5. Configure cluster resources for shared filesystems
- 4.6. Configure ASCS resource group
- 4.7. Configure ERS resource group
- 4.8. Create constraints
- 4.9 Add the ERS29 instance to ERS29 group
- 5. Test the cluster configuration
- 5.1. Check the constraints
- 5.2. Failover ASCS due to node crash
- 5.3. ERS moves to the previously failed node
- 6. Enable cluster to auto-start after reboot
Note this article is currently a preview that is subject to changes. For feedback on this article please use the comments below the article
SAP NetWeaver-based systems (like SAP ERP, SAP CRM, ...) play an important role in business processes, thus it's critical for SAP NetWeaver-based systems to be highly available. The underlying idea of Clustering is a fairly simple one: Not a single large machine bears all of the load and risk; but rather one or more machines automatically drop in as an instant full replacement for the service or the machine that has failed. In the best case, this replacement process causes no interruption to the systems' users.
This document is intended for SAP and Red Hat certified or trained administrators and consultants who already have experience setting up high available solutions using the RHEL HA add-on or other clustering solutions. Access to both SAP Service Marketplace and Red Hat Customer Portal is required to be able to download software and additional documentation.
Red Hat Consulting is highly recommend to set up the cluster and customize the solution to meet customers' data center requirements, that are normally more complex than the solution presented in this document.
This document describes how to set up a two-node-cluster solution that conforms to the guidelines for high availability that have been established by both SAP and Red Hat. It is based on SAP NetWeaver on top of Red Hat Enterprise Linux 7.5 or newer with RHEL HA Add-on. The reference architecture described in this document is under the tests of SAP High Availability Interface certification NW-HA-CLU 7.50. This document will be updated after the certification is obtained.
At application tier, the lock table of the enqueue server is the most critical piece. To protect it, SAP has developed the “Enqueue Replication Server” (ERS) which maintains a backup copy of the lock table. While the (A)SCS is running on node1, the ERS always needs to maintain a copy of the current enqueue table on node2. When the system needs to failover the (A)SCS to node2, it first starts on node2, and then shuts down the ERS, taking over its' shared memory segment and thereby acquiring an up-to-date enqueue table, thus the replicated enqueue table becomes the new primary enqueue table. ERS will start on node1 when it becomes available.
Required by the SAP HA-Interface certification, this document focuses on the high availability of (A)SCS and ERS, both are controlled by the cluster software. In normal mode, cluster ensures that ERS and (A)SCS always run on different nodes. In the event of failover, the (A)SCS needs to “follow” the ERS. (A)SCS needs to switch to the node where ERS is running. Cluster has to ensure that the ERS is already running when (A)SCS starts up, because (A)SCS needs to take over the replicated enqueue table from ERS.
1.4. Resources: Standalone vs. Master/Slave
There are two approaches to configure (A)SCS and ERS resources in Pacemaker: Master/Slave and Standalone. RHEL 7.4 and older supports Master/Slave approach which has been deprecated but still supported for existing customers. RHEL 7.5 and newer supports Standalone approach. In any new deployment, Standalone should be implemented because it meets the requirements of the SAP HA-Interface certification.
SAPInstance Standalone and Master/Slave:
- Binaries are on shared storage in Standalone approach, while some directories are on local storage in Master/Slave
- Additional constraints are needed for Standalone approach
- In Standalone approach, the fatal failure of ERS is more independent from ASCS than in the Master/Slave approach
This article outlines the configuration procedure of the
SAPinstance Standalone approach. For instructions on
SAPInstance Master/Slave configuration, please refer to kabse article SAP Netweaver in Pacemaker cluster with Master/Slave SAPInstance resource
1.5. Support Policies
It’s important to keep the subscription, kernel, and patch level identical on both nodes.
Please follow this kbase article to subscribe your systems to the Update Service for RHEL for SAP Solutions.
2.2. Pacemaker Resource Agents
- RHEL for SAP Solutions 7.5 or newer
- On RHEL versions older than 7.5, install
resource-agents-sap-3.9.5-124.el7.x86_64or newer that contains the
2.3. SAP NetWeaver High-Availability Architecture
A typical setup for SAP NetWeaver High-Availability consists of 3 distinctive components:
- SAP Netweaver ASCS/ERS cluster resources
- SAP Netweaver application servers - Primary application server (PAS) and additional application servers (AAS)
- Database: SAP HANA or Oracle/DB2/ASE/MaxDB
This article focuses on the configuration of SAP Netweaver ASCS and ERS in a pacemaker cluster. As the best practice, we recommend to install Application Servers and Database on separate nodes outside of the two-node cluster designated for (A)SCS and ERS.
Below is the architecture diagram of the example installation:
2.4. `SAPInstance` resource agent
SAPInstance is a pacemaker resource agent used for both ASCS and ERS resources. All operations of the
SAPInstance resource agent are done by using the SAP start service framework
2.5. Storage requirements
Directories created for Netweaver installation should be put on shared storage, following the rules:
2.5.1. Instance Specific Directory
The instance specific directory for 'ASCS' and 'ERS', respectively, must be present on the corresponding node. These directories must be available before the cluster is started.
- ASCS node: /usr/sap/SID/ASCS<Ins#>
- ERS node: /usr/sap/SID/ERS<Ins#>
For Application Servers, the following directory should be made available on corresponding node designated for the Application Server instance:
- App Server D<Ins#>: /usr/sap/SID/D<Ins#>
2.5.2. Shared Directories
The following mount points must be available on ASCS, ERS, and Application Servers nodes.
/sapmnt /usr/sap/trans /usr/sap/SID/SYS
2.5.3. Shared Directories on HANA
The following mount point(s) must be available on the HANA node.
Shared storage can be achieved by:
- use of the external NFS server (NFS server cannot run on any of nodes inside the cluster in which the shares would be mounted, more details about this limitation can be found in article Hangs occur if a Red Hat Enterprise Linux system is used as both NFS server and NFS client for the same mount)
- using the GFS2 filesystem (this requires all nodes to have
Resilient Storage Add-on)
- using the glusterfs filesystem (check the additional notes in article Can glusterfs be used for the SAP NetWeaver shared filesystems?)
These mount points must be either managed by cluster or
mounted before cluster is started.
3. Install SAP Netweaver
3.1. Configuration options used in this document
Below are configuration options that will be used for instances in this document:
Two nodes will be running the ASCS/ERS instances in pacemaker:
1st node hostname: node1 2nd node hostname: node2 SID: RH2 ASCS Instance number: 20 ASCS virtual hostname: rhascs ERS Instance number: 29 ERS virtual hostname: rhers
Outside the two-node cluster:
PAS Instance number: 21 AAS Instance number: 22
SID: RH0 HANA Instance number: 00 HANA virtual hostname: rhdb
3.2. Prepare hosts
Before starting installation ensure that:
- Install RHEL for SAP Solutions 7.x (latest is recommended)
- Register system to RHN or Satellite, enable RHEL for SAP Applications channel, or Update Services (E4S) channel
- Enable High Availability Add-on channel
- Shared storage and filesystems are present at correct mount points
- Virtual IP addresses used by instances are present and reachable
- Hostnames that will be used by instances can be resolved to IP addresses and back
- Installation medias are available
- System is configured according to the recommendation for running SAP Netweaver
- Red Hat Enterprise Linux 7.x: Installation and Upgrade - SAP Note 2002167
3.3. Install Netweaver
Using software provisioning manager (SWPM) install instances in the following order:
- ASCS instance
- ERS instance
- DB instance
- PAS instance
- AAS instances
3.3.1. Install ASCS on node1
The following file systems should be mounted on node1, where ASCS will be installed:
/usr/sap/RH2/ASCS20 /usr/sap/RH2/SYS /usr/sap/trans /sapmnt
Virtual IP for
rhascs should be enabled on node1.
Run the installer:
[root@node1]# ./sapinst SAPINST_USE_HOSTNAME=rhascs
High-Availability System option.
3.3.2. Install ERS on node2
The following file systems should be mounted on node2, where ERS will be installed:
/usr/sap/RH2/ERS29 /usr/sap/RH2/SYS /usr/sap/trans /sapmnt
Virtual IP for
rhers should be enabled on node2.
Run the installer:
[root@node2]# ./sapinst SAPINST_USE_HOSTNAME=rhers
High-Availability System option.
3.3.3. SAP HANA
In the example, SAP HANA will be using the following configuration. You can also use other supported database.
SAP HANA SID: RH0 SAP HANA Instance number: 00
SAP HANA should be installed on separate host. Optionally, Automated HANA System Replication can be installed in another pacemaker cluster by following document SAP HANA system replication in pacemaker cluster.
Run the installer on the HANA host:
[root]# ./sapinst SAPINST_USE_HOSTNAME=rhdb
3.3.4. Install Application Servers
The following file systems should be mounted on the host to run the Application Server instance. If you have multiple application servers, install each one on corresponding host:
/usr/sap/RH2/D<Ins#> /usr/sap/RH2/SYS /usr/sap/trans /sapmnt
Run the installer:
High-Availability System option.
3.4. Post Installation
3.4.1. (A)SCS profile modification
(A)SCS instance requires following modification in profile to prevent automatic restart of enqueue server as it will be managed by cluster. To apply the change run the following command at your ASCS profile
[root]# sed -i -e 's/Restart_Program_01/Start_Program_01/' /sapmnt/RH2/profile/RH2_ASCS20_rhascs
3.4.2. ERS profile modification
ERS instance requires following modification in profile to prevent automatic restart as it will be managed by cluster. To apply the change run the following command at your ERS profile
[root]# sed -i -e 's/Restart_Program_00/Start_Program_00/' /sapmnt/RH2/profile/RH2_ERS29_rhers
3.4.3. Update the `/usr/sap/sapservices` file
On both node1 and node2, make sure following two lines are commented out in
#LD_LIBRARY_PATH=/usr/sap/RH2/ASCS20/exe:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; /usr/sap/RH2/ASCS20/exe/sapstartsrv pf=/usr/sap/RH2/SYS/profile/RH2_ASCS20_rhascs -D -u rh2adm #LD_LIBRARY_PATH=/usr/sap/RH2/ERS29/exe:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; /usr/sap/RH2/ERS29/exe/sapstartsrv pf=/usr/sap/RH2/ERS29/profile/RH2_ERS29_rhers -D -u rh2adm
3.4.4. Create mount points for ASCS and ERS on the failover node, respectively:
[root@node1 ~]# mkdir /usr/sap/RH2/ERS29 [root@node1 ~]# chown rh2adm:sapsys /usr/sap/RH2/ERS29 [root@node2 ~]# mkdir /usr/sap/RH2/ASCS20 [root@node2 ~]# chown rh2adm:sapsys /usr/sap/RH2/ASCS20
3.4.5. Manual Testing Instance on Other Node
Stop ASCS and ERS instances. Move the instance specific directory to the other node:
[root@node1 ~]# umount /usr/sap/RH2/ASCS20 [root@node2 ~]# mount /usr/sap/RH2/ASCS20 [root@node2 ~]# umount /usr/sap/RH2/ERS29 [root@node1 ~]# mount /usr/sap/RH2/ERS29
Manually start ASCS and ERS instances on other cluster node, then manually stop them, respectively.
3.4.6. Check SAP HostAgent on all nodes
On all nodes check if SAP HostAgent has the same version and meets the minimum version requirement:
[root]# /usr/sap/hostctrl/exe/saphostexec -version
To upgrade/install SAP HostAgent, follow SAP note 1031096.
3.4.7. Install permanent SAP license keys
SAP hardware key determination in the high-availability scenario has been improved. It might be necessary to install several SAP license keys based on the hardware key of each cluster node. Please see SAP Note 1178686 - Linux: Alternative method to generate a SAP hardware key for more information.
4. Install Pacemaker
Follow Pacemaker documentation: HA Add-On Reference - RHEL 7.
Below is a sample procedure to install pacemaker. It's recommended to work with a Red Hat consultant to install and configure Pacemaker in your environment.
4.1. Install Pacemaker rpm's
# yum -y install pcs pacemaker # passwd hacluster [provide a password] # systemctl enable pcsd.service; systemctl start pcsd.service
4.2. Create a Cluster
Create a cluster named
nwha, consisting of
node2, and start the cluster. Please note that at this point, cluster is not yet configured to auto-start after reboot.
# pcs cluster auth node1 node2 # pcs cluster setup --name nwha node1 node2 # pcs cluster start --all
4.3. Configure STONITH
The fencing mechanism STONITH depends on the underneath platform. Please check the corresponding document to configure the STONITH. Support Policies for RHEL High Availability Clusters - General Requirements for Fencing/STONITH.
After configuring the STONITH, on
node1 test fencing
[root@node1]# pcs stonith fence node2
node2 should be properly fenced. After fencing, start cluster on node2 using the following command. This is because the cluster has not yet been enabled to auto-start. Auto-start will be enabled after initial testings showing the cluster is properly configured.
[root@node2 ~]# pcs cluster start
4.4. Install `resource-agents-sap` on all cluster nodes
[root]# yum install resource-agents-sap
4.5. Configure cluster resources for shared filesystems
Configure shared filesystem to provide following mount points on all of cluster nodes.
/sapmnt /usr/sap/trans /usr/sap/RH2/SYS
Filesystem cluster resource can be used to mount the shares from external NFS server on all cluster nodes as shown below.
[root]# pcs resource create rh2_fs_sapmnt Filesystem device='<NFS_Server>:<sapmnt_nfs_share>' directory='/sapmnt' fstype='nfs' --clone interleave=true [root]# pcs resource create rh2_fs_sap_trans Filesystem device='<NFS_Server>:<sap_trans_nfs_share>' directory='/usr/sap/trans' fstype='nfs' --clone interleave=true [root]# pcs resource create rh2_fs_sap_sys Filesystem device='<NFS_Server>:<rh2_sys_nfs_share>' directory='/usr/sap/RH2/SYS' fstype='nfs' --clone interleave=true
After creating the
Filesystem resources verify that they have started properly on all nodes.
[root]# pcs status ... Clone Set: rh2_fs_sapmnt-clone [rh2_fs_sapmnt] Started: [ node1 node2 ] Clone Set: rh2_fs_sap_trans-clone [rh2_fs_sap_trans] Started: [ node1 node2 ] Clone Set: rh2_fs_sys-clone [rh2_fs_sys] Started: [ node1 node2 ] ...
4.6. Configure ASCS resource group
Create group with ASCS filesystem, IP address and ASCS instance.
# pcs resource create rh2_fs_ascs20 Filesystem device='<NFS_Server>:<rh2_ascs20_nfs_share>' directory=/usr/sap/RH2/ASCS20 fstype=nfs force_unmount=safe --group rh2_ASCS20_group # pcs resource create rh2_vip_ascs20 IPaddr2 ip=192.168.200.101 --group rh2_ASCS20_group # pcs resource create rh2_ascs20 SAPInstance InstanceName="RH2_ASCS20_rhascs" START_PROFILE=/sapmnt/RH2/profile/RH2_ASCS20_rhascs AUTOMATIC_RECOVER=false meta resource-stickiness=5000 migration-threshold=1 failure-timeout=60 --group rh2_ASCS20_group
meta resource-stickiness=5000 is here to balance out the failover constraint with ERS so the resource stays on the node where it started and doesn't migrate around cluster uncontrollably.
migration-threshold=1 ensures that ASCS failover to another node when issue is detected instead of restarting on same node.
4.7. Configure ERS resource group
Create group with ERS filesystem and IP address.
# pcs resource create rh2_fs_ers29 Filesystem device='<NFS_Server>:<rh2_ers29_nfs_share>' directory='/usr/sap/RH2/ERS29' fstype=nfs force_unmount=safe --group rh2_ERS29_group # pcs resource create rh2_vip_ers29 IPaddr2 ip=192.168.200.102 --group rh2_ERS29_group
4.8. Create constraints
rh2_ASCS20_group and rh2_ERS29_group should try to avoid running on same node. Order of groups matters.
# pcs constraint colocation add rh2_ERS29_group with rh2_ASCS20_group -5000
ASCS20 instance (rh2_ascs20) prefers to run on node where ERS was running before when failover is happening
# pcs constraint location rh2_ascs20 rule score=2000 runs_ers_RH2 eq 1
Prefer to start ASCS20 before the ERS29 (optionally)
# pcs constraint order rh2_ASCS20_group then rh2_ERS29_group symmetrical=false kind=Optional
4.9 Add the ERS29 instance to ERS29 group
# pcs resource create rh2_ers29 SAPInstance InstanceName="RH2_ERS29_rhers" START_PROFILE=/sapmnt/RH2/profile/RH2_ERS29_rhers AUTOMATIC_RECOVER=false IS_ERS=true --group rh2_ERS29_group
5. Test the cluster configuration
5.1. Check the constraints
# pcs constraint Location Constraints: Resource: rh2_ascs20 Constraint: location-rh2_ascs20 Rule: score=2000 Expression: runs_ers_RH2 eq 1 Ordering Constraints: start rh2_ASCS20_group then start rh2_ERS29_group (kind:Optional) (non-symmetrical) Colocation Constraints: rh2_ERS29_group with rh2_ASCS20_group (score:-5000) Ticket Constraints:
5.2. Failover ASCS due to node crash
Before the crash, ASCS is running on node1 while ERS running on node2.
# pcs status ... Resource Group: rh2_ASCS20_group rh2_vip_ascs20 (ocf::heartbeat:aws-vpc-move-ip): Started node1 rh2_fs_ascs20 (ocf::heartbeat:Filesystem): Started node1 rh2_ascs20 (ocf::heartbeat:SAPInstance): Started node1 Resource Group: rh2_ERS29_group rh2_vip_ers29 (ocf::heartbeat:aws-vpc-move-ip): Started node2 rh2_fs_ers29 (ocf::heartbeat:Filesystem): Started node2 rh2_ers29 (ocf::heartbeat:SAPInstance): Started node2 ...
On node2, run the following command to monitor the status changes in the cluster:
[root@node2 ~]# crm_mon -Arf
Crash node1 by running the following command. Please note that connection to node1 will be lost after the command.
[root@node1 ~]# echo c > /proc/sysrq-trigger
On node2, monitor the failover process. After failover, cluster should be in such state, with ASCS and ERS both on node2.
[root@node2 ~]# pcs status ... Resource Group: rh2_ASCS20_group rh2_fs_ascs20 (ocf::heartbeat:Filesystem): Started node2 rh2_ascs20 (ocf::heartbeat:SAPInstance): Started node2 rh2_vip_ascs20 (ocf::heartbeat:aws-vpc-move-ip): Started node2 Resource Group: rh2_ERS29_group rh2_fs_ers29 (ocf::heartbeat:Filesystem): Started node2 rh2_vip_ers29 (ocf::heartbeat:aws-vpc-move-ip): Started node2 rh2_ers29 (ocf::heartbeat:SAPInstance): Started node2 ...
5.3. ERS moves to the previously failed node
Bring node1 back online, and start the cluster on node1:
[root@node1 ~]# pcs cluster start
ERS should move to node1, while ASCS remaining on node2. Wait for ERS to finish the migration, and at the end the cluster should be in such state:
[root@node1 ~]# pcs status ... Resource Group: rh2_ASCS20_group rh2_fs_ascs20 (ocf::heartbeat:Filesystem): Started node2 rh2_ascs20 (ocf::heartbeat:SAPInstance): Started node2 rh2_vip_ascs20 (ocf::heartbeat:aws-vpc-move-ip): Started node2 Resource Group: rh2_ERS29_group rh2_fs_ers29 (ocf::heartbeat:Filesystem): Started node1 rh2_vip_ers29 (ocf::heartbeat:aws-vpc-move-ip): Started node1 rh2_ers29 (ocf::heartbeat:SAPInstance): Started node1 ...
6. Enable cluster to auto-start after reboot
The cluster is not yet enabled to auto-start after reboot. System admin needs to manually start the cluster after the node is fenced and rebooted. After testing the previous section, when everything works fine, enable the cluster to auto-start after reboot:
# pcs cluster enable --all
Now please rerun the tests in previous section to make sure that cluster still works fine. Please note that in section 5.3., there is no need to run command
pcs cluster start after a node is rebooted. Cluster should automatically start after reboot.
[image=[src="images/nw-ha-architecture.png", title="NW HA ", alt="NW HA ", size="LG - Large", data-cp-size="100%", data-cp-align="left", caption="NW HA ", lightbox="true"]]