Deploying SAP Workloads with OpenShift Virtualization
NOTE: Red Hat does recommend customers to test their workloads on OpenShift Virtualization in test and development environments to gain experience. SAP does not support SAP HANA and SAP business applications deployed on OpenShift Virtualization. Red Hat fully supports OpenShift Virtualization independent from the workload perspective.
Contents
- 1. Prerequisites
- 2. Procedure
- 3. Ansible roles
- 4. Preparing the OS for the SAP workload
- 5. Comments regarding performance
- 6. References
1. Prerequisites
- A 4.12 or 4.14 OpenShift cluster.
- Worker nodes:
- At least 96GB of RAM (note: minimal memory checking can be disabled at your own risk).
- Identical worker nodes (same vendor, CPU, RAM, disks, network).
- Storage: OpenShift Data Foundation, NFS or local disk.
- Checkout
community.sap_install collection
from Ansible release 1.3.5. - sap_create_vm Ansible role.
2. Procedure
The following steps outline the entire process required in order to deploy a SAP workload on OpenShift Virtualization. This guide focuses on steps 2 and 3.
- Install the OpenShift cluster: This is not part of the document; see Installing a user-provisioned cluster on bare metal for more information on how to install OpenShift. An automated way of installing an OpenShift cluster is shown here: https://github.com/agonzalezrh/install_openshift.
- Configure worker nodes for SAP workloads with the
sap_hypervisor_preconfigure
role. - Create one or more VMs using the role
sap_create_vm
. - Make sure the
sap_*
roles are in the Ansible path. - Deploy the SAP workload:
a. NetWeaver usingcommunity.sap_install.sap_swpm
, or
b. SAP HANA usingcommunity.sap_install.sap_hana_install
.
3. Ansible roles
Install the Ansible collection from Ansible galaxy using version 1.3.5:
# ansible-galaxy collection install community.sap_install
3.1 Configuring a cluster for SAP workloads using sap_hypervisor_node_preconfigure
The role sap_hypervisor_node_preconfigure
is used to configure the OpenShift cluster. The steps automatically performed are:
- Set tuned profile to
virtual-host
. - Install CNV operator.
- Install SRIOV operator.
- Install
nmstate
operator. - Download and install
virtctl
. -
Setup worker nodes by executing the following steps:
a. Set the following kernel parameters:intel_iommu=on iommu=pt default_hugepagesz=1GB hugepagesz=1GB hugepages=<#1GBHugepages> tsx=on
Regarding huge pages: on every worker node defined, the amount of HP is determined by the following formula:
[Amount of RAM in worker] - 32 GB if [Amount of RAM] < 512 GB, 64 GB otherwise
b. Enable CPU Manager.
c. Configure network: SRIOV or Linux-Bridge (optional). - Install Trident and configure it (optional).
- Setup local storage (optional).
3.1.1. Requirements
- An OpenShift cluster.
- The worker nodes should have greater than 96GB of memory.
- Storage is required, e.g., via NFS, OpenShift Data Foundation, or local storage.
- This role can set up access to a NetApp Filer via the Trident storage connector.
- Local storage will be configured using the host path provisioner.
- Point the
KUBECONFIG
environment variable to yourkubeconfig
. - Required packages: This role uses the kubernetes Ansible module, which can be installed via the packages
ansible-collection-kubernetes-core
andpython3-kubernetes
. - Needs
oc
binary available in path. community.sap_install collection
from Ansible.
3.1.2. Role variables
Here are the variables shown that are used in this role. The defaults are set in the file sap_hypervisor_node_preconfigure/vars/platform_defaults_redhat_ocp_virt.yml
:
# Install and configure the host path provisioner (hpp) for a local storage disk
sap_hypervisor_node_preconfigure_install_hpp: False
# Install the trident NFS storage provider
sap_hypervisor_node_preconfigure_install_trident: False
# URL of the trident installer package to use
sap_hypervisor_node_preconfigure_install_trident_url: https://github.com/NetApp/trident/releases/download/v23.01.0/trident-installer-23.01.0.tar.gz
# should SRIOV be enabled for unsupported NICs
sap_hypervisor_node_preconfigure_sriov_enable_unsupported_nics: True
# Amount of memory [GB] to be reserved for the hypervisor on hosts >= 512GB
sap_hypervisor_node_preconfigure_hypervisor_reserved_ram_host_ge_512: 64 #GB
# Amount of memory [GB] to be reserved for the hypervisor on hosts < 512GB
sap_hypervisor_node_preconfigure_hypervisor_reserved_ram_host_lt_512: 32 #GB
# Should the check for the minimal amount of be ignored? Minimal amount is 96 GB
sap_hypervisor_node_preconfigure_ignore_minimal_memory_check: False
# Should the operators be installed
sap_hypervisor_node_preconfigure_install_operators: True
3.1.3. Configuration variables
The following variable structure is used to configure the cluster according to the actual requirements. The best practice is to create a copy of defaults/main.yml
and adjust it to your needs. For example, a minimal version (using host path provisioner) would look like this:
# Minimal configuration
sap_hypervisor_node_platform: redhat_ocp_virt
sap_hypervisor_node_preconfigure_install_hpp: True
sap_hypervisor_node_preconfigure_cluster_config:
# URL under which the OpenShift cluster is reachable
cluster_url: ocpcluster.domain.org
# namespace under which the VMs are created, note this has to be
# openshift-sriov-network-operator in case of using SRIOV network
# devices
vm_namespace: sap
# CPU cores reserved for kubernetes on worker node
worker_kubernetes_reserved_cpus: "0,1"
# Storage device which should be used if host path provisioner is used
worker_localstorage_device: /dev/sdb
# detailed configuration for every worker that should be configured
workers:
- name: worker-0 # name must match the node name
- name: worker-1 # second worker configuration
For a reference, a full example containing network configuration is in defaults/main.yml
:
# ibmpower_phyp, redhat_ocp_virt, redhat_rhel_kvm, vmware_vsphere
sap_hypervisor_node_platform: redhat_ocp_virt
# sap_hypervisor_node_preconfigure_cluster_config.trident, see example config.
sap_hypervisor_node_preconfigure_install_trident: True|False
# should SRIOV be enabled for unsupported NICs
sap_hypervisor_node_preconfigure_sriov_enable_unsupported_nics: True|False
# Define if the host path provisioner should be installed in order to use a local disk as storage device.
# Uses the following variable to be set to the storage device to be used, e.g.:
# sap_hypervisor_node_preconfigure_cluster_config.worker_localstorage_device: /dev/sdb
sap_hypervisor_node_preconfigure_install_hpp: True|False
# Example configuration
sap_hypervisor_node_preconfigure_cluster_config:
# URL under which the OpenShift cluster is reachable
cluster_url: ocpcluster.domain.org
# namespace under which the VMs are created, note this has to be
# openshift-sriov-network-operator in case of using SRIOV network
# devices
vm_namespace: sap
# Optional, configuration for trident driver for Netapp NFS filer
trident:
management: management.domain.org
data: datalif.netapp.domain.org
svm: sap_svm
backend: nas_backend
aggregate: aggregate_Name
username: admin
password: xxxxx
storage_driver: ontap-nas
storage_prefix: ocpv_sap_
# CPU cores reserved for kubernetes on worker node
worker_kubernetes_reserved_cpus: "0,1"
# Storage device which should be used if host path provisioner is used
worker_localstorage_device: /dev/vdb
# detailed configuration for every worker that should be configured
workers:
- name: worker-0 # name must match the node name
networks: # Example network config
- name: sapbridge # using a bridge
description: SAP bridge
state: up
type: linux-bridge
ipv4:
enabled: false
auto-gateway: false
auto-dns: false
bridge:
options:
stp:
enabled: false
port:
- name: ens1f0 # network IF name
- name: storage # an SRIOV device
interface: ens2f0 # network IF name
type: sriov
- bridge: # another bridge
options:
stp:
enabled: false
port:
- name: ens2f0 # network IF name
description: storage
mtu: 9000
ipv4:
address:
- ip: 192.168.1.51 # IP config
prefix-length: 24
auto-dns: false
auto-gateway: false
enabled: true
name: storagebridge
state: up
type: linux-bridge
- name: multi # another SRIOV device
interface: ens2f1 # network IF name
type: sriov
- name: worker-1 # second worker configuration
networks: # Example network config
- name: sapbridge # using a bridge
description: SAP bridge
state: up
type: linux-bridge
ipv4:
enabled: false
auto-gateway: false
auto-dns: false
bridge:
options:
stp:
enabled: false
port:
- name: ens1f0 # network IF name
- name: storage # an SRIOV device
interface: ens2f0 # network IF name
type: sriov
3.1.4. Example for playbook
See this example for a simple playbook that just calls the role. It’s in the file playbooks/sample-sap-hypervisor-redhat_ocp_virt-preconfigure.yml
:
- hosts: all
gather_facts: true
serial: 1
vars:
sap_hypervisor_node_platform: redhat_ocp_virt
sap_hypervisor_node_kubeconfig: "{{ lookup( 'ansible.builtin.env', 'KUBECONFIG') }}"
environment:
KUBECONFIG: "{{ sap_hypervisor_node_kubeconfig }}"
K8S_AUTH_KUBECONFIG: "{{ sap_hypervisor_node_kubeconfig }}"
roles:
- { role: community.sap_install.sap_hypervisor_node_preconfigure }
3.1.5. Running the playbook
In order to run the example, use the steps given below:
-
Make sure the
kubeconfig
file for your cluster is available through the environment variable:# export KUBECONFIG=path/to/kubeconfig
-
Go to the playbooks directory:
# cd community.sap_install/playbooks
-
To invoke the example playbook with the example configuration using your localhost as an Ansible host, use the following command line:
# ansible-playbook -v --connection=local -i localhost, sample-sap-hypervisor-redhat_ocp_virt-preconfigure.yml -e @<your copy of defaults/main.yml>
In case the entry Virtualization does not show up in the OpenShift UI, you have to clear the browser cache.
3.1.6. Setting up Storage
You need to set up a storage connection in order to continue installing VMs on your cluster. For example, you can use:
- NFS (with Trident)
- OpenShift Data Foundation
- local storage
Regarding NFS with NetApp Astra Trident, for additional information, refer to the blog Configure a NetApp SVM to expose NFS for OpenShift Virtualization. For the official NetApp documentation, refer to Learn about Astra Trident installation. Refer to Red Hat OpenShift Data Foundation for a starting point for setting up OpenShift Data Foundation.
3.2 Creating VMs using role sap_create_vm
Use the role sap_create_vm
from https://github.com/newkitlabs/sap_create_vm to create VMs with the performance tunings for SAP. The tunings are:
- 1G hugepages
- dedicated CPU placement
- NUMA CPU topology; see variables
sap_create_vm_cores
,sap_create_vm_threads
andsap_create_vm_sockets
- optional: use SRIOV network interfaces
3.2.1. Requirements
- An OpenShift Virtualization cluster configured with
sap_hypervisor_preconfigure_role
. - A default storage class setup along with enough storage space.
- The following packages:
python3-kubernetes
ansible-collection-kubernetes-core
They can be installed with the following command:
# yum install python3-kubernetes ansible-collection-kubernetes-core
3.2.2. Variables
The variables used in this role are as follows (see vars/main.yml
):
# Namespace the VMs are created in
sap_create_vm_namespace: sap
# Amount of memory [GB] the VM will have
sap_create_vm_memory: 32 # [GB]
######################################
# CPU Topology #######################
######################################
# The here defined virtual CPUs should be a subset of the physical CPUs.
# Ideally, the guest (VM) would have a full NUMA node (typically socket) minus two cores.
# E.g. a physical four socket host with 27 CPU cores (with 2 SMT threads)
# --> A guest with 25 CPU cores, 2 threads and 4 sockets. If the guest should be smaller, use less sockets.
# This settings will greatly effect performance.
#
# Number of CPU cores the VM will have
sap_create_vm_cores: 2
# Number of CPU threads
sap_create_vm_threads: 2
# Number of CPU sockets to use
sap_create_vm_sockets: 1
# Should the RHEL OS image be downloaded from Red Hat
sap_create_vm_download_rhel_images: true
# Version of the RHEL disk image to be downloaded and used as root disk, e.g. 8.6.0, 8.8.0
sap_create_vm_rhel_version: 8.8.0
# This amount of memory [GB] is reserved on top of the VM memory for the container to give it some headroom, only change with caution.
sap_create_vm_memory_overhead: 16 # [GB]
sap_create_vm_cloud_config:
userData: >+
#cloud-config
user: cloud-user
password: {{ sap_create_vm_cloud_user_passwd }}
chpasswd:
expire: false
network:
version: 2
ethernets:
eth0:
dhcp4: true
# Networks the VM is attached to, see below for an example
# The OpenShift Virtualization cluster has to have the according NetworkAttachmentDefinitions and NetworkPolicies
sap_create_vm_networks: []
# Interfaces the VM has, see below for an example.
# The OpenShift Virtualization cluster has to have the according NetworkAttachmentDefinitions and NetworkPolicies
sap_create_vm_interfaces: []
An example of a bridge and two SRIOV networks would look as shown below. Set sap_create_vm_interfaces
and sap_create_vm_networks
as follows:
sap_create_vm_interfaces:
- name: sapbridge-network-definition
bridge: {}
model: virtio
networkInterfaceMultiqueue: true
- name: storage-network-definition
sriov: {}
- name: multi-network-definition
sriov: {}
sap_create_vm_networks:
- multus:
networkName: sapbridge-network-definition
name: sapbridge-network-definition
- multus:
networkName: iface-storage-sriov
name: storage-network-definition
- multus:
networkName: iface-multi-sriov
name: multi-network-definition
The OS configuration in terms of existing users, passwords, SSH keys, etc. of the VM can be adjusted via the variable sap_create_vm_cloud_config
. Note the additional newlines in the definition that are required (for now). To find more information on what can be set through cloud-init, refer to the Cloud-init documentation.
3.2.3. Deploying a VM
In order to deploy a VM, perform the following steps:
- Get a checkout of the
sap_create_vm
role in your working directory:
# git clone https://github.com/newkitlabs/sap_create_vm.git
- Move or link that checkout to a directory in your Ansible path. Create
~/.ansible/roles
if it does not exist, for example:
# ln -sf ~/sap_create_vm/roles/sap_create_vm ~/.ansible/roles/
- Export your
kubeconfig
file, for example:
# export KUBECONFIG=$HOME/clusterconfigs-dciokd/kubeconfig
- To create a VM named
sapvm01
with 24GB of RAM and a password ofMyS5perS3cureP@assW0rD
run the following command:
# ansible-playbook -vvv --connection=local -i localhost, playbooks/sap-create-vm.yml -e sap_create_vm_names="['sapvm01']" -e sap_create_vm_memory=24 -e sap_create_vm_cloud_user_passwd=MyS5perS3cureP@assW0rD
- This will download the RHEL GA version specified in
sap_create_vm_rhel_version
, e.g.,8.8.0
. The behavior of whether the image will be downloaded can be controlled viasap_create_vm_download_rhel_images
.
4. Preparing the OS for the SAP workload
From the community.sap_install
repository, use the role sap_general_preconfigure
to configure your OS accordingly. For the installation of SAP HANA, use additionally the role sap_hana_preconfigure
and then sap_hana_install
. For Netweaver, use sap_netweaver_preconfigure
and then sap_swpm
.
5. Comments regarding performance
Comparing performance between bare metal and an OpenShift Virtualization VM shows a degradation of the performance of the virtual machine. This is due to the overhead caused by virtualization and is often seen in such environments. While some of the tests showed no impairment at all, others were impacted.
The impact is highly dependent on the workload. In certain mixed-olap/oltp scenarios, the delta is around 12%.
SAP BW edition for SAP HANA benchmarks (BWH) showed a slowdown of around 5% when running in OpenShift Virtualization.
Comments