Deploying SAP Workloads with OpenShift Virtualization

Updated -

NOTE: Red Hat does recommend customers to test their workloads on OpenShift Virtualization in test and development environments to gain experience. SAP does not support SAP HANA and SAP business applications deployed on OpenShift Virtualization. Red Hat fully supports OpenShift Virtualization independent from the workload perspective.

Contents

1. Prerequisites

  • A 4.12 or 4.14 OpenShift cluster.
  • Worker nodes:
    • At least 96GB of RAM (note: minimal memory checking can be disabled at your own risk).
    • Identical worker nodes (same vendor, CPU, RAM, disks, network).
  • Storage: OpenShift Data Foundation, NFS or local disk.
  • Checkout community.sap_install collection from Ansible release 1.3.5.
  • sap_create_vm Ansible role.

2. Procedure

The following steps outline the entire process required in order to deploy a SAP workload on OpenShift Virtualization. This guide focuses on steps 2 and 3.

  1. Install the OpenShift cluster: This is not part of the document; see Installing a user-provisioned cluster on bare metal for more information on how to install OpenShift. An automated way of installing an OpenShift cluster is shown here: https://github.com/agonzalezrh/install_openshift.
  2. Configure worker nodes for SAP workloads with the sap_hypervisor_preconfigure role.
  3. Create one or more VMs using the role sap_create_vm.
  4. Make sure the sap_* roles are in the Ansible path.
  5. Deploy the SAP workload:
    a. NetWeaver using community.sap_install.sap_swpm, or
    b. SAP HANA using community.sap_install.sap_hana_install.

3. Ansible roles

Install the Ansible collection from Ansible galaxy using version 1.3.5:

# ansible-galaxy collection install community.sap_install

3.1 Configuring a cluster for SAP workloads using sap_hypervisor_node_preconfigure

The role sap_hypervisor_node_preconfigure is used to configure the OpenShift cluster. The steps automatically performed are:

  1. Set tuned profile to virtual-host.
  2. Install CNV operator.
  3. Install SRIOV operator.
  4. Install nmstate operator.
  5. Download and install virtctl.

  6. Setup worker nodes by executing the following steps:
    a. Set the following kernel parameters:

    intel_iommu=on iommu=pt default_hugepagesz=1GB hugepagesz=1GB hugepages=<#1GBHugepages> tsx=on

    Regarding huge pages: on every worker node defined, the amount of HP is determined by the following formula:

    [Amount of RAM in worker] - 32 GB if [Amount of RAM] < 512 GB, 
                                    64 GB otherwise

    b. Enable CPU Manager.
    c. Configure network: SRIOV or Linux-Bridge (optional).

  7. Install Trident and configure it (optional).
  8. Setup local storage (optional).

3.1.1. Requirements

  • An OpenShift cluster.
  • The worker nodes should have greater than 96GB of memory.
  • Storage is required, e.g., via NFS, OpenShift Data Foundation, or local storage.
    • This role can set up access to a NetApp Filer via the Trident storage connector.
    • Local storage will be configured using the host path provisioner.
  • Point the KUBECONFIG environment variable to your kubeconfig.
  • Required packages: This role uses the kubernetes Ansible module, which can be installed via the packages ansible-collection-kubernetes-core and python3-kubernetes.
  • Needs oc binary available in path.
  • community.sap_install collection from Ansible.

3.1.2. Role variables

Here are the variables shown that are used in this role. The defaults are set in the file sap_hypervisor_node_preconfigure/vars/platform_defaults_redhat_ocp_virt.yml:

# Install and configure the host path provisioner (hpp) for a local storage disk
sap_hypervisor_node_preconfigure_install_hpp: False

# Install the trident NFS storage provider
sap_hypervisor_node_preconfigure_install_trident: False
# URL of the trident installer package to use
sap_hypervisor_node_preconfigure_install_trident_url: https://github.com/NetApp/trident/releases/download/v23.01.0/trident-installer-23.01.0.tar.gz

# should SRIOV be enabled for unsupported NICs
sap_hypervisor_node_preconfigure_sriov_enable_unsupported_nics: True

# Amount of memory [GB] to be reserved for the hypervisor on hosts >= 512GB
sap_hypervisor_node_preconfigure_hypervisor_reserved_ram_host_ge_512: 64 #GB
# Amount of memory [GB] to be reserved for the hypervisor on hosts < 512GB
sap_hypervisor_node_preconfigure_hypervisor_reserved_ram_host_lt_512: 32 #GB

# Should the check for the minimal amount of be ignored? Minimal amount is 96 GB
sap_hypervisor_node_preconfigure_ignore_minimal_memory_check: False

# Should the operators be installed
sap_hypervisor_node_preconfigure_install_operators: True

3.1.3. Configuration variables

The following variable structure is used to configure the cluster according to the actual requirements. The best practice is to create a copy of defaults/main.yml and adjust it to your needs. For example, a minimal version (using host path provisioner) would look like this:

# Minimal configuration

sap_hypervisor_node_platform: redhat_ocp_virt
sap_hypervisor_node_preconfigure_install_hpp: True

sap_hypervisor_node_preconfigure_cluster_config:
  # URL under which the OpenShift cluster is reachable
  cluster_url: ocpcluster.domain.org

  # namespace under which the VMs are created, note this has to be
  # openshift-sriov-network-operator in case of using SRIOV network
  # devices
  vm_namespace: sap

  # CPU cores reserved for kubernetes on worker node
  worker_kubernetes_reserved_cpus: "0,1"

  # Storage device which should be used if host path provisioner is used
  worker_localstorage_device: /dev/sdb

  # detailed configuration for every worker that should be configured
  workers:
    - name: worker-0                   # name must match the node name
    - name: worker-1                   # second worker configuration

For a reference, a full example containing network configuration is in defaults/main.yml:

# ibmpower_phyp, redhat_ocp_virt, redhat_rhel_kvm, vmware_vsphere
sap_hypervisor_node_platform: redhat_ocp_virt

# sap_hypervisor_node_preconfigure_cluster_config.trident, see example config.
sap_hypervisor_node_preconfigure_install_trident: True|False

# should SRIOV be enabled for unsupported NICs
sap_hypervisor_node_preconfigure_sriov_enable_unsupported_nics: True|False

# Define if the host path provisioner should be installed in order to use a local disk as storage device.
# Uses the following variable to be set to the storage device to be used, e.g.:
# sap_hypervisor_node_preconfigure_cluster_config.worker_localstorage_device: /dev/sdb 
sap_hypervisor_node_preconfigure_install_hpp: True|False


# Example configuration
sap_hypervisor_node_preconfigure_cluster_config:

  # URL under which the OpenShift cluster is reachable
  cluster_url: ocpcluster.domain.org

  # namespace under which the VMs are created, note this has to be
  # openshift-sriov-network-operator in case of using SRIOV network
  # devices
  vm_namespace: sap

  # Optional, configuration for trident driver for Netapp NFS filer
  trident:
    management: management.domain.org
    data: datalif.netapp.domain.org
    svm: sap_svm
    backend: nas_backend
    aggregate: aggregate_Name
    username: admin
    password: xxxxx
    storage_driver: ontap-nas
    storage_prefix: ocpv_sap_

  # CPU cores reserved for kubernetes on worker node
  worker_kubernetes_reserved_cpus: "0,1"

  # Storage device which should be used if host path provisioner is used
  worker_localstorage_device: /dev/vdb

  # detailed configuration for every worker that should be configured
  workers:
    - name: worker-0                   # name must match the node name
      networks:                        # Example network config
        - name: sapbridge              # using a bridge
          description: SAP bridge
          state: up
          type: linux-bridge
          ipv4:
            enabled: false
            auto-gateway: false
            auto-dns: false
          bridge:
            options:
              stp:
                enabled: false
            port:
              - name: ens1f0           # network IF name
        - name: storage                # an SRIOV device
          interface: ens2f0            # network IF name
          type: sriov

        - bridge:                      # another bridge
            options:
              stp:
                enabled: false
            port:
              - name: ens2f0           # network IF name
          description: storage
          mtu: 9000
          ipv4:
            address:
              - ip: 192.168.1.51         # IP config
                prefix-length: 24
            auto-dns: false
            auto-gateway: false
          enabled: true
          name: storagebridge
          state: up
          type: linux-bridge
        - name: multi                  # another SRIOV device
          interface: ens2f1            # network IF name
          type: sriov

    - name: worker-1                   # second worker configuration
      networks:                        # Example network config
        - name: sapbridge              # using a bridge
          description: SAP bridge
          state: up
          type: linux-bridge
          ipv4:
            enabled: false
            auto-gateway: false
            auto-dns: false
          bridge:
            options:
              stp:
                enabled: false
            port:
              - name: ens1f0           # network IF name
        - name: storage                # an SRIOV device
          interface: ens2f0            # network IF name
          type: sriov

3.1.4. Example for playbook

See this example for a simple playbook that just calls the role. It’s in the file playbooks/sample-sap-hypervisor-redhat_ocp_virt-preconfigure.yml:

- hosts: all
  gather_facts: true
  serial: 1
  vars:
    sap_hypervisor_node_platform: redhat_ocp_virt
    sap_hypervisor_node_kubeconfig: "{{ lookup( 'ansible.builtin.env', 'KUBECONFIG') }}"
  environment:
    KUBECONFIG: "{{ sap_hypervisor_node_kubeconfig }}"
    K8S_AUTH_KUBECONFIG: "{{ sap_hypervisor_node_kubeconfig }}"
  roles:
    - { role: community.sap_install.sap_hypervisor_node_preconfigure }

3.1.5. Running the playbook

In order to run the example, use the steps given below:

  1. Make sure the kubeconfig file for your cluster is available through the environment variable:

    # export KUBECONFIG=path/to/kubeconfig

  2. Go to the playbooks directory:

    # cd community.sap_install/playbooks

  3. To invoke the example playbook with the example configuration using your localhost as an Ansible host, use the following command line:

    # ansible-playbook -v --connection=local -i localhost,  sample-sap-hypervisor-redhat_ocp_virt-preconfigure.yml -e @<your copy of defaults/main.yml>

    In case the entry Virtualization does not show up in the OpenShift UI, you have to clear the browser cache.

3.1.6. Setting up Storage

You need to set up a storage connection in order to continue installing VMs on your cluster. For example, you can use:

  • NFS (with Trident)
  • OpenShift Data Foundation
  • local storage

Regarding NFS with NetApp Astra Trident, for additional information, refer to the blog Configure a NetApp SVM to expose NFS for OpenShift Virtualization. For the official NetApp documentation, refer to Learn about Astra Trident installation. Refer to Red Hat OpenShift Data Foundation for a starting point for setting up OpenShift Data Foundation.

3.2 Creating VMs using role sap_create_vm

Use the role sap_create_vm from https://github.com/newkitlabs/sap_create_vm to create VMs with the performance tunings for SAP. The tunings are:

  • 1G hugepages
  • dedicated CPU placement
  • NUMA CPU topology; see variables sap_create_vm_cores, sap_create_vm_threads and sap_create_vm_sockets
  • optional: use SRIOV network interfaces

3.2.1. Requirements

  • An OpenShift Virtualization cluster configured with sap_hypervisor_preconfigure_role.
  • A default storage class setup along with enough storage space.
  • The following packages:
    • python3-kubernetes
    • ansible-collection-kubernetes-core
      They can be installed with the following command:
# yum install python3-kubernetes ansible-collection-kubernetes-core

3.2.2. Variables

The variables used in this role are as follows (see vars/main.yml):

# Namespace the VMs are created in
sap_create_vm_namespace: sap

# Amount of memory [GB] the VM will have
sap_create_vm_memory: 32 # [GB]

######################################
# CPU Topology #######################
######################################
# The here defined virtual CPUs should be a subset of the physical CPUs.
# Ideally, the guest (VM) would have a full NUMA node (typically socket) minus two cores.
# E.g. a physical four socket host with 27 CPU cores (with 2 SMT threads)
# --> A guest with 25 CPU cores, 2 threads and 4 sockets. If the guest should be smaller, use less sockets.
# This settings will greatly effect performance.
#
# Number of CPU cores the VM will have
sap_create_vm_cores: 2

# Number of CPU threads
sap_create_vm_threads: 2

# Number of CPU sockets to use
sap_create_vm_sockets: 1

# Should the RHEL OS image be downloaded from Red Hat
sap_create_vm_download_rhel_images: true

# Version of the RHEL disk image to be downloaded and used as root disk, e.g. 8.6.0, 8.8.0
sap_create_vm_rhel_version: 8.8.0

# This amount of memory [GB] is reserved on top of the VM memory for the container to give it some headroom, only change with caution.
sap_create_vm_memory_overhead: 16 # [GB]

sap_create_vm_cloud_config:
      userData: >+
        #cloud-config

        user: cloud-user

        password: {{ sap_create_vm_cloud_user_passwd }}

        chpasswd:
          expire: false

        network:
          version: 2
          ethernets:
            eth0:
              dhcp4: true
# Networks the VM is attached to, see below for an example
# The OpenShift Virtualization cluster has to have the according NetworkAttachmentDefinitions and NetworkPolicies
sap_create_vm_networks: []

# Interfaces the VM has, see below for an example.
# The OpenShift Virtualization cluster has to have the according NetworkAttachmentDefinitions and NetworkPolicies
sap_create_vm_interfaces: []

An example of a bridge and two SRIOV networks would look as shown below. Set sap_create_vm_interfaces and sap_create_vm_networks as follows:

sap_create_vm_interfaces:
   - name: sapbridge-network-definition
     bridge: {}
     model: virtio
     networkInterfaceMultiqueue: true
   - name: storage-network-definition
     sriov: {}
   - name: multi-network-definition
      sriov: {}     

sap_create_vm_networks:
  - multus:
      networkName: sapbridge-network-definition
    name: sapbridge-network-definition
  - multus:
      networkName: iface-storage-sriov
    name: storage-network-definition
  - multus:
      networkName: iface-multi-sriov
    name: multi-network-definition

The OS configuration in terms of existing users, passwords, SSH keys, etc. of the VM can be adjusted via the variable sap_create_vm_cloud_config. Note the additional newlines in the definition that are required (for now). To find more information on what can be set through cloud-init, refer to the Cloud-init documentation.

3.2.3. Deploying a VM

In order to deploy a VM, perform the following steps:

  1. Get a checkout of the sap_create_vm role in your working directory:
  # git clone https://github.com/newkitlabs/sap_create_vm.git
  1. Move or link that checkout to a directory in your Ansible path. Create ~/.ansible/roles if it does not exist, for example:
 # ln -sf ~/sap_create_vm/roles/sap_create_vm ~/.ansible/roles/
  1. Export your kubeconfig file, for example:
  # export KUBECONFIG=$HOME/clusterconfigs-dciokd/kubeconfig
  1. To create a VM named sapvm01 with 24GB of RAM and a password of MyS5perS3cureP@assW0rD run the following command:
# ansible-playbook -vvv --connection=local -i localhost,  playbooks/sap-create-vm.yml -e sap_create_vm_names="['sapvm01']" -e sap_create_vm_memory=24 -e sap_create_vm_cloud_user_passwd=MyS5perS3cureP@assW0rD
  1. This will download the RHEL GA version specified in sap_create_vm_rhel_version, e.g., 8.8.0. The behavior of whether the image will be downloaded can be controlled via sap_create_vm_download_rhel_images.

4. Preparing the OS for the SAP workload

From the community.sap_install repository, use the role sap_general_preconfigure to configure your OS accordingly. For the installation of SAP HANA, use additionally the role sap_hana_preconfigure and then sap_hana_install. For Netweaver, use sap_netweaver_preconfigure and then sap_swpm.

5. Comments regarding performance

Comparing performance between bare metal and an OpenShift Virtualization VM shows a degradation of the performance of the virtual machine. This is due to the overhead caused by virtualization and is often seen in such environments. While some of the tests showed no impairment at all, others were impacted.

The impact is highly dependent on the workload. In certain mixed-olap/oltp scenarios, the delta is around 12%.

SAP BW edition for SAP HANA benchmarks (BWH) showed a slowdown of around 5% when running in OpenShift Virtualization.

6. References

Comments