Deploying SAP Workloads with OpenShift Virtualization

Updated -

NOTE: Red Hat does recommend customers to test their workloads on OpenShift Virtualization in test and development environments to gain experience. SAP does not support SAP HANA and SAP business applications deployed on OpenShift Virtualization. Red Hat fully supports OpenShift Virtualization independent from the workload perspective.

Contents

1. Prerequisites

  • You have an OpenShift cluster >= 4.18.
  • Worker nodes:
    • At least 96GB of RAM (note: minimal memory checking can be disabled at your own risk).
    • Identical worker nodes (same vendor, CPU, RAM, disks, network).
  • Storage: OpenShift Data Foundation, NFS or local disk.

  • A jumphost running RHEL 9.7 or RHEL 10.1. For now, there are dependencies on:

    • python3-jmespath
    • python3-netaddr
      But these dependencies will be removed in future versions.

  • Ansible collection redhat.sap_infrastructure and to install SAP workloads such as SAP HANA or NetWeaver use the collection redhat.sap_install.

2. Procedure

The following steps outline the entire process required in order to deploy a SAP workload on OpenShift Virtualization. This guide focuses on steps 2 and 3.

  1. Install the OpenShift cluster: This is not part of the document; see Installing a user-provisioned cluster on bare metal for more information on how to install OpenShift.
  2. Configure worker nodes for SAP workloads with the redhat.sap_infrastructure.sap_hypervisor_preconfigure role.
  3. Create one or more VMs using the role redhat.sap_infrastructure.sap_vm_provision.
  4. Deploy the SAP workload:
    a. NetWeaver using redhat.sap_install.sap_swpm, or
    b. SAP HANA using redhat.sap_install.sap_hana_install.

2.1. Deploying Scenarios

You need at least one machine besides the OpenShift cluster where the Ansible playbook is run. Additionally, there can be a bastion host in between the host which runs the playbook and the OpenShift cluster. This can be used, e.g., if the network access is not possible from the machine where you would like to run the playbook.

sap_vm_provision_execution_host: localhost

3. Ansible roles

Install the Ansible collection from Ansible galaxy using version 1.3.5:

# ansible-galaxy collection install redhat.sap_infrastructure
# ansible-galaxy collection install redhat.sap_install

As an alternative, the collection can also be installed via the following RPM, e.g., for offline installations. You need to be subscribed to the RHEL for SAP Solutions channel.

# dnf install rhel-system-roles-sap

Note that the sample playbooks and vars files are then installed here:
* /usr/share/ansible/collections/ansible_collections/redhat/sap_infrastructure/playbooks
* /usr/share/ansible/collections/ansible_collections/redhat/sap_infrastructure/playbooks/vars

Use those as a starting point for your configuration as described in this document.

3.1. Configuring a cluster for SAP workloads using sap_hypervisor_node_preconfigure

The role redhat.sap_infrastructure.sap_hypervisor_node_preconfigure is used to configure the OpenShift cluster. The steps automatically performed are:

  1. Set tuned profile to virtual-host.
  2. Install Operators:
    a. CNV operator,
    b. SRIOV operator,
    c. nmstate operator.

  3. Setup worker nodes by executing the following steps:
    a. Set the following kernel parameters:

    intel_iommu=on iommu=pt default_hugepagesz=1GB hugepagesz=1GB hugepages=<#1GBHugepages> tsx=on

    Regarding huge pages: on every worker node defined, the amount of HP is determined by the following formula:

    [Amount of RAM in worker] - 32 GB if [Amount of RAM] < 512 GB, 
                                    64 GB otherwise

    b. Enable CPU Manager.
    c. Configure network: SRIOV and/or Linux-Bridge.

  4. Enable downwardMetrics feature gate.
  5. Storage (at least one of them is needed):
    a. Install Trident Operator and configure it,
    b. Setup local storage.

3.1.1. Preparations

Create a copy of the variables example file and adjust it to the requirements. The following section shows the variables used to control which operators to install. At least one storage provider needs to be configured, the host path provisioner (HPP) can be used to configure local disks, the trident storage is used for Netapp NFS. Enable either one of them by setting the variable to true. Otherwise you will have to configure the storage manually.

Note that for the trident storage it is required that your kubeconfig file does contain a valid API token. You can update/get an API token by logging into your cluster, e.g., (put in your kubeadmin password and your cluster domain):

# oc login -u kubeadmin -p <your password> --server=https://api.<your cluster domain>:6443

Your kubeconfig will then be updated with the token, typically the location is ~/.kube/kubeconfig.

###########################################################
# Configuration of what should be preconfigured
###########################################################


# Install and configure the host path provisioner (hpp) for a local storage disk
sap_hypervisor_node_preconfigure_install_hpp: false


# Install the trident NFS storage provider
sap_hypervisor_node_preconfigure_install_trident: false


# Should the operators be installed
sap_hypervisor_node_preconfigure_install_operators: true


# Configure the workers?
sap_hypervisor_node_preconfigure_setup_worker_nodes: true

3.1.2. Configuration variables

At the end of the configuration template, pay attention to the section starting with the variable sap_hypervisor_node_preconfigure_cluster_config. This data structure describes the cluster. Make sure they match the exact worker node names, e.g.

# oc get nodes
NAME              STATUS   ROLES                  AGE   VERSION
…
dciokd-worker-0   Ready    worker                 75m   v1.29.14+7cf4c05
dciokd-worker-1   Ready    worker                 75m   v1.29.14+7cf4c05

In this example the worker nodes are called dciokd-worker-0 and dciokd-worker-1. Let’s have a look at a minimalistic example with two nodes, using the HPP with local storage on the disk named /dev/sdb and a network bridge named sapbridge using interface ens1f0. The storage device and the interface must be available on all nodes. Via this bridge the guest VMs can be reached later.

# Use local storage
sap_hypervisor_node_preconfigure_install_hpp: true

sap_hypervisor_node_preconfigure_cluster_config:


  # namespace under which the VMs are created, note this has to be
  # openshift-sriov-network-operator in case of using SR-IOV network
  # devices
  vm_namespace: sap

  # CPU cores which will be reserved for kubernetes
  worker_kubernetes_reserved_cpus: "0,1"


  # Storage device used for host path provisioner as local storage.
  worker_localstorage_device: /dev/sdb


  # detailed configuration for every worker that should be configured
  workers:
    - name: dciokd-worker-0            # name must match the node name
      networks:                        # Example network config
        - name: sapbridge              # using a bridge
          description: SAP bridge
          state: up
          type: linux-bridge
          ipv4:
            enabled: false
            auto-gateway: false
            auto-dns: false
          bridge:
            options:
              stp:
                enabled: false
            port:
              - name: ens1f0           # network IF name


    - name: dciokd-worker-1            # second worker configuration
      networks:                        # Example network config
        - name: sapbridge              # using a bridge
          description: SAP bridge
          state: up
          type: linux-bridge
          ipv4:
            enabled: false
            auto-gateway: false
            auto-dns: false
          bridge:
            options:
              stp:
                enabled: false
            port:
              - name: ens1f0           # network IF name

3.1.3. Example for playbook

See this example for a simple playbook that just calls the role. It’s in the file playbooks/sample-sap-hypervisor-redhat_ocp_virt-preconfigure.yml:

- hosts: all
  gather_facts: true
  serial: 1
  vars:
    sap_hypervisor_node_platform: redhat_ocp_virt
    sap_hypervisor_node_kubeconfig: "{{ lookup( 'ansible.builtin.env', 'KUBECONFIG') }}"
  environment:
    KUBECONFIG: "{{ sap_hypervisor_node_kubeconfig }}"
    K8S_AUTH_KUBECONFIG: "{{ sap_hypervisor_node_kubeconfig }}"
  roles:
    - { role: redhat.sap_install.sap_hypervisor_node_preconfigure }

3.1.4. Configure the OCP cluster for SAP workloads

Use the playbook sample-sap-hypervisor-redhat-ocp-virt-preconfigure.yml from the playbooks directory of the Ansible collection. See this for a reference: sample-sap-hypervisor-redhat-ocp-virt-preconfigure.yml. This playbook simply applies the sap_hypervisor_node_preconfigure role to your cluster. It expects a valid kubeconfig file in the environment variable KUBECONFIG or K8S_AUTH_KUBECONFIG. Or the file can be specified with the role variable sap_hypervisor_node_preconfigure_kubeconfig. If you are using the Trident storage operator for NFS, you also need a valid API token in the kubeconfig. Please refer to to see how to get one.

In order to set all the parameters in a way so that the OpenShift Cluster gives the best performance and ensures compatibility:

  1. Make sure the kubeconfig file for your cluster is available through the environment variable:

    # export KUBECONFIG=path/to/kubeconfig

  2. Go to the playbooks directory:

    # cd redaht.sap_install/playbooks

  3. To invoke the example playbook with the example configuration using your localhost as an Ansible host, use the following command line:

    # ansible-playbook -v --connection=local -i localhost,  sample-sap-hypervisor-redhat_ocp_virt-preconfigure.yml -e @<your copy of defaults/main.yml>

    In case the entry Virtualization does not show up in the OpenShift UI, you have to clear the browser cache.

3.1.5. Setting up Storage

You need to set up a storage connection in order to continue installing VMs on your cluster. For example, you can use:

  • NFS (with Trident)
  • OpenShift Data Foundation
  • local storage

Regarding NFS with NetApp Astra Trident, for additional information, refer to the blog Configure a NetApp SVM to expose NFS for OpenShift Virtualization. For the official NetApp documentation, refer to Learn about Astra Trident installation. Refer to Red Hat OpenShift Data Foundation for a starting point for setting up OpenShift Data Foundation.

3.2 Creating VMs using role sap_vm_provision

Use the role sap_vm_provision from the Ansible collection redhat.sap_infrastructure to create VMs with the correct setting and performance tunings for SAP workloads. The tunings are:
* 1G hugepages
* dedicated CPU placement
* NUMA CPU topology
* optional: use SRIOV network interfaces

3.2.1. Requirements

  • An OpenShift Virtualization cluster configured with sap_hypervisor_preconfigure_role.
  • A default storage class setup along with enough storage space.
  • An SSH keypair which will be set on the VM and used for authentication.
  • The following packages:
    • python3-kubernetes
    • python3-jmespath
    • python3-netaddr
    • ansible-collection-kubernetes-core
      They can be installed with the following command:
# dnf install python3-kubernetes python3-jmespath

python3-netaddr 
ansible-collection-kubernetes-core

3.2.2. Preparations

Create a copy of the variables file playbooks/vars/sample-variables-sap-vm-provision-redhat-ocpv.yml.

Either set the kubeconfig file there or provide it via the environment variables K8S_AUTH_KUBECONFIG or KUBECONFIG. The variable sap_vm_provision_kubevirt_vm_target_namespace specifies the namespace the VMs will be created in. This namespace must be identical to the namespace defined in the previous step for the sap_hypervisor_preconfigure role.

In case you plan to use SRIOV network interfaces, create the VM in the namespace openshift-sriov-network-operator.

Provide a SSH key pair with sap_vm_provision_ssh_host_public_key_file_path and sap_vm_provision_ssh_host_private_key_file_path. This key will be added to the authorized_keys file in the guest VM and also used to authenticate during deployment.

The variable sap_vm_provision_kubevirt_vm_host_specifications_dictionary is used to specify so-called host execution plans. They contain one or more guest VM definitions. In the example below there is one execution plan defined, named deploy_hana. Then there follows the definition for the VM called host1. This describes how many CPU and memory resources the guest will have:

      kubevirt_vm_cpu_smt: 2
      kubevirt_vm_cpu_cores: 2
      kubevirt_vm_memory_gib: 24

The os_image section defines what image or which PVC is used for the OS disk. To use a base image that is available on a registry, use this:

      os_image:
        url: "docker://registry.redhat.io/rhel8/rhel-guest-image:8.8.0"
        namespace: openshift-virtualization-os-images
        size: "50Gi

You could also specify a PVC that should be cloned:

      os_image: 
        source_pvc_name: "rhel10-beta-da1c0cdc24da"
        size: "50Gi"

The storage_definition section describes additional disks that should be created and attached to the VM, e.g.:

      storage_definition:
        - name: hana
          mountpoint: /hana
          disk_count: 1                  # default: 1
          disk_size: 2048                # size in GB, integer
          disk_type: nas                 # KubeVirt Storage Class

Then the cloud-init section describes a cloud-init disk for initial host bootstrapping. Refer to the Cloud-init documentation for more details. There is a user data section and a network config area. The example shown here sets things such as hostname user and password, timezone and authorized SSH keys:

      cloudinit:
        userData: |-
          #cloud-config
          timezone: Europe/Berlin
          hostname: lu0567
          user: {{ sap_vm_provision_kubevirt_vm_os_user if sap_vm_provision_kubevirt_vm_os_user is defined }}
          password: {{ sap_vm_provision_kubevirt_vm_os_user_password if sap_vm_provision_kubevirt_vm_os_user_password is defined }}
          chpasswd:
            expire: false
          ssh_authorized_keys:
            - "{{ lookup('ansible.builtin.file', sap_vm_provision_ssh_host_public_key_file ) if sap_vm_provision_ssh_host_public_key_file is defined }}"

The network configuration for a static IP address looks as follows:

        networkData: |-
          version: 2
          ethernets:
            eth0:
              addresses:
                - 10.10.1.1/16
              gateway4: 10.10.254.254
              nameservers:
                search:
                  - example-domain.com
                addresses:
                  - 10.10.254.254
                  - 10.10.254.252

Using DHCP for network setup can be done with:

        networkData: |-
          version: 2
          ethernets:
            eth0:
              dhcp4: true

3.2.3. Deploying a VM

Use the file playbooks/sample-sap-vm-provision-redhat-ocpv.yml together with the variables file prepared in the step before. Make sure the environment variable ANSIBLE_JINJA2_NATIVE=true is set, the kubeconfig file is provided and the host specification plan is selected as defined in your variables file, e.g.:

ANSIBLE_JINJA2_NATIVE=true KUBECONFIG=/path/to/kubeconfig ansible-playbook -i localhost, sample-sap-vm-provision-redhat-ocpv.yml -e @<your variables file> -e sap_vm_provision_host_specification_plan=deploy_hana

4. Preparing the OS for the SAP workload

From the redhat.sap_install repository, use the role sap_general_preconfigure to configure your OS accordingly. For the installation of SAP HANA, additionally use the role sap_hana_preconfigure and then sap_hana_install. For Netweaver, use sap_netweaver_preconfigure and then sap_swpm.

5. Comments regarding performance

Comparing performance between bare metal and an OpenShift Virtualization VM shows a degradation of the performance of the virtual machine. This is due to the overhead caused by virtualization and is often seen in such environments. While some of the tests showed no impairment at all, others were impacted.

The impact is highly dependent on the workload. In certain mixed-olap/oltp scenarios, the delta is around 12% slower than bare metal.

The SAP BW edition for SAP HANA benchmarks (BWH) showed a slowdown of around 5% when running in OpenShift Virtualization.

6. Live migration

Live migration is possible; please see this blog article for more details.

7. Support statement from Red Hat

From the collection redhat.sap_infrastructure, the following roles are fully supported by Red Hat when used against a Red Hat OpenShift cluster version >= 4.18 with the variables as shown here:

  • sap_hypervisor_node_preconfigure
    • sap_hypervisor_node_preconfigure_platform = redhat_ocp_virt
  • sap_vm_provision
    • sap_vm_provision_iac_type = ansible
    • sap_vm_provision_iac_platform = kubevirt_vm

7.1. Test scope

As part of Red Hat’s QA, these two supported roles were tested with:

  • RHEL 9.7 and the shipped Ansible version 2.14.18-1.el9
  • RHEL 10.1 and the shipped Ansible version 2.16.14-1.el10

8. References

Comments