In general, the installation of SAP Data Hub Foundation (SDH) follows these steps:

• Install Red Hat OpenShift Container Platform
• Configure the prerequisites for SAP Data Hub Foundation
• Install SAP Data Hub Foundation on OpenShift Container Platform

The last step has three different approaches listed below. Each approach is compatible with this guide. Please refer to the SAP's documentation for more information.

• mpsl - installation using the Maintenance Planner and SL Plugin (recommended by SAP)
• mpfree - installation using SL Plugin without Maintenance Planner
• manual - manual installation using an installation script

If you're interested in installation of older SDH or SAP Vora releases, please refer to the other installation guides:

• Install SAP Data Hub 1.X Distributed Runtime on OpenShift Container Platform
• Installing SAP Vora 2.1 on Red Hat OpenShift 3.7

1. OpenShift Container Platform validation version matrix

The following version combinations of SDH 2.X, OCP and RHEL have been validated:

SAP Data Hub	OpenShift Container Platform	RHEL for OCP Installation
2.3	3.9, 3.10	7.5

NOTE: so far only the following installations have been validated:

• Amazon AWS with GP2 dynamic storage provisioner
• RHV virtualized environment with:
  • Ceph RBD dynamic storage provisioning
  • Gluster dynamic storage provisioning

Please refer to compatibility matrix for version combinations that are considered as working.

2. Requirements

2.1. Hardware/VM and OS Requirements

2.1.1. OpenShift Cluster

The following are the minimum requirements for the OpenShift Cluster Nodes:

• OS: Red Hat Enterprise Linux 7.5, 7.4 or 7.3
• CPU: 4 cores
• Memory:
  • 16GB for the regular node and 32GB for the node hosting HANA pod
• Diskspace:
  • 50GiB for / including at least 30 GiB for /var/lib/docker
  • 500GiB for persistent volumes if hosting persistent storage

2.1.2. Jump host

It is recommended to do the installation of SAP Data Hub Foundation from an external Jump host and not from within the OpenShift Cluster.

The Jump host is used among other things for:

• running the OCP installation
• hosting the external image registry
• running the SDH installation
  • in case of mpsl and mpfree installations, it is used to host and run the SAP Host Agent

The hardware requirements for the Jump host can be:

• OS: Red Hat Enterprise Linux 7.5, 7.4 or 7.3
• CPU: 2 cores
• Memory: 4GB
• Diskspace:
  • 50GiB for /:
    • to store the work directory and the installation binaries of SAP Data Hub Foundation
    • including at least 20 GiB for /var/lib/docker
  • Additional 15 GiB for registry's storage if hosting image registry (by default at /var/lib/registry).

NOTE: It is of course possible not to have a dedicated Jump host and instead, run the installation from one of the OCP cluster hosts - ideally from one of the master nodes. In that case please run all the commands meant for the Jump host on the host of your choice.

2.2. Software Requirements

2.2.1. Compatibility Matrix

Later versions of SAP Data Hub support newer versions of Kubernetes and OpenShift Container Platform. Even if not listed in the OCP validation version matrix above, the following version combinations are considered fully working:

SAP Data Hub	OpenShift Container Platform	RHEL for OCP Installation
2.3	3.9	7.5, 7.4, 7.3
2.3	3.10	7.5, 7.4

2.2.2. Prepare the Subscription and Packages

• On each host of the OpenShift cluster, register system using subscription-manager. Look up and then and attach to the pool that provides the OpenShift Container Platform subscription.

  # subscription-manager register --username=UserName --password=Password
  your system is registered with ID: XXXXXXXXXXXXXXXX
  # subscription-manager list --available
  # subscription-manager attach --pool=Pool_Id_Identified_From_Previous_Command
  

• Subscribe each host only to the following repositories.

  # subscription-manager repos --disable='*' \
      --enable='rhel-7-server-ose-3.9-rpms' \
      --enable='rhel-7-server-rpms' --enable='rhel-7-fast-datapath-rpms' \
      --enable='rhel-7-server-extras-rpms' --enable='rhel-7-server-optional-rpms'
  

  For OCP 3.10, please use the following commands:

  # subscription-manager repos --disable='*' \
      --enable='rhel-7-server-ose-3.10-rpms' \
      --enable='rhel-7-server-rpms' \
      --enable='rhel-7-server-extras-rpms' --enable='rhel-7-server-optional-rpms'
  

• Install the following packages on each host.

  # yum -y install bash-completion bind-utils bridge-utils curl git \
      iptables-services kexec-tools net-tools psacct sos wget yum-utils
  # yum -y install docker
  

2.2.3. Persistent Volumes

Persistent storage is needed for SDH. It’s required to use storage that can be created dynamically. You can find more information in the Dynamic Provisioning and Creating Storage Classes document.

Recommended and tested storage options for on-premise installations are:

• Ceph RBD
• Gluster or CNS - supported for manual installation method only

For installations in cloud, please use the recommended provider for particular cloud platform.

2.2.3.1. Ceph RBD

To make use of an existing Ceph cluster in OCP, please follow the Using Ceph RBD for dynamic provisioning guide (3.9) / (3.10).

Please refer to Installing the Ceph Object Gateway on enabling S3 object interface for Checkpoint store enablement.

For small-to-medium sized test and pre-production deployments, Civetweb interface can be utilized. The S3 API endpoint must either be secured by a signed certificate (not self-signed) or insecure. In the latter case, scheme must be present in the endpoint URL specified during the installation (e.g. http://s3.172.18.0.85.nip.io:8080).

NOTE: Wildcard DNS does not need to be configured for the checkpoint store.

The following is an example configuration of a rados gateway residing in a Ceph configuration file (e.g. /etc/ceph/ceph.conf) for small-to-medium sized test and pre-production deployments.

[client.rgw.ip-172-18-0-85]
host = 0.0.0.0
keyring = /var/lib/ceph/radosgw/ceph-rgw.ip-172-18-0-85/keyring
log file = /var/log/ceph/ceph-rgw-ip-172-18-0-85.log
#rgw frontends = civetweb port=443s ssl_certificate=/etc/ceph/radosgw-172.18.0.85.nip.io.pem num_threads=100
rgw frontends = civetweb port=8080 num_threads=100
debug rgw = 1
rgw_enable_apis = s3
rgw_dns_name = ip-172-18-0-85.internal
rgw_resolve_cname = false

In this case the Amazon S3 Host input parameter shall be set to http://ip-172-18-0-85.internal:8080.

2.2.3.2. Gluster or CNS

Can be deployed into the OpenShift cluster at the time of OCP installation or afterwards (as a Container Native Storage - aka CNS) using either converged or independent mode. An existing standalone Gluster Storage deployed outside of the OpenShift cluster can be utilized as well. For various installation options, please refer to the OCP documentation 3.9 / 3.10.

There are certain options that need to be set on the GlusterFS Volumes to make them suitable for SDH. If set on the storage class, the volume options will propagate to the newly created volumes. If you have used the Advanced Installer (3.9) / (3.10) to install the Gluster Storage¹, the resulting storage class will be named glusterfs-storage and the following command will modify it to apply the recommended volume options.

# oc get -o json sc glusterfs-storage | \
    oc patch -o json --type=json --local -f - -p '[{
      "op": "add",
      "path": "/parameters/volumeoptions",
      "value": "group metadata-cache, nl-cache on"
    }]' | oc replace --force -f -

If a standalone Gluster Storage is being used, additional permissions need to be granted by enabling the following SELinux booleans on all the schedulable nodes:

# setsebool -P virt_sandbox_use_fusefs=on virt_use_fusefs=on

NOTE SDH can be installed only manually with this storage back-end due to an SGID bit problem described below.

Please refer to S3 COMPATIBLE OBJECT STORE IN A CONTAINER-NATIVE STORAGE ENVIRONMENT on enabling S3 interface for checkpoint store with Gluster.
IMPORTANT: this is still under technology preview as of version 3.3.

2.2.4. Checkpoint store enablement

In order to enable SAP Vora Database streaming tables, checkpoint store needs to be enabled. The store is an object storage on a particular storage back-end. Several back-end types are supported by the SDH installer that cover most of the storage cloud providers. For on-premise deployments, Ceph RBD or Gluster Storage can be utilized with their S3 interfaces. IMPORTANT: the latter is under tech preview as of now.

Make sure to create a desired bucket before the SDH Installation. If the checkpoint store shall reside in a directory on a bucket, the directory needs to exist as well.

2.2.5. External Image Registry

The SDH installation requires an Image Registry where images are first mirrored from an SAP Registry and then delivered to the OCP cluster nodes. The integrated OpenShift Container Registry is not appropriate for this purpose. An external image registry needs to be setup instead.

The Jump host is a feasible machine to host the registry. Please follow the steps in article How do I setup/install a Docker registry?.

After the setup you should have an external image registry up and running at the URL My_Image_Registry_FQDN:5000. You can verify that with the following command.

# curl http://My_Image_Registry_FQDN:5000/v2/
{}

Be sure to mark the address as insecure.

2.2.5.1. Update the list of insecure registries

• Since the external image registry deployed above is insecure by default, in order to push images to the image registry and pull them on nodes it must be listed as insecure in /etc/containers/registries.conf file on all the hosts, including the Jump host:

  # vi /etc/containers/registries.conf
  ...
  [registries.insecure]
  registries = [
    "My_Image_Registry_FQDN:5000"
  ]
  ...
  

• For the changes to become effective, restart the docker daemon:

  # systemctl restart docker
  

If you plan to run the installation as a non-root user, please check the instructions below for additional steps.

During the advanced installation of the OCP, be sure to include the My_Image_Registry_FQDN:5000 among openshift_docker_insecure_registries.

2.2.5.2. Update proxy settings

If there's a mandatory proxy in the cluster's network, make sure to include the My_Image_Registry_FQDN in the NO_PROXY settings in addition to the recommended NO_PROXY addresses (3.9) / (3.10).

Additionally, during the advanced installation (3.9) / (3.10) of the OCP, you should include My_Image_Registry_FQDN in openshift_no_proxy variable.

2.2.6. (Optional) Hadoop

It's optional to install the extensions to the Spark environment on Hadoop. Please refer to SAP Data Hub Spark Extensions on a Hadoop Cluster for details. This document doesn't cover the Hadoop part.

3. Install Red Hat OpenShift Container Platform

3.1. Prepare the Jump host

• Ideally, subscribe to the same repositories as on the cluster hosts.

• Install a helm client on the Jump host.

  • Download a script from https://github.com/kubernetes/helm and execute it.

    # curl --silent https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | \
        DESIRED_VERSION=v2.9.1 bash
    

  • See the blog Getting started with Helm on OpenShift for more information.

• Download and install kubectl.

  • Either via standard repositories by installing atomic-openshift-clients:

    # sudo yum install -y atomic-openshift-clients
    

    NOTE: rhel-7-server-ose-X.Y-rpms repositories corresponding to the same major release version (e.g. 3.10) as on the cluster nodes need to be enabled.

  • Or by downloading and installing the binary manually after determining the right version:

    # curl -LO https://dl.k8s.io/release/${ver:-v1.10.7}/bin/linux/amd64/kubectl
    # chmod +x ./kubectl
    # sudo mv ./kubectl /usr/local/bin/kubectl
    

    To determine the right version, look for the highest release name of the same minor version as the desired OCP cluster at the release page. That would be v1.9.10 for OCP 3.9 and v1.10.7 for OCP 3.10 as of Sep 13 2018. The command below will determine the version for you. Make sure to install jq RPM package before running it though.

    # OCPVER=3.10    # or OCPVER=3.9
    # ver="$(curl --silent https://api.github.com/repos/kubernetes/kubernetes/releases | \
       jq -r '.[] | .name' | sort -t '.' -n -k 3 | sort --version-sort | \
       grep -F "$(echo "$OCPVER" | sed -e 's/^\s*v\?3//' -e 's/^\([^.]\+\.[^.]\+\).*/\1/')" | \
       tail -n 1)"; echo "$ver"
    

• In case of mpsl and mpfree SDH installation methods, make sure to install and run the SAP Host Agent as well.
  However, instead of downloading a *.SAR archive, as suggested by the guide, it is recommended to download the latest RPM package (e.g. saphostagentrpm_39-20009394.rpm) and install it on the Jump host using a command like:

  # yum install saphostagentrpm_39-20009394.rpm
  

  This way, the installation of SAPCAR listed in prerequisites is not needed. Steps 6 (archive extraction) and 7 (installation) in the guide can be skipped as well.

3.2. Install OpenShift Container Platform

Install OpenShift Container Platform on your desired cluster hosts. Follow the OpenShift installation guide (3.9) / (3.10) or use the playbooks for a cloud reference architecture.

IMPORTANT: Make sure not to set the default node selector. Otherwise, SDH daemon sets will fail to deploy all their pods which will cause the installation to fail. For new advanced installations, comment out lines with osm_default_node_selector. For existing clusters, unset the node selector in /etc/origin/master/master-config.yaml with the following lines:

projectConfig:
  defaultNodeSelector: ''

NOTE: On AWS you have to label all nodes according to Labeling Clusters for AWS with openshift_clusterid="<clusterid>" where <clusterid> part matches the same part in the tag kubernetes.io/cluster/<clusterid>,Value=(owned|shared) of resources belonging to the cluster.

Important Advanced Installation Variables

Variable	Description
openshift_docker_insecure_registries	shall contain the URL of the external image registry (My_Image_Registry_FQDN:5000)2
openshift_https_proxy, openshift_http_proxy	shall be set up according to internal network policies
openshift_no_proxy	if the proxy settings are set and the registry is deployed in the internal network, it must contain My_Image_Registry_FQDN
osm_default_node_selected	needs to be unset
openshift_cloudprovider_kind	the name of the target cloud provider if deploying in cloud (e.g. aws, azure, openstack or vsphere)
openshift_clusterid	needs to be set only for AWS unless using IAM profiles3

3.3. OCP Post Installation Steps

3.3.1. Set up an External Image Registry

If you haven't done so already, please follow the External Image Registry prerequisite.

3.3.2. Configure the OpenShift Cluster for SDH

3.3.2.1. Becoming a cluster-admin

Many commands below require cluster admin privileges. To become a cluster-admin, you can do one of the following:

• Copy the admin.kubeconfig file from a remote master node to a local host and use that:

  # scp master.node:/etc/origin/master/admin.kubeconfig .
  # export KUBECONFIG=$(pwd)/admin.kubeconfig
  # oc login -u system:admin
  

  This is recommended for mpsl and mpfree installation methods when using the Jump host.

  NOTE the same file is used as the KUBECONFIG file input parameter for mpsl and mpfree installation methods.

• Log-in to any master node as the root user, execute the following command and continue the installation as system:admin user. In this case, the master node becomes a Jump host.

  # oc login -u system:admin
  

  Possible for all installation methods without a Jump host.

• (manual installation method only) Make any existing user (dhamin in this example) a cluster admin by doing the previous step followed by:

  # oc adm policy add-cluster-role-to-user cluster-admin dhadmin
  

You can learn more about the cluster-admin role in Cluster Roles and Local Roles article.

3.3.2.2. Project setup

• On every scheduledable node of the OpenShift cluster, make sure the NFS filesystem is ready for use by loading the nfsd kernel module and enabling it with SELinux boolean.

  # setsebool virt_use_nfs true
  

• On every schedulable node of the OpenShift cluster, change the SELinux security context of file /var/run/docker.sock.

  # semanage fcontext -m -t container_file_t -f s "/var/run/docker\.sock"
  # restorecon -v /var/run/docker.sock
  

  To make the change permanent, execute the following on all the nodes:

  # cat >/etc/systemd/system/docker.service.d/socket-context.conf <<EOF
  [Service]
  ExecStartPost=/sbin/restorecon /var/run/docker.sock
  EOF
  

• As a cluster-admin, create a tiller service account in the kube-system project (aka namespace) and grant it the necessary permissions:

  # oc create sa -n kube-system tiller
  # oc adm policy add-cluster-role-to-user cluster-admin -n kube-system -z tiller
  

• Set up helm and tiller for the deployment:

  # helm init --service-account=tiller --upgrade --wait
  

  Upon successful initialization, you should be able to see a tiller pod in the kube-system namespace:

  # oc get pods -n kube-system
  NAME                            READY     STATUS    RESTARTS   AGE
  tiller-deploy-551988758-dzjx5   1/1       Running   0          1m
  # helm ls
  [There should be no error in the output. If there is no output at all, it means good news, no error]
  

• Create a dedicated project in OpenShift for the SDH deployment. For example sdh. Login to OpenShift as a cluster-admin, and perform the following configurations for the installation:

  # oc new-project sdh
  # oc adm policy add-scc-to-group anyuid "system:serviceaccounts:$(oc project -q)"
  # oc adm policy add-scc-to-group hostmount-anyuid "system:serviceaccounts:$(oc project -q)"
  # oc adm policy add-scc-to-user privileged -z "vora-vsystem-$(oc project -q)"
  # oc adm policy add-scc-to-user privileged -z "vora-vflow-server"
  

• (Optional) In case of the manual installation, you may want to execute the installation as a regular user (not as a cluster-admin). You can either create a new OpenShift user or delegate the installation procedure to an existing OCP user. In any case, the user needs to be granted the following roles in the chosen target project (in this case dhadmin is the user name and sdh is the target project):

  # oc adm policy add-role-to-user -n sdh admin dhadmin
  # oc adm policy add-cluster-role-to-user system:node-reader dhadmin
  

• As a cluster-admin, allow the project administrator to manage SDH custom resources.

  # oc create -f - <<EOF
  kind: ClusterRole
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: aggregate-sapvc-admin-edit
    labels:
      rbac.authorization.k8s.io/aggregate-to-admin: "true"
      rbac.authorization.k8s.io/aggregate-to-edit: "true"
  rules:
  - apiGroups: ["sap.com"]
    resources: ["voraclusters"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"]
  ---
  kind: ClusterRole
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: aggregate-sapvc-view
    labels:
      # Add these permissions to the "view" default role.
      rbac.authorization.k8s.io/aggregate-to-view: "true"
  rules:
  - apiGroups: ["sap.com"]
    resources: ["voraclusters"]
    verbs: ["get", "list", "watch"]
  EOF
  

4. Install SDH on OpenShift

4.1. Installation using the Maintenance Planner and SL Plugin (mpsl)

Is a web-based installation method recommended by SAP offering you an option to send analytics data and feedback to SAP. All the necessary prerequisites have been satisfied by applying all the steps described above. The Installation using the Maintenance Planner and SL Plugin documentation will guide you through the process.

4.2. Installation using SL Plugin without Maintenance Planner (mpfree)

Is an alternative command-line-based installation method. Please refer to the SAP Data Hub documentation for more information and the exact procedure.

If you have prepared the installation host according to the prior instructions, you will have sapcar binary available at /usr/sap/hostctrl/exe/SAPCAR. Thus the extraction step 3 in the procedure will look like:

# /usr/sap/hostctrl/exe/SAPCAR -xvf /tmp/download/SLPUGIN<latest SP-version>.SAR /tmp/slplugin/bin

4.3. Manual Installation using an installation script (manual)

4.3.1. Download and unpack the SDH binaries

Download and unpack SDH installation binary onto the Jump host.

• Go to SAP Software Download Center, login with your SAP account and search for SAP DATA HUB 2 or access this link.

• Download the SAP Data Hub Foundation file, for example: DHFOUNDATION03_2-80004015.ZIP (SAP DATA HUB - FOUNDATION 2.3).

• Unpack the installer file. For example, when you unpack the DHFOUNDATION03_2-80004015.ZIP package, it will create the installation folder SAPDataHub-2.3.155-Foundation.

  # unzip DHFOUNDATION03_2-80004015.ZIP
  

4.3.1.1. Note on Installation on cluster with 3 nodes in total

This note is useful for just a small proof-of-concept, not for production deployment.

SDH's dlog pod expects at least 3 schedulable compute nodes that are neither master nor infra nodes. This requirement can be mitigated by reducing replication factor of dlog pod with the following patch applied to the foundation directory. See below for instructions on patching the deployment files.

4.3.2. Install SAP Data Hub

4.3.2.1. Remarks on the installation options

When there is no default dynamic storage provisioner defined, you need to specify the preferred one using the following parameters:

Installation method	Parameter
Manual	--pv-storage-class="$storage_class_name"
Advanced mpsl or mpfree installations	Default storage class

If the default dynamic storage provisioner has been defined, the parameter can be omitted. To define the default dynamic storage provisioner, please follow the document Changing the Default StorageClass.

4.3.2.2. Executing the installation script

• Run the SDH installer as described in Manually Installing SAP Data Hub on the Kubernetes Cluster.

4.4. SDH Post installation steps

4.4.1. (Optional) Create a route to a vsystem service

OpenShift allows you to access the Data Hub services via routes as opposed to regular node ports. For example, instead of accessing the vsystem service via https://master-node.example.com:32322, after the service exposure, you will be able to access it at https://vsystem-vora.wildcard-domain.

• After a successful installation, create a route for the vsystem service. You can find more information in OpenShift documentation Using Wildcard Routes (for a Subdomain).

  • Look up the vsystem service:

    # oc get services -n sdh
    vsystem      172.30.81.230    <nodes>       10002:31753/TCP,10000:32322/TCP     1d
    

  • Create the route:

    # oc create route passthrough --service=vsystem -n sdh
    # oc get route -n sdh
    NAME      HOST/PORT                      PATH     SERVICES  PORT         TERMINATION   WILDCARD
    vsystem   vsystem-vora.wildcard-domain   vsystem  vsystem   passthrough   None
    

• Access the SDH web console at https://vsystem-vora.wildcard-domain.

4.4.2. SDH Validation

Validate SDH installation on OCP to make sure everything works as expected. Please follow the instructions in Testing Your Installation.

5. Appendix

5.1. HANA template modification for Gluster Storage

NOTE this is applicable only to the manual installation method. The other installation methods does not allow for modification of deployment files.

The GlusterFS volumes set SGID bits on the mount point and newly created directories. Although it makes no harm to HANA database, the HANA pre-checks prevent the database from starting. It can be fixed using the following command executed in the root of the foundation directory:

# patch -p1 -B patchbaks -r - <<EOF
Index: SAPDataHub-2.3.155-Foundation/deployment/helm/hana/templates/hana-statefulset.yaml
===================================================================
--- SAPDataHub-2.3.155-Foundation.orig/deployment/helm/hana/templates/hana-statefulset.yaml
+++ SAPDataHub-2.3.155-Foundation/deployment/helm/hana/templates/hana-statefulset.yaml
@@ -51,6 +51,7 @@ spec:
         image: {{.Values.docker.registry}}/{{.Values.docker.image | default "hana"}}:{{.Values.docker.imageTag | default "latest"}}
         args:
             - --dont-check-version
+            - --dont-check-mount-points
             - --passwords-url
             - file:///var/run/hana/password.json
         ports:
EOF

5.2. SDH uninstallation

Choose one of the uninstallation methods based on what method has been used for the installation.

When done, you may continue with a new installation round in the same or another namespace.

5.2.1. Using the SL Plugin

Please follow the SAP documentation Uninstall SAP Data Hub Using the SL Plugin if you have installed SDH using either mpsl or mpfree methods.

5.2.2. Manual uninstallation

The installation script allows for project clean-up where all the deployed pods, persistent volume claims, secrets, etc. are deleted. If the manual installation method was used, the SDH can be uninstalled using the same script and different set of parameters. The snippet below is an example where the SDH installation resides in the project sdh. In addition to running this script, the project needs to be deleted as well if the same project shall host the new installation.

# ./install.sh --delete --purge --force-deletion --namespace=sdh \
    --docker-registry=registry.local.example:5000
# oc delete project sdh
# # start the new installation

The deletion of the project often takes quite a while. Until fully uninstalled, the project will be listed as Terminating in the output of oc get project. You may speed the process up with the following command. Again please mind the namespace.

# oc delete pods --all --grace-period=0 --force --namespace sdh

NOTE: Be sure not to run the same installation script more than once at the same time even when working with different OpenShift projects.

5.3. Uninstall Helm

# helm reset

5.4. Allow a non-root user to interact with Docker on Jump host

• Append -G dockerroot to OPTIONS= in /etc/sysconfig/docker file on your Jump host.

• Run the following commands on the Jump host, after you modify the /etc/sysconfig/docker file. Make sure to replace alice with your user name.

  # sudo usermod -a -G dockerroot alice
  # sudo chown root:dockerroot /var/run/docker.sock
  

• Log out and re-log-in to the Jump host for the changes to become effective.

5.5. Load nfsd kernel modules

Execute the following on all the schedulable nodes.

# echo "nfsd" > /etc/modules-load.d/nfsd.conf
# sudo mount -t nfsd nfsd /proc/fs/nfsd

5.6. Make the installer cope with just 3 nodes in cluster

IMPORTANT This hint is useful for just small PoCs, not for production deployment. For the latter, please increase the number of schedulable compute nodes.

SDH's dlog pod expects at least 3 schedulable compute nodes that are neither master nor infra nodes. This requirement can be mitigated by reducing replication factor of dlog pod with the following parameters passed either to the installation script (when installing manually) or as Additional Installation Parameters during mpsl or mpfree installation methods:

-e=vora-cluster.components.dlog.standbyFactor=0 -e=vora-cluster.components.dlog.replicationFactor=2

Alternatively, you may choose 1 for both standby and replication factors. The parameters are documented in the Installation Guide.

6. Troubleshooting Tips

6.1. HANA pod not able to start on GlusterFS

If the installation terminates during the deployment of the HANA pod, please inspect the failing pod and look for the cause.

# oc get pods -l vora-component=hana
NAME        READY     STATUS             RESTARTS   AGE
po/hana-0   0/1       CrashLoopBackOff   9          25m

If the log of the pod shows an error similar to:

======== Starting HANA container run script ========
Started at: Mon Aug 20 17:46:56 UTC 2018
Script parameters: --dont-check-version --passwords-url file:///var/run/hana/password.json
Image version:
HANA version: 2.00.030.00.1522209842
WARNING: Unsupported Linux kernel version 3.10. Supported are 4.4+.
New host: 'hana-0'
Setting HANAs host name to 'hana-0' ...
Checking allowed mountpoints ...
ERROR: Unallowed flag on mount point /hana/mounts: sgid

You probably use Gluster Storage as the dynamic storage provisioner and will need to patch the HANA deployment template to tolerate the SGID bit set.

NOTE the unsupported Linux kernel version message is considered harmless by SAP Data Hub team and can be ignored. The warning will be removed in future versions.

6.2. Vystem-vrep pod not starting

Upon inspection, it looks like the NFS filesystem is not supported:

# oc get pods -l vora-component=vsystem-vrep
NAME             READY     STATUS             RESTARTS   AGE
vsystem-vrep-0   0/1       CrashLoopBackOff   6          8m
# oc logs $(oc get pods -o name -l vora-component=vsystem-vrep)
2018-08-30 10:41:51.935459|+0000|INFO |Starting Kernel NFS Server||vrep|1|Start|server.go(53)
2018-08-30 10:41:52.014885|+0000|ERROR|service nfs-kernel-server start: Not starting NFS kernel daemon: no support in current kernel.||vrep|1|Start|server.go(80)
2018-08-30 10:41:52.014953|+0000|ERROR|error starting nfs-kernel-server: exit status 1
||vrep|1|Start|server.go(82)
2018-08-30 10:41:52.014976|+0000|FATAL|Error starting NFS server: NFSD error||vsystem|1|fail|server.go(145)

This means that the proper NFS kernel module hasn't been loaded yet. Make sure to load it permanently.

6.3. Graphs cannot be run in the pipeline modeler

If in the log of the vflow pod, there are problems with reaching outside of the private network like the following output shows, be sure to verify your proxy settings and make sure that the installation script is run with the following parameters:

# ./install.sh --cluster-http-proxy="${HTTP_PROXY}" --cluster-https-proxy="${HTTPS_PROXY}" --cluster-no-proxy="${NO_PROXY}"

vflow log viewable with a command like oc logs $(oc get pods -o name -l vora-component=vflow):

W: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease  Could not connect to deb.debian.org:80 (5.153.231.4), connection timed out [IP: 5.153.231.4 80]
W: Failed to fetch http://security.debian.org/debian-security/dists/stretch/updates/InRelease  Could not connect to security.debian.org:80 (217.196.149.233), connection timed out [IP: 217.196.149.233 80]
W: Failed to fetch http://deb.debian.org/debian/dists/stretch-updates/InRelease  Unable to connect to deb.debian.org:http: [IP: 5.153.231.4 80]

6.4. Vora Installation Error: timeout at “Deploying vora-consul”

Vora Installation Error: timeout at "Deploying vora-consul with: helm install --namespace vora -f values.yaml ..."

To view the log messages, you can login to the OpenShift web console, navigate to Applications -> Pods, select the failing pod e.g. vora-consul-2-0, and check the log under the Events tab.

A common error: if the external image registry is ininsecure, but the OpenShift cluster is configured to pull from a secure registry, you will see errors in the log. If secure registry is not feasible, follow the instructions on configuring the registry as insecure.

6.5. Too few worker nodes

If you see the installation failing with the following error, there are too few schedulable non-infra nodes in the cluster.

Status:
  Message:  Less available workers than Distributed Log requirements
  State:    Failed
Events:
  Type  Reason               Age   From                      Message
  ----  ------               ----  ----                      -------
        New Vora Cluster     11m   vora-deployment-operator  Started processing
        Update Vora Cluster  11m   vora-deployment-operator  Processing failed: less available workers than Distributeed Log requirements
2018-09-14T16:16:15+0200 [ERROR] Timeout waiting for vora cluster! Please check the status of the cluster from above logs and kubernetes dashboard...

If you have at least 2 schedulable non-infra nodes, you may still make the installation succeed by reducing the dlog's replication factor. After the patch is applied, make sure to uninstall the failed installation and start it anew.

6.6. Validation errors

May happen during the validation phase initiated by running the SDH installation script with the validate flag:

# ./install.sh --validate \
    --docker-registry="$DOCKER_REGISTRY_HOST_PORT" \
    --namespace=sdh

6.6.1. Services not installed

Failure description

vora-vsystem validation complains about services not installed even though deployed:

2018-08-28T13:32:19+0200 [INFO] Validating...
2018-08-28T13:32:19+0200 [INFO] Running validation for vora-cluster...OK!
2018-08-28T13:33:14+0200 [INFO] Running validation for vora-sparkonk8s...OK!
2018-08-28T13:34:56+0200 [INFO] Running validation for vora-vsystem...2018-08-28T13:35:01+0200 [ERROR] Failed! Please see the validation logs -> /home/miminar/SAPDataHub-2.3.144-Foundation/logs/20180828_133214/vora-vsystem_validation_log.txt
2018-08-28T13:35:01+0200 [INFO] Running validation for datahub-app-base-db...OK!
2018-08-28T13:35:01+0200 [ERROR] There is a failed validation. Exiting...


# cat /home/miminar/SAPDataHub-2.3.144-Foundation/logs/20180828_133214/vora-vsystem_validation_log.txt
2018-08-28T13:34:56+0200 [INFO] Connecting to vSystem ...
2018-08-28T13:34:57+0200 [INFO] Wait until pod vsystem-vrep-0 is running...
2018-08-28T13:34:57+0200 [INFO] Wait until containers in the pod vsystem-vrep-0 are ready...
2018-08-28T13:34:58+0200 [INFO] Wait until pod vsystem-7d7ffdd649-8mcxv is running...
2018-08-28T13:34:58+0200 [INFO] Wait until containers in the pod vsystem-7d7ffdd649-8mcxv are ready...
2018-08-28T13:35:01+0200 [INFO] Logged in!
2018-08-28T13:35:01+0200 [INFO] Installed services:

/home/miminar/SAPDataHub-2.3.144-Foundation/validation/vora-vsystem/vsystem-validation.sh: line 19: error: command not found

Alternatively, the following message may appear at the end of the vora-vsyste_validation_log.txt:

2018-08-31T08:59:52+0200 [ERROR] Connection Management is not installed

Resolution

If there are proxy environment variables set, make sure to include 127.0.0.1 and localhost in no_proxy and NO_PROXY environment variables. You may find Setting Proxy Overrides and Working with HTTP Proxies helpful. Apart from the recommended settings, do not forget to include My_Image_Registry_FQDN in the NO_PROXY settings if the registry is hosted inside the proxied network.