SAP Data Hub 2 on OpenShift Container Platform 4

Updated -

Table of Contents

In general, the installation of SAP Data Hub Foundation (SDH) follows these steps:

  • Install Red Hat OpenShift Container Platform
  • Configure the prerequisites for SAP Data Hub Foundation
  • Install SAP Data Hub Foundation on OpenShift Container Platform

The last step has three different approaches listed below. Each approach is compatible with this guide. Please refer to the SAP's Documentation for more information.

  • mpsl - installation using the Maintenance Planner and SL Plugin (recommended by SAP)
  • mpfree - installation using SL Plugin without Maintenance Planner
  • manual - manual installation using an installation script

If you're interested in installation of older SDH or SAP Vora releases, please refer to the other installation guides:

1. OpenShift Container Platform validation version matrix

The following version combinations of SDH 2.X, OCP, RHEL or RHCOS have been validated:

SAP Data Hub OpenShift Container Platform Operating System Infrastructure and (Storage)
2.6 4.1 RHCOS, RHEL 7.6 AWS (EBS)
2.6 Patch 1 4.1 RHCOS, RHEL 7.6 AWS (EBS)

Please refer to compatibility matrix for version combinations that are considered as working.

2. Requirements

2.1. Hardware/VM and OS Requirements

2.1.1. OpenShift Cluster

Make sure to consult the following official cluster requirements:

There are 4 kind of nodes:

Bootstrap Node - On AWS, one temporary bootstrap node is created for the OCP deployment. The node is then destroyed by the installer or can be deleted manually by the administrator. Please refer to the Installation process for more information.
Master Nodes - The control plane manages the OpenShift Container Platform cluster.
Compute Nodes - Run the actual workload - all SDH pods among others.
Jump host - The Jump host is used among other things for:

The hardware/software requirements for the Jump host can be:

  • OS: Red Hat Enterprise Linux 7.6
  • Diskspace:
    • 50GiB for /:
      • to store the work directory and the installation binaries of SAP Data Hub Foundation
      • including at least 20 GiB for /var/lib/docker
    • Additional 25 GiB for registry's storage if hosting image registry (by default at /var/lib/registry).
2.1.1.1. Minimum Hardware Requirements

The table below lists the minimum requirements and the minimum number of instances for each node type. This is sufficient of a PoC (Proof of Concept) environments.

Type Count Operating System vCPU RAM (GB) Storage (GB) AWS Instance Type
Bootstrap 1 RHCOS 2 16 120 i3.large
Master 3+ RHCOS 4 16 120 m4.xlarge
Compute 3+ RHEL 7.6 or RHCOS 4 32 120 m4.2xlarge
Jump host 1 RHEL 7.6 2 4 75 t2.medium
2.1.1.2. Minimum Production Hardware Requirements

The minimum production requirements for production systems are following:

Type Count Operating System vCPU RAM (GB) Storage (GB) AWS Instance Type
Bootstrap 1 RHCOS 2 16 120 i3.large
Master 3+ RHCOS 8 16 120 m4.xlarge
Compute 4+ RHEL 7.6 or RHCOS 8 64 120 m4.4xlarge
Jump host 1 RHEL 7.6 2 4 75 t2.medium

2.2. Software Requirements

2.2.1. Compatibility Matrix

Later versions of SAP Data Hub support newer versions of Kubernetes and OpenShift Container Platform. Even if not listed in the OCP validation version matrix above, the following version combinations are considered fully working:

SAP Data Hub OpenShift Container Platform RHEL or RHCOS Storage
2.6 3.11, 4.1 7.6 Ceph, OCS1 Block (3.11+), cloud2

Unless stated otherwise, the compatibility of a listed SDH version covers all its patch releases as well.

2.2.2. Persistent Volumes

Persistent storage is needed for SDH. It’s required to use storage that can be created dynamically. You can find more information in the Dynamic Provisioning and Creating Storage Classes document.

2.2.3. External Image Registry

The SDH installation requires an Image Registry where images are first mirrored from an SAP Registry and then delivered to the OCP cluster nodes. The integrated OpenShift Container Registry is not appropriate for this purpose or may require further analysis. For now an external image registry needs to be setup instead. On AWS, it is recommended to utilize Amazon Elastic Container Registry. Please refer to Using AWS ECR Registry for the Modeler for a post-configuration step to enable the registry for the Modeler.

The Jump host is a feasible machine to host the registry. Please follow the steps in article How do I setup/install a Docker registry?.

After the setup you should have an external image registry up and running at the URL My_Image_Registry_FQDN:5000. You can verify that with the following command.

# curl http://My_Image_Registry_FQDN:5000/v2/
{}

Make sure to mark the address as insecure.

Additionally, make sure to mark the registry as insecure within the Pipeline Modeler.

2.2.3.1. Update the list of insecure registries
  • Since the external image registry deployed above is insecure by default, in order to push images to the image registry and pull them on nodes it must be listed as insecure in /etc/containers/registries.conf file on all the hosts, including the Jump host:

    # vi /etc/containers/registries.conf
    ...
    [registries.insecure]
    registries = [
      "My_Image_Registry_FQDN:5000"
    ]
    ...
    
  • On the Jump host, the change is achieved simply by editing the file and restarting the host:

    # systemctl restart docker
    
  • On the OCP nodes, the configuration is achieved by adding the "My_image_REgistry_FQDN:5000" to the list of .spec.registryResources.insecureRegistries in the resource image.config.openshift.io/cluster. Please refer to Configuring image settings for more information.

If you plan to run the installation on the Jump host as a non-root user, please check the instructions below for additional steps.

NOTE These settings have no effect on the Kaniko Image Builder, which also needs to be aware of the insecure registry. Please refer to Marking the vflow registry as insecure for more information.

3. Install Red Hat OpenShift Container Platform

3.1. Prepare the Jump host

  1. Subscribe the Jump host at least to the following repositories:

    # sudo subscription-manager repos        \
        --enable=rhel-7-server-rpms          \
        --enable=rhel-7-server-extras-rpms   \
        --enable=rhel-7-server-optional-rpms \
        --enable=rhel-7-server-ose-4.1-rpms
    
  2. Install a helm client on the Jump host.

    • Download a script from https://github.com/kubernetes/helm and execute it with the desired version set to the latest major release of helm for your SDH release. That is v2.12 for SDH 2.6.

      # DESIRED_VERSION=v2.12.3
      # curl --silent https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | \
          DESIRED_VERSION="${DESIRED_VERSION:-v2.12.3}" bash
      
    • To determine the latest release for particular major version of helm, run the following command as a regular user. Note: make sure to install jq RPM package first though.

      # MAJOR=v2.12
      # curl -s https://api.github.com/repos/helm/helm/releases | \
          jq -r '.[] | select(.tag_name | test("'"$(echo "${MAJOR:-2.12}" | sed \
                  -e 's/^v\?\([0-9]\+\)\.\([0-9]\+\).*/v\1.\2/' \
                  -e 's/\./\\\\./')"'\\.\\d+$")) |
              .tag_name' | sort -V | tail -n 1
      v2.12.3
      
  3. Download and install kubectl.

    • Either via standard repositories by installing atomic-openshift-clients:

      # sudo yum install -y openshift-clients
      

      NOTE: rhel-7-server-ose-X.Y-rpms repositories corresponding to the same major release version (e.g. 4.1) as on the cluster nodes need to be enabled.

    • Or by downloading and installing the binary manually after determining the right version:

      # curl -LO https://dl.k8s.io/release/${ver:-v1.13.8}/bin/linux/amd64/kubectl
      # chmod +x ./kubectl
      # sudo mv ./kubectl /usr/local/bin/kubectl
      

      To determine the right version, look for the highest release name of the same minor version as the desired OCP cluster at the release page. That would be v1.13.8 for OCP 4.1 as of Jul 26 2019. The command below will determine the version for you. Make sure to install jq RPM package before running it though.

      # OCPVER=4.1
      # ver="$(curl --silent https://api.github.com/repos/kubernetes/kubernetes/releases | \
             jq -r '.[] | .name' | sort -t '.' -n -k 3 | sort --version-sort | \
             grep -F "$(echo "$OCPVER" | awk -F . '{print "v1."($2 + 12)"."}')" | tail -n 1)"; echo "$ver"
      v1.13.8
      
  4. In case of mpsl and mpfree SDH installation methods, make sure to install and run the SAP Host Agent as well.

    • However, in step 4, instead of downloading a *.SAR archive, as suggested by the guide, on RHEL it is recommended to download the latest RPM package (e.g. saphostagentrpm_43-20009394.rpm) and install it on the Jump host using a command like:

      # yum install saphostagentrpm_43-20009394.rpm
      
    • This way, the installation of SAPCAR listed in prerequisites is not needed.

    • Step 6 (SAR archive extraction) can then be skipped.
    • In the step 7, the command then needs to be modified to:

      # cd /usr/sap/hostctrl/exe
      # ./saphostexec -setup slplugin -passwd
      
    • Additionally, make sure to set the password for sapadm user. You will be prompted for the username and password by the maintenance planner.

      # passwd sapadm
      

3.2. Install OpenShift Container Platform

Install OpenShift Container Platform on your desired cluster hosts. Follow the OpenShift installation guide.

If you choose the Installer Provisioned Infrastructure (IPI), please follow the Installing a cluster on AWS with customizations methods to allow for customizations.

Several changes need to be done to the compute nodes running SDH workloads before SDH installation. These include:

  1. choose a sufficient number and type of compute instances for SDH workload
  2. pre-load needed kernel modules
  3. increasing the pids limit of CRI-O container engine
  4. configure insecure registry (if an insecure registry shall be used)

The first two items can be performed during or after OpenShift installation. The others only after the OpenShift installation.

3.2.1. Customizing IPI installation on AWS

In order to allow for customizations, the installation need to be performed in steps:

  1. create the installation configuration file
  2. create the ignition configuration files
  3. create the cluster
3.2.1.1. Modifying the installation configuration file

After the configuration file is created by the installer, you can specify the desired instance type of compute nodes by editing <installation_directory>/install-config.yaml. A shortened example could look like this:

apiVersion: v1
compute:
- hyperthreading: Enabled
  name: worker
  platform:
    aws:
      region: us-east-1
      type: m4.2xlarge
  replicas: 3

To satisfy the SDH's production requirements, you can change the compute.platform.aws.type to r5.2xlarge and compute.replicas to 4.

Then continue to generate the ignition configuration files from the installation file by running the following command. Note that the configuration file will be deleted. Therefor, it may make sense to back it up first.

# openshift-install create ignition-configs --dir <installation_directory>
3.2.1.2. Modifying the worker ignition configuration file

The following kernel modules need to be pre-loaded on compute nodes running SDH workloads: nfsd, ip_tables and ipt_REDIRECT.

That can be achieved by creating a file in /etc/modules-load.d on each node containing the list of the modules. The following snippet shows, how to do it in bash. Make sure to install jq application/RPM package and to change to the correct <installation_directory>.

# pushd <installation_directory>
# modules=( nfsd ip_tables ipt_REDIRECT )
# content="$(printf '%s\n' "${modules[@]}" | base64)"
# cp worker.{,bak.}ign                      # make a backup of the ignition file
# jq -c '.storage |= {"files":[{
    "contents": {
      "source": "data:text/plain;charset=utf-8;base64,'"${content}"'",
      "verification": {}
    },
    "filesystem": "root",
    "mode": 420,
    "path": "/etc/modules-load.d/sap-datahub-dependencies.conf"
}]} | .systemd |= {"units":[{
    "contents": "[Unit]\nDescription=Pre-load kernel modules for SAP Data Hub\nAfter=network.target\n\n[Service]\nType=simple\nExecStart=/usr/sbin/modprobe iptable_nat\nRestart=on-failure\nRestartSec=10\nRemainAfterExit=yes\n\n[Install]\nWantedBy=multi-user.target",
    "enabled": true,
    "name": "sdh-modules-load.service"
}]}' worker.bak.ign >worker.ign
# popd
3.2.1.3. Continue the installation by creating the cluster

To continue the IPI installation, execute the following command:

# openshift-install create cluster --dir <installation_directory>

3.3. OCP Post Installation Steps

3.3.1. Change the count and instance type of compute nodes

Please refer to Creating a MachineSet for changing an instance type and Manually scaling a MachineSet or Applying autoscaling to an OpenShift Container Platform cluster for information on scaling the nodes.

3.3.2. Pre-load needed kernel modules

To apply the desired changes to the existing compute nodes, all that is necessary is to create a new machine config that will be merged with the existing configuration:

# modules=( nfsd ip_tables ipt_REDIRECT )
# content="$(printf '%s\n' "${modules[@]}" | base64)"
# oc create -f - <<EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 75-worker-sap-datahub
spec:
  config:
    ignition:
      version: 2.2.0
    storage:
      files:
      - contents:
          source: "data:text/plain;charset=utf-8;base64,$content"
          verification: {}
        filesystem: root
        mode: 420
        path: /etc/modules-load.d/sap-datahub-dependencies.conf
    systemd:
      units:
      - contents: |
          [Unit]
          Description=Pre-load kernel modules for SAP Data Hub
          After=network.target

          [Service]
          Type=simple
          ExecStart=/usr/sbin/modprobe iptable_nat
          Restart=on-failure
          RestartSec=10
          RemainAfterExit=yes

          [Install]
          WantedBy=multi-user.target
        enabled: true
        name: sdh-modules-load.service
EOF

The changes will be rendered into machineconfigpool/worker. The workers will be restarted one-by-one until the changes are applied to all of them. See Applying configuration changes to the cluster for more information.

3.3.3. Change the maximum number of PIDs per Container

The process of configuring the nodes is described at Modifying Nodes. In SDH case, the required settings are .spec.containerRuntimeConfig.pidsLimit in a ContainerRuntimeConfig. The result is a modified /etc/crio/crio.conf configuration file on each affected worker node with pids_limit set to the desired value.

  1. Label the particular pool of worker nodes.

    # oc label machineconfigpool/worker workload=sapdatahub
    
  2. Create the following ContainerRuntimeConfig resource.

    # oc create -f - <<EOF
    apiVersion: machineconfiguration.openshift.io/v1
    kind: ContainerRuntimeConfig
    metadata:
     name: bumped-pid-limit
    spec:
     machineConfigPoolSelector:
       matchLabels:
         workload: sapdatahub
     containerRuntimeConfig:
       pidsLimit: 4096
    EOF
    
  3. Wait until the machineconfigpool/worker becomes updated.

    # watch oc get  machineconfigpool/worker 
    NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED
    worker   rendered-worker-8f91dd5fdd2f6c5555c405294ce5f83c   True      False      False
    

3.3.4. Set up an External Image Registry

If you haven't done so already, please follow the External Image Registry prerequisite.

3.3.5. Configure an insecure registry

In order for OpenShift to deploy SDH images from an insecure registry, the cluster needs to be told to treat the registry as insecure. Please follow Configuring image settings and the registry to the .spec.registrySources.insecureRegistries list. For example:

apiVersion: config.openshift.io/v1
kind: Image
metadata:
  annotations:
    release.openshift.io/create-only: "true"
  name: cluster
spec:
  registrySources:
    insecureRegistries:
    - My_Image_Registry_FQDN:5000

3.3.6. Configure the OpenShift Cluster for SDH

3.3.6.1. Becoming a cluster-admin

Many commands below require cluster admin privileges. To become a cluster-admin, you can do one of the following:

  • Copy the admin.kubeconfig file from a remote master node to a local host and use that:

    # scp master.node:/etc/origin/master/admin.kubeconfig .
    # export KUBECONFIG=$(pwd)/admin.kubeconfig
    # oc login -u system:admin
    

    This is recommended for mpsl and mpfree installation methods when using the Jump host.

    NOTE: the same file is used as the KUBECONFIG File input parameter for mpsl and mpfree installation methods.

  • Log-in to any master node as the root user, execute the following command and continue the installation as system:admin user. In this case, the master node becomes a Jump host.

    # oc login -u system:admin
    

    Possible for all installation methods without a Jump host.

  • (manual installation method only) Make any existing user (dhamin in this example) a cluster admin by doing the previous step followed by:

    # oc adm policy add-cluster-role-to-user cluster-admin dhadmin
    

You can learn more about the cluster-admin role in Cluster Roles and Local Roles article.

3.3.6.2. Project setup
  1. As a cluster-admin, create a tiller service account in the kube-system project (aka namespace) and grant it the necessary permissions:

    # oc create sa -n kube-system tiller
    # oc adm policy add-cluster-role-to-user cluster-admin -n kube-system -z tiller
    
  2. Set up helm and tiller for the deployment:

    # helm init --service-account=tiller --upgrade --wait
    

    Upon successful initialization, you should be able to see a tiller pod in the kube-system namespace:

    # oc get pods -n kube-system
    NAME                            READY     STATUS    RESTARTS   AGE
    tiller-deploy-551988758-dzjx5   1/1       Running   0          1m
    # helm ls
    [There should be no error in the output. If there is no output at all, it means good news, no error]
    
  3. Create a dedicated project in OpenShift for the SDH deployment. For example sdh. Login to OpenShift as a cluster-admin, and perform the following configurations for the installation:

    # oc new-project sdh
    # oc adm policy add-scc-to-group anyuid "system:serviceaccounts:$(oc project -q)"
    # oc adm policy add-scc-to-group hostmount-anyuid "system:serviceaccounts:$(oc project -q)"
    # oc adm policy add-scc-to-user privileged -z "vora-vsystem-$(oc project -q)"
    # oc adm policy add-scc-to-user privileged -z "$(oc project -q)-elasticsearch"
    # oc adm policy add-scc-to-user privileged -z "$(oc project -q)-fluentd"
    # oc adm policy add-scc-to-user privileged -z "default"
    # oc adm policy add-scc-to-user privileged -z "vora-vflow-server"
    
  4. As a cluster-admin, allow the project administrator to manage SDH custom resources.

    # oc create -f - <<EOF
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: aggregate-sapvc-admin-edit
      labels:
        rbac.authorization.k8s.io/aggregate-to-admin: "true"
        rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rules:
    - apiGroups: ["sap.com"]
      resources: ["voraclusters"]
      verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"]
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: aggregate-sapvc-view
      labels:
        # Add these permissions to the "view" default role.
        rbac.authorization.k8s.io/aggregate-to-view: "true"
    rules:
    - apiGroups: ["sap.com"]
      resources: ["voraclusters"]
      verbs: ["get", "list", "watch"]
    EOF
    
  5. When Kaniko Image Builder shall be enabled and the external image registry is not secured with a certificate signed by a trusted certificate authority, the registry needs to be marked as insecure.

  6. Deploy sdh-observer in the sdh namespace. Please follow the appendix Deploy SDH Observer .

4. Install SDH on OpenShift

4.1. Required Input Parameters

A few important installation parameters are described below. Please refer to the Required Input Parameters for their full description. Most of the parameters must be provided for the mpsl and mpfree installation methods. The Command line argument column describes corresponding options for the manual installation.

Name Condition Recommendation Command line argument
Kubernetes Namespace Always Must match the project name chosen in the Project Setup (e.g. sdh) -n sdh
Installation Type Installation or Update Choose Advanced Installation if you need to specify proxy settings or you want to choose particular storage class or there is no default storage class set. None
KUBECONFIG File Always The path to the kubeconfig file on the Jump host. It is the same file as described in Becoming a cluster-admin. If the SAP Host Agent is running on the master host, it can be set to /root/.kube/config. None3
Container Registry Installation Must be set to the external image registry. -r My_Image_Registry_FQDN:5000
Certificate Domain4 Installation Shall be set either to
1. the external FQDN of the OpenShift node used to access the vsystem service if using NodePort or
2. the FQDN of the vsystem route.
--cert-domain vsystem-sdh.wildcard-domain
Custom Certificate None5 Custom certificate for SDH services. Ideally a wildcard certificate for the master default subdomain resolving to the OpenShift Router. --provide-certs
Cluster Proxy Settings Advanced Installation or Advanced Updates Make sure to configure this if the traffic to internet needs to be routed through a proxy. None
Cluster HTTP Proxy Advanced Installation or Advanced Updates Make sure to set this if the traffic to internet needs to be routed through a proxy. --cluster-http-proxy "$HTTP_PROXY"
Cluster HTTPS Proxy Advanced Installation or Advanced Updates Make sure to set this if the traffic to internet needs to be routed through a proxy. --cluster-https-proxy "$HTTPS_PROXY"
Cluster No Proxy Advanced Installation or Advanced Updates Make sure to set this if the traffic to internet needs to be routed through a proxy. --cluster-https-proxy "$NO_PROXY"
StorageClass Configuration Advanced Installation Configure this if you want to choose different dynamic storage provisioners for different SDH components or if there's no default storage class set or you want to choose non-default storage class for the SDH components. None
Default StorageClass Advanced Installation and if storage classes are configured Set this if there's no default storage class set or you want to choose non-default storage class for the SDH components. --pv-storage-class ceph-rbd
Additional Installer Parameters Advanced Installation Useful for reducing the minimum memory requirements of the HANA pod, enabling Kaniko Image Builder and much more. -e vora-cluster.components.dlog.replicationFactor=2
--enable-kaniko=yes

4.2 Kaniko Image Builder

By default, Pipeline Modeler (vflow) pod uses Docker Daemon on the node, where it runs, to build container images before they are run. This was possible on OCP releases prior to 4.0. Since then, OCP uses CRI-O containter runtime.

To enable Pipeline Modeler to build images on recent OCP releases, it must be configured to use kaniko image builder. This is achieved by passing --enable-kaniko=yes parameter to the install.sh script during the manual installation. For the other installation methods, one can enable it by appending --enable-kaniko=yes to SLP_EXTRA_PARAMETERS (Additional Installation Parameters).

4.2.1. Registry requirements for the Kaniko Image Builder

The Kaniko Image Builder supports out-of-the-box only connections to secure image registries with a certificate signed by a trusted certificate authority.

In order to use an insecure image registry (e.g. the proposed external image registry) in combination with the builder, the registry must be whitelisted in Pipeline Modeler by marking it as insecure.

4.3. Installation using the Maintenance Planner and SL Plugin (mpsl)

Is a web-based installation method recommended by SAP offering you an option to send analytics data and feedback to SAP. All the necessary prerequisites have been satisfied by applying all the steps described above. The Installing SAP Data Hub using SLC Bridge (SL Plugin) with Maintenance Planner and SAP Host Agent documentation will guide you through the process.

NOTE: Make sure to enable Advanced Installation and to add --enable-kaniko=yes parameter to the Advanced Installer Parameters.

4.4. Installation using SL Plugin without Maintenance Planner (mpfree)

Is an alternative command-line-based installation method. Please refer to the SAP Data Hub documentation for more information and the exact procedure.

NOTE: Make sure to enable Advanced Installation and to add --enable-kaniko=yes parameter to the Advanced Installer Parameters.

4.5. Manual Installation using an installation script (manual)

4.5.1. Download and unpack the SDH binaries

Download and unpack SDH installation binary onto the Jump host.

  1. Go to SAP Software Download Center, login with your SAP account and search for DATA HUB 2 or access this link.

  2. Download the SAP Data Hub Foundation file, for example: DHFOUNDATION06_0-80004015.ZIP (SAP DATA HUB - FOUNDATION 2.6).

  3. Unpack the installer file. For example, when you unpack the DHFOUNDATION06_0-80004015.ZIP package, it will create the installation folder SAPDataHub-2.6.97-Foundation.

    # unzip DHFOUNDATION06_0-80004015.ZIP
    

4.5.2. Install SAP Data Hub

4.5.2.1. Remarks on the installation options
Feature Installation method Parameter Name
Storage Class Manual --pv-storage-class="$storage_class_name"
Advanced mpsl or mpfree installations Default storage class
Kaniko Image Builder Manual --enable-kaniko=yes
Advanced mpsl or mpfree installations Additional Installation Parameters or SLP_EXTRA_PARAMETERS

Storage Class

When there is no default dynamic storage provisioner defined, the preferred one needs to be specified explicitely.
If the default dynamic storage provisioner has been defined, the parameter can be omitted. To define the default dynamic storage provisioner, please follow the document Changing the Default StorageClass.

Kaniko Image Builder

Can be enabled starting from 2.5. It became necessary to enable on OCP releases 4.x. During mpsl and mpfree installation methods, one needs to append --enable-kaniko=yes to the list of Additional Installation Parameters or SLP_EXTRA_PARAMETERS.
See Kaniko Image Builder for more information.

4.5.2.2. Executing the installation script

Run the SDH installer as described in Manually Installing SAP Data Hub on the Kubernetes Cluster.

4.6. SDH Post installation steps

4.6.1. (Optional) Expose SDH services externally

There are multiple possibilities how to make SDH services accessible outside of the cluster. Compared to kubernetes, OpenShift offers additional method, which is recommended for most of the scenarios including SDH services. It's based on OpenShift Router and routes. The other methods documented in the official SAP Data Hub documentation are still available.

4.6.1.1. Using OpenShift Router and routes

OpenShift allows you to access the Data Hub services via routes as opposed to regular NodePorts. For example, instead of accessing the vsystem service via https://master-node.example.com:32322, after the service exposure, you will be able to access it at https://vsystem-sdh.wildcard-domain. This is an alternative to the official guide documentation to Expose the Service From Outside the Network.

NOTE: For this to work, a wildcard domain needs to be preconfigured in the local DNS server to resolve the desired wildcard-domain and all its subdomains (e.g. vsystem-sdh.wildcard-domain or hana-wire-sdh.wildcard-domain) to the node, where OpenShift Router (or its load balancer) runs. Please follow Using Wildcard Routes (for a Subdomain) for more information.

  1. After a successful installation, create a route for the vsystem and other services.

    1. Look up the vsystem service:

      # oc project sdh            # switch to the Data Hub project
      # oc get services | grep "vsystem "
      vsystem   ClusterIP   172.30.227.186   <none>   8797/TCP   19h
      
    2. Create the route:

      # oc create route passthrough --service=vsystem
      # oc get route
      NAME      HOST/PORT                     PATH  SERVICES  PORT      TERMINATION  WILDCARD
      vsystem   vsystem-sdh.wildcard-domain         vsystem   vsystem   passthrough  None
      
    3. (Optional) Expose the SAP Vora Transaction Coordinator for external access:

      # oc create route passthrough --service=vora-tx-coordinator-ext
      # oc get route
      NAME                     HOST/PORT                                    PATH  SERVICES                 PORT      TERMINATION  WILDCARD
      vora-tx-coordinator-ext  vora-tx-coordinator-ext-sdh.wildcard-domain        vora-tx-coordinator-ext  tc-ext    passthrough  None
      vsystem                  vsystem-sdh.wildcard-domain                        vsystem                  vsystem   passthrough  None
      

      Alternatively, you may want to specify a different hostname instead of the generated one (e.g. --hostname=vora-tx-coordinator.wildcard-domain).

    4. (Optional) Expose the SAP HANA Wire for external access:

      # oc create route passthrough --service=vora-tx-coordinator-ext --port=hana-wire --dry-run -o yaml | \
          oc patch -o yaml --local -p '{"metadata":{"name":"hana-wire"}}' -f - | oc create -f -
      # oc get route
      NAME                     HOST/PORT                                    PATH  SERVICES                 PORT       TERMINATION  WILDCARD
      hana-wire                hana-wire-sdh.wildcard-domain                      vora-tx-coordinator-ext  hana-wire  passthrough  None
      vora-tx-coordinator-ext  vora-tx-coordinator-ext-sdh.wildcard-domain        vora-tx-coordinator-ext  tc-ext     passthrough  None
      vsystem                  vsystem-sdh.wildcard-domain                        vsystem                  vsystem    passthrough  None
      
  2. Access the SDH web console at https://vsystem-sdh.wildcard-domain.

NOTE: The connection will be insecure unless one of the following holds true:

  • A signed certificate has been generated for the hostname and passed to the installer during the installation with --provide-certs.
  • The Certificate Domain has been set to the vsystem-sdh.wildcard-domain and the generated certificate has been imported to the browser.
4.6.1.2. Using NodePorts

NOTE For OpenShift, an exposure using routes is preferred.

Exposing SAP Data Hub vsystem

  • Either with an auto-generated node port:

    # oc expose service vsystem --type NodePort --name=vsystem-nodeport --generator=service/v2
    # oc get -o jsonpath=$'{.spec.ports[0].nodePort}\n' services vsystem-nodeport
    30617
    
  • Or with a specific node port (e.g. 32123):

    # oc expose service vsystem --type NodePort --name=vsystem-nodeport --generator=service/v2 --dry-run -o yaml | \
        oc patch -p '{"spec":{"ports":[{"port":8797, "nodePort": 32123}]}}' --local -f - -o yaml | oc create -f -
    

The original service remains accessible on the same ClusterIP:Port as before. Additionally, it is now accessible from outside of the cluster under the node port.

Exposing SAP Vora Transaction Coordinator and HANA Wire

# oc expose service vora-tx-coordinator-ext --type NodePort --name=vora-tx-coordinator-nodeport --generator=service/v2
# oc get -o jsonpath=$'tx-coordinator:\t{.spec.ports[0].nodePort}\nhana-wire:\t{.spec.ports[1].nodePort}\n' \
    services vora-tx-coordinator-nodeport
tx-coordinator: 32445
hana-wire:      32192

The output shows the generated node ports for the newly exposed services.

4.6.2. (AWS only) Configure registry secret for the Modeler

Please follow Using AWS ECR Registry for the Modeler if this registry shall be used for the Pipeline Modeler.

4.6.3. SDH Validation

Validate SDH installation on OCP to make sure everything works as expected. Please follow the instructions in Testing Your Installation.

6. Appendix

6.1. SDH uninstallation

Choose one of the uninstallation methods based on what method has been used for the installation.

When done, you may continue with a new installation round in the same or another namespace.

6.1.1. Using the SL Plugin

Please follow the SAP documentation Uninstall SAP Data Hub Using the SL Plugin if you have installed SDH using either mpsl or mpfree methods.

6.1.2. Manual uninstallation

The installation script allows for project clean-up where all the deployed pods, persistent volume claims, secrets, etc. are deleted. If the manual installation method was used, the SDH can be uninstalled using the same script and different set of parameters. The snippet below is an example where the SDH installation resides in the project sdh. In addition to running this script, the project needs to be deleted as well if the same project shall host the new installation.

# ./install.sh --delete --purge --force-deletion --namespace=sdh \
    --docker-registry=registry.local.example:5000
# oc delete project sdh
# # start the new installation

The deletion of the project often takes quite a while. Until fully uninstalled, the project will be listed as Terminating in the output of oc get project. You may speed the process up with the following command. Again please mind the namespace.

# oc delete pods --all --grace-period=0 --force --namespace sdh

NOTE: Make sure not to run the same installation script more than once at the same time even when working with different OpenShift projects.

6.2. Deploy SDH Observer

The sdh-observer observes the SDH namespace and applies fixes to deployments as they appear. It does the following:

  • adds additional emptyDir volume to vsystem-vrep Stateful Set to allow it to run on RHCOS system
  • enables the Pipeline Modeler (aka vflow) to talk to an insecure registry - needed only if the registry is insecure
  • grants fluentd pods permissions to logs
  • (optionally) marks containers manipulating iptables on RHCOS hosts as privileged when the modules are not pre-loaded and the nodes

Apart from accessing resources in sdh namespace, it also requires node-reader cluster role.

To deploy it, as a cluster-admin execute the following command in the SDH namespace before the SDH installation:

# OCPVER=4.1                 # this must match OCP minor release
# INSECURE_REGISTRY=false    # set to true if the registry is insecure
# oc process -f https://raw.githubusercontent.com/miminar/sdh-helpers/master/sdh-observer.yaml \
        NAMESPACE="$(oc project -q)" \
        BASE_IMAGE_TAG="${OCPVER:-4.1}" \
        MARK_REGISTRY_INSECURE=${INSECURE_REGISTRY:-0} | oc create -f -

NOTE: that the BASE_IMAGE_TAG must match one of the tags available in the quay.io/openshift/origin-cli repository. The difference between the client's minor release version and OCP server's minor release must not exceed 1.

6.3. Allow a non-root user to interact with Docker on Jump host

  • Append -G dockerroot to OPTIONS= in /etc/sysconfig/docker file on your Jump host.

  • Run the following commands on the Jump host, after you modify the /etc/sysconfig/docker file. Make sure to replace alice with your user name.

    # sudo usermod -a -G dockerroot alice
    # sudo chown root:dockerroot /var/run/docker.sock
    
  • Log out and re-log-in to the Jump host for the changes to become effective.

6.4. Grant fluentd pods permissions to logs

The diagnostics-fluentd-* pods need access to /var/log directories on nodes. For this to work. The pods need to be run as privileged. There are two steps necessary to make it happen:

  1. the ${SDH_PROJECT_NAME}-fluentd service account needs to be added on privileged scc list with the following command copied from the project setup:

    # oc project "${SDH_PROJECT_NAME}"
    # oc adm policy add-scc-to-user privileged -z "$(oc project -q)-fluentd"
    
  2. the daemonset diagnostics-fluentd need to be patched to request the privileged security context.

The recommended way to execute the second step is to deploy deploy SDH Observer. Alternatively, the patching of the daemonset can be done either before the SDH Installation (only applicable for manual installation) or afterwards.

6.4.1. Before the installation

Execute one of the patch commands below depending on the SDH version.

For SDH 2.6 during manual installation only:

# patch -p1 -B patchbaks/ -r - <<EOF
Index: foundation/deployment/helm/vora-diagnostic/templates/logging/fluentd.yaml
===================================================================
--- foundation.orig/deployment/helm/vora-diagnostic/templates/logging/fluentd.yaml
+++ foundation/deployment/helm/vora-diagnostic/templates/logging/fluentd.yaml
@@ -41,6 +41,7 @@ spec:
         - name: "FLUENT_ELASTICSEARCH_SCHEME"
           value: "http"
         securityContext:
+          privileged: true
           runAsUser: 0
           runAsNonRoot: false
         volumeMounts:
EOF

6.4.2. After the SDH installation

Execute the following command once the SDH installation is finished in the Data Hub's namespace.

# oc patch ds/diagnostics-fluentd  -p '{ "spec": { "template": { "spec": {
      "containers": [{ "name": "diagnostics-fluentd", "securityContext": { "privileged": true }}]
    }}}}'

The fluentd pods will get restarted automatically.

6.5. Marking the vflow registry as insecure

NOTE: applicable before, during or a after SDH installation.

In SAP Data Hub 2.6.x releases, it is possible to configure insecure registry for Pipeline Modeler (aka vflow pod) neither via installer nor in the UI.

The insecure registry needs to be set if the container registry listens on an insecure port (HTTP) or the communication is encrypted using a self-signed certificate.

Without the insecure registry set, kaniko builder cannot push built images into the configured registry for the Pipeline Modeler (see "Container Registry for Pipeline Modeler" Input Parameter at the official SAP Data Hub documentation).

To mark the configured vflow registry as insecure, the SDH Observer needs to be deployed with MARK_REGISTRY_INSECURE=true parameter. If it is already deployed, it can be re-configured to take care of insecure registries by executing the following command in the sdh namespace:

# oc set env dc/sdh-observer MARK_REGISTRY_INSECURE=true

Once deployed, all the existing pipeline modeler pods will be patched. It will take a couple of tens of seconds until all the modified pods become available.

For more information, take a look at SDH Helpers.

6.6. Using AWS ECR Registry for the Modeler

Post-sdh-installation step

SAP Data Hub installer allows to specify "AWS IAM Role for Pipeline Modeler" when AWS ECR Registry is used as the external registry. However, due to a bug in Data Hub, the Modeler cannot use it. In order to use AWS ECR Registry for Data Hub, one can follow the instructions at Provide Access Credentials for a Password Protected Container Registry with the following modification.

# cat >/tmp/vsystem-registry-secret.txt <<EOF
username: "AWS_ACCESS_KEY_ID"
password: "AWS_SECRET_ACCESS_KEY"
EOF

The AWS_* credentials must belong to a user that has the full access to the ECR registry. Please refer to Amazon ECR Repository Policies when you need fine-grained access control.

7. Troubleshooting Tips

7.1. Installation problems

7.1.1. Vora Installation Error: timeout at “Deploying vora-consul”

Vora Installation Error: timeout at "Deploying vora-consul with: helm install --namespace vora -f values.yaml ..."

To view the log messages, you can login to the OpenShift web console, navigate to Applications -> Pods, select the failing pod e.g. vora-consul-2-0, and check the log under the Events tab.

A common error: if the external image registry is insecure, but the OpenShift cluster is configured to pull from a secure registry, you will see errors in the log. If secure registry is not feasible, follow the instructions on configuring the registry as insecure.

7.1.2. Privileged security context unassigned

If there are pods, replicasets, or statefulsets not coming up and you can see an event similar to the one below, you need to add privileged security context constraint to its service account.

# oc get events | grep securityContext
1m          32m          23        diagnostics-elasticsearch-5b5465ffb.156926cccbf56887                          ReplicaSet                                                                            Warning   FailedCreate             replicaset-controller                  Error creating: pods "diagnostics-elasticsearch-5b5465ffb-" is forbidden: unable to validate against any security context constraint: [spec.initContainers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.initContainers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.initContainers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

Copy the name in the fourth column (the event name - diagnostics-elasticsearch-5b5465ffb.156926cccbf56887) and determine its corresponding service account name.

# eventname="diagnostics-elasticsearch-5b5465ffb.156926cccbf56887"
# oc get -o go-template=$'{{with .spec.template.spec.serviceAccountName}}{{.}}{{else}}default{{end}}\n' \
    "$(oc get events "${eventname}" -o jsonpath=$'{.involvedObject.kind}/{.involvedObject.name}\n')"
sdh-elasticsearch

The obtained service account name (sdh-elasticsearch) now needs to be assigned privileged scc:

# oc adm policy add-scc-to-user privileged -z sdh-elasticsearch

The pod then shall come up on its own unless this was the only problem.

7.1.3. No Default Storage Class set

If pods are failing because because of PVCs not being bound, the problem may be that the default storage class has not been set and no storage class was specified to the installer.

# oc get pods
NAME                                                  READY     STATUS    RESTARTS   AGE
hana-0                                                0/1       Pending   0          45m
vora-consul-0                                         0/1       Pending   0          45m
vora-consul-1                                         0/1       Pending   0          45m
vora-consul-2                                         0/1       Pending   0          45m

# oc describe pvc data-hana-0
Name:          data-hana-0
Namespace:     sdh
StorageClass:
Status:        Pending
Volume:
Labels:        app=vora
               datahub.sap.com/app=hana
               vora-component=hana
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
Events:
  Type    Reason         Age                  From                         Message
  ----    ------         ----                 ----                         -------
  Normal  FailedBinding  47s (x126 over 30m)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

To fix this, either make sure to set the Default StorageClass or provide the storace class name to the installer. For the manual installation, that would be ./install.sh --pv-storage-class STORAGECLASS.

7.1.4. Checkpoint store validation

If you see the following error during the installation for the checkpoint store validation, it means the S3 bucket or the given directory does not exist. Make sure to create them first.

2018-12-05T06:30:17-0500 [INFO] Validating checkpoint store...
2018-12-05T06:30:17-0500 [INFO] Checking connection...
2018-12-05T06:30:42-0500 [INFO] AFSI CLI ouput:
2018-12-05T06:30:42-0500 [INFO] Unknown error when executing operation: Couldn't open URL : Cannot open connection: file/directory 'bucket1/dir' does not exist.
pod sdh/checkpoint-store-administration terminated (Error)
2018-12-05T06:30:42-0500 [ERROR] Connection check failed!
2018-12-05T06:30:42-0500 [ERROR] Checkpoint store validation failed!

2018-12-05T06:30:42-0500 [ERROR] Please reconfigure your checkpoint store connection...

7.1.5. vsystem-app pods not coming up

If you have SELinux in enforcing mode you may see the pods launched by vsystem crash-looping because of the container named vsystem-iptables like this:

# oc get pods
NAME                                                          READY     STATUS             RESTARTS   AGE
auditlog-59b4757cb9-ccgwh                                     1/1       Running            0          40m
datahub-app-db-gzmtb-67cd6c56b8-9sm2v                         2/3       CrashLoopBackOff   11         34m
datahub-app-db-tlwkg-5b5b54955b-bb67k                         2/3       CrashLoopBackOff   10         30m
...
internal-comm-secret-gen-nd7d2                                0/1       Completed          0          36m
license-management-gjh4r-749f4bd745-wdtpr                     2/3       CrashLoopBackOff   11         35m
shared-k98sh-7b8f4bf547-2j5gr                                 2/3       CrashLoopBackOff   4          2m
...
vora-tx-lock-manager-7c57965d6c-rlhhn                         2/2       Running            3          40m
voraadapter-lsvhq-94cc5c564-57cx2                             2/3       CrashLoopBackOff   11         32m
voraadapter-qkzrx-7575dcf977-8x9bt                            2/3       CrashLoopBackOff   11         35m
vsystem-5898b475dc-s6dnt                                      2/2       Running            0          37m

When you ispect one of those pods, you can see an error message similar to the one below:

# oc logs voraadapter-lsvhq-94cc5c564-57cx2 -c vsystem-iptables
2018-12-06 11:45:16.463220|+0000|INFO |Execute: iptables -N VSYSTEM-AGENT-PREROUTING -t nat||vsystem|1|execRule|iptables.go(56)
2018-12-06 11:45:16.465087|+0000|INFO |Output: iptables: Chain already exists.||vsystem|1|execRule|iptables.go(62)
Error: exited with status: 1
Usage:
  vsystem iptables [flags]

Flags:
  -h, --help               help for iptables
      --no-wait            Exit immediately after applying the rules and don't wait for SIGTERM/SIGINT.
      --rule stringSlice   IPTables rule which should be applied. All rules must be specified as string and without the iptables command.

And in the audit log on the node, where the pod got scheduled, you should be able to find an avc denial similar to the following. On RHCOS nodes, you may need to inspect the output of dmesg command instead.

# grep 'denied.*iptab' /var/log/audit/audit.log
type=AVC msg=audit(1544115868.568:15632): avc:  denied  { module_request } for  pid=54200 comm="iptables" kmod="ipt_REDIRECT" scontext=system_u:system_r:container_t:s0:c826,c909 tcontext=system_u:system_r:kernel_t:s0 tclass=system permissive=0
...
# # on RHCOS
# dmesg | grep denied

To fix this, the ipt_REDIRECT kernel module needs to be loaded. Please refer to Pre-load needed kernel modules.

7.1.6. Fluentd pods cannot access /var/log

If you see errors like shown below in the logs of fluentd pods, make sure to follow the Grant fluentd pods permissions to logs to fix the problem.

# oc logs $(oc get pods -o name -l datahub.sap.com/app-component=fluentd | head -n 1) | tail -n 20
2019-04-15 18:53:24 +0000 [error]: unexpected error error="Permission denied @ rb_sysopen - /var/log/es-containers-sdh25-mortal-garfish.log.pos"
  2019-04-15 18:53:24 +0000 [error]: suppressed same stacktrace
  2019-04-15 18:53:25 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
  2019-04-15 18:53:26 +0000 [error]: unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/es-containers-sdh25-mortal-garfish.log.pos"
  2019-04-15 18:53:26 +0000 [error]: /usr/lib64/ruby/gems/2.5.0/gems/fluentd-0.14.8/lib/fluent/plugin/in_tail.rb:151:in `initialize'
  2019-04-15 18:53:26 +0000 [error]: /usr/lib64/ruby/gems/2.5.0/gems/fluentd-0.14.8/lib/fluent/plugin/in_tail.rb:151:in `open'
...

7.1.7. License Manager cannot be initialized

The installation may fail with the following error.

2019-07-22T15:07:29+0000 [INFO] Initializing system tenant...
2019-07-22T15:07:29+0000 [INFO] Initializing License Manager in system tenant...2019-07-22T15:07:29+0000 [ERROR] Couldn't start License Manager!
The response: {"status":500,"code":{"component":"router","value":8},"message":"Internal Server Error: see logs for more info"}Error: http status code 500 Internal Server Error (500)
2019-07-22T15:07:29+0000 [ERROR] Failed to initialize vSystem, will retry in 30 sec...

In the log of license management pod, you can find an error like this:

# oc logs deploy/license-management-l4rvh
Found 2 pods, using pod/license-management-l4rvh-74595f8c9b-flgz9
+ iptables -D PREROUTING -t nat -j VSYSTEM-AGENT-PREROUTING
+ true
+ iptables -F VSYSTEM-AGENT-PREROUTING -t nat
+ true
+ iptables -X VSYSTEM-AGENT-PREROUTING -t nat
+ true
+ iptables -N VSYSTEM-AGENT-PREROUTING -t nat
iptables v1.6.2: can't initialize iptables table `nat': Permission denied
Perhaps iptables or your kernel needs to be upgraded.

This means, the vsystem-iptables container in the pod lacks permissions to manipulabe iptables. It needs to be marked as privileged. Please follow the appendix Deploy SDH Observer and restart the installation.

7.2. Pipeline Modeler troubleshooting

7.2.1. Graphs cannot be run in the Pipeline Modeler

If in the log of the vflow pod, there are problems with reaching outside of the private network like the following output shows, make sure to verify your proxy settings and make sure that the installation script is run with the following parameters:

# ./install.sh \
    --cluster-http-proxy="${HTTP_PROXY}" \
    --cluster-https-proxy="${HTTPS_PROXY}" \
    --cluster-no-proxy="${NO_PROXY}"

vflow log can be displayed with a command like oc logs $(oc get pods -o name -l vora-component=vflow | head -n 1):

W: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease  Could not connect to deb.debian.org:80 (5.153.231.4), connection timed out [IP: 5.153.231.4 80]
W: Failed to fetch http://security.debian.org/debian-security/dists/stretch/updates/InRelease  Could not connect to security.debian.org:80 (217.196.149.233), connection timed out [IP: 217.196.149.233 80]
W: Failed to fetch http://deb.debian.org/debian/dists/stretch-updates/InRelease  Unable to connect to deb.debian.org:http: [IP: 5.153.231.4 80]

7.2.2. Pipeline Modeler cannot push images to the registry

If the SDH is configured to build images with kaniko and the vflow registry is not configured with a certificate signed by a trusted certificate authority, the builder will not be able to push the built images there. The Pipeline Modeler will then label the graphs as dead with a message like the following:

failed to prepare graph description: failed to prepare image: build failed for image: internal-registry.example.org:5000/vora/vflow-node-482f9340ff573d1a7a03108d18556792bb70ae2a:com.sap.debian

To determine the cause, the log of the vflow pod needs to be inspected. There, you can notice the root issue - in this case it is the insecure registry internal-registry.example.org:5000 accessible only via HTTP protocol.

# oc logs $(oc get pods -o name -l vora-component=vflow | head -n 1)
...
INFO[0019] Using files from context: [/workspace/vflow] 
INFO[0019] COPY /vflow /vflow                           
INFO[0019] Taking snapshot of files...                  
INFO[0023] ENTRYPOINT ["/vflow"]                        
error pushing image: failed to push to destination internal-registry.example.org:5000/vora/vflow-node-482f9340ff573d1a7a03108d18556792bb70ae2a:com.sap.debian: Get https://internal-registry.example.org:5000/v2/: http: server gave HTTP response to HTTPS client |vflow|container|192|getPodLogs|build.go(126)
...

To resolve it, you can:

7.2.3. Modeler does not run when AWS ECR registry is used

If the initialization of the vflow pod fails with a message like the one below, your SDH deployment suffers from a bug that prevents it from using the AWS IAM Role for authentication against the AWS ECR Registry.

# oc logs $(oc get pods -o name -l vora-component=vflow | head -n 1)
....
2019-07-15 12:23:03.147231|+0000|INFO |Statistics Publisher started with publication interval 30s ms|vflow|statistic|38|loop|statistics_monitor.go(89)
2019-07-15 12:23:30.446482|+0000|INFO |connecting to vrep at vsystem-vrep.sdh:8738|vflow|container|1|NewImageFactory|factory.go(131)
2019-07-15 12:23:30.446993|+0000|INFO |Creating AWS ECR Repository 'sdh26/vora/vflow-node-482f9340ff573d1a7a03108d18556792bb70ae2a'|vflow|container|1|assertRepositoryExists|ecr.go(106)
2019-07-15 12:23:35.001030|+0000|ERROR|API node execution is failed: cannot instantiate docker registry client: failed to assert repository existance: Error creating AWS ECR repository 'sdh26/vora/vflow-node-482f9340ff573d1a7a03108d18556792bb70ae2a': NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors
failed to create image factory
main.runMaster
        /data/xmake/prod-build7010/w/velocity/velocity-vflow-OD-linuxx86_64_indirectshipment-bdh-2.5-linuxx86_64/gen/out/build-hl-vflow-linuxx86_64/gows/src/github.wdf.sap.corp/velocity/vflow/src/cmd/vflow/main.go:386
main.main
        /data/xmake/prod-build7010/w/velocity/velocity-vflow-OD-linuxx86_64_indirectshipment-bdh-2.5-linuxx86_64/gen/out/build-hl-vflow-linuxx86_64/gows/src/github.wdf.sap.corp/velocity/vflow/src/cmd/vflow/main.go:357
runtime.main
        /data/xmake/tools/xmake-tools/FA/org.golang.download.go/go/1.11.4-bin/go/src/runtime/proc.go:201
runtime.goexit
        /data/xmake/tools/xmake-tools/FA/org.golang.download.go/go/1.11.4-bin/go/src/runtime/asm_amd64.s:1333|vflow|vflow|1|main|main.go(359)

The work-around is to use a registry pull secret.


  1. OCS stands for OpenShift Container Platform 

  2. any storage provisioned by your cloud provider listed in the Overview of Persistent Storage

  3. The environment variable $KUBECONFIG shall be set instead. 

  4. This setting assumes that all Data Hub services are accessed under the same name using NodePort. However, using OpenShift Routes, each service will be assigned a different hostname. Therefor, for production environment, it is necessary to provide signed certificates for these routes. You may consider configuring a custom wildcard certificate for master default subdomain

  5. Only available for the manual installation.