This chapter provides an overview and description of the reference architecture for a highly available Red Hat OpenShift Container Platform 3 environment deployed on a VMware private cloud.

The image shown above provides a high-level representation of the components within this reference architecture. Virtual machine (VM) resources are highly available using VMware technologies; VMware HA (high availability), storage IO (input/output) control, and resource allocation via hypervisor affinity and anti-affinity rules. The Ansible deployment host is a virtual machine and acts as the entry point for access to the hosts and performs configuration of the internal servers by ensuring that all Secure Shell (SSH) traffic passes through it.

Authentication is managed by Microsoft Active Directory via lightweight directory access protocol (LDAP) authentication. OpenShift on VMware has three cloud native storage options; virtual machine persistent storage, network file system (NFS) and Gluster file system (OCS).

Virtual machine persistent storage is housed on virtual machine disk VMDKs on datastores located on external logical unit numbers (LUNs) or NFS shares.

The other storage utilized is for container persistent storage including the OCP registry. The network is configured to leverage a single load balancer for access to the OpenShift API & Console (8443/tcp) and the OpenShift routers (80/tcp, 443/tcp).

Finally, the image shows that domain name system (DNS) is handled by an external DNS source. This DNS source should be pre-configured with the proper entries prior to deployment. In this case the solutions engineering team is managing all DNS entries through a BIND server and a conditional lookup zone in Microsoft DNS.

This reference architecture breaks down the deployment into three separate phases.

Provisioning of the VMware environment is a prerequisite, and outside the scope of this document. Phase 1 proceeds with the deployment of virtual machines, following requirements listed in the Section 2.10.1, “Virtual Machine Hardware Requirements”

Phase 2 is the installation of OpenShift Container Platform, which is done via the Ansible playbooks installed by the openshift-ansible-playbooks rpm package. During Phase 2 the router and registry are also deployed.

The last phase, Phase 3, concludes the deployment by confirming the environment was deployed properly. This is done by running some command line tools.

This reference architecture utilizes the following versions of VMware software:

This guide uses an external load balancer running HAproxy to offer a single entry point for the many Red Hat OpenShift Container Platform components. Organizations can provide their own currently deployed load balancers in the event that the service already exists.

The Red Hat OpenShift Container Platform console, provided by the Red Hat OpenShift Container Platform master nodes, can be spread across multiple instances to provide both load balancing and high availability properties.

Application traffic passes through the Red Hat OpenShift Container Platform Router on its way to the container processes. The Red Hat OpenShift Container Platform Router is a reverse proxy service container that multiplexes the traffic to multiple containers making up a scaled application running inside Red Hat OpenShift Container Platform. The load balancer used by infra nodes acts as the public view for the Red Hat OpenShift Container Platform applications.

The destination for the master and application traffic must be set in the load balancer configuration after each instance is created, the floating IP address is assigned and before the installation. A single HAproxy load balancer can forward both sets of traffic to different destinations.

DNS service is an important component in the Red Hat OpenShift Container Platform environment. Regardless of the provider of DNS, an organization is required to have certain records in place to serve the various Red Hat OpenShift Container Platform components.

Since the load balancer values for the Red Hat OpenShift Container Platform master service and infrastructure nodes running router pods are known beforehand, entries should be configured into the DNS prior to starting the deployment procedure.

Red Hat OpenShift Container Platform is comprised of multiple instances running on a VMware SDDC that allow for scheduled and configured OpenShift services and supplementary containers. These containers can have persistent storage, if required, by the application and integrate with optional OpenShift services such as logging and metrics.

... [OUTPUT ABBREVIATED] ...
[etcd]
master1.example.com
master2.example.com
master3.example.com

[masters]
master1.example.com
master2.example.com
master3.example.com

[nodes]
master1.example.com openshift_node_group_name="node-config-master"
master2.example.com openshift_node_group_name="node-config-master"
master3.example.com openshift_node_group_name="node-config-master"

... [OUTPUT ABBREVIATED] ...
[nodes]
infra1.example.com openshift_node_group_name="node-config-infra"
infra2.example.com openshift_node_group_name="node-config-infra"
infra3.example.com openshift_node_group_name="node-config-infra"

... [OUTPUT ABBREVIATED] ...

[nodes]
node1.example.com openshift_node_group_name="node-config-compute"
node2.example.com openshift_node_group_name="node-config-compute"
node3.example.com openshift_node_group_name="node-config-compute"

"labels": {
  "key1" : "value1",
  "key2" : "value2"
}

{"release" : "stable", "release" : "canary"}
{"environment" : "dev", "environment" : "qa", "environment" : "production"}
{"tier" : "frontend", "tier" : "backend", "tier" : "cache"}
{"partition" : "customerA", "partition" : "customerB"}
{"track" : "daily", "track" : "weekly"}

Red Hat OpenShift Container Platform offers the ability to specify how pods communicate with each other. This could be through the use of Red Hat provided Software-defined networks (SDN) or a third-party SDN.

Deciding on the appropriate internal network for an Red Hat OpenShift Container Platform step is a crucial step. Unfortunately, there is no right answer regarding the appropriate pod network to chose, as this varies based upon the specific scenario requirements on how a Red Hat OpenShift Container Platform environment is to be used.

For the purposes of this reference environment, the Red Hat OpenShift Container Platform ovs-networkpolicy SDN plug-in is chosen due to its ability to provide pod isolation using Kubernetes NetworkPolicy. The following section, “OpenShift SDN Plugins”, discusses important details when deciding between the three popular options for the internal networks - ovs-multitenant, ovs-networkpolicy and ovs-subnet.

Container images are stored locally on the nodes running Red Hat OpenShift Container Platform pods. The container-storage-setup script uses the /etc/sysconfig/docker-storage-setup file to specify the storage configuration.

The /etc/sysconfig/docker-storage-setup file should be created before starting the docker service, otherwise the storage would be configured using a loopback device. The container storage setup is performed on all hosts running containers, therefore masters, infrastructure, and application nodes.

Containers by default offer ephemeral storage, but some applications require the storage to persist between different container deployments or upon container migration. PersistentVolumeClaims (PVC) objects are used to store this persistent application data. These claims can either be added into the environment by hand or provisioned dynamically using a StorageClass object.

OpenShift can build container images from source code, deploy them, and manage their lifecycle. To enable this, OpenShift provides an internal, integrated registry that can be deployed in the OpenShift environment to manage images.

The registry stores images and metadata. For production environment, persistent storage should be used for the registry, otherwise any images that were built or pushed into the registry would disappear if the pod were to restart.

One of the Red Hat OpenShift Container Platform optional components named Red Hat OpenShift Container Platform aggregated logging collects and aggregates logs from the pods running in the Red Hat OpenShift Container Platform cluster as well as /var/log/messages on nodes enabling Red Hat OpenShift Container Platform users to view the logs of projects which they have view access using a web interface.

Red Hat OpenShift Container Platform aggregated logging component it is a modified version of the ELK stack composed by a few pods running on the Red Hat OpenShift Container Platform environment:

Once deployed in the cluster, Fluentd (deployed as a DaemonSet on any node with the right labels) gathers logs from all nodes and containers, enriches the log document with useful metadata (e.g. namespace, container_name, node) and forwards them into Elasticsearch, where Kibana provides a web interface to users to be able to view any logs. Cluster administrators can view all logs, but application developers can only view logs for projects they have permission to view. To avoid users seeing logs from pods in other projects, the Search Guard plugin for Elasticsearch is used.

A separate Elasticsearch cluster, a separate Kibana, and a separate Curator components can be deployed to form the OPS cluster where Fluentd send logs from the default, openshift, and openshift-infra projects as well as /var/log/messages on nodes into this different cluster. If the OPS cluster is not deployed those logs are hosted in the regular aggregated logging cluster.

Red Hat OpenShift Container Platform aggregated logging components can be customized for longer data persistence, pods limits, replicas of individual components, custom certificates, etc. The customization is provided by the Ansible variables as part of the deployment process.

The OPS cluster can be customized as well using the same variables using the suffix ops as in openshift_logging_es_ops_pvc_size.

Basic concepts for aggregated logging

By default every Elasticsearch pod of the Red Hat OpenShift Container Platform logging components have both master and data node. If only 2 Elasticsearch pods are deployed and one of the pods fails, all logging stops until the second master returns, so there is no availability advantage to deploy 2 Elasticsearch pods.

Below is an example of some of the best practices when deploying Red Hat OpenShift Container Platform aggregated logging. Elasticsearch, and Kibana are deployed on nodes with the role of "infras". Specifying the node role ensures that the Elasticsearch and Kibana components are not competing with applications for resources. A highly-available environment for Elasticsearch is deployed to avoid data loss, therefore, at least 3 Elasticsearch replicas are deployed and openshift_logging_es_number_of_replicas parameter is configured to be 1 at least. The settings below would be defined in a variable file or static inventory. The curator is now a scheduled job and no longer a deployment configuration.

openshift_logging_install_logging=true
openshift_logging_es_pvc_dynamic=true
openshift_logging_es_pvc_size=30Gi
openshift_logging_es_cluster_size=1
openshift_logging_es_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_logging_kibana_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_logging_fluentd_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_logging_es_number_of_replicas=1

Red Hat OpenShift Container Platform has the ability to gather metrics from kubelet and store the values in Heapster. Red Hat OpenShift Container Platform Metrics provide the ability to view CPU, memory, and network-based metrics and display the values in the user interface. These metrics can allow for the horizontal autoscaling of pods based on parameters provided by an Red Hat OpenShift Container Platform user. It is important to understand capacity planning when deploying metrics into an Red Hat OpenShift Container Platform environment.

Red Hat OpenShift Container Platform metrics is composed by a few pods running on the Red Hat OpenShift Container Platform environment:

Red Hat OpenShift Container Platform metrics components can be customized for longer data persistence, pods limits, replicas of individual components, custom certificates, etc. The customization is provided by the Ansible variables as part of the deployment process.

As best practice, when metrics are deployed, persistent storage should be used to allow for metric data to be preserved. Node selectors should be used to specify where the Metrics components should run. In the reference architecture environment, the components are deployed on nodes with the role of "infra".

openshift_hosted_metrics_deploy=true
openshift_hosted_metrics_storage_kind=dynamic
openshift_hosted_metrics_storage_volume_size=10Gi
openshift_metrics_hawkular_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_metrics_cassandra_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_metrics_heapster_nodeselector={"node-role.kubernetes.io/infra": "true"}

An existing port group and virtual LAN (VLAN) are required for deployment. The environment can utilize a vSphere Distributed Switch (vDS) or vSwitch. The specifics of that are unimportant. However, to utilize network IO control and some of the quality of service (QoS) technologies that VMware employs, a vDS is required.

The vSphere hosts should have shared storage for the VMware virtual machine disk files (VMDKs). A best practice recommendation is to enable storage I/O control (SIOC) to address any performance issues caused by latency. This article discusses in depth how to do this.

OpenShift Container Platform can be configured to access VMware vSphere VMDK Volumes, including using VMware vSphere VMDK Volumes as persistent storage for application data.

The vSphere Cloud Provider allows using vSphere managed storage within OpenShift Container Platform and supports:

$ curl -LO https://github.com/vmware/govmomi/releases/download/v0.15.0/govc_linux_amd64.gz
$ gunzip govc_linux_amd64.gz
$ chmod +x govc_linux_amd64
$ cp govc_linux_amd64 /usr/bin/govc

$ export GOVC_URL='vCenter IP OR FQDN'
$ export GOVC_USERNAME='vCenter User'
$ export GOVC_PASSWORD='vCenter Password'
$ export GOVC_INSECURE=1

$ govc ls /<datacenter>/vm/<vm-folder-name>

for VM in $(govc ls /<datacenter>/vm/<vm-folder-name>);do govc vm.change -e="disk.enableUUID=1" -vm="$VM";done

[Global]
        user = "username" 1
        password = "password" 2
        server = "10.10.0.2" 3
        port = "443" 4
        insecure-flag = "1" 5
        datacenter = "datacenter-name" 6
        datastore = "datastore-name" 7
        working-dir = "vm-folder-path" 8

[Disk]
        scsicontrollertype = pvscsi

[Network]
        network = "VM Network" 9

During the installation of Red Hat OpenShift Container Platform, the VMware VM instances created should include various VMDK volumes to ensure various OpenShift directories do not fill up the disk or cause disk contention in the /var partition.

Container images are stored locally on the nodes running Red Hat OpenShift Container Platform pods. The container-storage-setup script uses the /etc/sysconfig/docker-storage-setup file to specify the storage configuration.

The /etc/sysconfig/docker-storage-setup file must be created before starting the docker service, otherwise the storage is configured using a loopback device. The container storage setup is performed on all hosts running containers, therefore masters, infrastructure, and application nodes.

# cat /etc/sysconfig/docker-storage-setup
DEVS="/dev/sdb"
VG="docker-vol"
DATA_SIZE="95%VG"
STORAGE_DRIVER=overlay2
CONTAINER_ROOT_LV_NAME="dockerlv"
CONTAINER_ROOT_LV_MOUNT_PATH="/var/lib/docker"

A VMDK volume should be created on the master instances for the storage of /var/lib/etcd. Storing etcd allows the similar benefit of protecting /var but more importantly provides the ability to perform snapshots of the volume when performing etcd maintenance.

A VMDK volume should be created for the directory of /var/lib/origin/openshift.local.volumes that is used with the perFSGroup setting at installation and with the mount option of gquota. These settings and volumes set a quota to ensure that containers cannot grow to an unreasonable size.

# mkfs -t xfs -n -n ftype=1 /dev/sdc
# vi /etc/fstab
/dev/mapper/rhel-root   /                       xfs     defaults        0 0
UUID=8665acc0-22ee-4e45-970c-ae20c70656ef /boot                   xfs     defaults        0 0
/dev/sdc /var/lib/origin/openshift.local.volumes xfs gquota 0 0

Red Hat Enterprise Linux 7 is the only OS supported by the Red Hat OpenShift Container Platform installer therefore provider infrastructure deployment and installer must be run from one of the following locations:

This reference architecture focuses on deploying and installing Red Hat OpenShift Container Platform from local workstation/server/virtual machine. Jenkins CI/CD and Bastion are out of scope.

This reference environment should consist of the following instances:

The installation process for Red Hat OpenShift Container Platform depends on a reliable name service that contains an address record for each of the target instances.

An example DNS configuration is listed below:

$ORIGIN apps.example.com.
*           A       10.x.y.200
$ORIGIN example.com.
haproxy-0       A   10.x.y.200
infra-0         A   10.x.y.100
infra-1         A   10.x.y.101
infra-2         A   10.x.y.102
master-0        A   10.x.y.103
master-1        A   10.x.y.104
master-2        A   10.x.y.105
app-0           A   10.x.y.106
app-1           A   10.x.y.107
app-2           A   10.x.y.108

$ govc ls /<datacenter>/vm/<folder>/

$ ssh master-1

If an organization currently does not have a load balancer in place then HAProxy can be deployed. A load balancer such as HAProxy provides a single view of the Red Hat OpenShift Container Platform master services for the applications. The master services and the applications use different TCP ports so a single TCP load balancer can handle all of the inbound connections.

The load balanced DNS name that developers use must be in a DNS A record pointing to the HAProxy server before installation. For applications, a wildcard DNS entry must point to the HAProxy host.

The configuration of the HAProxy instance is completed within the subsequent steps as the deployment host configures the Red Hat subscriptions for all the instances and the Red Hat OpenShift Container Platform installer auto configures the HAProxy instance based upon the information found within the Red Hat OpenShift Container Platform inventory file.

Ensure connectivity to all instances via the deployment instance via:

$ ansible all -m ping

Once connectivity to all instances has been established, register the instances via Red Hat Subscription Manager. This is accomplished using credentials or an activation key.

Via credentials the ansible command is as follows:

$ ansible all -m command -a "subscription-manager register --username <user> --password '<password>'"

Via activation key, the ansible command is as follows:

$ ansible all -m command -a "subscription-manager register --org=<org_id> --activationkey=<keyname>"

where the following options:

Once all the instances have been successfully registered, enable all the required RHOCP repositories on all the instances via:

$ ansible all -m command -a "subscription-manager repos \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-server-ose-3.11-rpms" \
    --enable="rhel-7-server-ansible-2.7-rpms""

Red Hat OpenShift Container Platform provides the ability to use many different authentication platforms. For this reference architecture, LDAP is the preferred authentication mechanism. A listing of other authentication options are available at Configuring Authentication and User Agent.

When configuring LDAP as the authentication provider the following parameters can be added to the ansible inventory. An example is shown below.

openshift_master_identity_providers=[{'name': 'idm', 'challenge': 'true', 'login': 'true', 'kind': 'LDAPPasswordIdentityProvider', 'attributes': {'id': ['dn'], 'email': ['mail'], 'name': ['cn'], 'preferredUsername': ['uid']}, 'bindDN': 'uid=admin,cn=users,cn=accounts,dc=openshift,dc=com', 'bindPassword': 'ldapadmin', 'ca': '/etc/origin/master/ca.crt', 'insecure': 'false', 'url': 'ldap://ldap.example.com/cn=users,cn=accounts,dc=openshift,dc=com?uid?sub?(memberOf=cn=ose-user,cn=groups,cn=accounts,dc=openshift,dc=com)'}]

It can be useful to check for potential issues or misconfigurations in the nodes before continuing the installation process. Connect to every node using the deployment host and verify the disks are properly created and mounted.

$ ssh deployment.example.com
$ ssh <instance>
$ lsblk
$ sudo journalctl
$ free -m
$ sudo yum repolist

where node is for example master-0.example.com

For reference, below is example output of lsblk for the master nodes.

$ lsblk
NAME                     MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                        8:0    0   60G  0 disk
├─sda1                     8:1    0  500M  0 part /boot
├─sda2                     8:2    0 39.5G  0 part
│ ├─rhel-root            253:0    0   55G  0 lvm  /
│ └─rhel-swap            253:1    0  3.9G  0 lvm
└─sda3                     8:3    0   20G  0 part
  └─rhel-root            253:0    0   55G  0 lvm  /
sdb                        8:16   0   40G  0 disk
└─sdb1                     8:17   0   40G  0 part
  └─docker--vol-dockerlv 253:2    0   40G  0 lvm  /var/lib/docker
sdc                        8:32   0   40G  0 disk /var/lib/origin/openshift.local.volumes
sdd                        8:48   0   40G  0 disk /var/lib/etcd

For reference, below is an example of output of lsblk for the infra and app nodes.

$ lsblk
sda                        8:0    0   60G  0 disk
├─sda1                     8:1    0  500M  0 part /boot
├─sda2                     8:2    0 39.5G  0 part
│ ├─rhel-root            253:0    0   55G  0 lvm  /
│ └─rhel-swap            253:1    0  3.9G  0 lvm
└─sda3                     8:3    0   20G  0 part
  └─rhel-root            253:0    0   55G  0 lvm  /
sdb                        8:16   0   40G  0 disk
└─sdb1                     8:17   0   40G  0 part
  └─docker--vol-dockerlv 253:2    0   40G  0 lvm  /var/lib/docker
sdc                        8:32   0   40G  0 disk /var/lib/origin/openshift.local.volumes
sdd                        8:48   0  300G  0 disk

Prior to Chapter 4, Operational Management, create DRS anti-affinity rules to ensure maximum availability for the cluster.

The following VMware documentation goes over creating and configuring anti-affinity rules in depth.

Lastly, all of the VMs created should have their latency sensitivity set to High.

This enables some additional tuning recommended by VMware for latency sensitive workloads as described here.

Figure 2.1. VMware High Latency

The next section discusses VMware NSX integration with OpenShift Container Platform.

To skip this section, Section 2.18, “Red Hat OpenShift Container Platform Prequisites Playbook” discusses the required prerequisites for installation of OCP.

NSX-T (NSX Transformer) provides network virtualization for hypervisors. NSX Container Plug-in (NCP) provides integration between NSX-T Data Center and container orchestrators such as Kubernetes, as well as integration between NSX-T Data Center and container based PaaS (platform as a service) software products such as OpenShift. This guide describes setting up NCP with OpenShift.

The setup and configuration of VMware NSX is outside of the scope of this reference architecture. For more information on how to prepare NSX for the use of the VMware NSX Container Plug-in the following article describes in depth how to prepare the environment.

{'ncp/node_name': '<node_name>'}
{'ncp/cluster': '<cluster_name>'}

$ docker load -i nsx-ncp-rhel-xxx.yyyyyyyy.tar
$ docker image tag registry.local/xxx.yyyyyyyy/nsx-ncp-rhel nsx-ncp

openshift_use_nsx=True
openshift_use_openshift_sdn=False
os_sdn_network_plugin_name='cni'
nsx_openshift_cluster_name='ocp-cluster1'

# (Required) This is required because multiple Openshift/Kubernetes clusters can connect to the same
NSX Manager.

nsx_api_managers='10.10.10.10'

# (Required) IP address of NSX Manager. For an NSX Manager cluster, specify comma-separated IP
addresses.

nsx_tier0_router='MyT0Router'

# (Required) Name or UUID of the tier-0 router that the project's tier-1 routers will connect to.

nsx_overlay_transport_zone='my_overlay_tz'

# (Required) Name or UUID of the overlay transport zone that will be used to create logical switches.

nsx_container_ip_block='ip_block_for_my_ocp_cluster'

# (Required) Name or UUID of an IP block configured on NSX-T. There will be a subnet per project out
of this IP block. These networks will be behind SNAT and not routable.

nsx_ovs_uplink_port='ens224'

# (Required) If in HOSTVM mode. NSX-T needs second a vNIC for POD networking on the OCP nodes,
different from the management vNIC. It is highly recomended that both vNICs be connected to NSX-T
logical switches. The second (non-management) vNIC must be supplied here. For bare metal, this
parameter is not needed.

nsx_cni_url='http://myserver/nsx-cni.rpm'

# (Required) Temporary requirement until NCP can bootstrap the nodes. Nsx-cni needs to be on
an http server.

nsx_ovs_url='http://myserver/openvswitch.rpm'
nsx_kmod_ovs_url='http://myserver/kmod-openvswitch.rpm'
# (Required) Temporary parameters until NCP can bootstrap the nodes. Can be ignored in bare metal
setup.

nsx_node_type='HOSTVM'

# (Optional) Defaults to HOSTVM. Set to BAREMETAL if OpenShift is not running in VMs.

nsx_k8s_api_ip=192.168.10.10

# (Optional) If set, NCP will talk to this IP address, otherwise to Kubernetes service IP.

nsx_k8s_api_port=192.168.10.10

# (Optional) Default to 443 for Kubernetes service. Set to 8443 if you use it in combination with
nsx_k8s_api_ip to specify master node IP.

nsx_insecure_ssl=true

# (Optional) Default is true as NSX Manager comes with untrusted certificate. If you have changed the
certificate with a trusted one you can set it to false.

nsx_api_user='admin'
nsx_api_password='super_secret_password'
nsx_subnet_prefix=24

# (Optional) Defaults to 24. This is the subnet size that will be dedicated per Openshift project. If the
number of PODs exceeds the subnet size a new logical switch with the same subnet size will be
added to the project.

nsx_use_loadbalancer=true

# (Optional) Defaults to true. Set to false if you do not want to use NSX-T load balancers for
OpenShift routes and services of type LoadBalancer.

nsx_lb_service_size='SMALL'

# (Optional) Defaults to SMALL. Depending on the NSX Edge size MEDIUM or LARGE is also possible.

nsx_no_snat_ip_block='router_ip_block_for_my_ocp_cluster'

# (Optional) If the ncp/no_snat=true annotation is applied on a project or namespace the subnet will
be taken from this IP block and there will be no SNAT for it. It is expected to be routable.

nsx_external_ip_pool='external_pool_for_snat'

# (Requred) IP pool for SNAT and load balancer if nsx_external_ip_pool_lb is not defined.

nsx_external_ip_pool_lb='my_ip_pool_for_lb'

# (Optional) Set this if you want a distinct IP pool for Router and SvcTypeLB.

nsx_top_fw_section='top_section'

# (Optional) Kubernetes network policy rules will be translated to NSX-T firewall rules and placed below
this section.

nsx_bottom_fw_section='bottom_section'

# (Optional) Kubernetes network policy rules will be translated to NSX-T firewall rules and placed above
this section.

nsx_api_cert='/path/to/cert/nsx.crt'
nsx_api_private_key='/path/to/key/nsx.key'

# (Optional) If set, nsx_api_user and nsx_api_password will be ignored. The certificate must be
uploaded to NSX-T and a Principal Identity user authenticating with this certificate must be manually
created.

nsx_lb_default_cert='/path/to/cert/nsx.crt'
nsx_lb_default_key='/path/to/key/nsx.key'

# (Optional) NSX-T load balancer requires a default certificate in order to be able to crate SNIs for TLS
based Routes. This certificate will be presented only if there is no Route configured. If not provided, a
self-signed certificate will be generated.
[subs=+quotes]

[OSEv3:children]
masters
nodes
etcd

[OSEv3:vars]
ansible_ssh_user=root
openshift_deployment_type=openshift-enterprise
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true',
'kind': 'HTPasswdPasswordIdentityProvider'}]
openshift_master_htpasswd_users={'yasen' : 'password'}
openshift_master_default_subdomain=example.com
openshift_use_nsx=true
os_sdn_network_plugin_name=cni
openshift_use_openshift_sdn=false
openshift_node_sdn_mtu=1500
# NSX specific configuration
nsx_openshift_cluster_name='ocp-cluster1'
nsx_api_managers='192.168.110.201'
nsx_api_user='admin'
nsx_api_password='VMware1!'
nsx_tier0_router='DefaultT0Router'
nsx_overlay_transport_zone='overlay-tz'
nsx_container_ip_block='ocp-pod-networking'
nsx_no_snat_ip_block='ocp-nonat-pod-networking'
nsx_external_ip_pool='ocp-external'
nsx_top_fw_section='openshift-top'
nsx_bottom_fw_section='openshift-bottom'
nsx_ovs_uplink_port='ens224'
nsx_cni_url='http://1.1.1.1/nsx-cni-2.3.2.x86_64.rpm'
nsx_ovs_url='http://1.1.1.1/openvswitch-2.9.1.rhel75-1.x86_64.rpm'
nsx_kmod_ovs_url='http://1.1.1.1/kmod-openvswitch-2.9.1.rhel75-1.el7.x86_64.rpm'

[masters]
master-0.example.com

[etcd]
master-0.example.com

[nodes]
master-0.example.com ansible_ssh_host=10.x.y.103 openshift_node_group_name='node-config-master'
infra-0.example.com ansible_ssh_host=10.x.y.100 openshift_node_group_name='node-config-infra'
infra-1.example.com ansible_ssh_host=10.x.y.101 openshift_node_group_name='node-config-infra'
app-0.example.com ansible_ssh_host=10.x.y.106 openshift_node_group_name='node-config-compute'
app-1.example.com ansible_ssh_host=10.x.y.107 openshift_node_group_name='node-config-compute'

The Red Hat OpenShift Container Platform Ansible installation provides a playbook to ensure all prerequisites are met prior to the installation of Red Hat OpenShift Container Platform. This includes steps such as registering all the nodes with Red Hat Subscription Manager and setting up the docker on the docker volumes.

Via the ansible-playbook command on the deployment instance, ensure all the prerequisites are met using prerequisites.yml playbook:

$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml

In the event that OpenShift fails to install or the prerequisites playbook fails, follow the steps in Appendix Appendix F, Troubleshooting Ansible by Red Hat to troubleshoot Ansible.

The OpenShift image registry requires a volume to ensure that images are saved in the event that the registry needs to migrate to another node.

The initial installation of OpenShift will configure vSphere-volume and make it the default storage class.

Add the following lines to the /etc/ansible/hosts file in the [OSEv3:vars] section to allow for the default vSphere storage class to serve as the backend storage.

openshift_hosted_registry_storage_kind=vsphere
openshift_hosted_registry_storage_access_modes=['ReadWriteOnce']
openshift_hosted_registry_storage_annotations=['volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume']
openshift_hosted_registry_replicas=1

If registry storage was not configured during the installation. After the deployment is finished, the following steps reconfigure the registry to use the vSphere-volume storage class:

cat << EOF > pvc-registry.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: vsphere-registry-storage
  annotations:
    volume.beta.kubernetes.io/storage-class: vsphere-standard
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi
EOF

# submit the PVC
oc create -f pvc-registry.yaml

# update the volume config to use our new PVC
oc volume dc docker-registry --add --name=registry-storage -t pvc --claim-name=vsphere-registry-storage --overwrite

# rollout the new registry
oc rollout latest docker-registry

# verify the new volume
oc volume dc docker-registry

The oc adm diagnostics command runs a series of checks for error conditions in the host or cluster. Specifically, it:

Red Hat OpenShift Container Platform may be deployed in numerous scenarios including:

Each method implies a different configuration and environment. The diagnostics were included within openshift binary to minimize environment assumptions and provide the ability to run the diagnostics tool within an Red Hat OpenShift Container Platform server or client.

To use the diagnostics tool, preferably on a master host and as cluster administrator, run a sudo user:

$ sudo oc adm diagnostics

The above command runs all available diagnostic skipping any that do not apply to the environment.

The diagnostics tool has the ability to run all or specific diagnostics via name or as an enabler to address issues within the Red Hat OpenShift Container Platform environment. For example:

$ sudo oc adm diagnostics <name1>

The options provided by the diagnostics tool require working configuration files. For example, the NodeConfigCheck does not run unless a node configuration is readily available.

Diagnostics verifies that the configuration files reside in their standard locations unless specified with flags (respectively, --config, --master-config, and --node-config)

The standard locations are listed below:

If a configuration file is not found or specified, related diagnostics are skipped.

Available diagnostics include:

An Ansible-deployed cluster provides additional diagnostic benefits for nodes within Red Hat OpenShift Container Platform cluster due to:

Standard location of the configuration files placed by an Ansible-deployed cluster ensures that running sudo oc adm diagnostics works without any flags. In the event, the standard location of the configuration files is not used, options flags as those listed in the example below may be used.

$ sudo oc adm diagnostics --master-config=<file_path> --node-config=<file_path>

For proper usage of the log diagnostic, systemd units and log entries within journald are required. If log entries are not using the above method, log diagnostics won’t work as expected and are intentionally skipped.

The diagnostics runs using as much access as the existing user running the diagnostic has available. The diagnostic may run as an ordinary user, a cluster-admin user or cluster-admin user.

A client with ordinary access should be able to diagnose its connection to the master and run a diagnostic pod. If multiple users or masters are configured, connections are tested for all, but the diagnostic pod only runs against the current user, server, or project.

A client with cluster-admin access available (for any user, but only the current master) should be able to diagnose the status of the infrastructure such as nodes, registry, and router. In each case, running sudo oc adm diagnostics searches for the standard client configuration file location and uses it if available.

Additional diagnostic health checks are available through the Ansible-based tooling used to install and manage Red Hat OpenShift Container Platform clusters. The reports provide common deployment problems for the current Red Hat OpenShift Container Platform installation.

These checks can be run either using the ansible-playbook command (the same method used during Advanced Installation) or as a containerized version of openshift-ansible. For the ansible-playbook method, the checks are provided by the openshift-ansible RPM package.

For the containerized method, the openshift3/ose-ansible container image is distributed via the Red Hat Container Registry.

Example usage for each method are provided in subsequent sections.

The following health checks are a set of diagnostic tasks that are meant to be run against the Ansible inventory file for a deployed Red Hat OpenShift Container Platform cluster using the provided health.yml playbook.

# ansible-playbook -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/openshift-checks/health.yml

# ansible-playbook -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/openshift-checks/health.yml
    -e openshift_check_logging_index_timeout_seconds=45
    -e etcd_max_image_data_size_bytes=40000000000

openshift_disable_check=etcd_traffic,etcd_volume

The openshift-ansible playbooks may run in a Docker container avoiding the requirement for installing and configuring Ansible, on any host that can run the ose-ansible image via the Docker CLI.

This is accomplished by specifying the cluster’s inventory file and the health.yml playbook when running the following docker run command as a non-root user that has privileges to run containers:

# docker run -u `id -u` \ 1
    -v $HOME/.ssh/id_rsa:/opt/app-root/src/.ssh/id_rsa:Z,ro \ 2
    -v /etc/ansible/hosts:/tmp/inventory:ro \ 3
    -e INVENTORY_FILE=/tmp/inventory \
    -e PLAYBOOK_FILE=playbooks/openshift-checks/health.yml \ 4
    -e OPTS="-v -e openshift_check_logging_index_timeout_seconds=45 -e etcd_max_image_data_size_bytes=40000000000" \ 5
    openshift3/ose-ansible

In the above command, the SSH key is mounted with the :Z flag so that the container can read the SSH key from its restricted SELinux context. This ensures the original SSH key file is relabeled similarly to system_u:object_r:container_file_t:s0:c113,c247. For more details about :Z, see the docker-run(1) man page.

It is important to note these volume mount specifications because it could have unexpected consequences. For example, if one mounts (and therefore relabels) the $HOME/.ssh directory, sshd becomes unable to access the public keys to allow remote login. To avoid altering the original file labels, mounting a copy of the SSH key (or directory) is recommended.

It is plausible to want to mount an entire .ssh directory for various reasons. For example, this enables the ability to use an SSH configuration to match keys with hosts or modify other connection parameters. It could also allow a user to provide a known_hosts file and have SSH validate host keys, which is disabled by the default configuration and can be re-enabled with an environment variable by adding -e ANSIBLE_HOST_KEY_CHECKING=True to the docker command line.