Employing Red Hat Virtualization in a self-hosted configuration, this reference architecture provides a complete high-availability environment in three physical hosts. In production environments, best practices recommend separating the Red Hat Virtualization cluster running Red Hat OpenShift Container Platform from the cluster running the management engine. Proper set up of the Red Hat Virtualization environment is beyond the scope of this document. A guide to best practices in setting up Red Hat Virtualization can be found in the Appendix C, Links section of the Appendix at the end of this document.

DNS service is an important component in the Red Hat OpenShift Container Platform environment. Regardless of the provider of DNS, an organization is required to have certain records in place to serve the various Red Hat OpenShift Container Platform components.

Since the load balancer values for the Red Hat OpenShift Container Platform master service and infrastructure nodes running router pods are known beforehand, entries should be configured into the DNS prior to starting the deployment procedure.

Pods inside of an OpenShift cluster are only reachable via their IP addresses on the cluster network. An edge load balancer can be used to accept traffic from outside networks and proxy the traffic to pods inside the OpenShift cluster.

An OpenShift administrator can deploy routers to nodes in an OpenShift cluster. Routers enable routes created by developers to be used by external clients.

OpenShift routers provide external hostname mapping and load balancing to services over protocols that pass distinguishing information directly to the router; the hostname must be present in the protocol in order for the router to determine where to send it. Routers support the following protocols:

The router utilizes the wildcard zone specified during the installation and configuration of OpenShift. This wildcard zone is used by the router to create routes for a service running within the OpenShift environment to a publicly accessible URL. The wildcard zone itself is a wildcard entry in Route53 which is linked using a CNAME to an ELB which performs a health check and forwards traffic to router pods on port 80 and 443.

This guide uses an external load balancer running haproxy to offer a single entry point for the many Red Hat OpenShift Container Platform components. Organizations can provide their own currently deployed load balancers in the event that the service already exists.

The Red Hat OpenShift Container Platform console, provided by the Red Hat OpenShift Container Platform master nodes, can be spread across multiple instances to provide both load balancing and high availability properties.

Application traffic passes through the Red Hat OpenShift Container Platform Router on its way to the container processes. The Red Hat OpenShift Container Platform Router is a reverse proxy service container that multiplexes the traffic to multiple containers making up a scaled application running inside Red Hat OpenShift Container Platform. The load balancer used by infra nodes acts as the public view for the Red Hat OpenShift Container Platform applications.

The destination for the master and application traffic must be set in the load balancer configuration after each instance is created, the floating IP address is assigned and before the installation. A single haproxy load balancer can forward both sets of traffic to different destinations.

Best practices recommend minimizing attack vectors into a system by exposing only those services required by consumers of the system. In the event of failure or a need for manual configuration, systems administrators require further access to internal components in the form of secure administrative back-doors.

In the case of Red Hat OpenShift Container Platform running in a cloud provider context, the entry points to the Red Hat OpenShift Container Platform infrastructure such as the API, Web Console and routers are the only services exposed to the outside. The systems administrators' access from the public network space to the private network is possible with the use of a bastion instance.

A bastion instance is a non-OpenShift instance accessible from outside of the Red Hat OpenShift Container Platform environment, configured to allow remote access via secure shell (ssh). To remotely access an instance, the systems administrator first accesses the bastion instance, then "jumps" via another ssh connection to the intended OpenShift instance. The bastion instance may be referred to as a "jump host".

Depending on the environment, the bastion instance may be an ideal candidate for running administrative tasks such as the Red Hat OpenShift Container Platform installation playbooks. This reference environment uses the bastion instance for the installation of the Red Hat OpenShift Container Platform.

Red Hat OpenShift Container Platform comprises of multiple instances running on Red Hat Virtualization that allow for scheduled and configured OpenShift services and supplementary containers. These containers can have persistent storage, if required, by the application and integrate with optional OpenShift services such as logging and metrics.

... [OUTPUT ABBREVIATED] ...
[etcd]
master1.example.com
master2.example.com
master3.example.com

[masters]
master1.example.com
master2.example.com
master3.example.com

[nodes]
master1.example.com openshift_node_labels="{'region': 'master', 'masterlabel2': 'value2'}"
master2.example.com openshift_node_labels="{'region': 'master', 'masterlabel2': 'value2'}"
master3.example.com openshift_node_labels="{'region': 'master', 'masterlabel2': 'value2'}"

... [OUTPUT ABBREVIATED] ...
[nodes]
infra1.example.com openshift_node_labels="{'region': 'infra', 'infralabel1': 'value1'}"
infra2.example.com openshift_node_labels="{'region': 'infra', 'infralabel1': 'value1'}"
infra3.example.com openshift_node_labels="{'region': 'infra', 'infralabel1': 'value1'}"

... [OUTPUT ABBREVIATED] ...

[nodes]
node1.example.com openshift_node_labels="{'region': 'primary', 'nodelabel2': 'value2'}"
node2.example.com openshift_node_labels="{'region': 'primary', 'nodelabel2': 'value2'}"
node3.example.com openshift_node_labels="{'region': 'primary', 'nodelabel2': 'value2'}"

"labels": {
  "key1" : "value1",
  "key2" : "value2"
}

{"release" : "stable", "release" : "canary"}
{"environment" : "dev", "environment" : "qa", "environment" : "production"}
{"tier" : "frontend", "tier" : "backend", "tier" : "cache"}
{"partition" : "customerA", "partition" : "customerB"}
{"track" : "daily", "track" : "weekly"}

Red Hat OpenShift Container Platform offers the ability to specify how pods communicate with each other. This could be through the use of Red Hat provided Software-defined networks (SDN) or a third-party SDN.

Deciding on the appropriate internal network for an Red Hat OpenShift Container Platform step is a crucial step. Unfortunately, there is no right answer regarding the appropriate pod network to chose, as this varies based upon the specific scenario requirements on how a Red Hat OpenShift Container Platform environment is to be used.

For the purposes of this reference environment, the Red Hat OpenShift Container Platform ovs-networkpolicy SDN plug-in is chosen due to its ability to provide pod isolation using Kubernetes NetworkPolicy. The following section, “OpenShift SDN Plugins”, discusses important details when deciding between the three popular options for the internal networks - ovs-multitenant, ovs-networkpolicy and ovs-subnet.

Container images are stored locally on the nodes running Red Hat OpenShift Container Platform pods. The container-storage-setup script uses the /etc/sysconfig/docker-storage-setup file to specify the storage configuration.

The /etc/sysconfig/docker-storage-setup file should be created before starting the docker service, otherwise the storage would be configured using a loopback device. The container storage setup is performed on all hosts running containers, therefore masters, infrastructure, and application nodes.

Containers by default offer ephemeral storage but some applications require the storage to persist between different container deployments or upon container migration. Persistent Volume Claims (PVC) are used to store the application data. These claims can either be added into the environment by hand or provisioned dynamically using a StorageClass object.

OpenShift can build containerimages from source code, deploy them, and manage their lifecycle. To enable this, OpenShift provides an internal, integrated registry that can be deployed in the OpenShift environment to manage images.

The registry stores images and metadata. For production environment, persistent storage should be used for the registry, otherwise any images that were built or pushed into the registry would disappear if the pod were to restart.

One of the Red Hat OpenShift Container Platform optional components named Red Hat OpenShift Container Platform aggregated logging collects and aggregates logs from the pods running in the Red Hat OpenShift Container Platform cluster as well as /var/log/messages on nodes enabling Red Hat OpenShift Container Platform users to view the logs of projects which they have view access using a web interface.

Red Hat OpenShift Container Platform aggregated logging component it is a modified version of the ELK stack composed by a few pods running on the Red Hat OpenShift Container Platform environment:

Once deployed in the cluster, Fluentd (deployed as a DaemonSet on any node with the right labels) gathers logs from all nodes and containers, enriches the log document with useful metadata (e.g. namespace, container_name, node) and forwards them into Elasticsearch, where Kibana provides a web interface to users to be able to view any logs. Cluster administrators can view all logs, but application developers can only view logs for projects they have permission to view. To avoid users to see logs from pods in other projects, the Search Guard plugin for Elasticsearch is used.

A separate Elasticsearch cluster, a separate Kibana, and a separate Curator components can be deployed to form the OPS cluster where Fluentd send logs from the default, openshift, and openshift-infra projects as well as /var/log/messages on nodes into this different cluster. If the OPS cluster is not deployed those logs are hosted in the regular aggregated logging cluster.

Red Hat OpenShift Container Platform aggregated logging components can be customized for longer data persistence, pods limits, replicas of individual components, custom certificates, etc. The customization is provided by the Ansible variables as part of the deployment process.

The OPS cluster can be customized as well using the same variables using the suffix ops as in openshift_logging_es_ops_pvc_size.

Basic concepts for aggregated logging

By default every Elasticsearch pod of the Red Hat OpenShift Container Platform aggregated logging components has the role of Elasticsearch master and Elasticsearch data node. If only 2 Elasticsearch pods are deployed and one of the pods fails, all logging stops until the second master returns, so there is no availability advantage to deploy 2 Elasticsearch pods.

Below is an example of some of the best practices when deploying Red Hat OpenShift Container Platform aggregated logging. Elasticsearch, Kibana, and Curator are deployed on nodes with the label of "region=infra". Specifying the node label ensures that the Elasticsearch and Kibana components are not competing with applications for resources. A highly-available environment for Elasticsearch is deployed to avoid data loss, therefore, at least 3 Elasticsearch replicas are deployed and openshift_logging_es_number_of_replicas parameter is configured to be 1 at least. The settings below would be defined in a variable file or static inventory.

openshift_logging_install_logging=true
openshift_logging_es_pvc_dynamic=true
openshift_logging_es_pvc_size=100Gi
openshift_logging_es_cluster_size=3
openshift_logging_es_nodeselector={"region":"infra"}
openshift_logging_kibana_nodeselector={"region":"infra"}
openshift_logging_curator_nodeselector={"region":"infra"}
openshift_logging_es_number_of_replicas=1

Red Hat OpenShift Container Platform has the ability to gather metrics from kubelet and store the values in Heapster. Red Hat OpenShift Container Platform Metrics provide the ability to view CPU, memory, and network-based metrics and display the values in the user interface. These metrics can allow for the horizontal autoscaling of pods based on parameters provided by an Red Hat OpenShift Container Platform user. It is important to understand capacity planning when deploying metrics into an Red Hat OpenShift Container Platform environment.

Red Hat OpenShift Container Platform metrics is composed by a few pods running on the Red Hat OpenShift Container Platform environment:

Red Hat OpenShift Container Platform metrics components can be customized for longer data persistence, pods limits, replicas of individual components, custom certificates, etc. The customization is provided by the Ansible variables as part of the deployment process.

As best practices when metrics are deployed, persistent storage should be used to allow for metrics to be preserved. Node selectors should be used to specify where the Metrics components should run. In the reference architecture environment, the components are deployed on nodes with the label of "region=infra".

openshift_metrics_install_metrics=True
openshift_metrics_storage_volume_size=20Gi
openshift_metrics_cassandra_storage_type=dynamic
openshift_metrics_hawkular_nodeselector={"region":"infra"}
openshift_metrics_cassandra_nodeselector={"region":"infra"}
openshift_metrics_heapster_nodeselector={"region":"infra"}

Container-Native Storage (CNS) provides dynamically provisioned storage for containers on Red Hat OpenShift Container Platform across cloud providers, virtual and bare-metal deployments. CNS relies on block devices available on the OpenShift nodes and uses software-defined storage provided by Red Hat Gluster Storage. CNS runs Red Hat Gluster Storage containerized, allowing OpenShift storage pods to spread across the cluster and across different data centers if latency is low between them. CNS enables the requesting and mounting of Gluster storage across one or many containers with access modes of either ReadWriteMany(RWX), ReadOnlyMany(ROX) or ReadWriteOnce(RWO). CNS can also be used to host the OpenShift registry.

The bastion instance itself may have a relatively minimal installation of Red Hat Enterprise Linux 7. It should meet recommended requirements for the operating system itself with enough spare disk space to hold downloaded QCOW2 images (around 700 MiB).

The Ansible commands to set up the RHV instances and OpenShift may be run as a normal user. Several of the following prerequisite tasks, however, require root access, preferably through sudo.

$ ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa

Your identification has been saved in ~/.ssh/id_rsa.
Your public key has been saved in ~/.ssh/id_rsa.pub.
The key fingerprint is:
e7:97:c7:e2:0e:f9:0e:fc:c4:d7:cb:e5:31:11:92:14 USER@bastion.example.com
The key's randomart image is:
+--[ RSA 2048]----+
|             E.  |
|            . .  |
|             o . |
|              . .|
|        S .    . |
|         + o o ..|
|          * * +oo|
|           O +..=|
|           o*  o.|
+-----------------+

$ sudo subscription-manager register
$ sudo subscription-manager list --available

$ sudo subscription-manager attach --pool <RHV Pool ID>
$ sudo subscription-manager attach --pool <RHOCP Pool ID>

$ sudo subscription-manager repos --disable=*
$ sudo subscription-manager repos \
    --enable=rhel-7-server-rpms \
    --enable=rhel-7-server-extras-rpms \
    --enable=rhel-7-server-ose-3.9-rpms \
    --enable=rhel-7-server-rhv-4-mgmt-agent-rpms

$ sudo yum install -y \
    ansible \
    atomic-openshift-utils \
    git \
    python-ovirt-engine-sdk4 \
    ovirt-ansible-roles

Before running playbooks, modify ansible.cfg to reflect the deployed environment:

$ sudo vi /etc/ansible/ansible.cfg

The code block below implements the following changes:

[defaults]
forks = 20
host_key_checking = False
remote_user = root
gathering = smart

log_path=./ansible.log
retry_files_enabled=False

vault_password_file=~/.test_vault_pw

According to the Red Hat OpenShift Container Platform Advanced Installation Guide, cluster variables should be configured in the system inventory (/etc/ansible/hosts) within the [OSEv3:vars] section.

The oVirt Ansible roles may be configured in a separate variable file and included during ansible-playbook runs using the -e flag, however it is also possible to include these in the same inventory file under a group created just for localhost.

The following snippets of the inventory for this reference architecture define localhost as part of a group called workstation:

[workstation]
localhost ansible_connection=local

Variables assigned to localhost include:

Credentials for the RHV Engine

In the following snippet, the credentials for the RHV engine are assigned to special variables which are then encrypted by Ansible Vault because they contain sensitive information like administrative passwords. Creation of this Ansible Vault file is explained in the next section.

[workstation:vars]
# RHV Engine
engine_url="{{ vault_engine_url }}"
engine_user="{{ vault_engine_user }}"
engine_password="{{ vault_engine_password }}"
# CA file copied from engine:/etc/pki/ovirt-engine/ca.pem
# path is relative to playbook directory
engine_cafile=../ca.pem

Guest Image URL and Path

# QCOW2 KVM Guest Image
#qcow_url=https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2c
qcow_url=https://access.cdn.redhat.com//content/origin/files/XXXX/rhel-server-7.5-x86_64-kvm.qcow2?_auth_=XXXX
template_name=rhel75
image_path="{{ lookup('env', 'HOME') }}/Downloads/{{ template_name }}.qcow2"

RHV Cluster information and SSH key

# RHV VM Cluster Info
rhv_cluster=Default
rhv_data_storage=vmstore
root_ssh_key="{{ lookup('file', '~/.ssh/id_rsa.pub') }}"

Some variables pertain to all hosts, including localhost and the OpenShift instances. They belong in a special group called all:

[all:vars]
app_dns_prefix=apps
public_hosted_zone=example.com
load_balancer_hostname=lb.{{public_hosted_zone}}
openshift_master_cluster_hostname="{{ load_balancer_hostname }}"
openshift_master_cluster_public_hostname=openshift-master.{{ public_hosted_zone }}
openshift_master_default_subdomain="{{ app_dns_prefix }}.{{ public_hosted_zone }}"
openshift_public_hostname="{{openshift_master_cluster_public_hostname}}"

OpenShift specific variables belong in the previously mentioned [OSEv3:vars] section:

[OSEv3:vars]
# General variables
ansible_ssh_user=root
debug_level=2
deployment_type=openshift-enterprise
openshift_debug_level="{{ debug_level }}"
openshift_deployment_type="{{ deployment_type }}"
openshift_master_cluster_method=native
openshift_node_debug_level="{{ node_debug_level | default(debug_level, true) }}"
openshift_release=3.9

As Red Hat Virtualization does not yet provide dynamic storage allocation capabilities to Red Hat OpenShift Container Platform, the Service Catalog must be disabled at install time:

openshift_enable_service_catalog=False

For easier readability, the inventory is broken down further into logical sections:

# Docker
container_runtime_docker_storage_setup_device=/dev/vdb
container_runtime_docker_storage_type=overlay2
openshift_docker_use_system_container=False
openshift_node_local_quota_per_fsgroup=512Mi
openshift_use_system_containers=False

# Pod Networking
os_sdn_network_plugin_name=redhat/openshift-ovs-networkpolicy

Here, an external NFS server is used as the backing store for the OpenShift Registry.

# Registry
openshift_hosted_registry_replicas=1
openshift_hosted_registry_storage_kind=nfs
openshift_hosted_registry_storage_access_modes=['ReadWriteMany']
openshift_hosted_registry_selector='region=infra'
openshift_hosted_registry_storage_host=
openshift_hosted_registry_storage_nfs_directory=/var/lib/exports
openshift_hosted_registry_storage_volume_name=registryvol
openshift_hosted_registry_storage_volume_size=20Gi

# Red Hat Subscription Management
rhsub_pool=Red Hat OpenShift Container Platform*
rhsub_user="{{ vault_rhsub_user }}"
rhsub_password="{{ vault_rhsub_password }}"

The embedded json here was converted from yaml. The yaml version of the inventory is also in the OpenShift Ansible Contrib GitHub Repo.

# Load Balancer Config
openshift_loadbalancer_additional_frontends=[{"name":"apps-http","option":"tcplog","binds":["*:80"],"default_backend":"apps-http"},{"name":"apps-https","option":"tcplog","binds":["*:443"],"default_backend":"apps-http"}]
openshift_loadbalancer_additional_backends=[{"name":"apps-http","balance":"source","servers":[{"name":"infra0","address":"{{ groups['infras'].0 }}:80","opts":"check"},{"name":"infra1","address":"{{ groups['infras'].1 }}:80","opts":"check"},{"name":"infra2","address":"{{ groups['infras'].2 }}:80","opts":"check"}]},{"name":"apps-https","balance":"source","servers":[{"name":"infra0","address":"{{ groups['infras'].0 }}:443","opts":"check"},{"name":"infra1","address":"{{ groups['infras'].1 }}:443","opts":"check"},{"name":"infra2","address":"{{ groups['infras'].2 }}:443","opts":"check"}]}]

Finally, the host names for the OpenShift instances are grouped and provided below:

[OSEv3:children]
nodes
masters
etcd
lb

[masters]
master0.example.com
master1.example.com
master2.example.com

[etcd]
master0.example.com
master1.example.com
master2.example.com

[infras]
infra0.example.com
infra1.example.com
infra2.example.com

[lb]
lb.example.com

[nodes]
master0.example.com openshift_node_labels="{'region': 'master'}" openshift_hostname=master0.example.com
master1.example.com openshift_node_labels="{'region': 'master'}" openshift_hostname=master1.example.com
master2.example.com openshift_node_labels="{'region': 'master'}" openshift_hostname=master2.example.com
infra0.example.com openshift_node_labels="{'region': 'infra'}" openshift_hostname=infra0.example.com
infra1.example.com openshift_node_labels="{'region': 'infra'}" openshift_hostname=infra1.example.com
infra2.example.com openshift_node_labels="{'region': 'infra'}" openshift_hostname=infra2.example.com
app0.example.com openshift_node_labels="{'region': 'primary'}" openshift_hostname=app0.example.com
app1.example.com openshift_node_labels="{'region': 'primary'}" openshift_hostname=app1.example.com
app2.example.com openshift_node_labels="{'region': 'primary'}" openshift_hostname=app2.example.com

For the complete inventory, see https://github.com/openshift/openshift-ansible-contrib/blob/master/reference-architecture/rhv-ansible/example/inventory

To avoid leaving secret passwords for critical infrastructure in clear-text on the file system, create an encrypted Ansible Vault variable file for inclusion in the Ansible commands in the next chapter.

To create the encrypted file, for example in ~/vault.yaml, run the following command:

$ ansible-vault create ~/vault.yaml

To edit the newly created vault file, run the following command:

$ ansible-vault edit ~/vault.yaml

This reference architecture employs the following process:

engine_password: '{{ vault_engine_password }}'

If using a vault file to protect some values, it must be included alongside the variables file that references it, e.g. -e @~/vault.yaml

The oVirt Ansible roles responsible for setting up instances in Red Hat Virtualization require credentials in order to connect to the engine and perform operations.

engine_url: https://engine.example.com/ovirt-engine/api
engine_user: admin@internal
engine_password: '{{ vault_engine_password }}'

In order to establish a trusted SSL connection with the engine, the oVirt API requires a copy of the engine’s certificate authority in the form of a self-signed certificate PEM.

To download the certificate from the engine itself, run:

$ curl --output ca.pem 'http://engine.example.com/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA'

Provide the CA file to oVirt Ansible with the following variable:

engine_cafile: ../ca.pem

Red Hat OpenShift Container Platform provides the ability to use a number of different authentication platforms. For this reference architecture, LDAP is the preferred authentication mechanism. A listing of other authentication options is available at Configuring Authentication and User Agent.

When configuring LDAP as the authentication provider, the following parameters can be added to the Ansible inventory:

openshift_master_identity_providers=[{'name': 'idm', 'challenge': 'true', 'login': 'true', 'kind': 'LDAPPasswordIdentityProvider', 'attributes': {'id': ['dn'], 'email': ['mail'], 'name': ['cn'], 'preferredUsername': ['uid']}, 'bindDN': 'uid=admin,cn=users,cn=accounts,dc=ocp3,dc=openshift,dc=com', 'bindPassword': 'ldapadmin', 'ca': '/etc/origin/master/ca.crt', 'insecure': 'false', 'url': 'ldap://ldap.ocp3.example.com/cn=users,cn=accounts,dc=ocp3,dc=openshift,dc=com?uid?sub?(memberOf=cn=ose-user,cn=groups,cn=accounts,dc=ocp3,dc=openshift,dc=com)'}]

For more information about the required LDAP parameters, see LDAP Authentication

Creation of OpenShift instances in Red Hat Virtualization takes place in two oVirt Ansible roles: oVirt.image-template and oVirt.vm-infra. Documentation and examples of these roles can be found on their respective GitHub project pages.

The following is a playbook which calls the Image Template and VM Infrastructure roles from oVirt Ansible:

---
- name: oVirt 39 all-in-one playbook
  hosts: localhost
  connection: local
  gather_facts: false
  roles:
    - oVirt.image-template
    - oVirt.vm-infra
  vars:
    # Common
    compatibility_version: 4.2
    data_center_name: Default
    debug_vm_create: true
    wait_for_ip: true
    vm_infra_wait_for_ip_retries: 10
    vm_infra_wait_for_ip_delay: 40

    # oVirt Image Template vars
    template_cluster: "{{ rhv_cluster }}"
    template_memory: 8GiB
    template_cpu: 1
    template_disk_storage: "{{ rhv_data_storage }}"
    template_disk_size: 60GiB
    template_nics:
      - name: nic1
        profile_name: ovirtmgmt
        interface: virtio

    # Profiles
    master_vm:
      cluster: "{{ rhv_cluster }}"
      template: "{{ template_name }}"
      memory: 16GiB
      cores: 2
      high_availability: true
      disks:
        - size: 100GiB
          storage_domain: "{{ rhv_data_storage }}"
          name: docker_disk
          interface: virtio
        - size: 50GiB
          storage_domain: "{{ rhv_data_storage }}"
          name: localvol_disk
          interface: virtio
      state: running

    node_vm:
      cluster: "{{ rhv_cluster }}"
      template: "{{ template_name }}"
      memory: 8GiB
      cores: 2
      disks:
        - size: 100GiB
          storage_domain: "{{ rhv_data_storage }}"
          name: docker_disk
          interface: virtio
        - size: 50GiB
          storage_domain: "{{ rhv_data_storage }}"
          name: localvol_disk
          interface: virtio
      state: running

    # Cloud Init Script
    cloud_init_script: |
      runcmd:
        - mkdir -p '/var/lib/origin/openshift.local.volumes'
        - /usr/sbin/mkfs.xfs -L localvol /dev/vdc
        - sleep "$(($RANDOM % 60))"
        - sync
        - reboot
      mounts:
        - [ '/dev/vdc', '/var/lib/origin/openshift.local.volumes', 'xfs', 'defaults,gquota' ]
      rh_subscription:
        username: {{vault_rhsub_user}}
        password: {{vault_rhsub_password}}
        add-pool: [{{vault_rhsub_pool}}]
        server-hostname: {{vault_rhsub_server}}
        enable-repo: ['rhel-7-server-rpms', 'rhel-7-server-extras-rpms', 'rhel-7-fast-datapath-rpms', 'rhel-7-server-ose-3.9-rpms']
        disable-repo: []

    vms:
      # Master VMs
      - name: "master0.{{ public_hosted_zone }}"
        profile: "{{ master_vm }}"
        tag: openshift_master
        cloud_init:
          host_name: "master0.{{ public_hosted_zone }}"
          authorized_ssh_keys: "{{ root_ssh_key }}"
          custom_script: "{{ cloud_init_script }}"
      - name: "master1.{{ public_hosted_zone }}"
        tag: openshift_master
        profile: "{{ master_vm }}"
        cloud_init:
          host_name: "master1.{{ public_hosted_zone }}"
          authorized_ssh_keys: "{{ root_ssh_key }}"
          custom_script: "{{ cloud_init_script }}"
      - name: "master2.{{ public_hosted_zone }}"
        tag: openshift_master
        profile: "{{ master_vm }}"
        cloud_init:
          host_name: "master2.{{ public_hosted_zone }}"
          authorized_ssh_keys: "{{ root_ssh_key }}"
          custom_script: "{{ cloud_init_script }}"

      # Infra VMs
      - name: "infra0.{{ public_hosted_zone }}"
        tag: openshift_infra
        profile: "{{ node_vm }}"
        cloud_init:
          host_name: "infra0.{{ public_hosted_zone }}"
          authorized_ssh_keys: "{{ root_ssh_key }}"
          custom_script: "{{ cloud_init_script }}"
      - name: "infra1.{{ public_hosted_zone }}"
        tag: openshift_infra
        profile: "{{ node_vm }}"
        cloud_init:
          host_name: "infra1.{{ public_hosted_zone }}"
          authorized_ssh_keys: "{{ root_ssh_key }}"
          custom_script: "{{ cloud_init_script }}"
      - name: "infra2.{{ public_hosted_zone }}"
        tag: openshift_infra
        profile: "{{ node_vm }}"
        cloud_init:
          host_name: "infra2.{{ public_hosted_zone }}"
          authorized_ssh_keys: "{{ root_ssh_key }}"
          custom_script: "{{ cloud_init_script }}"

      # Node VMs
      - name: "app0.{{ public_hosted_zone }}"
        tag: openshift_node
        profile: "{{ node_vm }}"
        cloud_init:
          host_name: "app0.{{ public_hosted_zone }}"
          authorized_ssh_keys: "{{ root_ssh_key }}"
          custom_script: "{{ cloud_init_script }}"
      - name: "app1.{{ public_hosted_zone }}"
        tag: openshift_node
        profile: "{{ node_vm }}"
        cloud_init:
          host_name: "app1.{{ public_hosted_zone }}"
          authorized_ssh_keys: "{{ root_ssh_key }}"
          custom_script: "{{ cloud_init_script }}"
      - name: "app2.{{ public_hosted_zone }}"
        tag: openshift_node
        profile: "{{ node_vm }}"
        cloud_init:
          host_name: "app2.{{ public_hosted_zone }}"
          authorized_ssh_keys: "{{ root_ssh_key }}"
          custom_script: "{{ cloud_init_script }}"

      # Load balancer
      - name: "lb.{{ public_hosted_zone }}"
        tag: openshift_lb
        profile: "{{ node_vm }}"
        cloud_init:
          host_name: "lb.{{ public_hosted_zone }}"
          authorized_ssh_keys: "{{ root_ssh_key }}"
          custom_script: "{{ cloud_init_script }}"

    affinity_groups:
      - name: masters_ag
        cluster: "{{ rhv_cluster }}"
        vm_enforcing: false
        vm_rule: negative
        vms:
          - "master0.{{ public_hosted_zone }}"
          - "master1.{{ public_hosted_zone }}"
          - "master2.{{ public_hosted_zone }}"
        wait: true
      - name: infra_ag
        cluster: "{{ rhv_cluster }}"
        vm_enforcing: false
        vm_rule: negative
        vms:
          - "infra0.{{ public_hosted_zone }}"
          - "infra1.{{ public_hosted_zone }}"
          - "infra2.{{ public_hosted_zone }}"
        wait: true

      - name: app_ag
        cluster: "{{ rhv_cluster }}"
        vm_enforcing: false
        vm_rule: negative
        vms:
          - "app0.{{ public_hosted_zone }}"
          - "app1.{{ public_hosted_zone }}"
          - "app2.{{ public_hosted_zone }}"
  pre_tasks:
    - name: Log in to oVirt
      ovirt_auth:
        url: "{{ engine_url }}"
        username: "{{ engine_user }}"
        password: "{{ engine_password }}"
        ca_file: "{{ engine_cafile | default(omit) }}"
        insecure: "{{ engine_insecure | default(true) }}"
      tags:
        - always
  post_tasks:
    - name: Logout from oVirt
      ovirt_auth:
        state: absent
        ovirt_auth: "{{ ovirt_auth }}"
      tags:
        - always
...

Run this playbook from the bastion host as follows:

$ ansible-playbook -e@~vault.yaml ovirt-vm-infra.yaml

Upon successful completion of this playbook, ten new virtual machines will be running in Red Hat Virtualization, ready for deployment.

After deploying the instances, ensure they are properly updated in DNS.

Red Hat OpenShift Container Platform allows the option to deploy aggregate logging for containers running in the OpenShift environment. OpenShift uses the Elasticsearch, Fluentd, and Kibana (EFK) stack to collect logs from applications and present them to OpenShift users. Cluster administrators can view all logs but application developers can only view logs for projects they have permission to view. The EFK stack consists of the following components:

Below is an example of some of the best practices when deploying OpenShift logging.

Elasticsearch, Kibana, and Curator are deployed on nodes with the label of "region=infra". Specifying the node label ensures that the Elasticsearch and Kibana components are not competing with applications for resources. The Elasticsearch cluster size (openshift_logging_es_cluster_size) of three is the minimum number of nodes to ensure data durability and redundancy.

The logging project created during installation must be modified to bypass the default node selector. If this is not done then the Elasticsearch, Kibana, and Curator components cannot be started. There are two options in making the appropriate edit.

Option 1:

ssh into the first master instance (master1.example.com) or the bastion instance and modify the logging project.

Add the following line openshift.io/node-selector: "" under annotations but do not remove any lines.

$ oc edit project logging
... [OUTPUT ABBREVIATED] ...
  annotations:
    openshift.io/node-selector: ""
... [OUTPUT ABBREVIATED] ...
project "logging" edited

Option 2:

Using oc patch command

oc patch namespace logging \
    -p "{\"metadata\":{\"annotations\":{\"openshift.io/node-selector\":\"\"}}}"

Log out of the master1 instance and on the bastion instance add the following lines to the /etc/ansible/hosts file below the registry and above the [masters] entry.

openshift_hosted_logging_deploy=true
openshift_logging_es_pvc_dynamic=true
openshift_logging_es_pvc_size=10Gi
openshift_logging_es_cluster_size=3
openshift_logging_es_nodeselector={"region":"infra"}
openshift_logging_kibana_nodeselector={"region":"infra"}
openshift_logging_curator_nodeselector={"region":"infra"}

Run the deployment playbooks to deploy logging using the parameters defined in the inventory file.

$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-logging.yml

An example of a successful playbook run is shown below.

PLAY RECAP ********************************************************************
localhost                  : ok=11   changed=0    unreachable=0    failed=0
master1.example.com : ok=237  changed=94   unreachable=0    failed=0
master2.example.com : ok=16   changed=4    unreachable=0    failed=0
master3.example.com : ok=16   changed=4    unreachable=0    failed=0
Wednesday 28 March 2018  12:34:47 -0400 (0:00:00.415)       0:03:58.679 ******
===============================================================================
openshift_facts : Ensure various deps are installed -------------------- 46.27s
openshift_facts : Ensure various deps are installed -------------------- 38.13s
openshift_logging : Run JKS generation script --------------------------- 8.98s
openshift_logging : restart master api ---------------------------------- 3.98s
openshift_logging : Gather OpenShift Logging Facts ---------------------- 3.69s
openshift_facts : Gather Cluster facts and set is_containerized if needed --- 3.60s
openshift_facts : Gather Cluster facts and set is_containerized if needed --- 3.13s
openshift_logging : restart master api ---------------------------------- 2.90s
openshift_logging_elasticsearch : Set logging-es-cluster service -------- 1.74s
openshift_logging : include_role ---------------------------------------- 1.60s
openshift_logging_elasticsearch : Set logging-elasticsearch-view-role role --- 1.51s
openshift_logging_kibana : Set logging-kibana service ------------------- 1.46s
openshift_logging_elasticsearch : Set logging-es-cluster service -------- 1.45s
openshift_logging_elasticsearch : Set logging-es service ---------------- 1.43s
openshift_logging_elasticsearch : Set logging-es service ---------------- 1.41s
openshift_logging_elasticsearch : Set logging-elasticsearch-view-role role --- 1.40s
openshift_logging_elasticsearch : Create rolebinding-reader role -------- 1.40s
openshift_logging_elasticsearch : Set logging-es service ---------------- 1.39s
openshift_logging_elasticsearch : Set logging-es-cluster service -------- 1.37s
openshift_logging_elasticsearch : Create rolebinding-reader role -------- 1.36s

Once the playbook finishes ssh into the first master instance (master1.example.com) or the bastion host, and view the pods in the logging project.

$ oc get pods -n logging
NAME                                       READY     STATUS    RESTARTS   AGE
logging-curator-1-tzrsx                    1/1       Running   2          4m
logging-es-data-master-9uq6xi6z-1-dw6vq    1/1       Running   0          5m
logging-es-data-master-mcanh3m7-1-deploy   1/1       Running   0          5m
logging-es-data-master-mcanh3m7-1-vfkt2    1/1       Running   0          4m
logging-es-data-master-qbwcw4j6-1-227gj    1/1       Running   0          4m
logging-es-data-master-qbwcw4j6-1-deploy   1/1       Running   0          4m
logging-fluentd-49hfs                      1/1       Running   0          4m
logging-fluentd-4jwd0                      1/1       Running   0          4m
logging-fluentd-8wtph                      1/1       Running   0          4m
logging-fluentd-bnld6                      1/1       Running   0          4m
logging-fluentd-dp89n                      1/1       Running   0          4m
logging-fluentd-hsgl8                      1/1       Running   0          4m
logging-fluentd-htgx0                      1/1       Running   0          4m
logging-fluentd-s9jp0                      1/1       Running   0          4m
logging-kibana-1-76wxp                     2/2       Running   0          4m

Red Hat OpenShift Container Platform has the ability to gather metrics from running pods by querying kubelet and storing the values in Heapster. Cluster metrics add CPU, memory, and network-based metrics to the OpenShift administrative interface. Additionally, having cluster metrics configured in the cluster enables the creation of Horizontal Pod Autoscalers by the OpenShift user. It is important to understand capacity planning when deploying metrics into an OpenShift environment.

Persistent storage should be employed to prevent the loss of metrics in the event of a pod restart. Node selectors should be used to specify where the metrics components should run. In the reference architecture environment, the components are deployed on nodes with the label of "region=infra".

Add the following lines to the /etc/ansible/hosts file in the [OSEv3:vars] section.

openshift_hosted_metrics_deploy=true
openshift_hosted_metrics_storage_kind=dynamic
openshift_hosted_metrics_storage_volume_size=10Gi
openshift_metrics_hawkular_nodeselector={"region":"infra"}
openshift_metrics_cassandra_nodeselector={"region":"infra"}
openshift_metrics_heapster_nodeselector={"region":"infra"}

Run the deployment playbooks to deploy metrics using the parameters defined in the inventory file.

$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml

An example of a successful playbook run is shown below.

PLAY RECAP *********************************************************************
localhost                  : ok=11   changed=0    unreachable=0    failed=0
master1.example.com : ok=177  changed=47   unreachable=0    failed=0
master2.example.com : ok=16   changed=3    unreachable=0    failed=0
master3.example.com : ok=16   changed=3    unreachable=0    failed=0

Wednesday 30 August 2017  12:49:12 -0400 (0:00:00.296)       0:01:54.693 ******
===============================================================================
openshift_facts : Gather Cluster facts and set is_containerized if needed --- 3.87s
openshift_metrics : restart master api ---------------------------------- 3.33s
openshift_facts : Gather Cluster facts and set is_containerized if needed --- 3.19s
openshift_facts : Ensure various deps are installed --------------------- 3.09s
openshift_facts : Ensure various deps are installed --------------------- 2.74s
openshift_metrics : slurp ----------------------------------------------- 2.56s
openshift_metrics : Set serviceaccounts for hawkular metrics/cassandra --- 2.38s
openshift_metrics : restart master api ---------------------------------- 2.25s
openshift_metrics : Stop Heapster --------------------------------------- 1.89s
openshift_metrics : Stop Hawkular Metrics ------------------------------- 1.60s
openshift_metrics : Start Hawkular Metrics ------------------------------ 1.57s
openshift_metrics : Start Hawkular Cassandra ---------------------------- 1.56s
openshift_metrics : command --------------------------------------------- 1.53s
openshift_metrics : Start Heapster -------------------------------------- 1.51s
openshift_metrics : read files for the hawkular-metrics secret ---------- 1.46s
openshift_metrics : Generate services for cassandra --------------------- 1.24s
openshift_metrics : Generating serviceaccounts for hawkular metrics/cassandra --- 1.19s
openshift_metrics : Set hawkular cluster roles -------------------------- 1.14s
openshift_metrics : generate hawkular-cassandra keys -------------------- 1.02s
Gathering Facts --------------------------------------------------------- 0.83s

Once the playbook finishes ssh into the first master instance (master1.example.com) or the bastion and view the pods in the openshift-infra project.

$ oc get pods -n openshift-infra
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-4q46f   1/1       Running   0          3m
hawkular-metrics-0w46z       1/1       Running   0          3m
heapster-gnhsh               1/1       Running   0          3m

The steps defined below assume that Red Hat Cloudforms has been deployed and is accessible by the OpenShift environment.

oc sa get-token -n management-infra management-admin
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtYW5hZ2VtZW50LWluZnJhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1hbmFnZW1lbnQtYWRtaW4tdG9rZW4tdHM0cTIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibWFuYWdlbWVudC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImY0ZDlmMGMxLTEyY2YtMTFlOC1iNTgzLWZhMTYzZTEwNjNlYSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptYW5hZ2VtZW50LWluZnJhOm1hbmFnZW1lbnQtYWRtaW4ifQ.LwNm0652paGcJu7m63PxBhs4mjXwYcqMS5KD-0aWkEMCPo64WwNEawyyYH31SvuEPaE6qFxZwDdJHwdNsfq1CjUL4BtZHv1I2QZxpVl6gMBQowNf6fWSeGe1FDZ4lkLjzAoMOCFUWA0Z7lZM1FAlyjfz2LkPNKaFW0ffelSJ2SteuXB_4FNup-T5bKEPQf2pyrwvs2DadClyEEKpIrdZxuekJ9ZfIubcSc3pp1dZRu8wgmSQSLJ1N75raaUU5obu9cHjcbB9jpDhTW347oJOoL_Bj4bf0yyuxjuUCp3f4fs1qhyjHb5N5LKKBPgIKzoQJrS7j9Sqzo9TDMF9YQ5JLQ

The oc adm diagnostics command runs a series of checks for error conditions in the host or cluster. Specifically, it:

Red Hat OpenShift Container Platform may be deployed in numerous scenarios including:

Each method implies a different configuration and environment. The diagnostics were included within openshift binary to minimize environment assumptions and provide the ability to run the diagnostics tool within an Red Hat OpenShift Container Platform server or client.

To use the diagnostics tool, preferably on a master host and as cluster administrator, run a sudo user:

$ sudo oc adm diagnostics

The above command runs all available diagnostis skipping any that do not apply to the environment.

The diagnostics tool has the ability to run one or multiple specific diagnostics via name or as an enabler to address issues within the Red Hat OpenShift Container Platform environment. For example:

$ sudo oc adm diagnostics <name1> <name2>

The options provided by the diagnostics tool require working configuration files. For example, the NodeConfigCheck does not run unless a node configuration is readily available.

Diagnostics verifies that the configuration files reside in their standard locations unless specified with flags (respectively, --config, --master-config, and --node-config)

The standard locations are listed below:

If a configuration file is not found or specified, related diagnostics are skipped.

Available diagnostics include:

An Ansible-deployed cluster provides additional diagnostic benefits for nodes within Red Hat OpenShift Container Platform cluster due to:

Standard location of the configuration files placed by an Ansible-deployed cluster ensures that running sudo oc adm diagnostics works without any flags. In the event, the standard location of the configuration files is not used, options flags as those listed in the example below may be used.

$ sudo oc adm diagnostics --master-config=<file_path> --node-config=<file_path>

For proper usage of the log diagnostic, systemd units and log entries within journald are required. If log entries are not using the above method, log diagnostics won’t work as expected and are intentionally skipped.

The diagnostics runs using as much access as the existing user running the diagnostic has available. The diagnostic may run as an ordinary user, a cluster-admin user or cluster-admin user.

A client with ordinary access should be able to diagnose its connection to the master and run a diagnostic pod. If multiple users or masters are configured, connections are tested for all, but the diagnostic pod only runs against the current user, server, or project.

A client with cluster-admin access available (for any user, but only the current master) should be able to diagnose the status of the infrastructure such as nodes, registry, and router. In each case, running sudo oc adm diagnostics searches for the standard client configuration file location and uses it if available.

Additional diagnostic health checks are available through the Ansible-based tooling used to install and manage Red Hat OpenShift Container Platform clusters. The reports provide common deployment problems for the current Red Hat OpenShift Container Platform installation.

These checks can be run either using the ansible-playbook command (the same method used during Advanced Installation) or as a containerized version of openshift-ansible. For the ansible-playbook method, the checks are provided by the atomic-openshift-utils RPM package.

For the containerized method, the openshift3/ose-ansible container image is distributed via the Red Hat Container Registry.

Example usage for each method are provided in subsequent sections.

The following health checks are a set of diagnostic tasks that are meant to be run against the Ansible inventory file for a deployed Red Hat OpenShift Container Platform cluster using the provided health.yml playbook.

# ansible-playbook -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/openshift-checks/health.yml

# ansible-playbook -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/openshift-checks/health.yml
    -e openshift_check_logging_index_timeout_seconds=45
    -e etcd_max_image_data_size_bytes=40000000000

openshift_disable_check=etcd_traffic,etcd_volume

The openshift-ansible playbooks may run in a Docker container avoiding the requirement for installing and configuring Ansible, on any host that can run the ose-ansible image via the Docker CLI.

This is accomplished by specifying the cluster’s inventory file and the health.yml playbook when running the following docker run command as a non-root user that has privileges to run containers:

# docker run -u `id -u` \ 1
    -v $HOME/.ssh/id_rsa:/opt/app-root/src/.ssh/id_rsa:Z,ro \ 2
    -v /etc/ansible/hosts:/tmp/inventory:ro \ 3
    -e INVENTORY_FILE=/tmp/inventory \
    -e PLAYBOOK_FILE=playbooks/openshift-checks/health.yml \ 4
    -e OPTS="-v -e openshift_check_logging_index_timeout_seconds=45 -e etcd_max_image_data_size_bytes=40000000000" \ 5
    openshift3/ose-ansible

In the above command, the SSH key is mounted with the :Z flag so that the container can read the SSH key from its restricted SELinux context. This ensures the original SSH key file is relabeled similarly to system_u:object_r:container_file_t:s0:c113,c247. For more details about :Z, see the docker-run(1) man page.

It is important to note these volume mount specifications because it could have unexpected consequences. For example, if one mounts (and therefore relabels) the $HOME/.ssh directory, sshd becomes unable to access the public keys to allow remote login. To avoid altering the original file labels, mounting a copy of the SSH key (or directory) is recommended.

It is plausible to want to mount an entire .ssh directory for various reasons. For example, this enables the ability to use an SSH configuration to match keys with hosts or modify other connection parameters. It could also allow a user to provide a known_hosts file and have SSH validate host keys, which is disabled by the default configuration and can be re-enabled with an environment variable by adding -e ANSIBLE_HOST_KEY_CHECKING=True to the docker command line.