When using the native high availability method with HAProxy, master components have the following availability.

1: OpenShift Documentation - Kubernetes Infrastructure

The node instances run containers on behalf of its users and are separated into two functional classes: infrastructure and application nodes. The infrastructure (infra) nodes run the OpenShift router , OpenShift logging , OpenShift metrics , and the OpenShift registry while the application (app) nodes host the user container processes.

The Red Hat OpenShift Container Platform SDN requires Master instances to be considered nodes, therefore, all nodes run the atomic-openshift-node service.

The routers are an important component of Red Hat OpenShift Container Platform as they are the entry point to applications running in Red Hat OpenShift Container Platform. Developers can expose their applications running in Red Hat OpenShift Container Platform to be available from outside the Red Hat OpenShift Container Platform SDN creating routes that are published in the routers. Once the traffic reaches the router, the router forwards traffic to the containers on the app nodes using the Red Hat OpenShift Container Platform SDN.

In this reference architecture a set of three routers are deployed on the infra nodes for high availability purposes. In order to have a single entry point for applications, a load balancer is created as part of the installation in the same Red Hat OpenStack Platform tenant where it can reach the infrastructure nodes for load balancing the incoming traffic to the routers running in the infra nodes.

Cinder volumes attach to instances to provide additional disk space. Red Hat OpenShift Container Platform requires at least 40 GB of space available on the master nodes and at least 15GB on the infrastructure and application nodes for the /var partition. This reference architecture breaks out /var/lib/etcd, docker storage, and ephemeral pod storage to provide individual storage for those specific directories.

Each master node Red Hat OpenStack Platform instance creates and mounts cinder volumes using the steps provided in this reference architecture. One volume contains a volume for etcd storage. The master nodes while not schedulable by default, contain cinder volumes for both docker storage and Red Hat OpenShift Container Platform pod storage.

Each infra and app node Red Hat OpenStack Platform instance creates and mounts cinder volumes using the steps provided in this reference architecture. One volume contains the docker runtime disk overhead to store the docker container images. The other cinder volume is created for pod storage. Each node offers cinder access to the containers it hosts through Red Hat OpenShift Container Platform.

Red Hat OpenShift Container Platform offers two mechanisms to operate the internal network, OpenShift SDN and flannel.

The default mechanism is a software-defined network (SDN). This is a custom overlay network that uses Open vSwitch (OVS) that offers fine-grained dynamic routing. The SDN plug-in ovs-mulitenant allows for network isolation by default.

ovs-mulitenant SDN plug-in is a perfect choice as an internal network when traffic isolation between containers, services, and projects is important. However, this isolation comes at a cost in performance. Performance is degraded due to double encapsulation when deploying Red Hat OpenShift Container Platform on Red Hat OpenStack Platform because all Red Hat OpenShift Container Platform internode traffic is carried over a Red Hat OpenStack Platform neutron network. neutron is itself a SDN that provides network access to all the Red Hat OpenStack Platform instances.

flannel is a virtual networking layer designed for containers. Instead of having two SDN to route traffic, flannel runs in a mode labeled host-gw that routes packets by each node in the Red Hat OpenShift Container Platform environment running a daemon labeled flanneld that sends information to a centralized location (etcd store) thus allowing other nodes running the flanneld daemon to route packets to other containers as long as they are all on the flannel network. The main benefit to flannel is that it provides performance benefits over OVS double encapsulation since it does not create its own SDN on top of the existing RHOSP neutron network.

Why not just default to flannel?

flannel uses a single IP network space for all of the containers allocating a contiguous subset of the space to each instance. Consequently, nothing prevents a container from attempting to contact any IP address in the same network space. This hinders multi-tenancy because the network cannot be used to isolate containers in one application from another.

The key to making the right decision rests between mutli-tenancy isolation vs performance. Depending on the priority of each of these, should determine the appropriate choice when deciding between OpenShift SDN and flannel internal networks.

The steps to setup flannel instead of OpenShift ovs-multitenant can be found within the Appendix B, Using Flannel.

The public network is a network that contains external access and can be reached by the outside world. The public network creation can be only done by an OpenStack administrator.

The following commands provide an example of creating an OpenStack provider network for public network access.

As an OpenStack administrator,

$ source /path/to/examplerc
$ neutron net-create public_network \
    --router:external \
    --provider:network_type flat \
    --provider:physical_network datacentre
$ neutron subnet-create \
    --name public_network \
    --gateway <ip> \
    --allocation-pool start=<float_start>,end=<float_end> \
    public_network <CIDR>

The internal network is connected to the public network via a router during the network setup. This allows each Red Hat OpenStack Platform instance attached to the internal network the ability to request a floating IP from the public network for public access.

The following commands create the internal network.

$ source /path/to/examplerc
$ openstack router create <router-name>
$ openstack network create <private-net-name>
$ openstack subnet create --net <private-net-name> \
    --dns-nameserver <dns-ip> \
    --subnet-range <cidr> \
    <private-net-name>
$ openstack subnet list
$ openstack router add subnet <router-name> <private-subnet-uuid>
$ neutron router-gateway-set <router-name> <public-network-uuid>

The bastion instance only needs to allow inbound ssh. This instance exists to give operators a stable base to deploy, monitor and manage the Red Hat OpenShift Container Platform environment.

Creation of the above security group is as follows:

$ source /path/to/examplerc
$ openstack security group rule create \
    --ingress \
    --protocol icmp \
    <bastion-sg-name>
$ openstack security group rule create \
    --ingress \
    --protocol tcp \
    --dst-port 22 \
    <bastion-sg-name>

Verification of the security group is as follows:

$ openstack security group rule list <bastion-sg-name>

The Red Hat OpenShift Container Platform master security group requires the most complex network access controls. In addition to the ports used by the API and master console, these nodes contain the etcd servers that form the cluster.

Creation of the above security group is as follows:

$ source /path/to/examplerc
$ openstack security group rule create \
    --ingress \
    --protocol tcp \
    --dst-port 2379:2380 \
    --src-group <master-sg-name> \
    <master-sg-name>

$ for PORT in 8053 24224;
do
  for PROTO in tcp udp;
  do
    openstack security group rule create \
      --ingress \
      --protocol $PROTO \
      --dst-port $PORT \
      --src-group <node-sg-name> \
      <master-sg-name>;
  done;
done
$ openstack security group rule create \
    --ingress \
    --protocol tcp \
    --dst-port 8443 \
    <master-sg-name>

Verification of the security group is as follows:

$ openstack security group rule list <master-sg-name>

The infrastructure nodes run the Red Hat OpenShift Container Platform router and the local registry. It must accept inbound connections on the web ports that are forwarded to their destinations.

Creation of the above security group is as follows:

$ source /path/to/examplerc
$ for PORT in 80 443 9200 9300;
do
  openstack security group rule create \
    --ingress \
    --protocol tcp \
    --dst-port $PORT \
    <infra-sg-name>;
done

Verification of the security group is as follows:

$ openstack security group rule list <infra-sg-name>

All instances run OpenShift node service. All instances should only accept ICMP from any source, ssh traffic from the bastion host or other nodes, pod to pod communication via SDN traffic and kubelet communication via Kubernetes.

Creation of the above security group is as follows:

$ source /path/to/examplerc
$ openstack security group rule create \
    --ingress \
    --protocol icmp \
    <node-sg-name>
$ for SRC in <bastion-sg-name> <node-sg-name>;
do
  openstack security group rule create \
    --ingress \
    --protocol tcp \
    --dst-port 22 \
    --src-group $SRC \
    <node-sg-name>;
done
$ openstack security group rule create \
    --ingress \
    --protocol tcp \
    --dst-port 10250 \
    --src-group <node-sg-name> \
    <node-sg-name>
$ openstack security group rule create \
    --ingress \
    --protocol udp \
    --dst-port 4789 \
    --src-group <node-sg-name> \
    <node-sg-name>

Verification of the security group is as follows:

$ openstack security group rule list <node-sg-name>

Containers and images are stored locally in a dedicated cinder volume on each instance able to run containers. The reason it must be a dedicate volume is to ensure there is not disk contention in the /var partition. The docker volume is customized using the docker-storage-setup at boot time using cloud-init.

A cinder volume is created on the master instances for the storage of /var/lib/etcd. Storing etcd allows the similar benefit of protecting /var but more importantly provides the ability to perform snapshots of the volume when performing etcd maintenance. The cinder etcd volume is created only on the master instances.

A cinder volume is created for the directory of /var/lib/origin/openshift.local.volumes that is used with the perFSGroup setting at installation and with the mount option of gquota. These settings and volumes set a quota to ensure that containers cannot grow to an unreasonable size.

The Red Hat OpenShift Container Platform registry requires a cinder volume to ensure that images are saved in the event that the registry needs to migrate to another node. The volume is attached automatically by Red Hat OpenShift Container Platform to the node running the registry, there is no need to attach it manually.

$ source /path/to/examplerc
$ openstack volume create --size *<volume-size-in-GB>* openshift-registry
$ openstack volume list

Red Hat OpenStack Platform provides a way for users to pass in information to be applied when an instance boots. The --user-data switch to openstack server create command makes the contents of the provided file available to the instance through cloud-init. cloud-init is a set of init scripts for cloud instances. It is available via the rhel-7-server-rh-common-rpms repository and queries a standard URL for the user-data file and processes the contents to initialize the OS of the deployed Red Hat OpenStack Platform instance.

The user-data is placed in files named <hostname>.yaml where <hostname> is the name of the instance.

An example of the <hostname>.yaml file is shown below. This is required for every Red Hat OpenStack Platform instance that is to be used for the Red Hat OpenShift Container Platform deployment.

Create a user-data directory to store the <hostname>.yaml files.

$ mkdir /path/to/user-data

#cloud-config
cloud_config_modules:
- disk_setup
- mounts

hostname: <hostname>
fqdn: <hostname>.<domain>

write_files:
  - path: "/etc/sysconfig/docker-storage-setup"
    permissions: "0644"
    owner: "root"
    content: |
      DEVS='/dev/vdb'
      VG=docker_vol
      DATA_SIZE=95%VG
      STORAGE_DRIVER=overlay2
      CONTAINER_ROOT_LV_NAME=dockerlv
      CONTAINER_ROOT_LV_MOUNT_PATH=/var/lib/docker
      CONTAINER_ROOT_LV_SIZE=100%FREE

fs_setup:
- label: emptydir
  filesystem: xfs
  device: /dev/vdc
  partition: auto
- label: etcd_storage
  filesystem: xfs
  device: /dev/vdd
  partition: auto

runcmd:
- mkdir -p /var/lib/origin/openshift.local.volumes
- mkdir -p /var/lib/etcd

mounts:
- [ /dev/vdc, /var/lib/origin/openshift.local.volumes, xfs, "defaults,gquota" ]
- [ /dev/vdd, /var/lib/etcd, xfs, "defaults" ]

#cloud-config
cloud_config_modules:
- disk_setup
- mounts

hostname: <hostname>
fqdn: <hostname>.<domain>

write_files:
  - path: "/etc/sysconfig/docker-storage-setup"
    permissions: "0644"
    owner: "root"
    content: |
      DEVS='/dev/vdb'
      VG=docker_vol
      DATA_SIZE=95%VG
      STORAGE_DRIVER=overlay2
      CONTAINER_ROOT_LV_NAME=dockerlv
      CONTAINER_ROOT_LV_MOUNT_PATH=/var/lib/docker
      CONTAINER_ROOT_LV_SIZE=100%FREE

fs_setup:
- label: emptydir
  filesystem: xfs
  device: /dev/vdc
  partition: auto

runcmd:
- mkdir -p /var/lib/origin/openshift.local.volumes

mounts:
- [ /dev/vdc, /var/lib/origin/openshift.local.volumes, xfs, "defaults,gquota" ]

The bash lines below are a loop that allow for all of the master instances to be created and the specific mounts, instance size, and security groups are configured at launch time. Create the master instances by filling in the bold items relevant to the current OpenStack values.

$ domain=<domain>
$ netid1=$(openstack network show <internal-network-name> -f value -c id)
$ for node in master-{0..2};
do
  nova boot \
    --nic net-id=$netid1 \
    --flavor <flavor> \
    --image <image> \
    --key-name <keypair> \
    --security-groups <master-sg-name>,<node-sg-name> \
    --user-data=/path/to/user-data/$node.yaml \
    --block-device source=blank,dest=volume,device=vdb,size=<docker-volume-size>,shutdown=preserve \
    --block-device source=blank,dest=volume,device=vdc,size=<openshift-local-volume-size>,shutdown=preserve \
    --block-device source=blank,dest=volume,device=vdd,size=<etcd-volume-size>,shutdown=preserve \
    $node.$domain;
done

The bash lines below are a loop that allow for all of the infrastructure instances to be created and the specific mounts, instance size, and security groups are configured at launch time. Create the infrastructure instances by filling in the bold items relevant to the current OpenStack values.

$ domain = <domain>
$ netid1=$(openstack network show <internal-network-name> -f value -c id)
$ for node in infra-node-{0..2};
do
  nova boot \
    --nic net-id=$netid1 \
    --flavor <flavor> \
    --image <image> \
    --key-name <keypair> \
    --security-groups <infra-sg-name>,<node-sg-name> \
    --user-data=/path/to/user-data/$node.yaml \
    --block-device source=blank,dest=volume,device=vdb,size=<docker-volume-size>,shutdown=preserve \
    --block-device source=blank,dest=volume,device=vdc,size=<openshift-local-volume-size>,shutdown=preserve \
    $node.$domain;
done

The bash lines below are a loop that allow for all of the application instances to be created and the specific mounts, instance size, and security groups are configured at launch time. Create the application instances by filling in the bold items relevant to the current OpenStack values.

$ domain = <domain>
$ netid1=$(openstack network show <internal-network-name> -f value -c id)
$ for node in app-node-{0..2};
do
  nova boot \
    --nic net-id=$netid1 \
    --flavor <flavor> \
    --image <image> \
    --key-name <keypair> \
    --security-groups <node-sg-name> \
    --user-data=/path/to/user-data/$node.yaml \
    --block-device source=blank,dest=volume,device=vdb,size=<docker-volume-size>,shutdown=preserve \
    --block-device source=blank,dest=volume,device=vdc,size=<openshift-local-volume-size>,shutdown=preserve \
      $node.$domain;
done

The above creates 3 master, infra and app instances.

Verify the creation of the Red Hat OpenStack Platform instances via:

$ openstack server list

Using the values provided by the openstack server list command, update the DNS master zone.db file as shown in Section A.1, “Setting up the DNS Red Hat OpenStack Platform Instance” with the appropriate IP addresses. Do not proceed to the until the DNS resolution is configured.

cinder volumes are created using the instance id making them confusing to identify.

To rename the cinder volumes and make them human readable, use the following snippet:

$ for node in master-{0..2} app-node-{0..2} infra-node-{0..2};
do
  dockervol=$(openstack volume list -f value -c ID -c "Attached to" | awk "/$node/ && /vdb/ {print \$1}")
  ocplocalvol=$(openstack volume list -f value -c ID -c "Attached to" | awk "/$node/ && /vdc/ {print \$1}")
  openstack volume set --name $node-docker $dockervol
  openstack volume set --name $node-ocplocal $ocplocalvol
done
for master in master-{0..2};
do
  etcdvol=$(openstack volume list -f value -c ID -c "Attached to" | awk "/$master/ && /vdd/ {print \$1}")
  openstack volume set --name $master-etcd $etcdvol
done

Create the bastion instance via:

$ domain=<domain>
$ netid1=$(openstack network show <internal-network-name> -f value -c id)
$ openstack server create \
    --nic net-id=$netid1 \
    --flavor <flavor> \
    --image <image> \
    --key-name <keypair> \
    --security-group <bastion-sg-name> \
    bastion.$domain

Once the instances are created, floating IPs must be created and then can be allocated to the instance. The following shows an example

$ source /path/to/examplerc
$ openstack floating ip create <public-network-name>
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| created_at          | 2017-08-24T22:44:03Z                 |
| description         |                                      |
| fixed_ip_address    | None                                 |
| floating_ip_address | 10.20.120.150                       |
| floating_network_id | 084884f9-d9d2-477a-bae7-26dbb4ff1873 |
| headers             |                                      |
| id                  | 2bc06e39-1efb-453e-8642-39f910ac8fd1 |
| port_id             | None                                 |
| project_id          | ca304dfee9a04597b16d253efd0e2332     |
| project_id          | ca304dfee9a04597b16d253efd0e2332     |
| revision_number     | 1                                    |
| router_id           | None                                 |
| status              | DOWN                                 |
| updated_at          | 2017-08-24T22:44:03Z                 |
+---------------------+--------------------------------------+

Within the above output, the floating_ip_address field shows that the floating IP 10.20.120.150 is created. In order to assign this IP to one of the Red Hat OpenShift Container Platform instances, run the following command:

$ source /path/to/examplerc
$ openstack server add floating ip <instance-name> <ip>

For example, if instance bastion.ocp3.example.com is to be assigned IP 10.20.120.150 the command would be:

$ source /path/to/examplerc
$ openstack server add floating ip bastion.ocp3.example.com 10.20.120.150

The following steps configure a HAProxy server once logged into the instance.

Subscribe to the following repositories using subscription-manager

$ sudo subscription-manager register
$ sudo subscription-manager attach --pool <pool_id>
$ sudo subscription-manager repos --disable=*
$ sudo subscription-manager repos \
    --enable=rhel-7-server-rpms \
    --enable=rhel-7-server-extras-rpms \
    --enable rhel-7-server-rh-common-rpms

Install and enable the following packages and services:

$ sudo yum -y install haproxy firewalld
$ sudo systemctl enable firewalld
$ sudo systemctl start firewalld
$ sudo systemctl enable haproxy
$ sudo systemctl start haproxy

Enable and permanently set the required services within firewalld

$ sudo firewall-cmd --zone public --add-service http
$ sudo firewall-cmd --zone public --add-service https
$ sudo firewall-cmd --zone public --add-port=8443/tcp
$ sudo firewall-cmd --zone public --add-service http --permanent
$ sudo firewall-cmd --zone public --add-service https --permanent
$ sudo firewall-cmd --zone public --add-port=8443/tcp --permanent

Modify the /etc/haproxy/haproxy.cfg file with the appropriate Red Hat OpenShift Container Platform instance IPs

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

defaults
    log                     global
    option                  httplog
    option                  dontlognull
    option                  http-server-close
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
listen stats :9000
    stats enable
    stats realm Haproxy\ Statistics
    stats uri /haproxy_stats
    stats auth admin:password
    stats refresh 30
    mode http

frontend  main *:80
    default_backend router80

backend router80
    balance source
    mode tcp
    # INFRA_80
    server infra-node-0.internal.ocp3.example.com <IP>:80 check
    server infra-node-1.internal.ocp3.example.com <IP>:80 check
    server infra-node-2.internal.ocp3.example.com <IP>:80 check

frontend  main *:443
    default_backend router443

backend router443
    balance source
    mode tcp
    # INFRA_443
    server infra-node-0.internal.ocp3.example.com <IP>:443 check
    server infra-node-1.internal.ocp3.example.com <IP>:443 check
    server infra-node-2.internal.ocp3.example.com <IP>:443 check

frontend  main *:8443
    default_backend mgmt8443

backend mgmt8443
    balance source
    mode tcp
    # MASTERS 8443
    server master-0.internal.ocp3.example.com <IP>:8443 check
    server master-1.internal.ocp3.example.com <IP>:8443 check
    server master-2.internal.ocp3.example.com <IP>:8443 check

Restart the HAProxy service

$ sudo systemctl restart haproxy.service

To check connectivity to the instance services and DNS resolution:

$ for i in 0 1 2
do
  curl -v http://infra-node-$i.internal.ocp3.example.com
  curl -v -k https://infra-node-$i.internal.ocp3.example.com
  curl -v -k https://master-$i.internal.ocp3.example.com:8443
done
$ curl -v http://localhost
$ curl -v -k http://localhost:443
$ curl -v -k http://localhost:8443

The ssh private key is required to allow the bastion instance to connect to the other instances used for OpenShift. The following command copies the existing keypair from Section 3.4, “Creating an OpenStack Keypair” to the bastion instance and sets the appropriate permissions.

$ export KEY="/path/to/<keypair-name>.pem"
$ scp -i $KEY $KEY \
    cloud-user@bastion.internal.ocp3.example.com:~/.ssh/<keypair-name>.pem
$ ssh -i $KEY bastion.internal.ocp3.example.com \
    "chmod 0600 ~/.ssh/<keypair-name>.pem"

Log into the bastion instance and perform the following steps.

Register and subscribe to the following repositories using subscription-manager.

$ sudo subscription-manager register
$ sudo subscription-manager attach --pool <pool_id>
$ sudo subscription-manager repos --disable=*
$ sudo subscription-manager repos \
    --enable=rhel-7-server-rpms \
    --enable=rhel-7-server-extras-rpms \
    --enable=rhel-7-server-ose-3.6-rpms

Install the ansible, git, and atomic-openshift-utils packages that enable the installation and deployment of OpenShift.

$ sudo yum install -y ansible atomic-openshift-utils git

ansible is installed on the bastion instance to perform the registration, installation of packages, and the deployment of the Red Hat OpenShift Container Platform environment on the master and node instances.

Before running playbooks, it is important to modify ansible.cfg to reflect the deployed environment:

$ sudo vi /etc/ansible/ansible.cfg

The code block below can overwrite the default values in the file. Ensure to populate <keypair-name> with the keypair that was copied to the bastion instance.

[defaults]
forks = 20
host_key_checking = False
remote_user = cloud-user
private_key_file = /home/cloud-user/.ssh/<keypair-name>.pem
roles_path = roles/
gathering = smart
fact_caching = jsonfile
fact_caching_connection = $HOME/ansible/facts
fact_caching_timeout = 600
log_path = $HOME/ansible.log
nocows = 1
callback_whitelist = profile_tasks

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=600s -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=false
control_path = %(directory)s/%%h-%%r
pipelining = True
timeout = 10

[persistent_connection]
connect_timeout = 30
connect_retries = 30
connect_interval = 1

Red Hat OpenShift Container Platform provides the ability to use many different authentication platforms. For this reference architecture, LDAP is the preferred authentication mechanism. A listing of other authentication options are available at Configuring Authentication and User Agent.

When configuring LDAP as the authentication provider the following parameters can be added to the ansible inventory. An example is shown below.

openshift_master_identity_providers=[{'name': 'idm', 'challenge': 'true', 'login': 'true', 'kind': 'LDAPPasswordIdentityProvider', 'attributes': {'id': ['dn'], 'email': ['mail'], 'name': ['cn'], 'preferredUsername': ['uid']}, 'bindDN': 'uid=admin,cn=users,cn=accounts,dc=ocp3,dc=openshift,dc=com', 'bindPassword': 'ldapadmin', 'ca': '/etc/origin/master/ca.crt', 'insecure': 'false', 'url': 'ldap://ldap.ocp3.example.com/cn=users,cn=accounts,dc=ocp3,dc=openshift,dc=com?uid?sub?(memberOf=cn=ose-user,cn=groups,cn=accounts,dc=ocp3,dc=openshift,dc=com)'}]

For more information about the required LDAP parameters, see LDAP Authentication

The inventory file contains both variables and instances used for the configuration and deployment of Red Hat OpenShift Container Platform. In the example below, some values are bold and must reflect the deployed environment from the previous chapter.

To gather the openshift_hosted_registry_storage_openstack_volumeID value use:

$ openstack volume show openshift-registry -f value -c id

The openshift_cloudprovider_openstack_* values are required for Red Hat OpenShift Container Platform to be able to create OpenStack resources such as cinder volumes for persistent volumes. It is recommended to create a dedicated OpenStack user within the tenant instead of using a personal account.

$ sudo vi /etc/ansible/hosts

[OSEv3:children]
masters
etcd
nodes

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
ansible_ssh_user=cloud-user
deployment_type=openshift-enterprise
debug_level=2
ansible_become=true
console_port=8443
openshift_debug_level="{{ debug_level }}"
openshift_node_debug_level="{{ node_debug_level | default(debug_level, true) }}"
openshift_master_debug_level="{{ master_debug_level | default(debug_level, true) }}"

openshift_master_identity_providers=[{'name': 'idm', 'challenge': 'true', 'login': 'true', 'kind': 'LDAPPasswordIdentityProvider', 'attributes': {'id': ['dn'], 'email': ['mail'], 'name': ['cn'], 'preferredUsername': ['uid']}, 'bindDN': 'uid=admin,cn=users,cn=accounts,dc=ocp3,dc=example,dc=com', 'bindPassword': 'ldapadmin', 'ca': '/etc/origin/master/ca.crt', 'insecure': 'false', 'url': 'ldap://ldap.ocp3.example.com/cn=users,cn=accounts,dc=ocp3,dc=example,dc=com?uid?sub?(memberOf=cn=ose-user,cn=groups,cn=accounts,dc=ocp3,dc=openshift,dc=com)'}]

openshift_master_access_token_max_seconds=2419201
openshift_registry_selector="role=infra"
openshift_router_selector="role=infra"
openshift_hosted_router_replicas=3
openshift_hosted_registry_replicas=1
openshift_master_cluster_method=native
openshift_node_local_quota_per_fsgroup=512Mi
openshift_cloudprovider_kind=openstack

openshift_cloudprovider_openstack_auth_url=http://10.19.114.177:5000/v2.0
openshift_cloudprovider_openstack_username=openshift
openshift_cloudprovider_openstack_password=redhat
openshift_cloudprovider_openstack_tenant_name=openshift-tenant
openshift_master_cluster_hostname=openshift.ocp3.example.com
openshift_master_cluster_public_hostname=openshift.ocp3.example.com
openshift_master_default_subdomain=apps.ocp3.example.com

osm_default_node_selector="role=app"
os_sdn_network_plugin_name='redhat/openshift-ovs-multitenant'
osm_use_cockpit=true

containerized=false

# registry
openshift_hosted_registry_storage_kind=openstack
openshift_hosted_registry_storage_access_modes=['ReadWriteOnce']
openshift_hosted_registry_storage_openstack_filesystem=ext4

openshift_hosted_registry_storage_openstack_volumeID=3b5c9517-abd1-4364-80b9-a1469cae8fd8
openshift_hosted_registry_storage_volume_size=15Gi

# host group for masters
[masters]
master-0.internal.ocp3.example.com master-1.internal.ocp3.example.com master-2.internal.ocp3.example.com

# host group for etcd
[etcd]
master-0.internal.ocp3.example.com master-1.internal.ocp3.example.com master-2.internal.ocp3.example.com

# host group for nodes, includes region info
[nodes]
master-0.internal.ocp3.example.com openshift_node_labels="{'role': 'master'}" openshift_hostname=master-0.internal.ocp3.example.com openshift_schedulable=false master-1.internal.ocp3.example.com openshift_node_labels="{'role': 'master'}" openshift_hostname=master-1.internal.ocp3.example.com openshift_schedulable=false master-2.internal.ocp3.example.com openshift_node_labels="{'role': 'master'}" openshift_hostname=master-2.internal.ocp3.example.com openshift_schedulable=false app-node-0.internal.ocp3.example.com openshift_node_labels="{'role': 'app'}" openshift_hostname=app-node-0.internal.ocp3.example.com app-node-1.internal.ocp3.example.com openshift_node_labels="{'role': 'app'}" openshift_hostname=app-node-1.internal.ocp3.example.com app-node-2.internal.ocp3.example.com openshift_node_labels="{'role': 'app'}" openshift_hostname=app-node-2.internal.ocp3.example.com infra-node-0.internal.ocp3.example.com openshift_node_labels="{'role': 'infra'}" openshift_hostname=infra-node-0.internal.ocp3.example.com infra-node-1.internal.ocp3.example.com openshift_node_labels="{'role': 'infra'}" openshift_hostname=infra-node-1.internal.ocp3.example.com infra-node-2.internal.ocp3.example.com openshift_node_labels="{'role': 'infra'}" openshift_hostname=infra-node-2.internal.ocp3.example.com

It can be useful to check for potential issues or misconfigurations in the instances before continuing the installation process. Connect to every instance using the bastion host and verify the disks are properly created and mounted, the cloud-init process finished successfully and check for potential errors in the log files to ensure everything is ready for the Red Hat OpenShift Container Platform installation:

$ ssh -i $KEY bastion.internal.ocp3.example.com
$ ssh <instance>
$ df -h
$ mount
$ sudo journalctl
$ sudo less /var/log/cloud-init.log
$ free -m
$ sudo yum repolist

$ ansible all -b -i /etc/ansible/hosts -m command -a "subscription-manager register --username <user> --password '<password>'"

$ ansible all -b -i /etc/ansible/hosts -m command -a "subscription-manager register --org=<org_id> --activationkey=<keyname>"

$ ansible -i /etc/ansible/hosts all -b \
    -m command -a 'subscription-manager repos \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-server-ose-3.6-rpms" \
    --enable="rhel-7-fast-datapath-rpms"'

$ ansible -i /etc/ansible/hosts all -b -m yum -a "name=* state=latest"

$ ansible -i /etc/ansible/hosts all -b -a "reboot"

$ ansible -i /etc/ansible/hosts all -b \
    -m command -a 'yum -y install iptables NetworkManager iptables-services iptables docker'

$ ansible -i /etc/ansible/hosts all -b \
    -m service -a 'name=iptables state=started enabled=true'
$ ansible -i /etc/ansible/hosts all -b \
    -m service -a 'name=docker state=started enabled=true'
$ ansible -i /etc/ansible/hosts all -b \
    -m service -a 'name=NetworkManager state=started enabled=true'

To easily connect to the Red Hat OpenShift Container Platform environment, the following snippet can be added to the ~/.ssh/config file in the host used to administer the Red Hat OpenShift Container Platform platform:

Host osbastion
    HostName        <bastion-ip>
    User            cloud-user
    IdentityFile    /path/to/<keypair-name>.pem

Host osmaster-? osapp-node-? osinfra-node-?
    ProxyCommand    ssh osbastion -W $(sed -e "s/os//" <<< "%h").internal.ocp3.example.com:%p
    IdentityFile    /path/to/<keypair-name>.pem
    User            cloud-user

As an example, to access the second master, this can be used:

$ ssh osmaster-0

The Red Hat OpenShift Container Platform Registry Console deployment is deployed on any Red Hat OpenShift Container Platform node by default, so the container may end up running on any of the application nodes.

From the first master instance(master-0.internal.ocp3.example.com), ensure the Red Hat OpenShift Container Platform Registry Console pod runs on the infra nodes by modifying the nodeSelector as follows:

$ oc patch dc registry-console \
    -n default \
    -p '{"spec":{"template":{"spec":{"nodeSelector":{"role":"infra"}}}}}'

The StorageClass resource object describes and classifies storage that can be requested and provide a means for passing parameters for dynamically provisioned storage on demand. A StorageClass object can serve as a management mechanism for controlling various levels of access to storage that are defined and created by either a cluster or storage administrator.

With regards to Red Hat OpenStack Platform, the storage type that allows for dynamic provisioning is OpenStack cinder using the Provisioner Plug-in Name labeled kubernetes.io/cinder

The following is an example of storage-class.yaml that is required for dynamic provisioning using OpenStack cinder. ssh into the first master instance (master-0.internal.ocp3.example.com) and create the storage class file.

$ vi storage-class.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/cinder
parameters:
  availability: nova

Use the oc client to create the StorageClass.

$ oc create -f storage-class.yaml

In the storage-class.yaml file example, the cluster or storage administrator requires to input a name for the StorageClass. One parameter option not shown in the above example is type. Type refers to the volume type created in cinder. By default, the value for cinder is empty, thus it is not included in the above example.

However, if the cinder volumes created by Red Hat OpenStack Platform contain a volume type, a storage-class.yaml file with the additional type parameter is required as shown below:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: standard
provisioner: kubernetes.io/cinder
parameters:
  type: fast
  availability: nova

Red Hat OpenShift Container Platform allows the option to deploy aggregate logging for containers running in the OpenShift environment. OpenShift uses the EFK stack (Elasticsearch, Fluentd, Kibana) to collect logs from applications and present them to OpenShift users. Cluster administrators can view all logs, but application developers can only view logs for projects they have permission to view. The EFK stack consists of the following components:

Below is an example of some of the best practices when deploying OpenShift logging.

Elasticsearch, Kibana, and Curator are deployed on nodes with the label of "role=infra". Specifying the node label ensures that the Elasticsearch and Kibana components are not competing with applications for resources. The Elasticsearch cluster size (openshift_logging_es_cluster_size) of three is the minimum number of nodes to ensure data durability and redundancy.

The logging project created during installation must be modified to bypass the default node selector. If this is not done then the Elasticsearch, Kibana, and Curator components cannot be started. There are two options in making the appropriate edit.

Option 1:

ssh into the first master instance (master-0.internal.ocp3.example.com) or the bastion host and modify the logging project.

Add the following line openshift.io/node-selector: "" under annotations but do not remove any lines.

$ oc edit project logging
... omitted ...
  annotations:
    openshift.io/node-selector: ""
...ommitted...
project "logging" edited

Option 2:

Using oc patch command

oc patch namespace logging \
    -p "{\"metadata\":{\"annotations\":{\"openshift.io/node-selector\":\"\"}}}"

Log out of the master-0 instance and on the bastion instance add the following lines to the /etc/ansible/hosts file below the registry and above the [masters] entry.

openshift_hosted_logging_deploy=true
openshift_logging_es_pvc_dynamic=true
openshift_logging_es_pvc_size=10Gi
openshift_logging_es_cluster_size=3
openshift_logging_es_nodeselector={"role":"infra"}
openshift_logging_kibana_nodeselector={"role":"infra"}
openshift_logging_curator_nodeselector={"role":"infra"}

Run the deployment playbooks to deploy logging using the parameters defined in the inventory file.

$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-logging.yml

An example of a successful playbook run is shown below.

PLAY RECAP ********************************************************************
localhost                  : ok=11   changed=0    unreachable=0    failed=0
master-0.internal.ocp3.example.com : ok=237  changed=94   unreachable=0    failed=0
master-1.internal.ocp3.example.com : ok=16   changed=4    unreachable=0    failed=0
master-2.internal.ocp3.example.com : ok=16   changed=4    unreachable=0    failed=0
Wednesday 30 August 2017  12:34:47 -0400 (0:00:00.415)       0:03:58.679 ******
===============================================================================
openshift_facts : Ensure various deps are installed -------------------- 46.27s
openshift_facts : Ensure various deps are installed -------------------- 38.13s
openshift_logging : Run JKS generation script --------------------------- 8.98s
openshift_logging : restart master api ---------------------------------- 3.98s
openshift_logging : Gather OpenShift Logging Facts ---------------------- 3.69s
openshift_facts : Gather Cluster facts and set is_containerized if needed --- 3.60s
openshift_facts : Gather Cluster facts and set is_containerized if needed --- 3.13s
openshift_logging : restart master api ---------------------------------- 2.90s
openshift_logging_elasticsearch : Set logging-es-cluster service -------- 1.74s
openshift_logging : include_role ---------------------------------------- 1.60s
openshift_logging_elasticsearch : Set logging-elasticsearch-view-role role --- 1.51s
openshift_logging_kibana : Set logging-kibana service ------------------- 1.46s
openshift_logging_elasticsearch : Set logging-es-cluster service -------- 1.45s
openshift_logging_elasticsearch : Set logging-es service ---------------- 1.43s
openshift_logging_elasticsearch : Set logging-es service ---------------- 1.41s
openshift_logging_elasticsearch : Set logging-elasticsearch-view-role role --- 1.40s
openshift_logging_elasticsearch : Create rolebinding-reader role -------- 1.40s
openshift_logging_elasticsearch : Set logging-es service ---------------- 1.39s
openshift_logging_elasticsearch : Set logging-es-cluster service -------- 1.37s
openshift_logging_elasticsearch : Create rolebinding-reader role -------- 1.36s

Once the playbook finishes ssh into the first master instance (master-0.internal.ocp3.example.com) or the bastion host, and view the pods in the logging project.

$ oc get pods -n logging
NAME                                       READY     STATUS    RESTARTS   AGE
logging-curator-1-tzrsx                    1/1       Running   2          4m
logging-es-data-master-9uq6xi6z-1-dw6vq    1/1       Running   0          5m
logging-es-data-master-mcanh3m7-1-deploy   1/1       Running   0          5m
logging-es-data-master-mcanh3m7-1-vfkt2    1/1       Running   0          4m
logging-es-data-master-qbwcw4j6-1-227gj    1/1       Running   0          4m
logging-es-data-master-qbwcw4j6-1-deploy   1/1       Running   0          4m
logging-fluentd-49hfs                      1/1       Running   0          4m
logging-fluentd-4jwd0                      1/1       Running   0          4m
logging-fluentd-8wtph                      1/1       Running   0          4m
logging-fluentd-bnld6                      1/1       Running   0          4m
logging-fluentd-dp89n                      1/1       Running   0          4m
logging-fluentd-hsgl8                      1/1       Running   0          4m
logging-fluentd-htgx0                      1/1       Running   0          4m
logging-fluentd-s9jp0                      1/1       Running   0          4m
logging-kibana-1-76wxp                     2/2       Running   0          4m

Red Hat OpenShift Container Platform has the ability to gather metrics from the pods running by querying the kubelet and storing the values in Heapster. OpenShift Cluster Metrics provide the ability to view CPU, memory, and network-based metrics and display the values in the user interface. These metrics can allow for the horizontal autoscaling of pods based on parameters provided by an OpenShift user. It is important to understand capacity planning when deploying metrics into an OpenShift environment.

As best practices when metrics are deployed, persistent storage should be used to allow for metrics to be preserved. Node selectors should be used to specify where the metrics components should run. In the reference architecture environment, the components are deployed on nodes with the label of "role=infra".

Add the following lines to the /etc/ansible/hosts file below the registry and above the [masters] entry.

openshift_hosted_metrics_deploy=true
openshift_hosted_metrics_storage_kind=dynamic
openshift_hosted_metrics_storage_volume_size=10Gi
openshift_metrics_hawkular_nodeselector={"role":"infra"}
openshift_metrics_cassandra_nodeselector={"role":"infra"}
openshift_metrics_heapster_nodeselector={"role":"infra"}

Run the deployment playbooks to deploy metrics using the parameters defined in the inventory file.

$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml

An example of a successful playbook run is shown below.

PLAY RECAP *********************************************************************
localhost                  : ok=11   changed=0    unreachable=0    failed=0
master-0.internal.ocp3.example.com : ok=177  changed=47   unreachable=0    failed=0
master-1.internal.ocp3.example.com : ok=16   changed=3    unreachable=0    failed=0
master-2.internal.ocp3.example.com : ok=16   changed=3    unreachable=0    failed=0

Wednesday 30 August 2017  12:49:12 -0400 (0:00:00.296)       0:01:54.693 ******
===============================================================================
openshift_facts : Gather Cluster facts and set is_containerized if needed --- 3.87s
openshift_metrics : restart master api ---------------------------------- 3.33s
openshift_facts : Gather Cluster facts and set is_containerized if needed --- 3.19s
openshift_facts : Ensure various deps are installed --------------------- 3.09s
openshift_facts : Ensure various deps are installed --------------------- 2.74s
openshift_metrics : slurp ----------------------------------------------- 2.56s
openshift_metrics : Set serviceaccounts for hawkular metrics/cassandra --- 2.38s
openshift_metrics : restart master api ---------------------------------- 2.25s
openshift_metrics : Stop Heapster --------------------------------------- 1.89s
openshift_metrics : Stop Hawkular Metrics ------------------------------- 1.60s
openshift_metrics : Start Hawkular Metrics ------------------------------ 1.57s
openshift_metrics : Start Hawkular Cassandra ---------------------------- 1.56s
openshift_metrics : command --------------------------------------------- 1.53s
openshift_metrics : Start Heapster -------------------------------------- 1.51s
openshift_metrics : read files for the hawkular-metrics secret ---------- 1.46s
openshift_metrics : Generate services for cassandra --------------------- 1.24s
openshift_metrics : Generating serviceaccounts for hawkular metrics/cassandra --- 1.19s
openshift_metrics : Set hawkular cluster roles -------------------------- 1.14s
openshift_metrics : generate hawkular-cassandra keys -------------------- 1.02s
Gathering Facts --------------------------------------------------------- 0.83s

Once the playbook finishes ssh into the first master instance (master-0.internal.ocp3.example.com) or the bastion and view the pods in the openshift-infra project.

$ oc get pods -n openshift-infra
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-4q46f   1/1       Running   0          3m
hawkular-metrics-0w46z       1/1       Running   0          3m
heapster-gnhsh               1/1       Running   0          3m

List the router and registry by changing to the default project.

ssh into the first master instance (master-0.internal.ocp3.example.com) or the bastion instance to verify the router and registry in the default project.

The command oc get all is used to observe the values.

$ oc get all -n default
NAME                  DOCKER REPO                                                 TAGS      UPDATED
is/registry-console   docker-registry.default.svc:5000/default/registry-console   v3.6      5 hours ago

NAME                  REVISION   DESIRED   CURRENT   TRIGGERED BY
dc/docker-registry    1          1         1         config
dc/registry-console   1          1         1         config
dc/router             1          1         1         config

NAME                    DESIRED   CURRENT   READY     AGE
rc/docker-registry-1    1         1         1         4h
rc/registry-console-1   1         1         1         4h
rc/router-1             1         1         1         4h

NAME                      HOST/PORT                                         PATH      SERVICES           PORT      TERMINATION   WILDCARD
routes/docker-registry    docker-registry-default.apps.ocp3.example.com              docker-registry    <all>     passthrough   None
routes/registry-console   registry-console-default.apps.ocp3.example.com             registry-console   <all>     passthrough   None

NAME                   CLUSTER-IP      EXTERNAL-IP   PORT(S)                   AGE
svc/docker-registry    172.30.7.33     <none>        5000/TCP                  4h
svc/kubernetes         172.30.0.1      <none>        443/TCP,53/UDP,53/TCP     4h
svc/registry-console   172.30.174.69   <none>        9000/TCP                  4h
svc/router             172.30.83.208   <none>        80/TCP,443/TCP,1936/TCP   4h

NAME                          READY     STATUS    RESTARTS   AGE
po/docker-registry-1-ggpsg    1/1       Running   0          4h
po/registry-console-1-jcgn5   1/1       Running   0          4h
po/router-1-fhrj1             1/1       Running   0          4h
NAME                  DOCKER REPO                                                 TAGS      UPDATED
is/registry-console   docker-registry.default.svc:5000/default/registry-console   v3.6      5 hours ago

NAME                  REVISION   DESIRED   CURRENT   TRIGGERED BY
dc/docker-registry    1          1         1         config
dc/registry-console   1          1         1         config
dc/router             1          1         1         config

NAME                    DESIRED   CURRENT   READY     AGE
rc/docker-registry-1    1         1         1         4h
rc/registry-console-1   1         1         1         4h
rc/router-1             1         1         1         4h

NAME                      HOST/PORT                                         PATH      SERVICES           PORT      TERMINATION   WILDCARD
routes/docker-registry    docker-registry-default.apps.ocp3.example.com              docker-registry    <all>     passthrough   None
routes/registry-console   registry-console-default.apps.ocp3.example.com             registry-console   <all>     passthrough   None

NAME                   CLUSTER-IP      EXTERNAL-IP   PORT(S)                   AGE
svc/docker-registry    172.30.7.33     <none>        5000/TCP                  4h
svc/kubernetes         172.30.0.1      <none>        443/TCP,53/UDP,53/TCP     4h
svc/registry-console   172.30.174.69   <none>        9000/TCP                  4h
svc/router             172.30.83.208   <none>        80/TCP,443/TCP,1936/TCP   4h

NAME                          READY     STATUS    RESTARTS   AGE
po/docker-registry-1-ggpsg    1/1       Running   0          4h
po/registry-console-1-jcgn5   1/1       Running   0          4h
po/router-1-fhrj1             1/1       Running   0          4h

The OpenShift Ansible playbooks configure three infrastructure instances to have the ability to run the registry. Only one registry pod is running due to being backed by a cinder volume. In order to understand the configuration and mapping process of the registry pods, the command 'oc describe' is used. oc describe details how registries are configured and mapped to the Amazon S3 buckets for storage. Using oc describe should help explain how HA works in this environment.

ssh into the first master instance (master-0.internal.ocp3.example.com) or the bastion instance and use the default project to describe the registry.

$ oc describe svc/docker-registry -n default
Name:           docker-registry
Namespace:      default
Labels:         <none>
Annotations:        <none>
Selector:       docker-registry=default
Type:           ClusterIP
IP:         172.30.7.33
Port:           5000-tcp    5000/TCP
Endpoints:      172.45.16.10:5000
Session Affinity:   ClientIP
Events:         <none>

$ oc get pods -n default
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-ggpsg    1/1       Running   0          4h

$ oc describe pod docker-registry-1-ggpsg -n default
Name:           docker-registry-1-ggpsg
Namespace:      default
Security Policy:    hostnetwork
Node:           infra-node-1.internal.ocp3.example.com/172.16.77.13
Start Time:     Tue, 29 Aug 2017 10:32:43 -0400
Labels:         deployment=docker-registry-1
            deploymentconfig=docker-registry
            docker-registry=default
Annotations:        kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicationController","namespace":"default","name":"docker-registry-1","uid":"ea579212-8cc6-11e7-bd52-fa1...
            openshift.io/deployment-config.latest-version=1
            openshift.io/deployment-config.name=docker-registry
            openshift.io/deployment.name=docker-registry-1
            openshift.io/scc=hostnetwork
Status:         Running
IP:         172.45.16.10
Controllers:        ReplicationController/docker-registry-1
Containers:
  registry:
    Container ID:   docker://1ce05dc75aa9403a36f5239c49ab4c4ec11ec8dc58486d1744dd28d0ea44926e
    Image:      openshift3/ose-docker-registry:v3.6.173.0.5
    Image ID:       docker-pullable://registry.access.redhat.com/openshift3/ose-docker-registry@sha256:cb0a9039b7f037a3cca476f3b0f2fe32b9f21a37f15670e270bd01db21b66c19
    Port:       5000/TCP
    State:      Running
      Started:      Tue, 29 Aug 2017 10:34:11 -0400
    Ready:      True
    Restart Count:  0
    Requests:
      cpu:  100m
      memory:   256Mi
    Liveness:   http-get https://:5000/healthz delay=10s timeout=5s period=10s #success=1 #failure=3
    Readiness:  http-get https://:5000/healthz delay=0s timeout=5s period=10s #success=1 #failure=3
    Environment:
      REGISTRY_HTTP_ADDR:                   :5000
      REGISTRY_HTTP_NET:                    tcp
      REGISTRY_HTTP_SECRET:                 HQS/7MfynpqP911Mfqqrmw9rAA1W9DpwhqINZuGUzZk=
      REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA:    false
      OPENSHIFT_DEFAULT_REGISTRY:               docker-registry.default.svc:5000
      REGISTRY_HTTP_TLS_KEY:                    /etc/secrets/registry.key
      REGISTRY_HTTP_TLS_CERTIFICATE:                /etc/secrets/registry.crt
    Mounts:
      /etc/secrets from registry-certificates (rw)
      /registry from registry-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from registry-token-9379b (ro)
Conditions:
  Type      Status
  Initialized   True
  Ready     True
  PodScheduled  True
Volumes:
  registry-storage:
    Type:   PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  registry-claim
    ReadOnly:   false
  registry-certificates:
    Type:   Secret (a volume populated by a Secret)
    SecretName: registry-certificates
    Optional:   false
  registry-token-9379b:
    Type:   Secret (a volume populated by a Secret)
    SecretName: registry-token-9379b
    Optional:   false
QoS Class:  Burstable
Node-Selectors: role=infra
Tolerations:    <none>
Events:     <none>

This section explores the Docker storage on an infrastructure node.

The example below can be performed on any node but for this example the infrastructure instance(infra-node-0.internal.ocp3.example.com) is used.

The output below describing the Backing Filesystem: xfs states that the docker service is not using a loop-back device.

$ sudo docker info
Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 1.12.6
Storage Driver: overlay2
 Backing Filesystem: xfs
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host overlay null
 Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp selinux
Kernel Version: 3.10.0-693.el7.x86_64
Operating System: OpenShift Enterprise
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 4
Total Memory: 15.51 GiB
Name: ip-10-20-5-71.ec2.internal
ID: UJZA:WUBN:J6ZI:4ZJ2:WZVL:EVJX:3JXX:2ZN7:VGFM:UBTG:UMOJ:G54U
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://registry.access.redhat.com/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8
Registries: registry.access.redhat.com (secure), registry.access.redhat.com (secure), docker.io (secure)

Verify three disks are attached to the instance. The disk /dev/vda is used for the OS, /dev/vdb is used for docker storage, and /dev/vdc is used for emptyDir storage for containers that do not use a persistent volume.

$ fdisk -l

Disk /dev/vda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a73bb

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048    41943006    20970479+  83  Linux

Disk /dev/vdb: 32.2 GB, 32212254720 bytes, 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000093ab

   Device Boot      Start         End      Blocks   Id  System
/dev/vdb1            2048    62914559    31456256   8e  Linux LVM

Disk /dev/vdc: 16.1 GB, 16106127360 bytes, 31457280 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes