Red Hat OpenStack Platform director is a management and installation tool based on the OpenStack TripleO (OpenStack on OpenStack) project. This reference architecture uses director to install Red Hat OpenStack Platform.

The fundamental concept behind Red Hat OpenStack Platform director is that there are two clouds. The first cloud (called the undercloud) is a standalone OpenStack deployment. The undercloud can be deployed on a single physical server or virtual machine. The administrator uses the undercloud’s OpenStack services to define and deploy the production OpenStack cloud. Director is also used for day two management operations, such as applying software updates and upgrading between OpenStack versions.

Figure 3:

The second cloud (called the overcloud) is the full-featured production environment deployed by the undercloud. The overcloud is comprised of physical servers with various roles:

Figure 3 depicts the relationship between the undercloud, overcloud, and OpenStack nodes. Red Hat OpenShift Container Platform runs on the overcloud.

Once OpenStack is installed, virtual machines are built within OpenStack to run OpenShift. The virtual machines are configured using Ansible roles from openshift-ansible. Openshift-ansible is a set of Ansible playbooks that orchestrate complex deployment tasks including:

Figure 4:

Openshift-ansible deploys OpenShift on OpenStack using two playbooks:

More information on using openshift-ansible can be found in the official OpenShift Container Platform 3.11 documentation.

Openshift-ansible uses a dynamic Ansible inventory script to issue commands against roles. The inventory is automatically generated as output from provision.yml. The following command output shows an OpenShift host list generated from the dynamic inventory:

(shiftstack) [cloud-user@bastion openstack]$ ./inventory.py --list | jq .OSEv3.hosts

[

"master-1.openshift.example.io",

"app-node-1.openshift.example.io",

"app-node-2.openshift.example.io",

"infra-node-1.openshift.example.io",

"master-2.openshift.example.io",

"master-0.openshift.example.io",

"app-node-0.openshift.example.io",

"infra-node-0.openshift.example.io",

"infra-node-2.openshift.example.io"

]

Openshift-ansible can also deploy OpenShift directly onto physical servers using the OpenStack Ironic API. This reference architecture deploys to virtual machines as that is the more common deployment model.

OpenStack Platform Director deploys three controller nodes. Multiple instances of the OpenStack services and APIs run simultaneously on all three controllers. HAproxy load balances connections across the controller API endpoints to ensure service availability. The controllers also run the OpenStack state database and message bus. A Galera cluster protects the state database. RabbitMQ queues are duplicated across all nodes to protect the message bus. This is the default level of high availability enforced by Red Hat OpenStack Platform director.

OpenShift is also deployed for high availability. In this reference architecture the etcd state database is co-located across the master nodes. etcd requires a minimum of three nodes for high availability.

This reference architecture also uses three infrastructure nodes. Infrastructure nodes host OpenShift infrastructure components such as the registry, containers for log aggregation, and metrics. A minimum of three infrastructure nodes are needed for high availability when using a sharded aggregated logging database, and to ensure service interruptions do not occur during a reboot.

In deployments with three or more OpenStack Compute nodes, OpenShift should be configured to use node anti-affinity. This feature helps ensure high availability by preventing virtual machines of the same role from running on the same Compute node. Anti-affinity prevents a single Compute node failure from bringing down the OpenShift cluster.

Figure 5:

Figure 5 shows an example of a highly available OpenShift on OpenStack deployment. The OpenStack compute nodes and Ceph OSDs are grouped into availability zones on a per-rack basis. The virtual machines are all members of the same OpenStack tenant. Affinity rules spread the virtual machines across the physical compute nodes by role.

You should also take care to eliminate single points of failure at the hardware layer. Hardware fault tolerance recommendations implemented in this architecture include:

- type: ovs_bridge

name: br-vlan

members:

- type: linux_bond

name: bond2

bonding_options:

get_param: BondInterfaceOvsOptions

members:

- type: interface

name: nic4

primary: true

- type: interface

name: nic5

...

BondInterfaceOvsOptions: 'mode=1 miimon=150'

If the OpenStack deployment spans multiple racks, the racks should be connected to different PDUs. OpenStack Compute nodes should be divided into availability zones by PDU.

Red Hat Ceph Storage provides scalable cloud storage for this reference architecture. Ceph is tightly integrated with the complete Red Hat cloud software stack. It provides block, object, and file storage capabilities.

Ceph cluster servers are divided into Monitors and Object Storage Device (OSD) nodes. Ceph monitors run the monitor daemon, which keeps a master copy of the cluster topology. Clients query the Ceph monitors in order to read and write data to the cluster. Ceph OSD nodes store data. The data are replicated across the physical disks within the nodes.

Figure 6:

A minimum of three Ceph monitors and three or more Ceph OSD nodes are needed to ensure high availability in production. This is depicted in figure 6. Other than recommending the minimum number of Ceph nodes for high availability, Ceph hardware sizing guidelines are beyond the scope of this document. They vary depending on the workload size and characteristics. Refer to the Ceph Hardware Selection Guide for additional information.

This reference architecture also provides S3-compatible object storage using the Ceph Rados Gateway. A minimum of three Object Gateway nodes are required for external access to Ceph object storage.

The OpenStack Neutron API provides a common abstraction layer for various backend network plugins. Red Hat OpenStack Platform 13 supports multiple backend network plugins, including ML2 OVS, OVN, and commercial SDNs. This reference architecture uses the default ML2 OVS plugin. VXLAN encapsulation carries layer 2 traffic between nodes. All L3 traffic is routed through centralized Neutron agents running on the controller nodes.

A range of externally accessible floating IP addresses are bridged to the tenant instances using a provider network. The floating IP addresses are used by remote clients to access exposed OpenShift services, the OpenShift API endpoint, and the web console.

The OpenShift SDN connects pods across all node hosts. It provides a unified cluster network. OpenShift natively provides several options for inter-pod communication. These include OVS multi-tenant and network policy. More information on OpenShift 3 SDN concepts can be found in the OpenShift 3 SDN documentation.

This reference architecture uses Kuryr for OpenShift networking. Kuryr allows OpenShift pods direct access to the underlying OpenStack Neutron networks. It directly plumbs OpenShift pods and OpenStack virtual machines to the same software defined networks. Kuryr is described further in the integration section of this document.

Internally, OpenShift uses DNS to resolve container names. OpenShift nodes run local dnsmasq services to resolve the internal addresses.

OpenShift also requires external DNS; openshift-ansible resolves OpenShift node hostnames while validating certificate signing requests during installation. The OpenShift console and containerized applications exposed through the external router must also be externally resolvable through DNS.

Openshift-ansible uses floating IP addresses for the externally resolvable OpenShift addresses, this is not ideal for statically configured external DNS. Any delete or redeploy of the OpenShift environment requires a change to the external DNS records. Therefore, in this reference architecture openshift-ansible is configured to dynamically push the OpenShift console and application wildcard addresses to an external server using nsupdate.

OpenStack can provide DNS for OpenShift in multiple different ways. By default, OpenStack is configured for a provider-type environment. Tenants are expected to bring their own DNS. The external DNS server may exist outside of OpenStack, or it can be deployed within the tenant, forwarding addresses it cannot resolve to an external DNS server. The external DNS server address is pushed to the OpenStack instances using the Neutron subnet configuration for the tenant.

Red Hat OpenStack Platform 13 also supports native DNS resolution using Neutron; this is not enabled by default. When enabled, Neutron associates DNS names with ports. The port names are internally resolvable by dnsmasq running on the OpenStack hypervisors. Dnsmasq can be configured to forward addresses it cannot resolve to a remote DNS server or to whatever external DNS server is used by the OpenStack hypervisor. Note that when this approach is used, the default DNS suffix is changed for all OpenStack tenants, which does not cause a problem. Namespace isolation allows the same name to be used in multiple tenants without any issues.

This reference architecture implements the first approach. A BIND server was deployed within the tenant to provide local DNS resolution. “Bring your own DNS” is the more common case. The internal Neutron DNS approach involves changing the default OpenStack configuration deployed by Director. If internal Neutron DNS resolution is not configured during the initial OpenStack deployment, enabling it later can be an invasive process.

By default the OpenStack Identity service (keystone) stores user credentials in its state database, or it can use an LDAP-compliant directory server. Similarly, authentication and authorization in OpenStack can be delegated to another service.

OpenShift master nodes issue tokens to authenticate user requests with the API. OpenShift supports various identity providers including HTPassword and LDAP.

There is no formal integration between OpenShift and OpenStack identity providers and authentication. However, both services can be configured to use the same LDAP directory servers, providing a consistent authentication user experience across the platforms.

The Red Hat OpenStack Platform 13 Security and Hardening Guide recommends practices to help security harden OpenStack. Many of the security practices are embedded into a default Red Hat OpenStack Platform deployment. In the field, Red Hat OpenStack Platform has been security hardened to various levels of standard security compliance, such as ANSI and FedRamp. This reference architecture is not a comprehensive resource for securing OpenStack.

Multiple OpenStack Platform security features are used to help security harden OpenShift in this reference architecture. For example, openshift-ansible creates security groups for every role that only allow port and protocol level access to the services running on that role. In addition, TLS SSL certificates are used to encrypt the OpenStack public API endpoints. The certificate chain can be imported into OpenShift either during installation or manually to allow openshift-ansible to access the encrypted endpoints.

Note that if OpenStack TLS SSL public endpoint encryption is enabled, the certificates must be imported to any hosts issuing commands against the endpoints. This includes the deployment host where openshift-ansible is run.

Barbican is the OpenStack Key Manager service API. It is designed for the secure storage, provisioning, and management of secrets such as passwords, encryption keys, and X.509 certificates. Barbican does not appear in this reference architecture, but it was tested during reference architecture development. Potential use cases for Barbican with OpenShift on Openstack include:

Information related to installing and configuring Barbican can be found in the official OpenStack Platform 13 product documentation.

Chapter 9, Appendix A: Pod placement by role describes where the pods are scheduled to roles in this reference architecture. In some cases, multiple replicas of each pod are scheduled across all nodes of that role. In other cases there are single pods.

For example, the kuryr-controller pod runs on a single infrastructure node. Kuryr-cni (listener) pods run on all nodes regardless of role. They run as a daemonset that links the containers running on a node to the neutron network.

In addition, the service pods listed in the appendix, application nodes also run the application payload pods defined by user’s dpod definitions, deployment configs, and stateful sets.

A Cinder persistent volume is provisioned for each master and node by the openshift-ansible installer. The volume is mounted to /var/lib/docker, as the following example demonstrates.

[openshift@master-0 ~]$ mount | grep dockerlv

/dev/mapper/docker--vol-dockerlv on /var/lib/docker type xfs (rw,relatime,seclabel,attr2,inode64,prjquota)

The container storage volume in this reference architecture is 15 GB and mounted to /dev/vdb on all instances. This is handled automatically by openshift-ansible during instance provisioning. The following example shows the list of Cinder volumes attached to each OpenShift instance.

[cloud-user@bastion ~]$ openstack volume list -f table -c Size -c "Attached to"

+------+---------------------------------------------------------------+

| Size | Attached to |

+------+---------------------------------------------------------------+

| 15 | Attached to infra-node-2.openshift.example.io on /dev/vdb |

| 15 | Attached to infra-node-0.openshift.example.io on /dev/vdb |

| 15 | Attached to infra-node-1.openshift.example.io on /dev/vdb |

| 15 | Attached to master-1.openshift.example.io on /dev/vdb |

| 15 | Attached to master-0.openshift.example.io on /dev/vdb |

| 15 | Attached to master-2.openshift.example.io on /dev/vdb |

| 15 | Attached to app-node-1.openshift.example.io on /dev/vdb |

| 15 | Attached to app-node-0.openshift.example.io on /dev/vdb |

| 15 | Attached to app-node-2.openshift.example.io on /dev/vdb |

+------+---------------------------------------------------------------+

Adjust the size based on the number of size of containers each node will run. You can change the container volume size with the _`openshift_openstack_docker_volume_size Ansible variable in provision.yml.

The openshift-ansible installer automates the creation of a Cinder storage class for dynamic persistent volume creation. The following example shows the storage class:

[openshift@master-0 ~]$ oc get storageclass

NAME PROVISIONER AGE

standard (default) kubernetes.io/cinder 1d

In this reference architecture the storage class allocates persistent volumes for logging and metrics data storage. Persistent volume claims are issued to the storage class and fulfilled during the installation phase of the openshift-ansible installer. For example:

[openshift@master-0 ~]$ oc get pv

CAPACITY ACCESS MODES CLAIM STORAGECLASS

25Gi RWO openshift-infra/metrics-cassandra-1 standard

30Gi RWO openshift-logging/logging-es-0 standard

30Gi RWO openshift-logging/logging-es-1 standard

30Gi RWO openshift-logging/logging-es-2 standard0 ~]$

The Cinder storage class access mode is RWO only; this is sufficient for most use cases. RWX (shared) access mode will be addressed in a future reference architecture. There are multiple ways to address the shared use case with OpenShift Container Platform 3.11 and Red Hat OpenStack Platform 13 but they are beyond the scope of this document.

See the OpenStack Container Platform 3.11 Persistent Storage documentation for more information about storage classes and access modes.

OpenShift Container Platform can build container images from source code, deploy them, and manage their lifecycle. To enable this, OpenShift Container Platform provides an internal, integrated container image registry to locally manage images.

The openshift-ansible installer can configure a persistent volume for internal registry storage or use an S3-compatible object store. OpenStack can provide the persistent volume using Cinder, or object storage through Swift backed by Ceph Rados Gateway. The following code block shows the openshift-registry container automatically created by openshift-ansible when install.yml is run.

(shiftstack) [cloud-user@bastion ~]$ openstack container list -f value

openshift-registry

This reference architecture uses Ceph Rados Gateway. When a Cinder persistent volume is used it is attached to a single infrastructure node which can become a single point of failure.

Note that by default the OpenStack user specified to configure Ceph Rados Gateway in install.yml must have the admin or Member Keystone role.

The OpenShift image registry requires persistent storage to ensure that images are saved in the event that the registry pod needs to migrate to another node. The openshift-ansible installer can automatically create a Cinder volume for internal registry persistent storage, or it can point to a pre-created volume. Alternately, openshift-ansible can use an S-3 compatible object store to hold container images. Either OpenStack Swift or Ceph Rados Gateway are suitable object stores for the internal container registry.

Kuryr is a CNI plugin that uses OpenStack Neutron and Octavia to provide networking for pods and services. It is primarily designed for OpenShift clusters running on OpenStack virtual machines. Figure 11 is a block diagram of a Kuryr architecture.

Figure 11:

Kuryr components are deployed as pods in the kuryr namespace. The kuryr-controller is a single container service pod installed on an infrastructure node as a deployment OpenShift resource type. The kuryr-cni container installs and configures the kuryr CNI driver on each of the OpenShift masters, infrastructure nodes, and compute node as a daemonset.

The Kuryr controller watches the OpenShift API server for pod, service, and namespace create, update, and delete events. It maps the OpenShift API calls to corresponding objects in Neutron and Octavia. This means that every network solution that implements the Neutron trunk port functionality can be used to back OpenShift using Kuryr. This includes open source solutions such as OVS and OVN as well as Neutron-compatible commercial SDNs. That is also the reason the openvswitch firewall driver is a prerequisite instead of the ovs-hybrid firewall driver.

Kuryr is recommended for OpenShift deployments on encapsulated OpenStack tenant networks in order to avoid double encapsulation: running an encapsulated OpenShift SDN over top of an encapsulated OpenStack network, such as whenever VXLAN/GRE/GENEVE are required. Implementing Kuryr does not make sense when provider networks, tenant VLANs, or a third party commercial SDN such as Cisco ACI, or Juniper Contrail is implemented.

The OpenShift Network Policy SDN is recommended for unencapsulated OpenStack networks such as tenant VLANs or provider networks. Kuryr is also not needed when OpenStack Neutron is backed by a commercial SDN with hardware acceleration.

More information on OpenShift Container Platform 3.11 SDN can be found in the Configuring the SDN section of the official documentation.

Octavia is an operator-grade, open source, scalable load balancer. It implements load balancing functionality by launching virtual machine appliances (called amphora) in the OpenStack service project. The amphora run HAproxy services. Figure 12 depicts the Octavia architecture and its relationship to OpenShift.

Figure 12:

The load balancer is Octavia’s front end service. Each load balancer has a listener that maps a virtual IP address and port combination to a pool of pods. The openshift-ansible installer configures Octavia load balancers for the API server, cluster routers, and registry access.

(shiftstack) [stack@undercloud ~]$ openstack loadbalancer list -f table -c name -c vip_address -c provider

+-----------------------------------------------+----------------+----------+

| name | vip_address | provider |

+-----------------------------------------------+----------------+----------+

| openshift-ansible-openshift.example.io-api-lb | 172.30.0.1 | octavia |

| openshift-cluster-router_lb-gc2spptdjh46 | 172.16.1.5 | octavia |

| default/docker-registry | 172.30.95.176 | octavia |

| default/registry-console | 172.30.164.5 | octavia |

| default/router | 172.30.220.98 | octavia |

| openshift-monitoring/grafana | 172.30.242.217 | octavia |

| openshift-web-console/webconsole | 172.30.10.9 | octavia |

| openshift-monitoring/prometheus-k8s | 172.30.134.157 | octavia |

| openshift-console/console | 172.30.235.94 | octavia |

| openshift-metrics-server/metrics-server | 172.30.237.31 | octavia |

| openshift-monitoring/alertmanager-main | 172.30.189.57 | octavia |

| openshift-logging/logging-kibana | 172.30.226.107 | octavia |

| openshift-infra/hawkular-cassandra | 172.30.207.19 | octavia |

+-----------------------------------------------+----------------+----------+

This command output shows the OpenStack Octavia load balancers that correspond to the underlying OpenShift services. Notice that all VIP addresses belong to the OpenShift service network except for the cluster router load balancer, which has a VIP on the deployment network.

Octavia is also used by OpenShift to load balance across pod replica sets. When an OpenShift service is exposed as a LoadBalancer type, an Octavia load balancer is automatically created to load balance client connections to the service pods. This also happens when creating a ClusterIP service. Namespace isolation is enforced when kuryr ClusterIP services are created. It is not enforced when LoadBalancer services are created because they should be externally accessible. A floating IP is added to the load balancer VIP port.