Chapter 4. Design considerations
This section describes how this reference architecture addresses key design considerations. The recommendations in this reference architecture were developed in conjunction with Red Hat Field Consultants, Engineers, and Product teams.
4.1. Connectivity implementation
You can deploy Red Hat OpenShift Container Platform (RHOCP) to different on-premises architectures.
This reference architecture assumes a direct connection to the Internet, including full DNS resolution for director and all RHOCP nodes.
4.1.1. Deploying RHOCP with a direct Internet connection
Both the Red Hat OpenStack Platform (RHOSP) and RHOCP installation programs can take advantage of direct Internet access and an active subscription to Red Hat products. The installation processes are not self-contained and require access to external sources to retrieve core assets such as:
- RHOSP containers
- RHOCP containers
- RHOCP images
Therefore, an active, working DNS resolution is required.
4.1.2. Deploying RHOCP using a corporate proxy service
A RHOCP installer-provisioned infrastructure deployment supports the implementation of a HTTP or HTTPS cluster-wide proxy at both install time, by using the configuration file, and after an install, by using the cluster-wide egress proxy. For more information on setting up a cluster-wide proxy, see Configuring the cluster-wide proxy in the OpenShift Container Platform Networking guide.
4.1.3. Deploying RHOCP in a restricted network environment
While RHOCP 4 on RHOSP 13 fully supports a restricted network deployment, it requires additional steps and components not tested in this reference architecture. Before implementing this reference architecture in a restricted network environment, read Creating a mirror registry for installation in a restricted network in the OpenShift Container Platform Installing guide, and consult Red Hat support.
4.2. Installation methods and tooling
It is always best to install Red Hat products and product integrations with the tools recommended and supported by Red Hat.
For this reference architecture we install Red Hat OpenStack Platform (RHOSP) by using director. The installation includes the minimum number of Controller nodes, Compute nodes, and Storage nodes required for a high-availability (HA) control plane and storage replication.
We install Red Hat OpenShift Container Platform (RHOCP) using the installer-provisioned infrastructure method. We use the guided install to generate an installation configuration file that we can modify with some basic customisations.
We run the RHOCP installation from the director host by using a non-privileged RHOSP tenant.
4.2.1. Red Hat OpenStack Platform director
RHOSP director is a management and installation tool based on the OpenStack TripleO project. TripleO stands for “OpenStack on OpenStack.” This reference architecture uses director to install RHOSP.
The fundamental concept behind director is that there are two clouds. The first cloud, called the undercloud, is a standalone RHOSP deployment. The undercloud can be deployed on a single physical server or virtual machine. The administrator uses the undercloud’s RHOSP services to define and deploy the production RHOSP cloud. Director is also used for day two management operations, such as applying software updates and upgrading between RHOSP versions.
The second cloud, called the overcloud, is the full-featured production environment deployed by the undercloud. The overcloud is comprised of physical servers with various roles:
- Controller nodes run the OpenStack API endpoints. They also store RHOSP’s stateful configuration database and messaging queues.
- Compute nodes run virtual machine hypervisors. They host the computing resources allocated for user workloads.
- Storage nodes provide block, object, or software defined storage for the user workloads.
RHOCP runs within a project, or tenant, on the overcloud. Each tenant’s RHOCP cluster is isolated from other tenant clusters.
Director is required for all production RHOSP deployments. The configuration settings tested and validated by Red Hat Engineering are embedded into the deployment tooling provided by the director.
An administrator should only provision and manage their RHOSP cloud using director. Customizations should only be performed using director and never added manually.
4.2.2. Red Hat OpenShift Container Platform installation program
The RHOCP 4 installation program supports two methods of deployment: installer-provisioned infrastructure and user-provisioned infrastructure.
The RHOCP 4 installation program is primarily concerned with ensuring a simple path to configuring the infrastructure. Day 2 operations handle the bulk of the RHOCP configuration. To configure the infrastructure, the installation program creates a minimal RHOCP cluster.
22.214.171.124. Installer-provisioned infrastructure
With the introduction of the new RHOCP 4 installer-provisioned infrastructure deployment method, users no longer need to pre-provision RHOSP resources or describe them in Ansible playbooks.
Installer-provisioned infrastructure deployments are prescriptive and limit the amount of variance for the installation profile. This ensures an easier to manage and better supported deployment for the majority of use cases.
126.96.36.199. Generating the installation configuration file
The RHOCP installation program prompts you for your requirements, and uses your responses to create the install configuration file,
install-config.yaml. This configuration file describes the end state of the installation and allows an operator to add additional customizations to their installation, while still remaining within the installer-provisioned infrastructure footprint.
4.3. Implementing high availability
High availability (HA) is a requirement for any production deployment. A crucial consideration for HA is removing single points of failure. This reference architecture is highly available at the Red Hat OpenStack Platform (RHOSP), Red Hat OpenShift Container Platform (RHOCP), and Red Hat Ceph Storage layers.
This reference architecture uses RHOSP director to deploy:
- three RHOSP Controller nodes
- three Red Hat Ceph Storage nodes that use BlueStore as the OSD back end
- three RHOSP Compute nodes
The following diagram shows an example of a highly available RHOCP on RHOSP deployment. Each bare-metal server contains three RHOSP nodes: Controller, Storage, and Compute. RHOCP masters and workers are distributed across the Compute nodes using live migration. NIC’s are bonded and servers use RAID as required.
As shown in the above diagram, you cannot place the master nodes that constitute the control plane on the same RHOSP Compute nodes. To ensure this, you must provide at least three bare-metal servers to host three distinct Compute nodes. You must use live migration to ensure masters are never hosted on the same Compute node. For more information, see the following references:
- Live migration is an administrative function, therefore you need your cloud administrator to perform it.
- You cannot use ephemeral backing storage for master instances, as you cannot live migrate ephemeral backing storage.
This reference architecture represents a minimal starting point. It does not account for all HA possibilities. Therefore, when creating your own HA implementation consider all layers and plan accordingly. For instance, plan to have enough Compute nodes to ensure losing one does not compromise the RHOCP HA best practice of hosting master nodes on different Compute nodes.
4.3.1. RHOSP HA
The default level of HA enforced by the RHOSP director deploys three Controller nodes. Multiple RHOSP services and APIs run simultaneously on all three Controller nodes.
HAproxy load balances connections across the Controller API endpoints to ensure service availability. The Controller nodes also run the RHOSP state database and message bus. A Galera cluster protects the state database. RabbitMQ queues are duplicated across all nodes to protect the message bus.
4.3.2. RHOCP HA
RHOCP is also deployed for HA. The control plane manages the RHOCP cluster. This reference architecture colocates the
etcd state database across the master nodes. As
etcd requires a minimum of three nodes for HA, you must deploy three master nodes.
RHOCP hosts critical control plane components on three master nodes for HA. In RHOCP 4.x, the various infrastructure services, such as router pods, container image registry, metrics and monitoring services, and cluster aggregated logging, no longer run on separate infrastructure nodes by default. Instead, infrastructure services now run on the worker nodes. You can deploy the infrastructure services to separate infrastructure nodes on Day 2. For more information, see Creating infrastructure MachineSets.
Worker nodes are where the actual application workloads run. The masters manage the worker nodes. You should ensure that you have enough worker nodes to handle failures due to excessive load and lost infrastructure.
4.3.3. Storage HA
RHOCP and RHOSP services that use Red Hat Ceph Storage can make use of the Red Hat Ceph Storage built-in HA capabilities. By default, deploying RHOSP with Red Hat Ceph Storage results in integrated storage for Compute services, the Image Service, and Block Storage services.
You can add Ceph OSDs to increase performance and availability. Sizing and customizing your Red Hat Ceph Storage cluster requires special attention and is beyond the scope of this reference architecture. For information on customizing Red Hat Ceph Storage, see the following references:
Consult Red Hat support for guidance on implementing any Red Hat Ceph Storage customizations.
4.3.4. Hardware HA
You need to eliminate single points of failure at the hardware layer. Hardware fault tolerance may include the following:
- RAID on server internal hard disks.
- Redundant server power supplies connected to different power sources.
- Bonded network interfaces connected to redundant network switches.
- If the RHOSP deployment spans multiple racks, you should connect the racks to different PDUs.
A complete description of how to configure hardware fault tolerance is beyond the scope of this document.
Both Red Hat OpenStack Platform (RHOSP) and Red Hat OpenShift Container Platform (RHOCP) have independent and mature storage solutions. Combining these solutions can increase complexity and introduce performance overhead. You need to plan the storage carefully to avoid deploying duplicate storage components. Select storage backends that meet the needs of the RHOCP applications, while minimizing complexity.
4.4.1. Red Hat Ceph Storage
This reference architecture uses Red Hat Ceph Storage. Red Hat Ceph Storage provides scalable block, object, and file storage, and is tightly integrated with the Red Hat cloud software stack.
Red Hat Ceph Storage is a distributed data object store designed to provide excellent performance, reliability, and scalability. At the heart of every Red Hat Ceph Storage deployment is the Ceph Storage Cluster, which consists of two types of daemons:
- Ceph OSD (Object Storage Daemon)
- Ceph OSDs store data on behalf of Red Hat Ceph Storage clients. Ceph OSDs use the CPU and memory of Red Hat Ceph Storage nodes to perform data replication, rebalancing, recovery, monitoring and reporting functions.
- Ceph Monitor
- A Ceph monitor maintains a master copy of the Red Hat Ceph Storage cluster map with the current state of the storage cluster.
The RHOSP director deploys Red Hat Ceph Storage using the
ceph-ansible.yaml environment file. This environment file automatically configures Red Hat Ceph Storage as the backing store for OpenStack Compute (nova) ephemeral disks, OpenStack Block Storage (cinder) volumes, OpenStack Image Service (glance), and OpenStack Telemetry (ceilometer).
4.4.2. Red Hat Ceph Storage backends
The version of Red Hat Ceph Storage you are using determines the storage backend to use:
- For Red Hat Ceph Storage 3.1 or earlier, use FileStore as the backend.
- For Red Hat Ceph Storage 3.2 or later, use BlueStore as the backend.
You should use BlueStore for all new deployments, as it performs best. RHOSP 13 is integrated with Red Hat Ceph Storage 3.3, and director supports the creation of OSDs with BlueStore. RHOSP 16.0 implements Red Hat Ceph Storage 4 and uses BlueStore as the backend by default. RHOSP 16.0 discontinues support for the FileStore backend. All director deployed OSD nodes must run on dedicated physical servers.
For more information, see the following references:
4.4.3. Persistent volumes for RHOCP
The Kubernetes persistent volume (PV) framework allows RHOCP users to request block storage volumes without any knowledge of the underlying storage.
RHOCP supports the OpenStack Block Storage service (cinder) as the provider for this functionality. A RHOCP user can use a PersistentVolume object definition to access an OpenStack Block Storage volume in RHOSP.
The RHOCP installation program automates the creation of an OpenStack Block Storage storage class for dynamic persistent volume creation using the
$ oc get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE standard (default) kubernetes.io/cinder Delete WaitForFirstConsumer true 3d20h
The OpenStack Block Storage storage class access mode is ReadWriteOnce (
RWO). This is sufficient for most use cases. The ReadWriteMany (
RWX) access mode will be supported in a future reference architecture.
For more information, see Persistent storage using Cinder.
4.4.4. Object storage
Director does not consider object storage when installing RHOSP. By default, director installs OpenStack Object Storage (swift) on the Controller nodes to provide an easy-to-use, highly available object storage backend for the OpenStack Image Service (glance). This is not the preferred object storage solution for tenants to use, and access must be explicitly granted by the cloud administrator.
Instead, this reference architecture replaces the default OpenStack Object Storage with a Red Hat Ceph Object Gateway (RGW or
radosgw), which can provide a highly available object storage backend that is compatible with both Amazon S3 and OpenStack Object Storage RESTful APIs.
188.8.131.52. RHOSP registry and object storage
The RHOCP installation program follows the recommended practice for a scaled registry and attempts to use the RHOSP object storage service as the preferred location for the internal RHOSP image registry backend. To do this, the RHOCP installation program checks for an accessible object storage location and uses it. If the RHOCP installer cannot find an accessible object storage location, it follows the recommended best practice for a non-scaled registry and creates a ReadWriteOnce (RWO) OpenStack Block Storage (cinder) volume to use instead. For more information, see Optimizing storage.
Therefore, to ensure the best possible deployment pattern for the registry, Red Hat recommends deploying RHOSP with Red Hat Ceph Object Gateway (RGW).
4.4.5. Image storage
When deploying Red Hat Ceph Storage with director, the default installation of OpenStack Image Service (glance) is backed by Red Hat Ceph Block Devices (RBD). The RHOCP installation program uses the OpenStack Image Service for two purposes:
- To store the RHOCP Ignition files used to bring up the bootstrap node and cluster.
- To store the Red Hat Enterprise Linux CoreOS (RHCOS) image.
You need to plan the networking requirements of your deployment. Select networking backends that meet the needs of the applications running in the containers, while minimizing complexity. Both Red Hat OpenStack Platform (RHOSP) and Red Hat OpenShift Container Platform (RHOCP) have independent and mature networking solutions. However, naively combining the native solutions for each platform can increase complexity and unwanted performance overhead. Consider your workloads as you may want to avoid duplicate networking components. For example, consider the implications of running OpenShift SDN on top of an OpenStack SDN.
4.5.1. OpenStack Networking (neutron)
The OpenStack Networking (neutron) service provides a common abstraction layer for various backend network plugins. RHOSP 13 and 16.0 support the following backend network plugins.
- Both RHOSP 13 and 16.0 deploy an Open vSwitch (OVS) backend for the OpenStack Networking service. OVS is an open-source, multi-layer software switch designed as a virtual switch for use in virtualized server environments. It is used across multiple Red Hat products. OVS is fully tested and supported.
- RHOSP 13 uses the ML2 plugin by default. When using the ML2/OVS plugin, VXLAN encapsulation carries layer 2 traffic between nodes. All L3 traffic is routed through the VXLAN tunnel, to the centralized OpenStack Networking agents running on the Controller nodes.
RHOSP 16.0 uses the Open Virtual Network (OVN) component of Open vSwitch (OVS) by default. OVN is a network virtualization solution that is built into the OVS project. OVN supports the Neutron API, and offers a clean and distributed implementation of the most common networking capabilities, such as bridging, routing, security groups, NAT, and floating IPs.
When using OVN as a backend, GENEVE (Generic Network Virtualization Encapsulation) is used for network encapsulation. The details of GENEVE are beyond the scope of this document, but key advantages of using OVN and GENEVE include increased flexibility through an extendable header format and distribution of L3 traffic by default.
OVS-OVN has been tested and is supported for RHOCP on RHOSP 16.0 deployments, including for this reference architecture when using RHOSP 16.0.
- Third-party SDN solution
- Multiple networking vendors support their own plugins for OpenStack Networking. For future iterations of RHOCP on RHOSP there will be increased partnerships and testing to support as many third party solutions as possible.
RHOSP 13 and 16.0 have different default backend network plugins. This reference architecture uses the default backend for the RHOSP version implemented:
- RHOSP 13: ML2/OVS
- RHOSP 16.0: OVS-OVN
Moving from OVS to OVN on a running RHOSP 13 cloud is only supported when upgrading to RHOSP 16.0.
A provider network bridges a range of externally accessible floating IP addresses to the tenant instances. Remote clients use the floating IP addresses to access exposed RHOCP services, the OpenShift API endpoint, and the web console.
A detailed explanation of the OpenStack Networking service is beyond the scope of this document. For more information, see the following references:
4.5.2. Networking in Red Hat OpenShift Container Platform
Networking in Red Hat OpenShift Container Platform (RHOCP) is a complex topic and a detailed review is beyond the scope of this document. For more information, see Networking.
184.108.40.206. OpenShift SDN
The OpenShift SDN is a fully functioning, mature, and supported SDN solution for RHOCP on RHOSP. For more information, see About OpenShift SDN.
When choosing OpenShift SDN for a RHOCP on RHOSP deployment, consider the general network requirements for the workloads. When running OpenShift SDN on a standard RHOSP deployment, the network will be subject to "double encapsulation". This happens when running an encapsulated OpenShift SDN over an already encapsulated RHOSP network. This is true for both OVS and OVN implementations.
In some cases, running under double encapsulation can cause performance and complexity issues. You should therefore perform a detailed review of the requirements of your workloads to best understand the true impact of double encapsulation as it relates to your own workloads.
This reference architecture uses OpenShift SDN for RHOCP networking.
The Kubernetes Container Networking Interface (CNI) specification provides a mechanism for generic plugin-based networking solutions to be implemented for RHOCP application containers. The default RHOCP networking plugin can be replaced with Kuryr, a kubernetes plugin, which enables the OpenStack Networking service solution to configure networking for OpenShift’s Kubernetes containers, rather than layer the two SDNs on top of each other. The Kuryr plugin makes it possible to run both RHOSP virtual machines and Kubernetes containers on the same RHOSP network. Kuryr achieves this by working with the OpenStack Networking component, along with Load Balancing as a Service (LBaaS) with Octavia. This provides combined networking for pods and services. It is primarily designed for RHOCP clusters running on RHOSP virtual machines. Kuryr aims to improve network performance and has shown positive performance results in testing. For more information, see Accelerate your OpenShift Network Performance on OpenStack with Kuryr.
Kuryr is a complex topic and beyond the scope of this document. For more information, see About Kuryr SDN.
While Kuryr is fully supported for RHOCP 4.4 on RHOSP installations, this reference architecture does not use it. When using RHOSP 13, Kuryr uses many Octavia load balancers, which adds extra overhead to the Compute nodes. You need to consider the following to determine if your installation should use Kuryr:
- Do your workloads require the benefits of removing double encapsulation?
- You need Octavia deployed to your RHOSP cloud.
- Your RHOSP infrastructure has the necessary scale to support Kuryr. For more information, see Resource guidelines for installing OpenShift Container Platform on OpenStack with Kuryr.
RHOSP 16.0 provides a plugin for Octavia,
octavia-ovn, which integrates Octavia with OVN allowing for reduced resource requirements when using Kuryr with RHOCP on RHOSP. A future planned supplement to this reference architecture will cover Kuryr in more detail.
The RHOCP 4 installation program greatly simplifies the DNS requirements seen in previous RHOCP on RHOSP installations. All internal DNS resolution for the cluster, certificate validation, and bootstrapping is provided through a self-hosted, installer-controlled solution that uses the mDNS plugin for CoreDNS. This solution was developed to perform DNS lookups based on discoverable information from mDNS. This plugin will resolve both the etcd-NNN records, as well as the
_etcd-server-ssl._tcp.SRV record. It is also able to resolve the name of the nodes.
With this solution, you do not need to add the IP addresses of the master nodes, worker nodes, and the bootstrap node, either manually or dynamically, to any form of public DNS. The RHOCP on RHOSP installation is entirely self-contained in this respect.
This reference architecture uses the RHOCP installation program parameter
externalDNS to allow the installer-built subnets to offer external DNS resolution to their instances.
externalDNS parameter did not exist for RHOCP 4.3 on RHOSP implementations. If using those versions, you need to provide cloud-wide default DNS resolution, or manually update the subnet after creating it.
For more information, see OpenStack IPI Networking Infrastructure.
With this new solution there is no need to run a self-hosted name server, which was required with the RHOCP 3.11 on RHOSP reference architecture. There is also no requirement for external solutions such as the Designate DNSaaS project.
4.6.1. DNS setup
The RHOCP on RHOSP deployment has the following DNS requirements, both before and after installation:
The installation host must be able to resolve the OpenShift API address, for example, api.ocpra.example.com, to the RHOSP floating IP defined by using the
install-config.yaml. The value you enter for
APIFloatingIPAddressduring the RHOCP 4 guided installation populates the
- The bootstrap node must be able to resolve external, public domains.
- A wild card domain must resolve an additional floating IP. That floating IP must be manually allocated to the ingress port of the installation to allow for access to deployed applications.
220.127.116.11. OpenShift API DNS
The OpenShift API floating IP address needs to be in place for installing the cluster, and to ensure the
api.<cluster name>.<base domain>. address space resolves to it.
Assign this floating IP by using the
lbFloatingIP parameter in the RHOSP section of
install-config.yaml. The value you enter for
APIFloatingIPAddress during the RHOCP 4 installation populates the
platform: openstack: cloud: openstack computeFlavor: m1.large externalNetwork: public lbFloatingIP: 192.168.122.152 octaviaSupport: "1" region: "" trunkSupport: "1"
18.104.22.168. Application DNS
The RHOCP 4 installation program does not configure the floating IP address for the applications running on RHOCP containers. You must manually assign the floating IP address for applications to the correct port after installation.
You need a wildcard entry in your DNS for this IP to resolve the following naming structure:
*.apps.<cluster name>.<base domain>.
22.214.171.124. Bootstrap node
The bootstrap node must be able to resolve external domain names directly. The RHOCP 4 installation program uses this name resolution to connect externally and retrieve the containers required to stand up the bootstrap cluster used to instantiate the production cluster. No other RHOCP nodes require this external resolution.
4.7. Security and authentication
Red Hat OpenStack Platform (RHOSP) and Red Hat OpenShift Container Platform (RHOCP) are open source enterprise software. Both support Role Based Access Control and flexible options for integrating with existing user authentication systems. Additionally, both inherit the Red Hat Enterprise Linux security features, such as SELinux.
Several RHOSP security features are used to secure RHOCP in this reference architecture.
The installer-provisioned infrastructure method creates and manages all RHOSP security groups necessary to secure a RHOSP tenant’s installation from other tenants. These groups isolate the tenant’s RHOCP install from others on the cloud.
A TLS SSL certificate is used to encrypt the OpenStack public API endpoints. This reference architecture uses a self-signed certificate and a local certificate authority on the installation host. This means that when RHOSP TLS SSL public endpoint encryption is enabled, the certificates must be imported to any hosts issuing commands against the endpoints. This reference architecture uses the director host to run the installation program, meaning all interaction with encrypted endpoints is from the director host. This follows the officially supported and documented procedures for RHOSP 13 and 16.0:
At this time Internal TLS (“TLS Everywhere”) is not tested for RHOCP on RHOSP installations.
By default, the RHOSP identity service stores user credentials in its state database, or it can use an LDAP-compliant directory server. Similarly, authentication and authorization in RHOSP can be delegated to another service.
RHOCP master nodes issue tokens to authenticate user requests with the API. RHOCP supports various identity providers, including HTPassword and LDAP.
Beginning with RHOCP 4.x, the RHOCP and RHOSP identity providers are integrated. Administrators can configure the RHOCP keystone identity provider with the OpenStack Identity (keystone) service. This configuration allows users to log in to RHOCP with their OpenStack Identity (keystone) credentials. For more information, see Configuring a Keystone identity provider.
126.96.36.199. RHOSP security
The RHOSP Security and Hardening Guide recommends best practices for securing RHOSP. For more information, see the document for the version of RHOSP you are using:
Many of the security best practices are embedded into a default RHOSP deployment. In the field, we have secured RHOSP to various levels of standard security compliance, such as ANSSI and FedRamp. This reference architecture is not a comprehensive resource for securing RHOSP.
188.8.131.52. RHOCP security
The OpenShift Security Guide provides a comprehensive and detailed look into the many challenges related to security in the cloud. This guide provides assistance in the triaging of security trade-offs and risk, policy enforcement, reporting, and the validation of system configuration. The guide is available to download from the Red Hat Customer Portal.