RHOCP 4 nodes run on Red Hat Enterprise Linux CoreOS (RHCOS). RHCOS provides over-the-air updates on a Red Hat Enterprise Linux (RHEL) kernel to deliver a secure, easily managed container host. In an installer-provisioned infrastructure deployment, and for this reference architecture, RHCOS is the supported operating system for all the RHOCP nodes, and is used by default for worker and master nodes. It is also a RHOCP requirement that the master nodes run RHCOS.

Both the Red Hat OpenStack Platform (RHOSP) and RHOCP installation programs can take advantage of direct Internet access and an active subscription to Red Hat products. The installation processes are not self-contained and require access to external sources to retrieve core assets such as:

Therefore, an active, working DNS resolution is required.

A RHOCP installer-provisioned infrastructure deployment supports the implementation of a HTTP or HTTPS cluster-wide proxy at both install time, by using the configuration file, and after an install, by using the cluster-wide egress proxy. For more information on setting up a cluster-wide proxy, see Configuring the cluster-wide proxy in the OpenShift Container Platform Networking guide.

While RHOCP 4 on RHOSP 13 fully supports a restricted network deployment, it requires additional steps and components not tested in this reference architecture. Before implementing this reference architecture in a restricted network environment, read Creating a mirror registry for installation in a restricted network in the OpenShift Container Platform Installing guide, and consult Red Hat support.

RHOSP director is a management and installation tool based on the OpenStack TripleO project. TripleO stands for “OpenStack on OpenStack.” This reference architecture uses director to install RHOSP.

The fundamental concept behind director is that there are two clouds. The first cloud, called the undercloud, is a standalone RHOSP deployment. The undercloud can be deployed on a single physical server or virtual machine. The administrator uses the undercloud’s RHOSP services to define and deploy the production RHOSP cloud. Director is also used for day two management operations, such as applying software updates and upgrading between RHOSP versions.

The second cloud, called the overcloud, is the full-featured production environment deployed by the undercloud. The overcloud is comprised of physical servers with various roles:

RHOCP runs within a project, or tenant, on the overcloud. Each tenant’s RHOCP cluster is isolated from other tenant clusters.

Director is required for all production RHOSP deployments. The configuration settings tested and validated by Red Hat Engineering are embedded into the deployment tooling provided by the director.

An administrator should only provision and manage their RHOSP cloud using director. Customizations should only be performed using director and never added manually.

The RHOCP 4 installation program supports two methods of deployment: installer-provisioned infrastructure and user-provisioned infrastructure.

The RHOCP 4 installation program is primarily concerned with ensuring a simple path to configuring the infrastructure. Day 2 operations handle the bulk of the RHOCP configuration. To configure the infrastructure, the installation program creates a minimal RHOCP cluster.

The default level of HA enforced by the RHOSP director deploys three Controller nodes. Multiple RHOSP services and APIs run simultaneously on all three Controller nodes.

HAproxy load balances connections across the Controller API endpoints to ensure service availability. The Controller nodes also run the RHOSP state database and message bus. A Galera cluster protects the state database. RabbitMQ queues are duplicated across all nodes to protect the message bus.

RHOCP is also deployed for HA. The control plane manages the RHOCP cluster. This reference architecture colocates the etcd state database across the master nodes. As etcd requires a minimum of three nodes for HA, you must deploy three master nodes.

RHOCP hosts critical control plane components on three master nodes for HA. In RHOCP 4.x, the various infrastructure services, such as router pods, container image registry, metrics and monitoring services, and cluster aggregated logging, no longer run on separate infrastructure nodes by default. Instead, infrastructure services now run on the worker nodes. You can deploy the infrastructure services to separate infrastructure nodes on Day 2. For more information, see Creating infrastructure MachineSets.

Worker nodes are where the actual application workloads run. The masters manage the worker nodes. You should ensure that you have enough worker nodes to handle failures due to excessive load and lost infrastructure.

RHOCP and RHOSP services that use Red Hat Ceph Storage can make use of the Red Hat Ceph Storage built-in HA capabilities. By default, deploying RHOSP with Red Hat Ceph Storage results in integrated storage for Compute services, the Image Service, and Block Storage services.

You can add Ceph OSDs to increase performance and availability. Sizing and customizing your Red Hat Ceph Storage cluster requires special attention and is beyond the scope of this reference architecture. For information on customizing Red Hat Ceph Storage, see the following references:

Consult Red Hat support for guidance on implementing any Red Hat Ceph Storage customizations.

You need to eliminate single points of failure at the hardware layer. Hardware fault tolerance may include the following:

A complete description of how to configure hardware fault tolerance is beyond the scope of this document.

This reference architecture uses Red Hat Ceph Storage. Red Hat Ceph Storage provides scalable block, object, and file storage, and is tightly integrated with the Red Hat cloud software stack.

Red Hat Ceph Storage is a distributed data object store designed to provide excellent performance, reliability, and scalability. At the heart of every Red Hat Ceph Storage deployment is the Ceph Storage Cluster, which consists of two types of daemons:

The RHOSP director deploys Red Hat Ceph Storage using the ceph-ansible.yaml environment file. This environment file automatically configures Red Hat Ceph Storage as the backing store for OpenStack Compute (nova) ephemeral disks, OpenStack Block Storage (cinder) volumes, OpenStack Image Service (glance), and OpenStack Telemetry (ceilometer).

The version of Red Hat Ceph Storage you are using determines the storage backend to use:

You should use BlueStore for all new deployments, as it performs best. RHOSP 13 is integrated with Red Hat Ceph Storage 3.3, and director supports the creation of OSDs with BlueStore. RHOSP 16.0 implements Red Hat Ceph Storage 4 and uses BlueStore as the backend by default. RHOSP 16.0 discontinues support for the FileStore backend. All director deployed OSD nodes must run on dedicated physical servers.

For more information, see the following references:

The Kubernetes persistent volume (PV) framework allows RHOCP users to request block storage volumes without any knowledge of the underlying storage.

RHOCP supports the OpenStack Block Storage service (cinder) as the provider for this functionality. A RHOCP user can use a PersistentVolume object definition to access an OpenStack Block Storage volume in RHOSP.

The RHOCP installation program automates the creation of an OpenStack Block Storage storage class for dynamic persistent volume creation using the kubernetes.io/cinder driver:

$ oc get storageclass
NAME                 PROVISIONER            RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
standard (default)   kubernetes.io/cinder   Delete          WaitForFirstConsumer   true                   3d20h

For more information, see Persistent storage using Cinder.

Director does not consider object storage when installing RHOSP. By default, director installs OpenStack Object Storage (swift) on the Controller nodes to provide an easy-to-use, highly available object storage backend for the OpenStack Image Service (glance). This is not the preferred object storage solution for tenants to use, and access must be explicitly granted by the cloud administrator.

Instead, this reference architecture replaces the default OpenStack Object Storage with a Red Hat Ceph Object Gateway (RGW or radosgw), which can provide a highly available object storage backend that is compatible with both Amazon S3 and OpenStack Object Storage RESTful APIs.

When deploying Red Hat Ceph Storage with director, the default installation of OpenStack Image Service (glance) is backed by Red Hat Ceph Block Devices (RBD). The RHOCP installation program uses the OpenStack Image Service for two purposes:

The OpenStack Networking (neutron) service provides a common abstraction layer for various backend network plugins. RHOSP 13 and 16.0 support the following backend network plugins.

A provider network bridges a range of externally accessible floating IP addresses to the tenant instances. Remote clients use the floating IP addresses to access exposed RHOCP services, the OpenShift API endpoint, and the web console.

A detailed explanation of the OpenStack Networking service is beyond the scope of this document. For more information, see the following references:

Networking in Red Hat OpenShift Container Platform (RHOCP) is a complex topic and a detailed review is beyond the scope of this document. For more information, see Networking.

The RHOCP on RHOSP deployment has the following DNS requirements, both before and after installation:

platform:
  openstack:
    cloud: openstack
    computeFlavor: m1.large
    externalNetwork: public
    lbFloatingIP: 192.168.122.152
    octaviaSupport: "1"
    region: ""
    trunkSupport: "1"

By default, the RHOSP identity service stores user credentials in its state database, or it can use an LDAP-compliant directory server. Similarly, authentication and authorization in RHOSP can be delegated to another service.

RHOCP master nodes issue tokens to authenticate user requests with the API. RHOCP supports various identity providers, including HTPassword and LDAP.

Beginning with RHOCP 4.x, the RHOCP and RHOSP identity providers are integrated. Administrators can configure the RHOCP keystone identity provider with the OpenStack Identity (keystone) service. This configuration allows users to log in to RHOCP with their OpenStack Identity (keystone) credentials. For more information, see Configuring a Keystone identity provider.

As the control plane virtual machines run the etcd key-value store, they need to meet the known resource requirements. Fast disks are the most critical for etcd stability and performance. etcd is sensitive to disk write latency.

For more information on the etcd disk requirements, see Hardware recommendations - Disks.

For this reference architecture, all masters need access to fast disk (SSD or better) to ensure stability and guarantee supportability. Contact your Red Hat account team for assistance on reviewing your disk performance needs.

The full range of options for providing storage to your cluster is outside the scope of this document. The following sections provide a simple analysis of the most popular options.

$ openstack volume type list
+---------------------------------------+---------+-------------------+
 | ID                                   | Name    | Is Public         |
+---------------------------------------+---------+-------------------+
 | 37c2ed76-c9f7-4b0d-9a35-ab02ca6bcbb2 | tripleo | True              |
+---------------------------------------+---------+-------------------+

controlPlane:
  hyperthreading: Enabled
  name: master
  platform:
    openstack:
      type: m1.large
      rootVolume:
        size: 25
        type: tripleo
  replicas: 1

$ openstack volume list
---------------------------------------------------------------------------------------------------------------------------
| ID                                   | Name                  | Status | Size | Attached to                                    |
---------------------------------------------------------------------------------------------------------------------------
| b555cc99-3317-4334-aa19-1184a78ee3f8 | ostest-8vt7s-master-0 | in-use |   25 | Attached to ostest-8vt7s-master-0 on /dev/vda  |
---------------------------------------------------------------------------+-------------------------------------------------

[core@ostest-8vt7s-master-0 ~]$ df -h
Filesystem                            Size  Used Avail Use% Mounted on
devtmpfs                              7.8G     0  7.8G   0% /dev
tmpfs                                 7.9G   84K  7.9G   1% /dev/shm
tmpfs                                 7.9G   18M  7.9G   1% /run
tmpfs                                 7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/mapper/coreos-luks-root-nocrypt   25G   19G  6.2G  75% /sysroot
none                                  2.0G  435M  1.6G  22% /var/lib/etcd
/dev/vda1                             364M   84M  257M  25% /boot
/dev/vda2                             127M  3.0M  124M   3% /boot/efi
tmpfs                                 1.6G  4.0K  1.6G   1% /run/user/1000

compute:
- hyperthreading: Enabled
  name: worker
  platform:
    openstack:
      type: m1.large
      rootVolume:
        size: 25
        type: tripleo-ssd
  replicas: 3
controlPlane:
  hyperthreading: Enabled
  name: master
  platform:
    openstack:
      type: m1.large
      rootVolume:
        size: 25
        type: tripleo-nvme
  replicas: 3

This reference architecture has not been tested with the steps documented in the following procedures:

Contact Red Hat Support if you want to use this functionality with RHOCP on RHOSP.

When deploying RHOCP on RHOSP there is no support for ReadWriteMany (RWX) PersistentVolumes. An RWX volume is a volume that can be mounted as read-write by many nodes. For more information, see Understanding persistent storage.

While there are many PersistentVolume plugins for RHOCP, the only one currently tested for RHOCP on RHOSP is Red Hat OpenStack Block Storage (cinder). OpenStack Block Storage provides a convenient front end for many different storage providers. Using OpenStack Block Storage ensures that the underlying storage requests are managed using the OpenStack APIs, like all other cloud resources. This provides consistent management for infrastructure administrators across all elements of the on-demand infrastructure, and allows easier management and tracking of resource allocations.

OpenStack Block Storage does not support RWX, therefore RHOCP 4.4 on RHOSP deployments are not be able to offer RWX PersistentVolumes. Support for RWX Persistent Volumes is due in a future release of RHOCP on RHOSP through the OpenStack Shared-Filesystems-as-a-Service (manila) CSI plugin.

Red Hat OpenShift Container Storage is persistent software-defined storage, integrated with and optimized for RHOCP. OpenShift Container Storage offers storage to a RHOCP installation through a containerized solution that is run within RHOCP directly.

For RHOCP, OpenShift Container Storage provides a fully self-contained containerized Red Hat Ceph Storage deployment by using underlying storage to create a unique Red Hat Ceph Storage cluster for the RHOCP installation it runs within. OpenShift Container Storage uses the upstream Rook-Ceph operator to do this.

At the time of this writing, using OpenShift Container Storage in a RHOCP on RHOSP environment is not supported. Support for this scenario is planned for a future release of OpenShift Container Storage.

The overcloud deployment consists of:

The endpoint does not use a DNS hostname to access the overcloud through SSL/TLS, as described in the following documents:

The RHOCP installation program requires these certificates in front of IP-based endpoints to be part of the certificate’s Subject Alternative Name (SAN).

The reference architecture also adds the ca.crt.pem file to the local CA trust on the director host, as described in the following reference:

This allows the undercloud to communicate with the overcloud endpoints’ self-signed certificate, and to allow the RHOCP installation program to share the private CA with the necessary components of RHOCP during installation.

This reference architecture deploys Red Hat Ceph Storage across three dedicated, entirely SSD-backed Storage nodes. This is done to ensure that any storage provided to RHOCP nodes in this reference architecture is guaranteed to be fast without any extra configuration requirements. As mentioned earlier, your storage set up may differ and should be considered based on individual requirements and your unique hardware.

This reference architecture deploys Red Hat Ceph Storage with BlueStore as the OSD backend. Red Hat recommends using Red Hat Ceph Storage with BlueStore as the OSD backend for all new deployments of RHOSP 13 using Red Hat Ceph Storage 3.3 and later. Red Hat Ceph Storage with BlueStore as the OSD backend is the default for RHOSP 16.0 using Ceph 4.x, therefore no specific actions are required.

/usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-rgw.yaml

Using this template automatically disables the default OpenStack Object Storage (swift) installation on the Controller nodes.

A RHOSP cloud administrator must explicitly allow access to Object Storage. To allow access to RGW, the administrator grants the tenant the “Member” role.

Director deploys the default Open vSwitch/OVS plugin backend.

The external network is a provider network offering a range of routable addresses that can be added to DNS and are accessible from a client web browser.

This reference architecture uses a HostnameMap and pre-assigns IPs for all node types by using the configuration file ips-from-pool-all.yaml.

However, this also means that using QCOW2 formatted images is not advised. For more information, see Converting an image to RAW format. Instead, to boot VMs from an ephemeral back end or from a volume, the image format must be RAW.

The RHOCP installation program automatically downloads and uses a publicly available QCOW2-formatted image. You can change this by setting the clusterOSImage installation variable to the URL of an external RAW-formatted image, or to the name of an existing, pre-deployed RAW-formatted image already stored in the OpenStack Image Service.

The clusterOSImage variable is not available from the guided installation. You must manually add it to the install-config.yaml file.

We performed the following actions to prepare the environment after overcloud deployment.

$ openstack network create public --external --provider-network-type flat --provider-physical-network datacentre
$ openstack subnet create --dhcp --gateway 192.168.122.1 --network public --subnet-range 192.168.122.0/24 --allocation-pool start=192.168.122.151,end=192.168.122.200 public

$ openstack flavor create --ram 16384 --disk 25 --vcpu 4 --public m1.large

$ openstack project create shiftstack
$ openstack user create --password 'redhat' shiftstack_user

$ openstack role add --user shiftstack_user --project shiftstack _member_

$ openstack role add --user shiftstack_user --project shiftstack Member

$ openstack quota set --cores 28 --ram 120000 shiftstack

The ingress floating IP, for use by the applications running on RHOCP, was allocated to 192.168.122.167 as part of the installation and has an entry in DNS.

To make the RHOCP ingress access available, you need to manually attach the ingress floating IP to the ingress port once the cluster is created, following the guidance in Configuring application access with floating IP addresses.

Run the following command to check the ingress port ID:

$ openstack port list | grep ingress
| 28282230-f90e-4b63-a5c3-a6e2faddbd15 | ocpra-6blbm-ingress-port  | fa:16:3e:4e:b8:bc | ip_address='10.0.0.7', subnet_id='cb3dbf4a-8fb3-4b2e-bc2d-ad12606d849a'  | DOWN   |

Run the following command to attach the port to the IP address:

$ openstack floating ip set --port 28282230-f90e-4b63-a5c3-a6e2faddbd15 192.168.122.167

The RHOCP installation program does not include support for RHOSP anti-affinity rules or availability zones. Therefore, you need to move each master node to its own Compute node after installation. This reference architecture uses live migration to ensure one master per Compute node, as detailed in the following references:

Future releases are planned to support RHOSP anti-affinity rules and availability zones.

This reference architecture follows the procedure described in Verifying cluster status to verify that the cluster is running correctly.

You can access the RHOCP console using the URL associated with the DNS. For this reference architecture the console is deployed to: https://console-openshift-console.apps.ocpra.example.com/