Installation and Configuration

OpenShift Container Platform 3.4

OpenShift Container Platform 3.4 Installation and Configuration

Red Hat OpenShift Documentation Team

Abstract

OpenShift Installation and Configuration topics cover the basics of installing and configuring OpenShift in your environment. Use these topics for the one-time tasks required to get OpenShift up and running.

Chapter 1. Overview

OpenShift Container Platform Installation and Configuration topics cover the basics of installing and configuring OpenShift Container Platform in your environment. Configuration, management, and logging are also covered. Use these topics for the one-time tasks required quickly set up your OpenShift Container Platform environment and configure it based on your organizational needs.

For day to day cluster administrator tasks, see Cluster Administration.

Chapter 2. Installing a Cluster

2.1. Planning

2.1.1. Initial Planning

For production environments, several factors influence installation. Consider the following questions as you read through the documentation:

  • Which installation method do you want to use? The Installation Methods section provides some information about the quick and advanced installation methods.
  • How many hosts do you require in the cluster? The Environment Scenarios section provides multiple examples of Single Master and Multiple Master configurations.
  • How many pods are required in your cluster? The Sizing Considerations section provides limits for nodes and pods so you can calculate how large your environment needs to be.
  • Is high availability required? High availability is recommended for fault tolerance. In this situation, you might aim to use the Multiple Masters Using Native HA example as a basis for your environment.
  • Which installation type do you want to use: RPM or containerized? Both installations provide a working OpenShift Container Platform environment, but you might have a preference for a particular method of installing, managing, and updating your services.

2.1.2. Installation Methods

Both the quick and advanced installations methods are supported for development and production environments. If you want to quickly get OpenShift Container Platform up and running to try out for the first time, use the quick installer and let the interactive CLI guide you through the configuration options relevant to your environment.

For the most control over your cluster’s configuration, you can use the advanced installation method. This method is particularly suited if you are already familiar with Ansible. However, following along with the OpenShift Container Platform documentation should equip you with enough information to reliably deploy your cluster and continue to manage its configuration post-deployment using the provided Ansible playbooks directly.

If you install initially using the quick installer, you can always further tweak your cluster’s configuration and adjust the number of hosts in the cluster using the same installer tool. If you wanted to later switch to using the advanced method, you can create an inventory file for your configuration and carry on that way.

2.1.3. Sizing Considerations

Determine how many nodes and pods you require for your OpenShift Container Platform cluster. Cluster scalability correlates to the number of pods in a cluster environment. That number influences the other numbers in your setup.

The following table provides the maximum sizing limits for nodes and pods:

TypeMaximum

Maximum nodes per cluster

1000

Maximum pods per cluster

120,000

Maximum pods per node

250

Maximum pods per core

10

Important

Oversubscribing the physical resources on a node affects resource guarantees the Kubernetes scheduler makes during pod placement. Learn what measures you can take to avoid memory swapping.

Determine how many pods are expected to fit per node:

Maximum Pods per Cluster / Expected Pods per Node = Total Number of Nodes

Example Scenario

If you want to scope your cluster for 2200 pods per cluster, you would need at least 9 nodes, assuming that there are 250 maximum pods per node:

2200 / 250 = 8.8

If you increase the number of nodes to 20, then the pod distribution changes to 110 pods per node:

2200 / 20 = 110

2.1.4. Environment Scenarios

This section outlines different examples of scenarios for your OpenShift Container Platform environment. Use these scenarios as a basis for planning your own OpenShift Container Platform cluster.

Note

Moving from a single master cluster to multiple masters after installation is not supported.

2.1.4.1. Single Master and Multiple Nodes

The following table describes an example environment for a single master (with embedded etcd) and two nodes:

Host NameInfrastructure Component to Install

master.example.com

Master and node

node1.example.com

Node

node2.example.com

2.1.4.2. Single Master, Multiple etcd, and Multiple Nodes

The following table describes an example environment for a single master, three etcd hosts, and two nodes:

Host NameInfrastructure Component to Install

master.example.com

Master and node

etcd1.example.com

etcd

etcd2.example.com

etcd3.example.com

node1.example.com

Node

node2.example.com

Note

When specifying multiple etcd hosts, external etcd is installed and configured. Clustering of OpenShift Container Platform’s embedded etcd is not supported.

2.1.4.3. Multiple Masters Using Native HA

The following describes an example environment for three masters, one HAProxy load balancer, three etcd hosts, and two nodes using the native HA method:

Host NameInfrastructure Component to Install

master1.example.com

Master (clustered using native HA) and node

master2.example.com

master3.example.com

lb.example.com

HAProxy to load balance API master endpoints

etcd1.example.com

etcd

etcd2.example.com

etcd3.example.com

node1.example.com

Node

node2.example.com

Note

When specifying multiple etcd hosts, external etcd is installed and configured. Clustering of OpenShift Container Platform’s embedded etcd is not supported.

2.1.4.4. Stand-alone Registry

You can also install OpenShift Container Platform to act as a stand-alone registry using the OpenShift Container Platform’s integrated registry. See Installing a Stand-alone Registry for details on this scenario.

2.1.5. RPM vs Containerized

An RPM installation installs all services through package management and configures services to run within the same user space, while a containerized installation installs services using container images and runs separate services in individual containers.

See the Installing on Containerized Hosts topic for more details on configuring your installation to use containerized services.

2.2. Prerequisites

2.2.1. System Requirements

The following sections identify the hardware specifications and system-level requirements of all hosts within your OpenShift Container Platform environment.

2.2.1.1. Red Hat Subscriptions

You must have an active OpenShift Container Platform subscription on your Red Hat account to proceed. If you do not, contact your sales representative for more information.

Important

OpenShift Container Platform 3.4 requires Docker 1.12.

2.2.1.2. Minimum Hardware Requirements

The system requirements vary per host type:

Masters

  • Physical or virtual system, or an instance running on a public or private IaaS.
  • Base OS: RHEL 7.3 or later with "Minimal" installation option, or RHEL Atomic Host 7.3.2 or later. RHEL 7.2 is also supported using Docker 1.12 and its dependencies.
  • 2 vCPU.
  • Minimum 16 GB RAM.
  • Minimum 40 GB hard disk space for the file system containing /var/.

Nodes

  • Physical or virtual system, or an instance running on a public or private IaaS.
  • Base OS: RHEL 7.3 or later with "Minimal" installation option, or RHEL Atomic Host 7.3.2 or later. RHEL 7.2 is also supported using Docker 1.12 and its dependencies.
  • NetworkManager 1.0 or later.
  • 1 vCPU.
  • Minimum 8 GB RAM.
  • Minimum 15 GB hard disk space for the file system containing /var/.
  • An additional minimum 15 GB unallocated space to be used for Docker’s storage back end; see Configuring Docker Storage.

External etcd Nodes

  • Minimum 20 GB hard disk space for etcd data.
  • Consult Hardware Recommendations to properly size your etcd nodes.
  • Currently, OpenShift Container Platform stores image, build, and deployment metadata in etcd. You must periodically prune old resources. If you are planning to leverage a large number of images/builds/deployments, place etcd on machines with large amounts of memory and fast SSD drives.
Important

OpenShift Container Platform only supports servers with x86_64 architecture.

Note

Meeting the /var/ file system sizing requirements in RHEL Atomic Host requires making changes to the default configuration. See Managing Storage in Red Hat Enterprise Linux Atomic Host for instructions on configuring this during or after installation.

2.2.1.3. Production Level Hardware Requirements

Test or sample environments function with the minimum requirements. For production environments, the following recommendations apply:

Master Hosts
In a highly available OpenShift Container Platform cluster with external etcd, a master host should have 1 CPU core and 1.5 GB of memory, on top of the defaults in the table above, for each 1000 pods. Therefore, the recommended size of master host in an OpenShift Container Platform cluster of 2000 pods would be 2 CPU cores and 5 GB of RAM, in addition to the minimum requirements for a master host of 2 CPU cores and 8 GB of RAM.

When planning an environment with multiple masters, a minimum of three etcd hosts as well as a load-balancer between the master hosts, is required.

The OpenShift Container Platform master caches deserialized versions of resources aggressively to ease CPU load. However, in smaller clusters of less than 1000 pods, this cache can waste a lot of memory for negligible CPU load reduction. The default cache size is 50000 entries, which, depending on the size of your resources, can grow to occupy 1 to 2 GB of memory. This cache size can be reduced using the following setting the in /etc/origin/master/master-config.yaml:

kubernetesMasterConfig:
  apiServerArguments:
    deserialization-cache-size:
    - "1000"
Node Hosts
The size of a node host depends on the expected size of its workload. As an OpenShift Container Platform cluster administrator, you will need to calculate the expected workload, then add about 10 percent for overhead. For production environments, allocate enough resources so that node host failure does not affect your maximum capacity.

Use the above with the following table to plan the maximum loads for nodes and pods:

HostSizing Recommendation

Maximum nodes per cluster

1000

Maximum pods per cluster

120000

Maximum pods per nodes

250

Maximum pods per core

10

Important

Oversubscribing the physical resources on a node affects resource guarantees the Kubernetes scheduler makes during pod placement. Learn what measures you can take to avoid memory swapping.

2.2.1.4. Configuring Core Usage

By default, OpenShift Container Platform masters and nodes use all available cores in the system they run on. You can choose the number of cores you want OpenShift Container Platform to use by setting the GOMAXPROCS environment variable.

For example, run the following before starting the server to make OpenShift Container Platform only run on one core:

# export GOMAXPROCS=1

2.2.1.5. SELinux

Security-Enhanced Linux (SELinux) must be enabled on all of the servers before installing OpenShift Container Platform or the installer will fail. Also, configure SELINUXTYPE=targeted in the /etc/selinux/config file:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

2.2.1.6. NTP

You must enable Network Time Protocol (NTP) to prevent masters and nodes in the cluster from going out of sync. Set openshift_clock_enabled to true in the Ansible playbook to enable NTP on masters and nodes in the cluster during Ansible installation.

# openshift_clock_enabled=true

2.2.1.7. Security Warning

OpenShift Container Platform runs containers on your hosts, and in some cases, such as build operations and the registry service, it does so using privileged containers. Furthermore, those containers access your host’s Docker daemon and perform docker build and docker push operations. As such, you should be aware of the inherent security risks associated with performing docker run operations on arbitrary images as they effectively have root access.

For more information, see these articles:

To address these risks, OpenShift Container Platform uses security context constraints that control the actions that pods can perform and what it has the ability to access.

2.2.2. Environment Requirements

The following section defines the requirements of the environment containing your OpenShift Container Platform configuration. This includes networking considerations and access to external services, such as Git repository access, storage, and cloud infrastructure providers.

2.2.2.1. DNS

OpenShift Container Platform requires a fully functional DNS server in the environment. This is ideally a separate host running DNS software and can provide name resolution to hosts and containers running on the platform.

Important

Adding entries into the /etc/hosts file on each host is not enough. This file is not copied into containers running on the platform.

Key components of OpenShift Container Platform run themselves inside of containers and use the following process for name resolution:

  1. By default, containers receive their DNS configuration file (/etc/resolv.conf) from their host.
  2. OpenShift Container Platform then inserts one DNS value into the pods (above the node’s nameserver values). That value is defined in the /etc/origin/node/node-config.yaml file by the dnsIP parameter, which by default is set to the address of the host node because the host is using dnsmasq.
  3. If the dnsIP parameter is omitted from the node-config.yaml file, then the value defaults to the kubernetes service IP, which is the first nameserver in the pod’s /etc/resolv.conf file.

As of OpenShift Container Platform 3.2, dnsmasq is automatically configured on all masters and nodes. The pods use the nodes as their DNS, and the nodes forward the requests. By default, dnsmasq is configured on the nodes to listen on port 53, therefore the nodes cannot run any other type of DNS application.

Note

NetworkManager is required on the nodes in order to populate dnsmasq with the DNS IP addresses.

The following is an example set of DNS records for the Single Master and Multiple Nodes scenario:

master    A   10.64.33.100
node1     A   10.64.33.101
node2     A   10.64.33.102

If you do not have a properly functioning DNS environment, you could experience failure with:

  • Product installation via the reference Ansible-based scripts
  • Deployment of the infrastructure containers (registry, routers)
  • Access to the OpenShift Container Platform web console, because it is not accessible via IP address alone
2.2.2.1.1. Configuring Hosts to Use DNS

Make sure each host in your environment is configured to resolve hostnames from your DNS server. The configuration for hosts' DNS resolution depend on whether DHCP is enabled. If DHCP is:

  • Disabled, then configure your network interface to be static, and add DNS nameservers to NetworkManager.
  • Enabled, then the NetworkManager dispatch script automatically configures DNS based on the DHCP configuration. Optionally, you can add a value to dnsIP in the node-config.yaml file to prepend the pod’s resolv.conf file. The second nameserver is then defined by the host’s first nameserver. By default, this will be the IP address of the node host.

    Note

    For most configurations, do not set the openshift_dns_ip option during the advanced installation of OpenShift Container Platform (using Ansible), because this option overrides the default IP address set by dnsIP.

    Instead, allow the installer to configure each node to use dnsmasq and forward requests to SkyDNS or the external DNS provider. If you do set the openshift_dns_ip option, then it should be set either with a DNS IP that queries SkyDNS first, or to the SkyDNS service or endpoint IP (the Kubernetes service IP).

To properly check that hosts are correctly configured to resolved to your DNS server:

  1. Check the contents of /etc/resolv.conf:

    $ cat /etc/resolv.conf
    # Generated by NetworkManager
    search example.com
    nameserver 10.64.33.1
    # nameserver updated by /etc/NetworkManager/dispatcher.d/99-origin-dns.sh

    In this example, 10.64.33.1 is the address of our DNS server.

  2. Test the DNS servers listed in /etc/resolv.conf are able to resolve to the addresses of all the masters and nodes in your OpenShift Container Platform environment:

    $ dig <node_hostname> @<IP_address> +short

    For example:

    $ dig master.example.com @10.64.33.1 +short
    10.64.33.100
    $ dig node1.example.com @10.64.33.1 +short
    10.64.33.101
2.2.2.1.2. Disabling dnsmasq

If you want to disable dnsmasq (for example, if your /etc/resolv.conf is managed by a configuration tool other than NetworkManager), then set openshift_use_dnsmasq to false in the Ansible playbook.

However, certain containers do not properly move to the next nameserver when the first issues SERVFAIL. Red Hat Enterprise Linux (RHEL)-based containers do not suffer from this, but certain versions of uclibc and musl do.

2.2.2.1.3. Configuring a DNS Wildcard

Optionally, configure a wildcard for the router to use, so that you do not need to update your DNS configuration when new routes are added.

A wildcard for a DNS zone must ultimately resolve to the IP address of the OpenShift Container Platform router.

For example, create a wildcard DNS entry for cloudapps that has a low time-to-live value (TTL) and points to the public IP address of the host where the router will be deployed:

*.cloudapps.example.com. 300 IN  A 192.168.133.2

In almost all cases, when referencing VMs you must use host names, and the host names that you use must match the output of the hostname -f command on each node.

Warning

In your /etc/resolv.conf file on each node host, ensure that the DNS server that has the wildcard entry is not listed as a nameserver or that the wildcard domain is not listed in the search list. Otherwise, containers managed by OpenShift Container Platform may fail to resolve host names properly.

2.2.2.2. Network Access

A shared network must exist between the master and node hosts. If you plan to configure multiple masters for high-availability using the advanced installation method, you must also select an IP to be configured as your virtual IP (VIP) during the installation process. The IP that you select must be routable between all of your nodes, and if you configure using a FQDN it should resolve on all nodes.

2.2.2.2.1. NetworkManager

NetworkManager, a program for providing detection and configuration for systems to automatically connect to the network, is required.

2.2.2.2.2. Required Ports

The OpenShift Container Platform installation automatically creates a set of internal firewall rules on each host using iptables. However, if your network configuration uses an external firewall, such as a hardware-based firewall, you must ensure infrastructure components can communicate with each other through specific ports that act as communication endpoints for certain processes or services.

Ensure the following ports required by OpenShift Container Platform are open on your network and configured to allow access between hosts. Some ports are optional depending on your configuration and usage.

Table 2.1. Node to Node

4789

UDP

Required for SDN communication between pods on separate hosts.

Table 2.2. Nodes to Master

53 or 8053

TCP/UDP

Required for DNS resolution of cluster services (SkyDNS). Installations prior to 3.2 or environments upgraded to 3.2 use port 53. New installations will use 8053 by default so that dnsmasq may be configured.

4789

UDP

Required for SDN communication between pods on separate hosts.

443 or 8443

TCP

Required for node hosts to communicate to the master API, for the node hosts to post back status, to receive tasks, and so on.

Table 2.3. Master to Node

4789

UDP

Required for SDN communication between pods on separate hosts.

10250

TCP

The master proxies to node hosts via the Kubelet for oc commands.

Note

In the following table, (L) indicates the marked port is also used in loopback mode, enabling the master to communicate with itself.

In a single-master cluster:

  • Ports marked with (L) must be open.
  • Ports not marked with (L) need not be open.

In a multiple-master cluster, all the listed ports must be open.

Table 2.4. Master to Master

53 (L) or 8053 (L)

TCP/UDP

Required for DNS resolution of cluster services (SkyDNS). Installations prior to 3.2 or environments upgraded to 3.2 use port 53. New installations will use 8053 by default so that dnsmasq may be configured.

2049 (L)

TCP/UDP

Required when provisioning an NFS host as part of the installer.

2379

TCP

Used for standalone etcd (clustered) to accept changes in state.

2380

TCP

etcd requires this port be open between masters for leader election and peering connections when using standalone etcd (clustered).

4001 (L)

TCP

Used for embedded etcd (non-clustered) to accept changes in state.

4789 (L)

UDP

Required for SDN communication between pods on separate hosts.

Table 2.5. External to Load Balancer

9000

TCP

If you choose the native HA method, optional to allow access to the HAProxy statistics page.

Table 2.6. External to Master

443 or 8443

TCP

Required for node hosts to communicate to the master API, for node hosts to post back status, to receive tasks, and so on.

Table 2.7. IaaS Deployments

22

TCP

Required for SSH by the installer or system administrator.

53 or 8053

TCP/UDP

Required for DNS resolution of cluster services (SkyDNS). Installations prior to 3.2 or environments upgraded to 3.2 use port 53. New installations will use 8053 by default so that dnsmasq may be configured. Only required to be internally open on master hosts.

80 or 443

TCP

For HTTP/HTTPS use for the router. Required to be externally open on node hosts, especially on nodes running the router.

1936

TCP

For router statistics use. Required to be open when running the template router to access statistics, and can be open externally or internally to connections depending on if you want the statistics to be expressed publicly.

4001

TCP

For embedded etcd (non-clustered) use. Only required to be internally open on the master host. 4001 is for server-client connections.

2379 and 2380

TCP

For standalone etcd use. Only required to be internally open on the master host. 2379 is for server-client connections. 2380 is for server-server connections, and is only required if you have clustered etcd.

4789

UDP

For VxLAN use (OpenShift SDN). Required only internally on node hosts.

8443

TCP

For use by the OpenShift Container Platform web console, shared with the API server.

10250

TCP

For use by the Kubelet. Required to be externally open on nodes.

Notes

  • In the above examples, port 4789 is used for User Datagram Protocol (UDP).
  • When deployments are using the SDN, the pod network is accessed via a service proxy, unless it is accessing the registry from the same node the registry is deployed on.
  • OpenShift Container Platform internal DNS cannot be received over SDN. Depending on the detected values of openshift_facts, or if the openshift_ip and openshift_public_ip values are overridden, it will be the computed value of openshift_ip. For non-cloud deployments, this will default to the IP address associated with the default route on the master host. For cloud deployments, it will default to the IP address associated with the first internal interface as defined by the cloud metadata.
  • The master host uses port 10250 to reach the nodes and does not go over SDN. It depends on the target host of the deployment and uses the computed values of openshift_hostname and openshift_public_hostname.

Table 2.8. Aggregated Logging

9200

TCP

For Elasticsearch API use. Required to be internally open on any infrastructure nodes so Kibana is able to retrieve logs for display. It can be externally opened for direct access to Elasticsearch by means of a route. The route can be created using oc expose.

9300

TCP

For Elasticsearch inter-cluster use. Required to be internally open on any infrastructure node so the members of the Elasticsearch cluster may communicate with each other.

2.2.2.3. Persistent Storage

The Kubernetes persistent volume framework allows you to provision an OpenShift Container Platform cluster with persistent storage using networked storage available in your environment. This can be done after completing the initial OpenShift Container Platform installation depending on your application needs, giving users a way to request those resources without having any knowledge of the underlying infrastructure.

The Installation and Configuration Guide provides instructions for cluster administrators on provisioning an OpenShift Container Platform cluster with persistent storage using NFS, GlusterFS, Ceph RBD, OpenStack Cinder, AWS Elastic Block Store (EBS), GCE Persistent Disks, and iSCSI.

2.2.2.4. Cloud Provider Considerations

There are certain aspects to take into consideration if installing OpenShift Container Platform on a cloud provider.

2.2.2.4.1. Configuring a Security Group

When installing on AWS or OpenStack, ensure that you set up the appropriate security groups. These are some ports that you should have in your security groups, without which the installation will fail. You may need more depending on the cluster configuration you want to install. For more information and to adjust your security groups accordingly, see Required Ports for more information.

All OpenShift Container Platform Hosts

  • tcp/22 from host running the installer/Ansible

etcd Security Group

  • tcp/2379 from masters
  • tcp/2380 from etcd hosts

Master Security Group

  • tcp/8443 from 0.0.0.0/0
  • tcp/53 from all OpenShift Container Platform hosts for environments installed prior to or upgraded to 3.2
  • udp/53 from all OpenShift Container Platform hosts for environments installed prior to or upgraded to 3.2
  • tcp/8053 from all OpenShift Container Platform hosts for new environments installed with 3.2
  • udp/8053 from all OpenShift Container Platform hosts for new environments installed with 3.2

Node Security Group

  • tcp/10250 from masters
  • tcp/4789 from nodes

Infrastructure Nodes (ones that can host the OpenShift Container Platform router)

  • tcp/443 from 0.0.0.0/0
  • tcp/80 from 0.0.0.0/0

If configuring ELBs for load balancing the masters and/or routers, you also need to configure Ingress and Egress security groups for the ELBs appropriately.

2.2.2.4.2. Overriding Detected IP Addresses and Host Names

Some deployments require that the user override the detected host names and IP addresses for the hosts. To see the default values, run the openshift_facts playbook:

# ansible-playbook playbooks/byo/openshift_facts.yml

Now, verify the detected common settings. If they are not what you expect them to be, you can override them.

The Advanced Installation topic discusses the available Ansible variables in greater detail.

VariableUsage

hostname

  • Should resolve to the internal IP from the instances themselves.
  • openshift_hostname overrides.

ip

  • Should be the internal IP of the instance.
  • openshift_ip will overrides.

public_hostname

  • Should resolve to the external IP from hosts outside of the cloud.
  • Provider openshift_public_hostname overrides.

public_ip

  • Should be the externally accessible IP associated with the instance.
  • openshift_public_ip overrides.

use_openshift_sdn

  • Should be true unless the cloud is GCE.
  • openshift_use_openshift_sdn overrides.
Warning

If openshift_hostname is set to a value other than the metadata-provided private-dns-name value, the native cloud integration for those providers will no longer work.

In AWS, situations that require overriding the variables include:

VariableUsage

hostname

The user is installing in a VPC that is not configured for both DNS hostnames and DNS resolution.

ip

Possibly if they have multiple network interfaces configured and they want to use one other than the default. You must first set openshift_set_node_ip to True. Otherwise, the SDN would attempt to use the hostname setting or try to resolve the host name for the IP.

public_hostname

  • A master instance where the VPC subnet is not configured for Auto-assign Public IP. For external access to this master, you need to have an ELB or other load balancer configured that would provide the external access needed, or you need to connect over a VPN connection to the internal name of the host.
  • A master instance where metadata is disabled.
  • This value is not actually used by the nodes.

public_ip

  • A master instance where the VPC subnet is not configured for Auto-assign Public IP.
  • A master instance where metadata is disabled.
  • This value is not actually used by the nodes.

If setting openshift_hostname to something other than the metadata-provided private-dns-name value, the native cloud integration for those providers will no longer work.

For EC2 hosts in particular, they must be deployed in a VPC that has both DNS host names and DNS resolution enabled, and openshift_hostname should not be overridden.

2.2.2.4.3. Post-Installation Configuration for Cloud Providers

Following the installation process, you can configure OpenShift Container Platform for AWS, OpenStack, or GCE.

2.3. Host Preparation

2.3.1. Operating System Requirements

A base installation of RHEL 7.3 or later or RHEL Atomic Host 7.3.2 or later is required for master and node hosts. RHEL 7.2 is also supported using Docker 1.12 and its dependencies. See the following documentation for the respective installation instructions, if required:

2.3.2. Host Registration

Each host must be registered using Red Hat Subscription Manager (RHSM) and have an active OpenShift Container Platform subscription attached to access the required packages.

  1. On each host, register with RHSM:

    # subscription-manager register --username=<user_name> --password=<password>
  2. List the available subscriptions:

    # subscription-manager list --available --matches '*OpenShift*'
  3. In the output for the previous command, find the pool ID for an OpenShift Container Platform subscription and attach it:

    # subscription-manager attach --pool=<pool_id>
  4. Disable all yum repositories:

    1. Disable all the enabled RHSM repositories:

      # subscription-manager repos --disable="*"
    2. List the remaining yum repositories and note their names under repo id, if any:

      # yum repolist
    3. Use yum-config-manager to disable the remaining yum repositories:

      # yum-config-manager --disable <repo_id>

      Alternatively, disable all repositories:

       yum-config-manager --disable \*

      Note that this could take a few minutes if you have a large number of available repositories

  5. Enable only the repositories required by OpenShift Container Platform 3.4:

    # subscription-manager repos \
        --enable="rhel-7-server-rpms" \
        --enable="rhel-7-server-extras-rpms" \
        --enable="rhel-7-server-ose-3.4-rpms"

2.3.3. Installing Base Packages

For RHEL 7 systems:

  1. Install the following base packages:

    # yum install wget git net-tools bind-utils iptables-services bridge-utils bash-completion
  2. Update the system to the latest packages:

    # yum update
  3. Install the following package, which provides OpenShift Container Platform utilities and pulls in other tools required by the quick and advanced installation methods, such as Ansible and related configuration files:

    # yum install atomic-openshift-utils
  4. Install the following *-excluder packages on each RHEL 7 system, which helps ensure your systems stay on the correct versions of atomic-openshift and docker packages when you are not trying to upgrade, according to the OpenShift Container Platform version:

    # yum install atomic-openshift-excluder atomic-openshift-docker-excluder
  5. The *-excluder packages add entries to the exclude directive in the host’s /etc/yum.conf file when installed. Run the following command on each host to remove the atomic-openshift packages from the list for the duration of the installation.

    # atomic-openshift-excluder unexclude

For RHEL Atomic Host 7 systems:

  1. Ensure the host is up to date by upgrading to the latest Atomic tree if one is available:

    # atomic host upgrade
  2. After the upgrade is completed and prepared for the next boot, reboot the host:

    # systemctl reboot

2.3.4. Installing Docker

At this point, you should install Docker on all master and node hosts. This allows you to configure your Docker storage options before installing OpenShift Container Platform.

  1. For RHEL 7 systems, install Docker 1.12.

    Note

    On RHEL Atomic Host 7 systems, Docker should already be installed, configured, and running by default.

    The atomic-openshift-docker-excluder package that was installed in Installing Base Packages should ensure that the correct version of Docker is installed in this step:

    # yum install docker

    After the package installation is complete, verify that version 1.12 was installed:

    # docker version
  2. Edit the /etc/sysconfig/docker file and add --insecure-registry 172.30.0.0/16 to the OPTIONS parameter. For example:

    OPTIONS='--selinux-enabled --insecure-registry 172.30.0.0/16'

    If using the Quick Installation method, you can easily script a complete installation from a kickstart or cloud-init setup, change the default configuration file:

    # sed -i '/OPTIONS=.*/c\OPTIONS="--selinux-enabled --insecure-registry 172.30.0.0/16"' \
    /etc/sysconfig/docker
    Note

    The Advanced Installation method automatically changes /etc/sysconfig/docker.

    The --insecure-registry option instructs the Docker daemon to trust any Docker registry on the indicated subnet, rather than requiring a certificate.

    Important

    172.30.0.0/16 is the default value of the servicesSubnet variable in the master-config.yaml file. If this has changed, then the --insecure-registry value in the above step should be adjusted to match, as it is indicating the subnet for the registry to use. Note that the openshift_master_portal_net variable can be set in the Ansible inventory file and used during the advanced installation method to modify the servicesSubnet variable.

    Note

    After the initial OpenShift Container Platform installation is complete, you can choose to secure the integrated Docker registry, which involves adjusting the --insecure-registry option accordingly.

2.3.5. Configuring Docker Storage

Containers and the images they are created from are stored in Docker’s storage back end. This storage is ephemeral and separate from any persistent storage allocated to meet the needs of your applications.

For RHEL Atomic Host

The default storage back end for Docker on RHEL Atomic Host is a thin pool logical volume, which is supported for production environments. You must ensure that enough space is allocated for this volume per the Docker storage requirements mentioned in System Requirements.

If you do not have enough allocated, see Managing Storage with Docker Formatted Containers for details on using docker-storage-setup and basic instructions on storage management in RHEL Atomic Host.

For RHEL

The default storage back end for Docker on RHEL 7 is a thin pool on loopback devices, which is not supported for production use and only appropriate for proof of concept environments. For production environments, you must create a thin pool logical volume and re-configure Docker to use that volume.

You can use the docker-storage-setup script included with Docker to create a thin pool device and configure Docker’s storage driver. This can be done after installing Docker and should be done before creating images or containers. The script reads configuration options from the /etc/sysconfig/docker-storage-setup file and supports three options for creating the logical volume:

  • Option A) Use an additional block device.
  • Option B) Use an existing, specified volume group.
  • Option C) Use the remaining free space from the volume group where your root file system is located.

Option A is the most robust option, however it requires adding an additional block device to your host before configuring Docker storage. Options B and C both require leaving free space available when provisioning your host.

  1. Create the docker-pool volume using one of the following three options:

    • Option A) Use an additional block device.

      In /etc/sysconfig/docker-storage-setup, set DEVS to the path of the block device you wish to use. Set VG to the volume group name you wish to create; docker-vg is a reasonable choice. For example:

      # cat <<EOF > /etc/sysconfig/docker-storage-setup
      DEVS=/dev/vdc
      VG=docker-vg
      EOF

      Then run docker-storage-setup and review the output to ensure the docker-pool volume was created:

      # docker-storage-setup                                                                                                                                                                                                                                [5/1868]
      0
      Checking that no-one is using this disk right now ...
      OK
      
      Disk /dev/vdc: 31207 cylinders, 16 heads, 63 sectors/track
      sfdisk:  /dev/vdc: unrecognized partition table type
      
      Old situation:
      sfdisk: No partitions found
      
      New situation:
      Units: sectors of 512 bytes, counting from 0
      
         Device Boot    Start       End   #sectors  Id  System
      /dev/vdc1          2048  31457279   31455232  8e  Linux LVM
      /dev/vdc2             0         -          0   0  Empty
      /dev/vdc3             0         -          0   0  Empty
      /dev/vdc4             0         -          0   0  Empty
      Warning: partition 1 does not start at a cylinder boundary
      Warning: partition 1 does not end at a cylinder boundary
      Warning: no primary partition is marked bootable (active)
      This does not matter for LILO, but the DOS MBR will not boot this disk.
      Successfully wrote the new partition table
      
      Re-reading the partition table ...
      
      If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
      to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
      (See fdisk(8).)
        Physical volume "/dev/vdc1" successfully created
        Volume group "docker-vg" successfully created
        Rounding up size to full physical extent 16.00 MiB
        Logical volume "docker-poolmeta" created.
        Logical volume "docker-pool" created.
        WARNING: Converting logical volume docker-vg/docker-pool and docker-vg/docker-poolmeta to pool's data and metadata volumes.
        THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
        Converted docker-vg/docker-pool to thin pool.
        Logical volume "docker-pool" changed.
    • Option B) Use an existing, specified volume group.

      In /etc/sysconfig/docker-storage-setup, set VG to the desired volume group. For example:

      # cat <<EOF > /etc/sysconfig/docker-storage-setup
      VG=docker-vg
      EOF

      Then run docker-storage-setup and review the output to ensure the docker-pool volume was created:

      # docker-storage-setup
        Rounding up size to full physical extent 16.00 MiB
        Logical volume "docker-poolmeta" created.
        Logical volume "docker-pool" created.
        WARNING: Converting logical volume docker-vg/docker-pool and docker-vg/docker-poolmeta to pool's data and metadata volumes.
        THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
        Converted docker-vg/docker-pool to thin pool.
        Logical volume "docker-pool" changed.
    • Option C) Use the remaining free space from the volume group where your root file system is located.

      Verify that the volume group where your root file system resides has the desired free space, then run docker-storage-setup and review the output to ensure the docker-pool volume was created:

      # docker-storage-setup
        Rounding up size to full physical extent 32.00 MiB
        Logical volume "docker-poolmeta" created.
        Logical volume "docker-pool" created.
        WARNING: Converting logical volume rhel/docker-pool and rhel/docker-poolmeta to pool's data and metadata volumes.
        THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
        Converted rhel/docker-pool to thin pool.
        Logical volume "docker-pool" changed.
  2. Verify your configuration. You should have a dm.thinpooldev value in the /etc/sysconfig/docker-storage file and a docker-pool logical volume:

    # cat /etc/sysconfig/docker-storage
    DOCKER_STORAGE_OPTIONS=--storage-opt dm.fs=xfs --storage-opt
    dm.thinpooldev=/dev/mapper/docker--vg-docker--pool
    
    # lvs
      LV          VG   Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
      docker-pool rhel twi-a-t---  9.29g             0.00   0.12
    Important

    Before using Docker or OpenShift Container Platform, verify that the docker-pool logical volume is large enough to meet your needs. The docker-pool volume should be 60% of the available volume group and will grow to fill the volume group via LVM monitoring.

  3. Check if Docker is running:

    # systemctl is-active docker
  4. If Docker has not yet been started on the host, enable and start the service:

    # systemctl enable docker
    # systemctl start docker

    If Docker is already running, re-initialize Docker:

    Warning

    This will destroy any containers or images currently on the host.

    # systemctl stop docker
    # rm -rf /var/lib/docker/*
    # systemctl restart docker

    If there is any content in /var/lib/docker/, it must be deleted. Files will be present if Docker has been used prior to the installation of OpenShift Container Platform.

2.3.5.1. Reconfiguring Docker Storage

Should you need to reconfigure Docker storage after having created the docker-pool, you should first remove the docker-pool logical volume. If you are using a dedicated volume group, you should also remove the volume group and any associated physical volumes before reconfiguring docker-storage-setup according to the instructions above.

See Logical Volume Manager Administration for more detailed information on LVM management.

2.3.5.2. Managing Container Logs

Sometimes a container’s log file (the /var/lib/docker/containers/<hash>/<hash>-json.log file on the node where the container is running) can increase to a problematic size. You can manage this by configuring Docker’s json-file logging driver to restrict the size and number of log files.

OptionPurpose

--log-opt max-size

Sets the size at which a new log file is created.

--log-opt max-file

Sets the file on each host to configure the options.

For example, to set the maximum file size to 1MB and always keep the last three log files, edit the /etc/sysconfig/docker file to configure max-size=1M and max-file=3:

OPTIONS='--insecure-registry=172.30.0.0/16 --selinux-enabled --log-opt max-size=1M --log-opt max-file=3'

Next, restart the Docker service:

# systemctl restart docker

2.3.5.3. Viewing Available Container Logs

Container logs are stored in the /var/lib/docker/containers/<hash>/ directory on the node where the container is running. For example:

# ls -lh /var/lib/docker/containers/f088349cceac173305d3e2c2e4790051799efe363842fdab5732f51f5b001fd8/
total 2.6M
-rw-r--r--. 1 root root 5.6K Nov 24 00:12 config.json
-rw-r--r--. 1 root root 649K Nov 24 00:15 f088349cceac173305d3e2c2e4790051799efe363842fdab5732f51f5b001fd8-json.log
-rw-r--r--. 1 root root 977K Nov 24 00:15 f088349cceac173305d3e2c2e4790051799efe363842fdab5732f51f5b001fd8-json.log.1
-rw-r--r--. 1 root root 977K Nov 24 00:15 f088349cceac173305d3e2c2e4790051799efe363842fdab5732f51f5b001fd8-json.log.2
-rw-r--r--. 1 root root 1.3K Nov 24 00:12 hostconfig.json
drwx------. 2 root root    6 Nov 24 00:12 secrets

See Docker’s documentation for additional information on how to Configure Logging Drivers.

2.3.6. Ensuring Host Access

The quick and advanced installation methods require a user that has access to all hosts. If you want to run the installer as a non-root user, passwordless sudo rights must be configured on each destination host.

For example, you can generate an SSH key on the host where you will invoke the installation process:

# ssh-keygen

Do not use a password.

An easy way to distribute your SSH keys is by using a bash loop:

# for host in master.example.com \
    node1.example.com \
    node2.example.com; \
    do ssh-copy-id -i ~/.ssh/id_rsa.pub $host; \
    done

Modify the host names in the above command according to your configuration.

2.3.7. What’s Next?

If you are interested in installing OpenShift Container Platform using the containerized method (optional for RHEL but required for RHEL Atomic Host), see Installing on Containerized Hosts to prepare your hosts.

When you are ready to proceed, you can install OpenShift Container Platform using the quick installation or advanced installation method.

If you are installing a stand-alone registry, continue with Installing a Stand-alone Registry.

2.4. Installing on Containerized Hosts

2.4.1. Overview

An RPM installation installs all services through package management and configures services to run within the same user space, while a containerized installation installs services using container images and runs separate services in individual containers.

The default method for installing OpenShift Container Platform on Red Hat Enterprise Linux (RHEL) uses RPMs.

Important

When targeting a Red Hat Atomic Host system, the containerized method is the only available option, and is automatically selected for you based on the detection of the /run/ostree-booted file.

The following table outlines the differences between the RPM and Containerized methods:

 RPMContainerized

Installation Method

Packages via yum

Container images via docker

Service Management

systemd

docker and systemd units

Operating System

Red Hat Enterprise Linux

Red Hat Enterprise Linux or Red Hat Atomic Host

2.4.2. Install Methods for Containerized Hosts

As with the RPM installation, you can choose between the quick and advanced install methods for the containerized install.

For the quick installation method, you can choose between the RPM or containerized method on a per host basis during the interactive installation, or set the values manually in an installation configuration file.

For the advanced installation method, you can set the Ansible variable containerized=true in an inventory file on a cluster-wide or per host basis.

For the disconnected installation method, to install the etcd container, you can set the Ansible variable osm_etcd_image to be the fully qualified name of the etcd image on your local registry, for example, registry.example.com/rhel7/etcd.

Note

When installing an environment with multiple masters, the load balancer cannot be deployed by the installation process as a container. See Advanced Installation for load balancer requirements using the native HA method.

2.4.3. Required Images

Containerized installations make use of the following images:

  • openshift3/ose
  • openshift3/node
  • openshift3/openvswitch
  • registry.access.redhat.com/rhel7/etcd

By default, all of the above images are pulled from the Red Hat Registry at registry.access.redhat.com.

If you need to use a private registry to pull these images during the installation, you can specify the registry information ahead of time. For the advanced installation method, you can set the following Ansible variables in your inventory file, as required:

cli_docker_additional_registries=<registry_hostname>
cli_docker_insecure_registries=<registry_hostname>
cli_docker_blocked_registries=<registry_hostname>

For the quick installation method, you can export the following environment variables on each target host:

# export OO_INSTALL_ADDITIONAL_REGISTRIES=<registry_hostname>
# export OO_INSTALL_INSECURE_REGISTRIES=<registry_hostname>

Blocked Docker registries cannot currently be specified using the quick installation method.

The configuration of additional, insecure, and blocked Docker registries occurs at the beginning of the installation process to ensure that these settings are applied before attempting to pull any of the required images.

2.4.4. CLI Wrappers

When using containerized installations, a CLI wrapper script is deployed on each master at /usr/local/bin/openshift. The following set of symbolic links are also provided to ease administrative tasks:

Symbolic LinkUsage

/usr/local/bin/oc

Developer CLI

/usr/local/bin/oadm

Administrative CLI

/usr/local/bin/kubectl

Kubernetes CLI

The wrapper spawns a new container on each invocation, so you may notice it run slightly slower than native CLI operations.

The wrapper scripts mount a limited subset of paths:

  • ~/.kube
  • /etc/origin/
  • /tmp/

Be mindful of this when passing in files to be processed by the oc or oadm commands. You may find it easier to redirect the input, for example:

# oc create -f - < my-file.json
Note

The wrapper is intended only to be used to bootstrap an environment. You should install the CLI tools on another host after you have granted cluster-admin privileges to a user. See Managing Role Bindings and Get Started with the CLI for more information.

2.4.5. Starting and Stopping Containers

The installation process creates relevant systemd units which can be used to start, stop, and poll services using normal systemctl commands. For containerized installations, these unit names match those of an RPM installation, with the exception of the etcd service which is named etcd_container.

This change is necessary as currently RHEL Atomic Host ships with the etcd package installed as part of the operating system, so a containerized version is used for the OpenShift Container Platform installation instead. The installation process disables the default etcd service. The etcd package is slated to be removed from RHEL Atomic Host in the future.

2.4.6. File Paths

All OpenShift Container Platform configuration files are placed in the same locations during containerized installation as RPM based installations and will survive os-tree upgrades.

However, the default image stream and template files are installed at /etc/origin/examples/ for containerized installations rather than the standard /usr/share/openshift/examples/, because that directory is read-only on RHEL Atomic Host.

2.4.7. Storage Requirements

RHEL Atomic Host installations normally have a very small root file system. However, the etcd, master, and node containers persist data in the /var/lib/ directory. Ensure that you have enough space on the root file system before installing OpenShift Container Platform. See the System Requirements section for details.

2.4.8. Open vSwitch SDN Initialization

OpenShift SDN initialization requires that the Docker bridge be reconfigured and that Docker is restarted. This complicates the situation when the node is running within a container. When using the Open vSwitch (OVS) SDN, you will see the node start, reconfigure Docker, restart Docker (which restarts all containers), and finally start successfully.

In this case, the node service may fail to start and be restarted a few times, because the master services are also restarted along with Docker. The current implementation uses a workaround which relies on setting the Restart=always parameter in the Docker based systemd units.

2.5. Quick Installation

2.5.1. Overview

The quick installation method allows you to use an interactive CLI utility, the atomic-openshift-installer command, to install OpenShift Container Platform across a set of hosts. This installer can deploy OpenShift Container Platform components on targeted hosts by either installing RPMs or running containerized services.

Important

While RHEL Atomic Host is supported for running containerized OpenShift Container Platform services, the installer is provided by an RPM and not available by default in RHEL Atomic Host. Therefore, it must be run from a Red Hat Enterprise Linux 7 system. The host initiating the installation does not need to be intended for inclusion in the OpenShift Container Platform cluster, but it can be.

This installation method is provided to make the installation experience easier by interactively gathering the data needed to run on each host. The installer is a self-contained wrapper intended for usage on a Red Hat Enterprise Linux (RHEL) 7 system.

In addition to running interactive installations from scratch, the atomic-openshift-installer command can also be run or re-run using a predefined installation configuration file. This file can be used with the installer to:

Alternatively, you can use the advanced installation method for more complex environments.

Note

To install OpenShift Container Platform as a stand-alone registry, see Installing a Stand-alone Registry.

2.5.2. Before You Begin

The installer allows you to install OpenShift Container Platform master and node components on a defined set of hosts.

Note

By default, any hosts you designate as masters during the installation process are automatically also configured as nodes so that the masters are configured as part of the OpenShift Container Platform SDN. The node component on the masters, however, are marked unschedulable, which blocks pods from being scheduled on it. After the installation, you can mark them schedulable if you want.

Before installing OpenShift Container Platform, you must first satisfy the prerequisites on your hosts, which includes verifying system and environment requirements and properly installing and configuring Docker. You must also be prepared to provide or validate the following information for each of your targeted hosts during the course of the installation:

  • User name on the target host that should run the Ansible-based installation (can be root or non-root)
  • Host name
  • Whether to install components for master, node, or both
  • Whether to use the RPM or containerized method
  • Internal and external IP addresses
Important

If you are installing OpenShift Container Platform using the containerized method (optional for RHEL but required for RHEL Atomic Host), see the Installing on Containerized Hosts topic to ensure that you understand the differences between these methods, then return to this topic to continue.

After following the instructions in the Prerequisites topic and deciding between the RPM and containerized methods, you can continue to running an interactive or unattended installation.

2.5.3. Running an Interactive Installation

Note

Ensure you have read through Before You Begin.

You can start the interactive installation by running:

$ atomic-openshift-installer install

Then follow the on-screen instructions to install a new OpenShift Container Platform cluster.

After it has finished, ensure that you back up the ~/.config/openshift/installer.cfg.ymlinstallation configuration file that is created, as it is required if you later want to re-run the installation, add hosts to the cluster, or upgrade your cluster. Then, verify the installation.

2.5.4. Defining an Installation Configuration File

The installer can use a predefined installation configuration file, which contains information about your installation, individual hosts, and cluster. When running an interactive installation, an installation configuration file based on your answers is created for you in ~/.config/openshift/installer.cfg.yml. The file is created if you are instructed to exit the installation to manually modify the configuration or when the installation completes. You can also create the configuration file manually from scratch to perform an unattended installation.

Example 2.1. Installation Configuration File Specification

version: v2 1
variant: openshift-enterprise 2
variant_version: 3.4 3
ansible_log_path: /tmp/ansible.log 4
deployment:
  ansible_ssh_user: root 5
  hosts: 6
  - ip: 10.0.0.1 7
    hostname: master-private.example.com 8
    public_ip: 24.222.0.1 9
    public_hostname: master.example.com 10
    roles: 11
      - master
      - node
    containerized: true 12
    connect_to: 24.222.0.1 13
  - ip: 10.0.0.2
    hostname: node1-private.example.com
    public_ip: 24.222.0.2
    public_hostname: node1.example.com
    node_labels: {'region': 'infra'} 14
    roles:
      - node
    connect_to: 10.0.0.2
  - ip: 10.0.0.3
    hostname: node2-private.example.com
    public_ip: 24.222.0.3
    public_hostname: node2.example.com
    roles:
      - node
    connect_to: 10.0.0.3
  roles: 15
    master:
      <variable_name1>: "<value1>" 16
      <variable_name2>: "<value2>"
    node:
      <variable_name1>: "<value1>" 17
1
The version of this installation configuration file. As of OpenShift Container Platform 3.3, the only valid version here is v2.
2
The OpenShift Container Platform variant to install. For OpenShift Container Platform, set this to openshift-enterprise.
3
A valid version of your selected variant: 3.4, 3.3, 3.2, or 3.1. If not specified, this defaults to the latest version for the specified variant.
4
Defines where the Ansible logs are stored. By default, this is the /tmp/ansible.log file.
5
Defines which user Ansible uses to SSH in to remote systems for gathering facts and for the installation. By default, this is the root user, but you can set it to any user that has sudo privileges.
6
Defines a list of the hosts onto which you want to install the OpenShift Container Platform master and node components.
7 8
Required. Allows the installer to connect to the system and gather facts before proceeding with the install.
9 10
Required for unattended installations. If these details are not specified, then this information is pulled from the facts gathered by the installer, and you are asked to confirm the details. If undefined for an unattended installation, the installation fails.
11
Determines the type of services that are installed. Specified as a list.
12
If set to true, containerized OpenShift Container Platform services are run on target master and node hosts instead of installed using RPM packages. If set to false or unset, the default RPM method is used. RHEL Atomic Host requires the containerized method, and is automatically selected for you based on the detection of the /run/ostree-booted file. See Installing on Containerized Hosts for more details.
13
The IP address that Ansible attempts to connect to when installing, upgrading, or uninstalling the systems. If the configuration file was auto-generated, then this is the value you first enter for the host during that interactive install process.
14
Node labels can optionally be set per-host.
15
Defines a dictionary of roles across the deployment.
16 17
Any ansible variables that should only be applied to hosts assigned a role can be defined. For examples, see Configuring Ansible.

2.5.5. Running an Unattended Installation

Note

Ensure you have read through the Before You Begin.

Unattended installations allow you to define your hosts and cluster configuration in an installation configuration file before running the installer so that you do not have to go through all of the interactive installation questions and answers. It also allows you to resume an interactive installation you may have left unfinished, and quickly get back to where you left off.

To run an unattended installation, first define an installation configuration file at ~/.config/openshift/installer.cfg.yml. Then, run the installer with the -u flag:

$ atomic-openshift-installer -u install

By default in interactive or unattended mode, the installer uses the configuration file located at ~/.config/openshift/installer.cfg.yml if the file exists. If it does not exist, attempting to start an unattended installation fails.

Alternatively, you can specify a different location for the configuration file using the -c option, but doing so will require you to specify the file location every time you run the installation:

$ atomic-openshift-installer -u -c </path/to/file> install

After the unattended installation finishes, ensure that you back up the ~/.config/openshift/installer.cfg.yml file that was used, as it is required if you later want to re-run the installation, add hosts to the cluster, or upgrade your cluster. Then, verify the installation.

2.5.6. Verifying the Installation

After the installation completes:

  1. Verify that the master is started and nodes are registered and reporting in Ready status. On the master host, run the following as root:
# oc get nodes

NAME                        STATUS                     AGE
master.example.com          Ready,SchedulingDisabled   165d
node1.example.com           Ready                      165d
node2.example.com           Ready                      165d

To verify that the web console is installed correctly, use the master host name and the console port number to access the console with a web browser.

For example, for a master host with a hostname of master.openshift.com and using the default port of 8443, the web console would be found at:

https://master.openshift.com:8443/console

Now that the install has been verified, run the following command on each master and node host to add the atomic-openshift packages back to the list of yum excludes on the host:

# atomic-openshift-excluder exclude

Then, see What’s Next for the next steps on configuring your OpenShift Container Platform cluster.

2.5.7. Uninstalling OpenShift Container Platform

You can uninstall OpenShift Container Platform from all hosts in your cluster using the installer’s uninstall command. By default, the installer uses the installation configuration file located at ~/.config/openshift/installer.cfg.yml if the file exists:

$ atomic-openshift-installer uninstall

Alternatively, you can specify a different location for the configuration file using the -c option:

$ atomic-openshift-installer -c </path/to/file> uninstall

See the advanced installation method for more options.

2.5.8. What’s Next?

Now that you have a working OpenShift Container Platform instance, you can:

2.6. Advanced Installation

2.6.1. Overview

A reference configuration implemented using Ansible playbooks is available as the advanced installation method for installing a OpenShift Container Platform cluster. Familiarity with Ansible is assumed, however you can use this configuration as a reference to create your own implementation using the configuration management tool of your choosing.

Important

While RHEL Atomic Host is supported for running containerized OpenShift Container Platform services, the advanced installation method utilizes Ansible, which is not available in RHEL Atomic Host, and must therefore be run from a RHEL 7 system. The host initiating the installation does not need to be intended for inclusion in the OpenShift Container Platform cluster, but it can be.

Alternatively, you can use the quick installation method if you prefer an interactive installation experience.

Note

To install OpenShift Container Platform as a stand-alone registry, see Installing a Stand-alone Registry.

2.6.2. Before You Begin

Before installing OpenShift Container Platform, you must first see the Prerequisites topic to prepare your hosts, which includes verifying system and environment requirements per component type and properly installing and configuring Docker. It also includes installing Ansible version 2.2.0 or later, as the advanced installation method is based on Ansible playbooks and as such requires directly invoking Ansible.

If you are interested in installing OpenShift Container Platform using the containerized method (optional for RHEL but required for RHEL Atomic Host), see Installing on Containerized Hosts to ensure that you understand the differences between these methods, then return to this topic to continue.

After following the instructions in the Prerequisites topic and deciding between the RPM and containerized methods, you can continue in this topic to Configuring Ansible.

2.6.3. Configuring Ansible

The /etc/ansible/hosts file is Ansible’s inventory file for the playbook to use during the installation. The inventory file describes the configuration for your OpenShift Container Platform cluster. You must replace the default contents of the file with your desired configuration.

The following sections describe commonly-used variables to set in your inventory file during an advanced installation, followed by example inventory files you can use as a starting point for your installation. The examples describe various environment topographies, including using multiple masters for high availability. You can choose an example that matches your requirements, modify it to match your own environment, and use it as your inventory file when running the advanced installation.

2.6.3.1. Configuring Host Variables

To assign environment variables to hosts during the Ansible installation, indicate the desired variables in the /etc/ansible/hosts file after the host entry in the [masters] or [nodes] sections. For example:

[masters]
ec2-52-6-179-239.compute-1.amazonaws.com openshift_public_hostname=ose3-master.public.example.com

The following table describes variables for use with the Ansible installer that can be assigned to individual host entries:

Table 2.9. Host Variables

VariablePurpose

openshift_hostname

This variable overrides the internal cluster host name for the system. Use this when the system’s default IP address does not resolve to the system host name.

openshift_public_hostname

This variable overrides the system’s public host name. Use this for cloud installations, or for hosts on networks using a network address translation (NAT).

openshift_ip

This variable overrides the cluster internal IP address for the system. Use this when using an interface that is not configured with the default route.

openshift_public_ip

This variable overrides the system’s public IP address. Use this for cloud installations, or for hosts on networks using a network address translation (NAT).

containerized

If set to true, containerized OpenShift Container Platform services are run on the target master and node hosts instead of installed using RPM packages. If set to false or unset, the default RPM method is used. RHEL Atomic Host requires the containerized method, and is automatically selected for you based on the detection of the /run/ostree-booted file. See Installing on Containerized Hosts for more details. Containerized installations are supported starting in OpenShift Container Platform 3.1.1.

openshift_node_labels

This variable adds labels to nodes during installation. See Configuring Node Host Labels for more details.

openshift_node_kubelet_args

This variable is used to configure kubeletArguments on nodes, such as arguments used in container and image garbage collection, and to specify resources per node. kubeletArguments are key value pairs that are passed directly to the Kubelet that match the Kubelet’s command line arguments. kubeletArguments are not migrated or validated and may become invalid if used. These values override other settings in node configuration which may cause invalid configurations. Example usage: {'image-gc-high-threshold': ['90'],'image-gc-low-threshold': ['80']}.

openshift_hosted_router_selector

Default node selector for automatically deploying router pods. See Configuring Node Host Labels for details.

openshift_registry_selector

Default node selector for automatically deploying registry pods. See Configuring Node Host Labels for details.

openshift_docker_options

This variable configures additional Docker options within /etc/sysconfig/docker, such as options used in Managing Container Logs. Example usage: "--log-driver json-file --log-opt max-size=1M --log-opt max-file=3".

2.6.3.2. Configuring Cluster Variables

To assign environment variables during the Ansible install that apply more globally to your OpenShift Container Platform cluster overall, indicate the desired variables in the /etc/ansible/hosts file on separate, single lines within the [OSEv3:vars] section. For example:

[OSEv3:vars]

openshift_master_identity_providers=[{'name': 'htpasswd_auth',
'login': 'true', 'challenge': 'true',
'kind': 'HTPasswdPasswordIdentityProvider',
'filename': '/etc/origin/master/htpasswd'}]

openshift_master_default_subdomain=apps.test.example.com

The following table describes variables for use with the Ansible installer that can be assigned cluster-wide:

Table 2.10. Cluster Variables

VariablePurpose

ansible_ssh_user

This variable sets the SSH user for the installer to use and defaults to root. This user should allow SSH-based authentication without requiring a password. If using SSH key-based authentication, then the key should be managed by an SSH agent.

ansible_become

If ansible_ssh_user is not root, this variable must be set to true and the user must be configured for passwordless sudo.

containerized

If set to true, containerized OpenShift Container Platform services are run on all target master and node hosts in the cluster instead of installed using RPM packages. If set to false or unset, the default RPM method is used. RHEL Atomic Host requires the containerized method, and is automatically selected for you based on the detection of the /run/ostree-booted file. See Installing on Containerized Hosts for more details. Containerized installations are supported starting in OpenShift Container Platform 3.1.1.

openshift_master_cluster_hostname

This variable overrides the host name for the cluster, which defaults to the host name of the master.

openshift_master_cluster_public_hostname

This variable overrides the public host name for the cluster, which defaults to the host name of the master.

openshift_master_cluster_method

Optional. This variable defines the HA method when deploying multiple masters. Supports the native method. See Multiple Masters for more information.

openshift_rolling_restart_mode

This variable enables rolling restarts of HA masters (i.e., masters are taken down one at a time) when running the upgrade playbook directly. It defaults to services, which allows rolling restarts of services on the masters. It can instead be set to system, which enables rolling, full system restarts and also works for single master clusters.

os_sdn_network_plugin_name

This variable configures which OpenShift SDN plug-in to use for the pod network, which defaults to redhat/openshift-ovs-subnet for the standard SDN plug-in. Set the variable to redhat/openshift-ovs-multitenant to use the multitenant plug-in.

openshift_master_identity_providers

This variable overrides the identity provider, which defaults to Deny All.

openshift_master_named_certificates

These variables are used to configure custom certificates which are deployed as part of the installation. See Configuring Custom Certificates for more information.

openshift_master_overwrite_named_certificates

openshift_master_session_name

These variables override defaults for session options in the OAuth configuration. See Configuring Session Options for more information.

openshift_master_session_max_seconds

openshift_master_session_auth_secrets

openshift_master_session_encryption_secrets

openshift_master_portal_net

This variable configures the subnet in which services will be created within the OpenShift Container Platform SDN. This network block should be private and must not conflict with any existing network blocks in your infrastructure to which pods, nodes, or the master may require access to, or the installation will fail. Defaults to 172.30.0.0/16, and cannot be re-configured after deployment. If changing from the default, avoid 172.16.0.0/16, which the docker0 network bridge uses by default, or modify the docker0 network.

openshift_master_default_subdomain

This variable overrides the default subdomain to use for exposed routes.

openshift_node_proxy_mode

This variable specifies the service proxy mode to use: either iptables for the default, pure-iptables implementation, or userspace for the user space proxy.

osm_default_node_selector

This variable overrides the node selector that projects will use by default when placing pods.

osm_cluster_network_cidr

This variable overrides the SDN cluster network CIDR block. This is the network from which pod IPs are assigned. This network block should be a private block and must not conflict with existing network blocks in your infrastructure to which pods, nodes, or the master may require access. Defaults to 10.128.0.0/14 and cannot be arbitrarily re-configured after deployment, although certain changes to it can be made in the SDN master configuration.

osm_host_subnet_length

This variable specifies the size of the per host subnet allocated for pod IPs by OpenShift Container Platform SDN. Defaults to 9 which means that a subnet of size /23 is allocated to each host; for example, given the default 10.128.0.0/14 cluster network, this will allocate 10.128.0.0/23, 10.128.2.0/23, 10.128.4.0/23, and so on. This cannot be re-configured after deployment.

openshift_use_flannel

This variable enables flannel as an alternative networking layer instead of the default SDN. If enabling flannel, disable the default SDN with the openshift_use_openshift_sdn variable. For more information, see Using Flannel.

openshift_docker_additional_registries

OpenShift Container Platform adds the specified additional registry or registries to the Docker configuration.

openshift_docker_insecure_registries

OpenShift Container Platform adds the specified additional insecure registry or registries to the Docker configuration.

openshift_docker_blocked_registries

OpenShift Container Platform adds the specified blocked registry or registries to the Docker configuration.

openshift_hosted_metrics_public_url

This variable sets the host name for integration with the metrics console. The default is https://hawkular-metrics.{{openshift_master_default_subdomain}}/hawkular/metrics If you alter this variable, ensure the host name is accessible via your router.

2.6.3.3. Configuring a Registry Location

If you are using an image registry other than the default at registry.access.redhat.com, specify the desired registry within the /etc/ansible/hosts file.

oreg_url=example.com/openshift3/ose-${component}:${version}
openshift_examples_modify_imagestreams=true

Table 2.11. Registry Variables

VariablePurpose

oreg_url

Set to the alternate image location. Necessary if you are not using the default registry at registry.access.redhat.com.

openshift_examples_modify_imagestreams

Set to true if pointing to a registry other than the default. Modifies the image stream location to the value of oreg_url.

2.6.3.4. Configuring Global Proxy Options

If your hosts require use of a HTTP or HTTPS proxy in order to connect to external hosts, there are many components that must be configured to use the proxy, including masters, Docker, and builds. Node services only connect to the master API requiring no external access and therefore do not need to be configured to use a proxy.

In order to simplify this configuration, the following Ansible variables can be specified at a cluster or host level to apply these settings uniformly across your environment.

Note

See Configuring Global Build Defaults and Overrides for more information on how the proxy environment is defined for builds.

Table 2.12. Cluster Proxy Variables

VariablePurpose

openshift_http_proxy

This variable specifies the HTTP_PROXY environment variable for masters and the Docker daemon.

openshift_https_proxy

This variable specifices the HTTPS_PROXY environment variable for masters and the Docker daemon.

openshift_no_proxy

This variable is used to set the NO_PROXY environment variable for masters and the Docker daemon. This value should be set to a comma separated list of host names or wildcard host names that should not use the defined proxy. This list will be augmented with the list of all defined OpenShift Container Platform host names by default.

openshift_generate_no_proxy_hosts

This boolean variable specifies whether or not the names of all defined OpenShift hosts and *.cluster.local should be automatically appended to the NO_PROXY list. Defaults to true; set it to false to override this option.

openshift_builddefaults_http_proxy

This variable defines the HTTP_PROXY environment variable inserted into builds using the BuildDefaults admission controller. If openshift_http_proxy is set, this variable will inherit that value; you only need to set this if you want your builds to use a different value.

openshift_builddefaults_https_proxy

This variable defines the HTTPS_PROXY environment variable inserted into builds using the BuildDefaults admission controller. If openshift_https_proxy is set, this variable will inherit that value; you only need to set this if you want your builds to use a different value.

openshift_builddefaults_no_proxy

This variable defines the NO_PROXY environment variable inserted into builds using the BuildDefaults admission controller. If openshift_no_proxy is set, this variable will inherit that value; you only need to set this if you want your builds to use a different value.

openshift_builddefaults_git_http_proxy

This variable defines the HTTP proxy used by git clone operations during a build, defined using the BuildDefaults admission controller. If openshift_builddefaults_http_proxy is set, this variable will inherit that value; you only need to set this if you want your git clone operations to use a different value.

openshift_builddefaults_git_https_proxy

This variable defines the HTTPS proxy used by git clone operations during a build, defined using the BuildDefaults admission controller. If openshift_builddefaults_https_proxy is set, this variable will inherit that value; you only need to set this if you want your git clone operations to use a different value.

2.6.3.5. Configuring Node Host Labels

You can assign labels to node hosts during the Ansible install by configuring the /etc/ansible/hosts file. Labels are useful for determining the placement of pods onto nodes using the scheduler. Other than region=infra (discussed below), the actual label names and values are arbitrary and can be assigned however you see fit per your cluster’s requirements.

To assign labels to a node host during an Ansible install, use the openshift_node_labels variable with the desired labels added to the desired node host entry in the [nodes] section. In the following example, labels are set for a region called primary and a zone called east:

[nodes]
node1.example.com openshift_node_labels="{'region': 'primary', 'zone': 'east'}"

The openshift_router_selector and openshift_registry_selector Ansible settings are set to region=infra by default:

# default selectors for router and registry services
# openshift_router_selector='region=infra'
# openshift_registry_selector='region=infra'

The default router and registry will be automatically deployed if nodes exist that match the selector settings above. For example:

[nodes]
node1.example.com openshift_node_labels="{'region':'infra','zone':'default'}"

2.6.3.6. Marking Masters as Unschedulable Nodes

Any hosts you designate as masters during the installation process should also be configured as nodes by adding them to the [nodes] section so that the masters are configured as part of the OpenShift Container Platform SDN.

However, in order to ensure that your masters are not burdened with running pods, you can make them unschedulable by adding the openshift_scheduleable=false option any node that is also a master. For example:

[nodes]
master.example.com openshift_node_labels="{'region':'infra','zone':'default'}" openshift_schedulable=false

2.6.3.7. Configuring Session Options

Session options in the OAuth configuration are configurable in the inventory file. By default, Ansible populates a sessionSecretsFile with generated authentication and encryption secrets so that sessions generated by one master can be decoded by the others. The default location is /etc/origin/master/session-secrets.yaml, and this file will only be re-created if deleted on all masters.

You can set the session name and maximum number of seconds with openshift_master_session_name and openshift_master_session_max_seconds:

openshift_master_session_name=ssn
openshift_master_session_max_seconds=3600

If provided, openshift_master_session_auth_secrets and openshift_master_encryption_secrets must be equal length.

For openshift_master_session_auth_secrets, used to authenticate sessions using HMAC, it is recommended to use secrets with 32 or 64 bytes:

openshift_master_session_auth_secrets=['DONT+USE+THIS+SECRET+b4NV+pmZNSO']

For openshift_master_encryption_secrets, used to encrypt sessions, secrets must be 16, 24, or 32 characters long, to select AES-128, AES-192, or AES-256:

openshift_master_session_encryption_secrets=['DONT+USE+THIS+SECRET+b4NV+pmZNSO']

2.6.3.8. Configuring Custom Certificates

Custom serving certificates for the public host names of the OpenShift Container Platform API and web console can be deployed during an advanced installation and are configurable in the inventory file.

Note

Custom certificates should only be configured for the host name associated with the publicMasterURL which can be set using openshift_master_cluster_public_hostname. Using a custom serving certificate for the host name associated with the masterURL (openshift_master_cluster_hostname) will result in TLS errors as infrastructure components will attempt to contact the master API using the internal masterURL host.

Certificate and key file paths can be configured using the openshift_master_named_certificates cluster variable:

openshift_master_named_certificates=[{"certfile": "/path/to/custom1.crt", "keyfile": "/path/to/custom1.key"}]

File paths must be local to the system where Ansible will be run. Certificates are copied to master hosts and are deployed within the /etc/origin/master/named_certificates/ directory.

Ansible detects a certificate’s Common Name and Subject Alternative Names. Detected names can be overridden by providing the "names" key when setting openshift_master_named_certificates:

openshift_master_named_certificates=[{"certfile": "/path/to/custom1.crt", "keyfile": "/path/to/custom1.key", "names": ["public-master-host.com"]}]

Certificates configured using openshift_master_named_certificates are cached on masters, meaning that each additional Ansible run with a different set of certificates results in all previously deployed certificates remaining in place on master hosts and within the master configuration file.

If you would like openshift_master_named_certificates to be overwritten with the provided value (or no value), specify the openshift_master_overwrite_named_certificates cluster variable:

openshift_master_overwrite_named_certificates=true

For a more complete example, consider the following cluster variables in an inventory file:

openshift_master_cluster_method=native
openshift_master_cluster_hostname=lb.openshift.com
openshift_master_cluster_public_hostname=custom.openshift.com

To overwrite the certificates on a subsequent Ansible run, you could set the following:

openshift_master_named_certificates=[{"certfile": "/root/STAR.openshift.com.crt", "keyfile": "/root/STAR.openshift.com.key", "names": ["custom.openshift.com"]}]
openshift_master_overwrite_named_certificates=true

2.6.4. Configuring Cluster Metrics

Cluster metrics are not set to automatically deploy by default. Set the following to enable cluster metrics when using the advanced install:

[OSEv3:vars]

openshift_hosted_metrics_deploy=true

2.6.4.1. Metrics Storage

The openshift_hosted_metrics_storage_kind variable must be set in order to use persistent storage. If openshift_hosted_metrics_storage_kind is not set, then cluster metrics data is stored in an EmptyDir volume, which will be deleted when the Cassandra pod terminates.

There are three options for enabling cluster metrics storage when using the advanced install:

Option A - NFS Host Group

When the following variables are set, an NFS volume is created during an advanced install with path <nfs_directory>/<volume_name> on the host within the [nfs] host group. For example, the volume path using these options would be /exports/metrics:

[OSEv3:vars]

openshift_hosted_metrics_storage_kind=nfs
openshift_hosted_metrics_storage_access_modes=['ReadWriteOnce']
openshift_hosted_metrics_storage_nfs_directory=/exports
openshift_hosted_metrics_storage_nfs_options='*(rw,root_squash)'
openshift_hosted_metrics_storage_volume_name=metrics
openshift_hosted_metrics_storage_volume_size=10Gi

Option B - External NFS Host

To use an external NFS volume, one must already exist with a path of <nfs_directory>/<volume_name> on the storage host.

[OSEv3:vars]

openshift_hosted_metrics_storage_kind=nfs
openshift_hosted_metrics_storage_access_modes=['ReadWriteOnce']
openshift_hosted_metrics_storage_host=nfs.example.com
openshift_hosted_metrics_storage_nfs_directory=/exports
openshift_hosted_metrics_storage_volume_name=metrics
openshift_hosted_metrics_storage_volume_size=10Gi

The remote volume path using the following options would be nfs.example.com:/exports/metrics.

Option C - Dynamic

Use the following variable if your OpenShift Container Platform environment supports dynamic volume provisioning for your cloud platform:

[OSEv3:vars]

#openshift_hosted_metrics_storage_kind=dynamic

2.6.5. Single Master Examples

You can configure an environment with a single master and multiple nodes, and either a single embedded etcd or multiple external etcd hosts.

Note

Moving from a single master cluster to multiple masters after installation is not supported.

Single Master and Multiple Nodes

The following table describes an example environment for a single master (with embedded etcd) and two nodes:

Host NameInfrastructure Component to Install

master.example.com

Master and node

node1.example.com

Node

node2.example.com

You can see these example hosts present in the [masters] and [nodes] sections of the following example inventory file:

Example 2.2. Single Master and Multiple Nodes Inventory File

# Create an OSEv3 group that contains the masters and nodes groups
[OSEv3:children]
masters
nodes

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
# SSH user, this user should allow ssh based auth without requiring a password
ansible_ssh_user=root

# If ansible_ssh_user is not root, ansible_become must be set to true
#ansible_become=true

deployment_type=openshift-enterprise

# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider
#openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]

# host group for masters
[masters]
master.example.com

# host group for nodes, includes region info
[nodes]
master.example.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
node1.example.com openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
node2.example.com openshift_node_labels="{'region': 'primary', 'zone': 'west'}"

To use this example, modify the file to match your environment and specifications, and save it as /etc/ansible/hosts.

Single Master, Multiple etcd, and Multiple Nodes

The following table describes an example environment for a single master, three etcd hosts, and two nodes:

Host NameInfrastructure Component to Install

master.example.com

Master and node

etcd1.example.com

etcd

etcd2.example.com

etcd3.example.com

node1.example.com

Node

node2.example.com

Note

When specifying multiple etcd hosts, external etcd is installed and configured. Clustering of OpenShift Container Platform’s embedded etcd is not supported.

You can see these example hosts present in the [masters], [nodes], and [etcd] sections of the following example inventory file:

Example 2.3. Single Master, Multiple etcd, and Multiple Nodes Inventory File

# Create an OSEv3 group that contains the masters, nodes, and etcd groups
[OSEv3:children]
masters
nodes
etcd

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
ansible_ssh_user=root
deployment_type=openshift-enterprise

# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider
#openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]

# host group for masters
[masters]
master.example.com

# host group for etcd
[etcd]
etcd1.example.com
etcd2.example.com
etcd3.example.com

# host group for nodes, includes region info
[nodes]
master.example.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
node1.example.com openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
node2.example.com openshift_node_labels="{'region': 'primary', 'zone': 'west'}"

To use this example, modify the file to match your environment and specifications, and save it as /etc/ansible/hosts.

2.6.6. Multiple Masters Examples

You can configure an environment with multiple masters, multiple etcd hosts, and multiple nodes. Configuring multiple masters for high availability (HA) ensures that the cluster has no single point of failure.

Note

Moving from a single master cluster to multiple masters after installation is not supported.

When configuring multiple masters, the advanced installation supports the following high availability (HA) method:

native

Leverages the native HA master capabilities built into OpenShift Container Platform and can be combined with any load balancing solution. If a host is defined in the [lb] section of the inventory file, Ansible installs and configures HAProxy automatically as the load balancing solution. If no host is defined, it is assumed you have pre-configured a load balancing solution of your choice to balance the master API (port 8443) on all master hosts.

For your pre-configured load balancing solution, you must have:

  • A pre-created load balancer VIP configured for SSL passthrough.
  • A domain name for VIP registered in DNS.

    • The domain name will become the value of both openshift_master_cluster_public_hostname and openshift_master_cluster_hostname in the OpenShift Container Platform installer.

See External Load Balancer Integrations for more information.

Note

For more on the high availability master architecture, see Kubernetes Infrastructure.

Note the following when using the native HA method:

  • The advanced installation method does not currently support multiple HAProxy load balancers in an active-passive setup. See the Load Balancer Administration documentation for post-installation amendments.
  • In a HAProxy setup, controller manager servers run as standalone processes. They elect their active leader with a lease stored in etcd. The lease expires after 30 seconds by default. If a failure happens on an active controller server, it will take up to this number of seconds to elect another leader. The interval can be configured with the osm_controller_lease_ttl variable.

To configure multiple masters, refer to the following section.

Multiple Masters with Multiple etcd, and Using Native HA

The following describes an example environment for three masters, one HAProxy load balancer, three etcd hosts, and two nodes using the native HA method:

Host NameInfrastructure Component to Install

master1.example.com

Master (clustered using native HA) and node

master2.example.com

master3.example.com

lb.example.com

HAProxy to load balance API master endpoints

etcd1.example.com

etcd

etcd2.example.com

etcd3.example.com

node1.example.com

Node

node2.example.com

Note

When specifying multiple etcd hosts, external etcd is installed and configured. Clustering of OpenShift Container Platform’s embedded etcd is not supported.

You can see these example hosts present in the [masters], [etcd], [lb], and [nodes] sections of the following example inventory file:

Example 2.4. Multiple Masters Using HAProxy Inventory File

# Create an OSEv3 group that contains the master, nodes, etcd, and lb groups.
# The lb group lets Ansible configure HAProxy as the load balancing solution.
# Comment lb out if your load balancer is pre-configured.
[OSEv3:children]
masters
nodes
etcd
lb

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
ansible_ssh_user=root
deployment_type=openshift-enterprise

# Uncomment the following to enable htpasswd authentication; defaults to
# DenyAllPasswordIdentityProvider.
#openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]

# Native high availbility cluster method with optional load balancer.
# If no lb group is defined installer assumes that a load balancer has
# been preconfigured. For installation the value of
# openshift_master_cluster_hostname must resolve to the load balancer
# or to one or all of the masters defined in the inventory if no load
# balancer is present.
openshift_master_cluster_method=native
openshift_master_cluster_hostname=openshift-cluster.example.com
openshift_master_cluster_public_hostname=openshift-cluster.example.com

# apply updated node defaults
openshift_node_kubelet_args={'pods-per-core': ['10'], 'max-pods': ['250'], 'image-gc-high-threshold': ['90'], 'image-gc-low-threshold': ['80']}

# override the default controller lease ttl
#osm_controller_lease_ttl=30

# enable ntp on masters to ensure proper failover
openshift_clock_enabled=true

# host group for masters
[masters]
master1.example.com
master2.example.com
master3.example.com

# host group for etcd
[etcd]
etcd1.example.com
etcd2.example.com
etcd3.example.com

# Specify load balancer host
[lb]
lb.example.com

# host group for nodes, includes region info
[nodes]
master[1:3].example.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
node1.example.com openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
node2.example.com openshift_node_labels="{'region': 'primary', 'zone': 'west'}"

To use this example, modify the file to match your environment and specifications, and save it as /etc/ansible/hosts.

Multiple Masters with Master and etcd on the Same Host, and Using Native HA

The following describes an example environment for three masters with etcd on each host, one HAProxy load balancer, and two nodes using the native HA method:

Host NameInfrastructure Component to Install

master1.example.com

Master (clustered using native HA) and node with etcd on each host

master2.example.com

master3.example.com

lb.example.com

HAProxy to load balance API master endpoints

node1.example.com

Node

node2.example.com

You can see these example hosts present in the [masters], [etcd], [lb], and [nodes] sections of the following example inventory file:

# Create an OSEv3 group that contains the master, nodes, etcd, and lb groups.
# The lb group lets Ansible configure HAProxy as the load balancing solution.
# Comment lb out if your load balancer is pre-configured.
[OSEv3:children]
masters
nodes
etcd
lb

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
ansible_ssh_user=root
deployment_type=openshift-enterprise

# Uncomment the following to enable htpasswd authentication; defaults to
# DenyAllPasswordIdentityProvider.
#openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]

# Native high availbility cluster method with optional load balancer.
# If no lb group is defined installer assumes that a load balancer has
# been preconfigured. For installation the value of
# openshift_master_cluster_hostname must resolve to the load balancer
# or to one or all of the masters defined in the inventory if no load
# balancer is present.
openshift_master_cluster_method=native
openshift_master_cluster_hostname=openshift-cluster.example.com
openshift_master_cluster_public_hostname=openshift-cluster.example.com

# override the default controller lease ttl
#osm_controller_lease_ttl=30

# host group for masters
[masters]
master1.example.com
master2.example.com
master3.example.com

# host group for etcd
[etcd]
master1.example.com
master2.example.com
master3.example.com

# Specify load balancer host
[lb]
lb.example.com

# host group for nodes, includes region info
[nodes]
master[1:3].example.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
node1.example.com openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
node2.example.com openshift_node_labels="{'region': 'primary', 'zone': 'west'}"

To use this example, modify the file to match your environment and specifications, and save it as /etc/ansible/hosts.

2.6.7. Running the Advanced Installation

After you have configured Ansible by defining an inventory file in /etc/ansible/hosts, you can run the advanced installation using the following playbook:

# ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml

If for any reason the installation fails, before re-running the installer, see Known Issues to check for any specific instructions or workarounds.

2.6.8. Verifying the Installation

After the installation completes:

  1. Verify that the master is started and nodes are registered and reporting in Ready status. On the master host, run the following as root:
# oc get nodes

NAME                        STATUS                     AGE
master.example.com          Ready,SchedulingDisabled   165d
node1.example.com           Ready                      165d
node2.example.com           Ready                      165d

To verify that the web console is installed correctly, use the master host name and the console port number to access the console with a web browser.

For example, for a master host with a hostname of master.openshift.com and using the default port of 8443, the web console would be found at:

https://master.openshift.com:8443/console

Now that the install has been verified, run the following command on each master and node host to add the atomic-openshift packages back to the list of yum excludes on the host:

# atomic-openshift-excluder exclude
Note

The default port for the console is 8443. If this was changed during the installation, the port can be found at openshift_master_console_port in the /etc/ansible/hosts file.

Multiple etcd Hosts

If you installed multiple etcd hosts:

  1. On a master host, verify the etcd cluster health, substituting for the FQDNs of your etcd hosts in the following:

    # etcdctl -C \
        https://etcd1.example.com:2379,https://etcd2.example.com:2379,https://etcd3.example.com:2379 \
        --ca-file=/etc/origin/master/master.etcd-ca.crt \
        --cert-file=/etc/origin/master/master.etcd-client.crt \
        --key-file=/etc/origin/master/master.etcd-client.key cluster-health
  2. Also verify the member list is correct:

    # etcdctl -C \
        https://etcd1.example.com:2379,https://etcd2.example.com:2379,https://etcd3.example.com:2379 \
        --ca-file=/etc/origin/master/master.etcd-ca.crt \
        --cert-file=/etc/origin/master/master.etcd-client.crt \
        --key-file=/etc/origin/master/master.etcd-client.key member list

Multiple Masters Using HAProxy

If you installed multiple masters using HAProxy as a load balancer, browse to the following URL according to your [lb] section definition and check HAProxy’s status:

http://<lb_hostname>:9000

You can verify your installation by consulting the HAProxy Configuration documentation.

2.6.9. Uninstalling OpenShift Container Platform

You can uninstall OpenShift Container Platform hosts in your cluster by running the uninstall.yml playbook. This playbook deletes OpenShift Container Platform content installed by Ansible, including:

  • Configuration
  • Containers
  • Default templates and image streams
  • Images
  • RPM packages

The playbook will delete content for any hosts defined in the inventory file that you specify when running the playbook. If you want to uninstall OpenShift Container Platform across all hosts in your cluster, run the playbook using the inventory file you used when installing OpenShift Container Platform initially or ran most recently:

# ansible-playbook [-i /path/to/file] \
    /usr/share/ansible/openshift-ansible/playbooks/adhoc/uninstall.yml

2.6.9.1. Uninstalling Nodes

You can also uninstall node components from specific hosts using the uninstall.yml playbook while leaving the remaining hosts and cluster alone:

Warning

This method should only be used when attempting to uninstall specific node hosts and not for specific masters or etcd hosts, which would require further configuration changes within the cluster.

  1. First follow the steps in Deleting Nodes to remove the node object from the cluster, then continue with the remaining steps in this procedure.
  2. Create a different inventory file that only references those hosts. For example, to only delete content from one node:

    [OSEv3:children]
    nodes 1
    
    [OSEv3:vars]
    ansible_ssh_user=root
    deployment_type=openshift-enterprise
    
    [nodes]
    node3.example.com openshift_node_labels="{'region': 'primary', 'zone': 'west'}" 2
    1
    Only include the sections that pertain to the hosts you are interested in uninstalling.
    2
    Only include hosts that you want to uninstall.
  3. Specify that new inventory file using the -i option when running the uninstall.yml playbook:

    # ansible-playbook -i /path/to/new/file \
        /usr/share/ansible/openshift-ansible/playbooks/adhoc/uninstall.yml

When the playbook completes, all OpenShift Container Platform content should be removed from any specified hosts.

2.6.10. Known Issues

The following are known issues for specified installation configurations.

Multiple Masters

  • On failover, it is possible for the controller manager to overcorrect, which causes the system to run more pods than what was intended. However, this is a transient event and the system does correct itself over time. See https://github.com/kubernetes/kubernetes/issues/10030 for details.
  • On failure of the Ansible installer, you must start from a clean operating system installation. If you are using virtual machines, start from a fresh image. If you are using bare metal machines, run the following on all hosts:

    # yum -y remove openshift openshift-* etcd docker
    
    # rm -rf /etc/origin /var/lib/openshift /etc/etcd \
        /var/lib/etcd /etc/sysconfig/atomic-openshift* /etc/sysconfig/docker* \
        /root/.kube/config /etc/ansible/facts.d /usr/share/openshift

2.6.11. What’s Next?

Now that you have a working OpenShift Container Platform instance, you can:

2.7. Disconnected Installation

2.7.1. Overview

Frequently, portions of a datacenter may not have access to the Internet, even via proxy servers. Installing OpenShift Container Platform in these environments is considered a disconnected installation.

An OpenShift Container Platform disconnected installation differs from a regular installation in two primary ways:

  • The OpenShift Container Platform software channels and repositories are not available via Red Hat’s content distribution network.
  • OpenShift Container Platform uses several containerized components. Normally, these images are pulled directly from Red Hat’s Docker registry. In a disconnected environment, this is not possible.

A disconnected installation ensures the OpenShift Container Platform software is made available to the relevant servers, then follows the same installation process as a standard connected installation. This topic additionally details how to manually download the container images and transport them onto the relevant servers.

Once installed, in order to use OpenShift Container Platform, you will need source code in a source control repository (for example, Git). This topic assumes that an internal Git repository is available that can host source code and this repository is accessible from the OpenShift Container Platform nodes. Installing the source control repository is outside the scope of this document.

Also, when building applications in OpenShift Container Platform, your build may have some external dependencies, such as a Maven Repository or Gem files for Ruby applications. For this reason, and because they might require certain tags, many of the Quickstart templates offered by OpenShift Container Platform may not work on a disconnected environment. However, while Red Hat container images try to reach out to external repositories by default, you can configure OpenShift Container Platform to use your own internal repositories. For the purposes of this document, we assume that such internal repositories already exist and are accessible from the OpenShift Container Platform nodes hosts. Installing such repositories is outside the scope of this document.

Note

You can also have a Red Hat Satellite server that provides access to Red Hat content via an intranet or LAN. For environments with Satellite, you can synchronize the OpenShift Container Platform software onto the Satellite for use with the OpenShift Container Platform servers.

Red Hat Satellite 6.1 also introduces the ability to act as a Docker registry, and it can be used to host the OpenShift Container Platform containerized components. Doing so is outside of the scope of this document.

2.7.2. Prerequisites

This document assumes that you understand OpenShift Container Platform’s overall architecture and that you have already planned out what the topology of your environment will look like.

2.7.3. Required Software and Components

In order to pull down the required software repositories and container images, you will need a Red Hat Enterprise Linux (RHEL) 7 server with access to the Internet and at least 100GB of additional free space. All steps in this section should be performed on the Internet-connected server as the root system user.

2.7.3.1. Syncing Repositories

Before you sync with the required repositories, you may need to import the appropriate GPG key:

# rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

If the key is not imported, the indicated package is deleted after syncing the repository.

To sync the required repositories:

  1. Register the server with the Red Hat Customer Portal. You must use the login and password associated with the account that has access to the OpenShift Container Platform subscriptions:

    # subscription-manager register
  2. Attach to a subscription that provides OpenShift Container Platform channels. You can find the list of available subscriptions using:

    # subscription-manager list --available --matches '*OpenShift*'

    Then, find the pool ID for the subscription that provides OpenShift Container Platform, and attach it:

    # subscription-manager attach --pool=<pool_id>
    # subscription-manager repos --disable="*"
    # subscription-manager repos \
        --enable="rhel-7-server-rpms" \
        --enable="rhel-7-server-extras-rpms" \
        --enable="rhel-7-server-ose-3.4-rpms"
  3. The yum-utils command provides the reposync utility, which lets you mirror yum repositories, and createrepo can create a usable yum repository from a directory:

    # yum -y install yum-utils createrepo docker git

    You will need up to 110GB of free space in order to sync the software. Depending on how restrictive your organization’s policies are, you could re-connect this server to the disconnected LAN and use it as the repository server. You could use USB-connected storage and transport the software to another server that will act as the repository server. This topic covers these options.

  4. Make a path to where you want to sync the software (either locally or on your USB or other device):

    # mkdir -p </path/to/repos>
  5. Sync the packages and create the repository for each of them. You will need to modify the command for the appropriate path you created above:

    # for repo in \
    rhel-7-server-rpms rhel-7-server-extras-rpms \
    rhel-7-server-ose-3.4-rpms
    do
      reposync --gpgcheck -lm --repoid=${repo} --download_path=/path/to/repos
      createrepo -v </path/to/repos/>${repo} -o </path/to/repos/>${repo}
    done

2.7.3.2. Syncing Images

To sync the container images:

  1. Start the Docker daemon:

    # systemctl start docker
  2. Pull all of the required OpenShift Container Platform containerized components. Replace <tag> with v3.4.1.10 for the latest version.

    # docker pull registry.access.redhat.com/openshift3/ose-haproxy-router:<tag>
    # docker pull registry.access.redhat.com/openshift3/ose-deployer:<tag>
    # docker pull registry.access.redhat.com/openshift3/ose-sti-builder:<tag>
    # docker pull registry.access.redhat.com/openshift3/ose-docker-builder:<tag>
    # docker pull registry.access.redhat.com/openshift3/ose-pod:<tag>
    # docker pull registry.access.redhat.com/openshift3/ose-docker-registry:<tag>
  3. Pull all of the required OpenShift Container Platform containerized components for the additional centralized log aggregation and metrics aggregation components. Replace <tag> with 3.4.1 for the latest version.

    # docker pull registry.access.redhat.com/openshift3/logging-deployer:<tag>
    # docker pull registry.access.redhat.com/openshift3/logging-elasticsearch:<tag>
    # docker pull registry.access.redhat.com/openshift3/logging-kibana:<tag>
    # docker pull registry.access.redhat.com/openshift3/logging-fluentd:<tag>
    # docker pull registry.access.redhat.com/openshift3/logging-curator:<tag>
    # docker pull registry.access.redhat.com/openshift3/logging-auth-proxy:<tag>
    # docker pull registry.access.redhat.com/openshift3/metrics-deployer:<tag>
    # docker pull registry.access.redhat.com/openshift3/metrics-hawkular-metrics:<tag>
    # docker pull registry.access.redhat.com/openshift3/metrics-cassandra:<tag>
    # docker pull registry.access.redhat.com/openshift3/metrics-heapster:<tag>
  4. Pull the Red Hat-certified Source-to-Image (S2I) builder images that you intend to use in your OpenShift environment. You can pull the following images:

    • jboss-eap70-openshift
    • jboss-amq-62
    • jboss-datagrid65-openshift
    • jboss-decisionserver62-openshift
    • jboss-eap64-openshift
    • jboss-eap70-openshift
    • jboss-webserver30-tomcat7-openshift
    • jboss-webserver30-tomcat8-openshift
    • mongodb
    • mysql
    • nodejs
    • perl
    • php
    • postgresql
    • python
    • redhat-sso70-openshift
    • ruby

      Make sure to indicate the correct tag specifying the desired version number. For example, to pull both the previous and latest version of the Tomcat image:

      # docker pull \
      registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat7-openshift:latest
      # docker pull \
      registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat7-openshift:1.1
  5. If you are using a stand-alone registry or plan to enable the registry console with the integrated registry, you must pull the registry-console image.

    Replace <tag> with 3.4 for the latest version.

    # docker pull registry.access.redhat.com/openshift3/registry-console:<tag>

2.7.3.3. Preparing Images for Export

Container images can be exported from a system by first saving them to a tarball and then transporting them:

  1. Make and change into a repository home directory:

    # mkdir </path/to/repos/images>
    # cd </path/to/repos/images>
  2. Export the OpenShift Container Platform containerized components:

    # docker save -o ose3-images.tar \
        registry.access.redhat.com/openshift3/ose-haproxy-router \
        registry.access.redhat.com/openshift3/ose-deployer \
        registry.access.redhat.com/openshift3/ose-sti-builder \
        registry.access.redhat.com/openshift3/ose-docker-builder \
        registry.access.redhat.com/openshift3/ose-pod \
        registry.access.redhat.com/openshift3/ose-docker-registry
  3. If you synchronized the metrics and log aggregation images, export:

    # docker save -o ose3-logging-metrics-images.tar \
        registry.access.redhat.com/openshift3/logging-deployer \
        registry.access.redhat.com/openshift3/logging-elasticsearch \
        registry.access.redhat.com/openshift3/logging-kibana \
        registry.access.redhat.com/openshift3/logging-fluentd \
        registry.access.redhat.com/openshift3/logging-auth-proxy \
        registry.access.redhat.com/openshift3/metrics-deployer \
        registry.access.redhat.com/openshift3/metrics-hawkular-metrics \
        registry.access.redhat.com/openshift3/metrics-cassandra \
        registry.access.redhat.com/openshift3/metrics-heapster
  4. Export the S2I builder images that you synced in the previous section. For example, if you synced only the Tomcat image:

    # docker save -o ose3-builder-images.tar \
        registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat7-openshift:latest \
        registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat7-openshift:1.1

2.7.4. Repository Server

During the installation (and for later updates, should you so choose), you will need a webserver to host the repositories. RHEL 7 can provide the Apache webserver.

Option 1: Re-configuring as a Web server

If you can re-connect the server where you synchronized the software and images to your LAN, then you can simply install Apache on the server:

# yum install httpd

Skip to Placing the Software.

Option 2: Building a Repository Server

If you need to build a separate server to act as the repository server, install a new RHEL 7 system with at least 110GB of space. On this repository server during the installation, make sure you select the Basic Web Server option.

2.7.4.1. Placing the Software

  1. If necessary, attach the external storage, and then copy the repository files into Apache’s root folder. Note that the below copy step (cp -a) should be substituted with move (mv) if you are repurposing the server you used to sync:

    # cp -a /path/to/repos /var/www/html/
    # chmod -R +r /var/www/html/repos
    # restorecon -vR /var/www/html
  2. Add the firewall rules:

    # firewall-cmd --permanent --add-service=http
    # firewall-cmd --reload
  3. Enable and start Apache for the changes to take effect:

    # systemctl enable httpd
    # systemctl start httpd

2.7.5. OpenShift Container Platform Systems

2.7.5.1. Building Your Hosts

At this point you can perform the initial creation of the hosts that will be part of the OpenShift Container Platform environment. It is recommended to use the latest version of RHEL 7 and to perform a minimal installation. You will also want to pay attention to the other OpenShift Container Platform-specific prerequisites.

Once the hosts are initially built, the repositories can be set up.

2.7.5.2. Connecting the Repositories

On all of the relevant systems that will need OpenShift Container Platform software components, create the required repository definitions. Place the following text in the /etc/yum.repos.d/ose.repo file, replacing <server_IP> with the IP or host name of the Apache server hosting the software repositories:

[rhel-7-server-rpms]
name=rhel-7-server-rpms
baseurl=http://<server_IP>/repos/rhel-7-server-rpms
enabled=1
gpgcheck=0
[rhel-7-server-extras-rpms]
name=rhel-7-server-extras-rpms
baseurl=http://<server_IP>/repos/rhel-7-server-extras-rpms
enabled=1
gpgcheck=0
[rhel-7-server-ose-3.4-rpms]
name=rhel-7-server-ose-3.4-rpms
baseurl=http://<server_IP>/repos/rhel-7-server-ose-3.4-rpms
enabled=1
gpgcheck=0

2.7.5.3. Host Preparation

At this point, the systems are ready to continue to be prepared following the OpenShift Container Platform documentation.

Skip the section titled Host Registration and start with Installing Base Packages.

2.7.6. Installing OpenShift Container Platform

2.7.6.1. Importing OpenShift Container Platform Containerized Components

To import the relevant components, securely copy the images from the connected host to the individual OpenShift Container Platform hosts:

# scp /var/www/html/repos/images/ose3-images.tar root@<openshift_host_name>:
# ssh root@<openshift_host_name> "docker load -i ose3-images.tar"

If you prefer, you could use wget on each OpenShift Container Platform host to fetch the tar file, and then perform the Docker import command locally. Perform the same steps for the metrics and logging images, if you synchronized them.

On the host that will act as an OpenShift Container Platform master, copy and import the builder images:

# scp /var/www/html/images/ose3-builder-images.tar root@<openshift_master_host_name>:
# ssh root@<openshift_master_host_name> "docker load -i ose3-builder-images.tar"

2.7.6.2. Running the OpenShift Container Platform Installer

You can now choose to follow the quick or advanced OpenShift Container Platform installation instructions in the documentation.

2.7.6.3. Creating the Internal Docker Registry

You now need to create the internal Docker registry.

If you want to install a stand-alone registry, you must pull the registry-console container image and set deployment_subtype=registry in the inventory file.

2.7.7. Post-Installation Changes

In one of the previous steps, the S2I images were imported into the Docker daemon running on one of the OpenShift Container Platform master hosts. In a connected installation, these images would be pulled from Red Hat’s registry on demand. Since the Internet is not available to do this, the images must be made available in another Docker registry.

OpenShift Container Platform provides an internal registry for storing the images that are built as a result of the S2I process, but it can also be used to hold the S2I builder images. The following steps assume you did not customize the service IP subnet (172.30.0.0/16) or the Docker registry port (5000).

2.7.7.1. Re-tagging S2I Builder Images

  1. On the master host where you imported the S2I builder images, obtain the service address of your Docker registry that you installed on the master:

    # export REGISTRY=$(oc get service docker-registry -t '{{.spec.clusterIP}}{{"\n"}}')
  2. Next, tag all of the builder images that you synced and exported before pushing them into the OpenShift Container Platform Docker registry. For example, if you synced and exported only the Tomcat image:

    # docker tag \
    registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat7-openshift:1.1 \
    $REGISTRY:5000/openshift/webserver30-tomcat7-openshift:1.1
    # docker tag \
    registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat7-openshift:latest \
    $REGISTRY:5000/openshift/webserver30-tomcat7-openshift:1.2
    # docker tag \
    registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat7-openshift:latest \
    $REGISTRY:5000/openshift/webserver30-tomcat7-openshift:latest

2.7.7.2. Creating an Administrative User

Pushing the container images into OpenShift Container Platform’s Docker registry requires a user with cluster-admin privileges. Because the default OpenShift Container Platform system administrator does not have a standard authorization token, they cannot be used to log in to the Docker registry.

To create an administrative user:

  1. Create a new user account in the authentication system you are using with OpenShift Container Platform. For example, if you are using local htpasswd-based authentication:

    # htpasswd -b /etc/openshift/openshift-passwd <admin_username> <password>
  2. The external authentication system now has a user account, but a user must log in to OpenShift Container Platform before an account is created in the internal database. Log in to OpenShift Container Platform for this account to be created. This assumes you are using the self-signed certificates generated by OpenShift Container Platform during the installation:

    # oc login --certificate-authority=/etc/origin/master/ca.crt \
        -u <admin_username> https://<openshift_master_host>:8443
  3. Get the user’s authentication token:

    # MYTOKEN=$(oc whoami -t)
    # echo $MYTOKEN
    iwo7hc4XilD2KOLL4V1O55ExH2VlPmLD-W2-JOd6Fko

2.7.7.3. Modifying the Security Policies

  1. Using oc login switches to the new user. Switch back to the OpenShift Container Platform system administrator in order to make policy changes:

    # oc login -u system:admin
  2. In order to push images into the OpenShift Container Platform Docker registry, an account must have the image-builder security role. Add this to your OpenShift Container Platform administrative user:

    # oadm policy add-role-to-user system:image-builder <admin_username>
  3. Next, add the administrative role to the user in the openshift project. This allows the administrative user to edit the openshift project, and, in this case, push the container images:

    # oadm policy add-role-to-user admin <admin_username> -n openshift

2.7.7.4. Editing the Image Stream Definitions

The openshift project is where all of the image streams for builder images are created by the installer. They are loaded by the installer from the /usr/share/openshift/examples directory. Change all of the definitions by deleting the image streams which had been loaded into OpenShift Container Platform’s database, then re-create them:

  1. Delete the existing image streams:

    # oc delete is -n openshift --all
  2. Make a backup of the files in /usr/share/openshift/examples/ if you desire. Next, edit the file image-streams-rhel7.json in the /usr/share/openshift/examples/image-streams folder. You will find an image stream section for each of the builder images. Edit the spec stanza to point to your internal Docker registry.

    For example, change:

    "spec": {
      "dockerImageRepository": "registry.access.redhat.com/rhscl/mongodb-26-rhel7",

    to:

    "spec": {
      "dockerImageRepository": "172.30.69.44:5000/openshift/mongodb-26-rhel7",

    In the above, the repository name was changed from rhscl to openshift. You will need to ensure the change, regardless of whether the repository is rhscl, openshift3, or another directory. Every definition should have the following format:

    <registry_ip>:5000/openshift/<image_name>

    Repeat this change for every image stream in the file. Ensure you use the correct IP address that you determined earlier. When you are finished, save and exit. Repeat the same process for the JBoss image streams in the /usr/share/openshift/examples/xpaas-streams/jboss-image-streams.json file.

  3. Load the updated image stream definitions:

    # oc create -f /usr/share/openshift/examples/image-streams/image-streams-rhel7.json -n openshift
    # oc create -f /usr/share/openshift/examples/xpaas-streams/jboss-image-streams.json -n openshift

2.7.7.5. Loading the Container Images

At this point the system is ready to load the container images.

  1. Log in to the Docker registry using the token and registry service IP obtained earlier:

    # docker login -u adminuser -e mailto:adminuser@abc.com \
       -p $MYTOKEN $REGISTRY:5000
  2. Push the Docker images:

    # docker push $REGISTRY:5000/openshift/webserver30-tomcat7-openshift:1.1
    # docker push $REGISTRY:5000/openshift/webserver30-tomcat7-openshift:1.2
    # docker push $REGISTRY:5000/openshift/webserver30-tomcat7-openshift:latest
  3. Verify that all the image streams now have the tags populated:

    # oc get imagestreams -n openshift
    NAME                                 DOCKER REPO                                                      TAGS                                     UPDATED
    jboss-webserver30-tomcat7-openshift  $REGISTRY/jboss-webserver-3/webserver30-jboss-tomcat7-openshift  1.1,1.1-2,1.1-6 + 2 more...              2 weeks ago
    ...

2.7.8. Installing a Router

At this point, the OpenShift Container Platform environment is almost ready for use. It is likely that you will want to install and configure a router.

2.8. Installing a Stand-alone Registry

2.8.1. Overview

OpenShift Container Platform is a fully-featured platform-as-a-service (PaaS) enterprise solution that includes an embedded container registry. However, it can alternatively be installed as a stand-alone container registry to run on-premise or in the cloud.

The stand-alone registry from OpenShift Container Platform provides the following capabilities:

2.8.2. Minimum Hardware Requirements

Installing OpenShift Container Platform as a stand-alone registry has the following hardware requirements:

  • Physical or virtual system, or an instance running on a public or private IaaS.
  • Base OS: RHEL 7.1 or later with "Minimal" installation option, or RHEL Atomic Host 7.3.2 or later.
  • NetworkManager 1.0 or later
  • 2 vCPU.
  • Minimum 16 GB RAM.
  • Minimum 15 GB hard disk space for the file system containing /var/.
  • An additional minimum 15 GB unallocated space to be used for Docker’s storage back end; see Configuring Docker Storage for details.
Important

OpenShift Container Platform only supports servers with x86_64 architecture.

Note

Meeting the /var/ file system sizing requirements in RHEL Atomic Host requires making changes to the default configuration. See Managing Storage in Red Hat Enterprise Linux Atomic Host for instructions on configuring this during or after installation.

2.8.3. Supported System Topologies

The following system topologies are supported when running OpenShift Container Platform as a stand-alone registry:

All-in-one

A single host that includes the master, node, etcd, and registry components.

Multiple Masters (Highly-Available)

Three hosts with all components included on each (master, node, etcd, and registry), with the masters configured for native high-availability.

2.8.4. Host Preparation

Before installing a stand-alone registry, all of the same steps detailed in the Host Preparation topic for installing a full OpenShift Container Platform PaaS must be performed. This includes registering and subscribing the host(s) to the proper repositories, installing or updating certain packages, and setting up Docker and its storage requirements.

Follow the steps in the Host Preparation topic, then continue to Installation Methods.

2.8.5. Installation Methods

To install a stand-alone registry, use either of the standard installation methods (quick or advanced) used to install any variant of OpenShift Container Platform.

2.8.5.1. Quick Installation for Stand-alone Registries

When using the quick installation method to install a stand-alone registry, start the interactive installation by running:

$ atomic-openshift-installer install

Then follow the on-screen instructions to install a new registry. The installation questions will be largely the same as if you were installing a full OpenShift Container Platform PaaS, but when you reach the following screen:

Which variant would you like to install?


(1) OpenShift Container Platform
(2) Registry

Be sure to choose 2 to follow the registry installation path.

Note

For further usage details on the quick installer in general, see the full topic at Quick Installation.

2.8.5.2. Advanced Installation for Stand-alone Registries

When using the advanced installation method to install a stand-alone registry, use the same steps for installing a full OpenShift Container Platform PaaS using Ansible described in the full Advanced Installation topic. The main difference is that you must set deployment_subtype=registry in the inventory file within the [OSEv3:vars] section for the playbooks to follow the registry installation path.

See the following example inventory files for the different supported system topologies:

Example 2.5. All-in-one Stand-alone Registry Inventory File

# Create an OSEv3 group that contains the masters and nodes groups
[OSEv3:children]
masters
nodes

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
# SSH user, this user should allow ssh based auth without requiring a password
ansible_ssh_user=root

openshift_master_default_subdomain=apps.test.example.com

# If ansible_ssh_user is not root, ansible_become must be set to true
#ansible_become=true

deployment_type=openshift-enterprise
deployment_subtype=registry 1

# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider
#openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]

# host group for masters
[masters]
registry.example.com

# host group for nodes, includes region info
[nodes]
registry.example.com openshift_schedulable=true openshift_node_labels="{'region': 'infra', 'zone': 'default'}" 2
1
Set deployment_subtype=registry to ensure installation of the stand-alone registry.
2
Set openshift_schedulable=true on the node entry to make the single node schedulable for pod placement.

Example 2.6. Multiple Masters (Highly-Available) Stand-alone Registry Inventory File

# Create an OSEv3 group that contains the master, nodes, etcd, and lb groups.
# The lb group lets Ansible configure HAProxy as the load balancing solution.
# Comment lb out if your load balancer is pre-configured.
[OSEv3:children]
masters
nodes
etcd
lb

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
ansible_ssh_user=root
deployment_type=openshift-enterprise
deployment_subtype=registry 1

openshift_master_default_subdomain=apps.test.example.com

# Uncomment the following to enable htpasswd authentication; defaults to
# DenyAllPasswordIdentityProvider.
#openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]

# Native high availability cluster method with optional load balancer.
# If no lb group is defined installer assumes that a load balancer has
# been preconfigured. For installation the value of
# openshift_master_cluster_hostname must resolve to the load balancer
# or to one or all of the masters defined in the inventory if no load
# balancer is present.
openshift_master_cluster_method=native
openshift_master_cluster_hostname=openshift-cluster.example.com
openshift_master_cluster_public_hostname=openshift-cluster.example.com

# apply updated node defaults
openshift_node_kubelet_args={'pods-per-core': ['10'], 'max-pods': ['250'], 'image-gc-high-threshold': ['90'], 'image-gc-low-threshold': ['80']}

# override the default controller lease ttl
#osm_controller_lease_ttl=30

# enable ntp on masters to ensure proper failover
openshift_clock_enabled=true

# host group for masters
[masters]
master1.example.com
master2.example.com
master3.example.com

# host group for etcd
[etcd]
etcd1.example.com
etcd2.example.com
etcd3.example.com

# Specify load balancer host
[lb]
lb.example.com

# host group for nodes, includes region info
[nodes]
master[1:3].example.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
node1.example.com openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
node2.example.com openshift_node_labels="{'region': 'primary', 'zone': 'west'}"
1
Setting deployment_subtype=registry ensures installation of the stand-alone registry.

After you have configured Ansible by defining an inventory file in /etc/ansible/hosts, you can run the advanced installation using the following playbook:

# ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml
Note

For more detailed usage information on the advanced installation method, including a comprehensive list of available Ansible variables, see the full topic at Advanced Installation.

Chapter 3. Setting up the Registry

3.1. Registry Overview

3.1.1. About the Registry

OpenShift Container Platform can build container images from your source code, deploy them, and manage their lifecycle. To enable this, OpenShift Container Platform provides an internal, integrated Docker registry that can be deployed in your OpenShift Container Platform environment to locally manage images.

3.1.2. Integrated or Stand-alone Registries

During an initial installation of a full OpenShift Container Platform cluster, it is likely that the registry was deployed automatically during the installation process. If it was not, or if you want to further customize the configuration of your registry, see Deploying a Registry on Existing Clusters.

While it can be deployed to run as an integrated part of your full OpenShift Container Platform cluster, the OpenShift Container Platform registry can alternatively be installed separately as a stand-alone container image registry.

To install a stand-alone registry, follow Installing a Stand-alone Registry. This installation path deploys an all-in-one cluster running a registry and specialized web console.

3.2. Deploying a Registry on Existing Clusters

3.2.1. Overview

If the integrated registry was not previously deployed automatically during the initial installation of your OpenShift Container Platform cluster, or if it is no longer running successfully and you need to redeploy it on your existing cluster, see the following sections for options on deploying a new registry.

Note

This topic is not required if you installed a stand-alone registry.

3.2.2. Deploying the Registry

To deploy the integrated Docker registry, use the oadm registry command as a user with cluster administrator privileges. For example:

$ oadm registry --config=/etc/origin/master/admin.kubeconfig \1
    --service-account=registry \2
    --images='registry.access.redhat.com/openshift3/ose-${component}:${version}' 3
1
--config is the path to the CLI configuration file for the cluster administrator.
2
--service-account is the service account used to run the registry’s pod.
3
Required to pull the correct image for OpenShift Container Platform.

This creates a service and a deployment configuration, both called docker-registry. Once deployed successfully, a pod is created with a name similar to docker-registry-1-cpty9.

To see a full list of options that you can specify when creating the registry:

$ oadm registry --help

3.2.3. Deploying the Registry as a DaemonSet

Use the oadm registry command to deploy the registry as a DaemonSet with the --daemonset option.

Daemonsets ensure that when nodes are created, they contain copies of a specified pod. When the nodes are removed, the pods are garbage collected.

For more information on DaemonSets, see the DaemonSet documentation.

Note

Using DaemonSets with OpenShift Container Platform is a work in progress, and the only supported process is creating a registry. Currently, using deployment configurations to deploy a registry is recommended.

3.2.4. Registry Compute Resources

By default, the registry is created with no settings for compute resource requests or limits. For production, it is highly recommended that the deployment configuration for the registry be updated to set resource requests and limits for the registry pod. Otherwise, the registry pod will be considered a BestEffort pod.

See Compute Resources for more information on configuring requests and limits.

3.2.5. Storage for the Registry

The registry stores container images and metadata. If you simply deploy a pod with the registry, it uses an ephemeral volume that is destroyed if the pod exits. Any images anyone has built or pushed into the registry would disappear.

3.2.5.1. Production Use

For production use, attach a remote volume or define and use the persistent storage method of your choice.

For example, to use an existing persistent volume claim:

$ oc volume deploymentconfigs/docker-registry --add --name=registry-storage -t pvc \
     --claim-name=<pvc_name> --overwrite
Note

See Known Issues if using a scaled registry with a shared NFS volume.

3.2.5.1.1. Use Amazon S3 as a Storage Back-end

There is also an option to use Amazon Simple Storage Service storage with the internal Docker registry. It is a secure cloud storage manageable through AWS Management Console. To use it, the registry’s configuration file must be manually edited and mounted to the registry pod. However, before you start with the configuration, look at upstream’s recommended steps.

Take a default YAML configuration file as a base and replace the filesystem entry in the storage section with s3 entry such as below. The resulting storage section may look like this:

storage:
  cache:
    layerinfo: inmemory
  delete:
    enabled: true
  s3:
    accesskey: awsaccesskey 1
    secretkey: awssecretkey 2
    region: us-west-1
    regionendpoint: http://myobjects.local
    bucket: bucketname
    encrypt: true
    keyid: mykeyid
    secure: true
    v4auth: false
    chunksize: 5242880
    rootdirectory: /s3/object/name/prefix
1
Replace with your Amazon access key.
2
Replace with your Amazon secret key.

All of the s3 configuration options are documented in upstream’s driver reference documentation.

Overriding the registry configuration will take you through the additional steps on mounting the configuration file into pod.

Warning

When the registry runs on the S3 storage back-end, there are reported issues.

3.2.5.2. Non-Production Use

For non-production use, you can use the --mount-host=<path> option to specify a directory for the registry to use for persistent storage. The registry volume is then created as a host-mount at the specified <path>.

Important

The --mount-host option mounts a directory from the node on which the registry container lives. If you scale up the docker-registry deployment configuration, it is possible that your registry pods and containers will run on different nodes, which can result in two or more registry containers, each with its own local storage. This will lead to unpredictable behavior, as subsequent requests to pull the same image repeatedly may not always succeed, depending on which container the request ultimately goes to.

The --mount-host option requires that the registry container run in privileged mode. This is automatically enabled when you specify --mount-host. However, not all pods are allowed to run privileged containers by default. If you still want to use this option, create the registry and specify that it use the registry service account that was created during installation:

$ oadm registry --service-account=registry \
    --config=/etc/origin/master/admin.kubeconfig \
    --images='registry.access.redhat.com/openshift3/ose-${component}:${version}' \
    --mount-host=<path>
Important

The Docker registry pod runs as user 1001. This user must be able to write to the host directory. You may need to change directory ownership to user ID 1001 with this command:

$ sudo chown 1001:root <path>

3.2.6. Enabling the Registry Console

OpenShift Container Platform provides a web-based interface to the integrated registry. This registry console is an optional component for browsing and managing images. It is deployed as a stateless service running as a pod.

Note

If you installed OpenShift Container Platform as a stand-alone registry, the registry console is already deployed and secured automatically during installation.

Important

If Cockpit is already running, you’ll need to shut it down before proceeding in order to avoid a port conflict (9090 by default) with the registry console.

3.2.6.1. Deploying the Registry Console

Important

You must first have exposed the registry.

  1. Create a passthrough route in the default project. You will need this when creating the registry console application in the next step.

    $ oc create route passthrough --service registry-console \
        --port registry-console \
        -n default
  2. Deploy the registry console application. Replace <openshift_oauth_url> with the URL of the OpenShift Container Platform OAuth provider, which is typically the master.

    $ oc new-app -n default --template=registry-console \
        -p OPENSHIFT_OAUTH_PROVIDER_URL="https://<openshift_oauth_url>:8443" \
        -p REGISTRY_HOST=$(oc get route docker-registry -n default --template='{{ .spec.host }}') \
        -p COCKPIT_KUBE_URL=$(oc get route registry-console -n default --template='https://{{ .spec.host }}')
Note

If the redirection URL is wrong when you are trying to log in to the registry console, check your OAuth client with oc get oauthclients.

  1. Finally, use a web browser to view the console using the route URI.

3.2.6.2. Securing the Registry Console

By default, the registry console generates self-signed TLS certificates if deployed manually per the steps in Deploying the Registry Console. See Troubleshooting the Registry Console for more information.

Use the following steps to add your organization’s signed certificates as a secret volume. This assumes your certificates are available on the the oc client host.

  1. Create a .cert file containing the certificate and key. Format the file with:

    • One or more BEGIN CERTIFICATE blocks for the server certificate and the intermediate certificate authorities
    • A block containing a BEGIN PRIVATE KEY or similar for the key. The key must not be encrypted

      For example:

      -----BEGIN CERTIFICATE-----
      MIIDUzCCAjugAwIBAgIJAPXW+CuNYS6QMA0GCSqGSIb3DQEBCwUAMD8xKTAnBgNV
      BAoMIGI0OGE2NGNkNmMwNTQ1YThhZTgxOTEzZDE5YmJjMmRjMRIwEAYDVQQDDAls
      ...
      -----END CERTIFICATE-----
      -----BEGIN CERTIFICATE-----
      MIIDUzCCAjugAwIBAgIJAPXW+CuNYS6QMA0GCSqGSIb3DQEBCwUAMD8xKTAnBgNV
      BAoMIGI0OGE2NGNkNmMwNTQ1YThhZTgxOTEzZDE5YmJjMmRjMRIwEAYDVQQDDAls
      ...
      -----END CERTIFICATE-----
      -----BEGIN PRIVATE KEY-----
      MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCyOJ5garOYw0sm
      8TBCDSqQ/H1awGMzDYdB11xuHHsxYS2VepPMzMzryHR137I4dGFLhvdTvJUH8lUS
      ...
      -----END PRIVATE KEY-----

      The registry console loads a certificate from the /etc/cockpit/ws-certs.d directory. It uses the last file with a .cert extension in alphabetical order. Therefore, the .cert file should contain at least two PEM blocks formatted in the OpenSSL style.

      If no certificate is found, a self-signed certificate is created using the openssl command and stored in the 0-self-signed.cert file.

  2. Create the secret:

    $ oc secrets new console-secret \
        /path/to/console.cert
  3. Add the secrets to the registry-console deployment configuration:

    $ oc volume dc/registry-console --add --type=secret \
        --secret-name=console-secret -m /etc/cockpit/ws-certs.d

    This triggers a new deployment of the registry console to include your signed certificates.

3.2.6.3. Troubleshooting the Registry Console

3.2.6.3.1. Debug Mode

The registry console debug mode is enabled using an environment variable. The following command redeploys the registry console in debug mode:

$ oc set env dc registry-console G_MESSAGES_DEBUG=cockpit-ws,cockpit-wrapper

Enabling debug mode allows more verbose logging to appear in the registry console’s pod logs.

3.2.6.3.2. Display SSL Certificate Path

To check which certificate the registry console is using, a command can be run from inside the console pod.

  1. List the pods in the default project and find the registry console’s pod name:

    $ oc get pods -n default
    NAME                       READY     STATUS    RESTARTS   AGE
    registry-console-1-rssrw   1/1       Running   0          1d
  2. Using the pod name from the previous command, get the certificate path that the cockpit-ws process is using. This example shows the console using the auto-generated certificate:

    $ oc exec registry-console-1-rssrw remotectl certificate
    certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert

3.3. Accessing the Registry

3.3.1. Viewing Logs

To view the logs for the Docker registry, use the oc logs command with the deployment config:

$ oc logs dc/docker-registry
2015-05-01T19:48:36.300593110Z time="2015-05-01T19:48:36Z" level=info msg="version=v2.0.0+unknown"
2015-05-01T19:48:36.303294724Z time="2015-05-01T19:48:36Z" level=info msg="redis not configured" instance.id=9ed6c43d-23ee-453f-9a4b-031fea646002
2015-05-01T19:48:36.303422845Z time="2015-05-01T19:48:36Z" level=info msg="using inmemory layerinfo cache" instance.id=9ed6c43d-23ee-453f-9a4b-031fea646002
2015-05-01T19:48:36.303433991Z time="2015-05-01T19:48:36Z" level=info msg="Using OpenShift Auth handler"
2015-05-01T19:48:36.303439084Z time="2015-05-01T19:48:36Z" level=info msg="listening on :5000" instance.id=9ed6c43d-23ee-453f-9a4b-031fea646002

3.3.2. File Storage

Tag and image metadata is stored in OpenShift Container Platform, but the registry stores layer and signature data in a volume that is mounted into the registry container at /registry. As oc exec does not work on privileged containers, to view a registry’s contents you must manually SSH into the node housing the registry pod’s container, then run docker exec on the container itself:

  1. List the current pods to find the pod name of your Docker registry:

    # oc get pods

    Then, use oc describe to find the host name for the node running the container:

    # oc describe pod <pod_name>
  2. Log into the desired node:

    # ssh node.example.com
  3. List the running containers on the node host and identify the container ID for the Docker registry:

    # docker ps | grep ose-docker-registry
  4. List the registry contents using the docker exec command:

    # docker exec -it 4c01db0b339c find /registry
    /registry/docker
    /registry/docker/registry
    /registry/docker/registry/v2
    /registry/docker/registry/v2/blobs 1
    /registry/docker/registry/v2/blobs/sha256
    /registry/docker/registry/v2/blobs/sha256/ed
    /registry/docker/registry/v2/blobs/sha256/ed/ede17b139a271d6b1331ca3d83c648c24f92cece5f89d95ac6c34ce751111810
    /registry/docker/registry/v2/blobs/sha256/ed/ede17b139a271d6b1331ca3d83c648c24f92cece5f89d95ac6c34ce751111810/data 2
    /registry/docker/registry/v2/blobs/sha256/a3
    /registry/docker/registry/v2/blobs/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
    /registry/docker/registry/v2/blobs/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4/data
    /registry/docker/registry/v2/blobs/sha256/f7
    /registry/docker/registry/v2/blobs/sha256/f7/f72a00a23f01987b42cb26f259582bb33502bdb0fcf5011e03c60577c4284845
    /registry/docker/registry/v2/blobs/sha256/f7/f72a00a23f01987b42cb26f259582bb33502bdb0fcf5011e03c60577c4284845/data
    /registry/docker/registry/v2/repositories 3
    /registry/docker/registry/v2/repositories/p1
    /registry/docker/registry/v2/repositories/p1/pause 4
    /registry/docker/registry/v2/repositories/p1/pause/_manifests
    /registry/docker/registry/v2/repositories/p1/pause/_manifests/revisions
    /registry/docker/registry/v2/repositories/p1/pause/_manifests/revisions/sha256
    /registry/docker/registry/v2/repositories/p1/pause/_manifests/revisions/sha256/e9a2ac6418981897b399d3709f1b4a6d2723cd38a4909215ce2752a5c068b1cf
    /registry/docker/registry/v2/repositories/p1/pause/_manifests/revisions/sha256/e9a2ac6418981897b399d3709f1b4a6d2723cd38a4909215ce2752a5c068b1cf/signatures 5
    /registry/docker/registry/v2/repositories/p1/pause/_manifests/revisions/sha256/e9a2ac6418981897b399d3709f1b4a6d2723cd38a4909215ce2752a5c068b1cf/signatures/sha256
    /registry/docker/registry/v2/repositories/p1/pause/_manifests/revisions/sha256/e9a2ac6418981897b399d3709f1b4a6d2723cd38a4909215ce2752a5c068b1cf/signatures/sha256/ede17b139a271d6b1331ca3d83c648c24f92cece5f89d95ac6c34ce751111810
    /registry/docker/registry/v2/repositories/p1/pause/_manifests/revisions/sha256/e9a2ac6418981897b399d3709f1b4a6d2723cd38a4909215ce2752a5c068b1cf/signatures/sha256/ede17b139a271d6b1331ca3d83c648c24f92cece5f89d95ac6c34ce751111810/link 6
    /registry/docker/registry/v2/repositories/p1/pause/_uploads 7
    /registry/docker/registry/v2/repositories/p1/pause/_layers 8
    /registry/docker/registry/v2/repositories/p1/pause/_layers/sha256
    /registry/docker/registry/v2/repositories/p1/pause/_layers/sha256/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
    /registry/docker/registry/v2/repositories/p1/pause/_layers/sha256/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4/link 9
    /registry/docker/registry/v2/repositories/p1/pause/_layers/sha256/f72a00a23f01987b42cb26f259582bb33502bdb0fcf5011e03c60577c4284845
    /registry/docker/registry/v2/repositories/p1/pause/_layers/sha256/f72a00a23f01987b42cb26f259582bb33502bdb0fcf5011e03c60577c4284845/link
    1
    This directory stores all layers and signatures as blobs.
    2
    This file contains the blob’s contents.
    3
    This directory stores all the image repositories.
    4
    This directory is for a single image repository p1/pause.
    5
    This directory contains signatures for a particular image manifest revision.
    6
    This file contains a reference back to a blob (which contains the signature data).
    7
    This directory contains any layers that are currently being uploaded and staged for the given repository.
    8
    This directory contains links to all the layers this repository references.
    9
    This file contains a reference to a specific layer that has been linked into this repository via an image.

3.3.3. Accessing the Registry Directly

For advanced usage, you can access the registry directly to invoke docker commands. This allows you to push images to or pull them from the integrated registry directly using operations like docker push or docker pull. To do so, you must be logged in to the registry using the docker login command. The operations you can perform depend on your user permissions, as described in the following sections.

3.3.3.1. User Prerequisites

To access the registry directly, the user that you use must satisfy the following, depending on your intended usage:

  • For any direct access, you must have a regular user, if one does not already exist, for your preferred identity provider. A regular user can generate an access token required for logging in to the registry. System users, such as system:admin, cannot obtain access tokens and, therefore, cannot access the registry directly.

    For example, if you are using HTPASSWD authentication, you can create one using the following command:

    # htpasswd /etc/origin/openshift-htpasswd <user_name>
  • The user must have the system:registry role. To add this role:

    # oadm policy add-role-to-user system:registry <user_name>
  • Have the admin role for the project associated with the Docker operation. For example, if accessing images in the global openshift project:

     $ oadm policy add-role-to-user admin <user_name> -n openshift
  • For writing or pushing images, for example when using the docker push command, the user must have the system:image-builder role. To add this role:

    $ oadm policy add-role-to-user system:image-builder <user_name>

For more information on user permissions, see Managing Role Bindings.

3.3.3.2. Logging in to the Registry

Note

Ensure your user satisfies the prerequisites for accessing the registry directly.

To log in to the registry directly:

  1. Ensure you are logged in to OpenShift Container Platform as a regular user:

    $ oc login
  2. Get your access token:

    $ oc whoami -t
  3. Log in to the Docker registry:

    $ docker login -u <username> -e <any_email_address> \
        -p <token_value> <registry_ip>:<port>

3.3.3.3. Pushing and Pulling Images

After logging in to the registry, you can perform docker pull and docker push operations against your registry.

Important

You can pull arbitrary images, but if you have the system:registry role added, you can only push images to the registry in your project.

In the following examples, we use:

Component

Value

<registry_ip>

172.30.124.220

<port>

5000

<project>

openshift

<image>

busybox

<tag>

omitted (defaults to latest)

  1. Pull an arbitrary image:

    $ docker pull docker.io/busybox
  2. Tag the new image with the form <registry_ip>:<port>/<project>/<image>. The project name must appear in this pull specification for OpenShift Container Platform to correctly place and later access the image in the registry.

    $ docker tag docker.io/busybox 172.30.124.220:5000/openshift/busybox
    Note

    Your regular user must have the system:image-builder role for the specified project, which allows the user to write or push an image. Otherwise, the docker push in the next step will fail. To test, you can create a new project to push the busybox image.

  3. Push the newly-tagged image to your registry:

    $ docker push 172.30.124.220:5000/openshift/busybox
    ...
    cf2616975b4a: Image successfully pushed
    Digest: sha256:3662dd821983bc4326bee12caec61367e7fb6f6a3ee547cbaff98f77403cab55

3.4. Securing and Exposing the Registry

3.4.1. Securing the Registry

Optionally, you can secure the registry so that it serves traffic via TLS:

  1. Deploy the registry.
  2. Fetch the service IP and port of the registry:

    $ oc get svc/docker-registry
    NAME              LABELS                                    SELECTOR                  IP(S)            PORT(S)
    docker-registry   docker-registry=default                   docker-registry=default   172.30.124.220   5000/TCP
  3. You can use an existing server certificate, or create a key and server certificate valid for specified IPs and host names, signed by a specified CA. To create a server certificate for the registry service IP and the docker-registry.default.svc.cluster.local host name:

    $ oadm ca create-server-cert \
        --signer-cert=/etc/origin/master/ca.crt \
        --signer-key=/etc/origin/master/ca.key \
        --signer-serial=/etc/origin/master/ca.serial.txt \
        --hostnames='docker-registry.default.svc.cluster.local,172.30.124.220' \
        --cert=/etc/secrets/registry.crt \
        --key=/etc/secrets/registry.key

    If the router will be exposed externally, add the public route host name in the --hostnames flag:

    --hostnames='mydocker-registry.example.com,docker-registry.default.svc.cluster.local,172.30.124.220
    Note

    The oadm ca create-server-cert command generates a certificate that is valid for two years. This can be altered with the --expire-days option, but for security reasons, it is recommended to not make it greater than this value.

  4. Create the secret for the registry certificates:

    $ oc secrets new registry-secret \
        /etc/secrets/registry.crt \
        /etc/secrets/registry.key
  5. Add the secret to the registry pod’s service accounts (including the default service account):

    $ oc secrets link registry registry-secret
    $ oc secrets link default  registry-secret
    Note

    Limiting secrets to only the service accounts that reference them is disabled by default. This means that if serviceAccountConfig.limitSecretReferences is set to false (the default setting) in the master configuration file, linking secrets to a service is not required.

  6. Add the secret volume to the registry deployment configuration:

    $ oc volume dc/docker-registry --add --type=secret \
        --secret-name=registry-secret -m /etc/secrets
  7. Enable TLS by adding the following environment variables to the registry deployment configuration:

    $ oc set env dc/docker-registry \
        REGISTRY_HTTP_TLS_CERTIFICATE=/etc/secrets/registry.crt \
        REGISTRY_HTTP_TLS_KEY=/etc/secrets/registry.key

    See more details on overriding registry options.

  8. Update the scheme used for the registry’s liveness probe from HTTP to HTTPS:

    $ oc patch dc/docker-registry -p '{"spec": {"template": {"spec": {"containers":[{
        "name":"registry",
        "livenessProbe":  {"httpGet": {"scheme":"HTTPS"}}
      }]}}}}'
  9. If your registry was initially deployed on OpenShift Container Platform 3.2 or later, update the scheme used for the registry’s readiness probe from HTTP to HTTPS:

    $ oc patch dc/docker-registry -p '{"spec": {"template": {"spec": {"containers":[{
        "name":"registry",
        "readinessProbe":  {"httpGet": {"scheme":"HTTPS"}}
      }]}}}}'
  10. Validate the registry is running in TLS mode. Wait until the latest docker-registry deployment completes and verify the Docker logs for the registry container. You should find an entry for listening on :5000, tls.

    $ oc logs dc/docker-registry | grep tls
    time="2015-05-27T05:05:53Z" level=info msg="listening on :5000, tls" instance.id=deeba528-c478-41f5-b751-dc48e4935fc2
  11. Copy the CA certificate to the Docker certificates directory. This must be done on all nodes in the cluster:

    $ dcertsdir=/etc/docker/certs.d
    $ destdir_addr=$dcertsdir/172.30.124.220:5000
    $ destdir_name=$dcertsdir/docker-registry.default.svc.cluster.local:5000
    
    $ sudo mkdir -p $destdir_addr $destdir_name
    $ sudo cp ca.crt $destdir_addr    1
    $ sudo cp ca.crt $destdir_name
    1
    The ca.crt file is a copy of /etc/origin/master/ca.crt on the master.
  12. Remove the --insecure-registry option only for this particular registry in the /etc/sysconfig/docker file. Then, reload the daemon and restart the docker service to reflect this configuration change:

    $ sudo systemctl daemon-reload
    $ sudo systemctl restart docker
  13. Validate the docker client connection. Running docker push to the registry or docker pull from the registry should succeed. Make sure you have logged into the registry.

    $ docker tag|push <registry/image> <internal_registry/project/image>

    For example:

    $ docker pull busybox
    $ docker tag docker.io/busybox 172.30.124.220:5000/openshift/busybox
    $ docker push 172.30.124.220:5000/openshift/busybox
    ...
    cf2616975b4a: Image successfully pushed
    Digest: sha256:3662dd821983bc4326bee12caec61367e7fb6f6a3ee547cbaff98f77403cab55

3.4.2. Exposing the Registry

To expose your internal registry externally, it is recommended that you run a secure registry. To expose the registry you must first have deployed a router.

  1. Deploy the registry.
  2. Secure the registry.
  3. Deploy a router.
  4. Create a passthrough route via the oc create route passthrough command, specifying the registry as the route’s service. By default, the name of the created route is the same as the service name.

    For example:

    $ oc get svc
    NAME              CLUSTER_IP       EXTERNAL_IP   PORT(S)                 SELECTOR                  AGE
    docker-registry   172.30.69.167    <none>        5000/TCP                docker-registry=default   4h
    kubernetes        172.30.0.1       <none>        443/TCP,53/UDP,53/TCP   <none>                    4h
    router            172.30.172.132   <none>        80/TCP                  router=router             4h
    
    $ oc create route passthrough    \
        --service=docker-registry    \1
        --hostname=<host>
    route "docker-registry" created     2
    1
    Specify the registry as the route’s service.
    2
    The route name is identical to the service name.
    $ oc get route/docker-registry -o yaml
    apiVersion: v1
    kind: Route
    metadata:
      name: docker-registry
    spec:
      host: <host> 1
      to:
        kind: Service
        name: docker-registry 2
      tls:
        termination: passthrough 3
    1
    The host for your route. You must be able to resolve this name externally via DNS to the router’s IP address.
    2
    The service name for your registry.
    3
    Specify this route as a passthrough route.
    Note

    Passthrough is currently the only type of route supported for exposing the secure registry.

  5. Next, you must trust the certificates being used for the registry on your host system. The certificates referenced were created when you secured your registry.

    $ sudo mkdir -p /etc/docker/certs.d/<host>
    $ sudo cp <ca certificate file> /etc/docker/certs.d/<host>
    $ sudo systemctl restart docker
  6. Log in to the registry using the information from securing the registry. However, this time point to the host name used in the route rather than your service IP. You should now be able to tag and push images using the route host.

    $ oc get imagestreams -n test
    NAME      DOCKER REPO   TAGS      UPDATED
    
    $ docker pull busybox
    $ docker tag busybox <host>/test/busybox
    $ docker push <host>/test/busybox
    The push refers to a repository [<host>/test/busybox] (len: 1)
    8c2e06607696: Image already exists
    6ce2e90b0bc7: Image successfully pushed
    cf2616975b4a: Image successfully pushed
    Digest: sha256:6c7e676d76921031532d7d9c0394d0da7c2906f4cb4c049904c4031147d8ca31
    
    $ docker pull <host>/test/busybox
    latest: Pulling from <host>/test/busybox
    cf2616975b4a: Already exists
    6ce2e90b0bc7: Already exists
    8c2e06607696: Already exists
    Digest: sha256:6c7e676d76921031532d7d9c0394d0da7c2906f4cb4c049904c4031147d8ca31
    Status: Image is up to date for <host>/test/busybox:latest
    
    $ oc get imagestreams -n test
    NAME      DOCKER REPO                       TAGS      UPDATED
    busybox   172.30.11.215:5000/test/busybox   latest    2 seconds ago
    Note

    Your image streams will have the IP address and port of the registry service, not the route name and port. See oc get imagestreams for details.

    Note

    In the <host>/test/busybox example above, test refers to the project name.

3.4.2.1. Exposing a Secure Registry

Instead of logging in to the registry from within the OpenShift Container Platform cluster, you can gain external access to it by first securing the registry and then exposing the registry. This allows you to log in to the registry from outside the cluster using the route address, and to tag and push images using the route host.

When logging in to the secured and exposed registry, make sure you specify the registry in the login command. For example:

docker login -e user@company.com -u f83j5h6 -p Ju1PeM47R0B92Lk3AZp-bWJSck2F7aGCiZ66aFGZrs2 registry.example.com:80

3.4.2.2. Exposing a Non-Secure Registry

Instead of securing the registry in order to expose the registry, you can simply expose a non-secure registry for non-production OpenShift Container Platform environments. This allows you to have an external route to the registry without using SSL certificates.

Warning

Only non-production environments should expose a non-secure registry to external access.

To expose a non-secure registry:

  1. Expose the registry:

    # oc expose service docker-registry --hostname=<hostname> -n default

    This creates the following JSON file:

    apiVersion: v1
    kind: Route
    metadata:
      creationTimestamp: null
      labels:
        docker-registry: default
      name: docker-registry
    spec:
      host: registry.example.com
      port:
        targetPort: "5000"
      to:
        kind: Service
        name: docker-registry
    status: {}
  2. Verify that the route has been created successfully:

    # oc get route
    NAME              HOST/PORT                    PATH      SERVICE           LABELS                    INSECURE POLICY   TLS TERMINATION
    docker-registry   registry.example.com            docker-registry   docker-registry=default
  3. Check the health of the registry:

    $ curl -v http://registry.example.com/healthz

    Expect an HTTP 200/OK message.

    After exposing the registry, update your /etc/sysconfig/docker file by adding the port number to the OPTIONS entry. For example:

    OPTIONS='--selinux-enabled --insecure-registry=172.30.0.0/16 --insecure-registry registry.example.com:80'
    Important

    The above options should be added on the client from which you are trying to log in.

    Also, ensure that Docker is running on the client.

When logging in to the non-secured and exposed registry, make sure you specify the registry in the login command. For example:

docker login -e user@company.com -u f83j5h6 -p Ju1PeM47R0B92Lk3AZp-bWJSck2F7aGCiZ66aFGZrs2 registry.example.com:80

3.5. Extended Registry Configuration

3.5.1. Maintaining the Registry IP Address

OpenShift Container Platform refers to the integrated registry by its service IP address, so if you decide to delete and recreate the docker-registry service, you can ensure a completely transparent transition by arranging to re-use the old IP address in the new service. If a new IP address cannot be avoided, you can minimize cluster disruption by rebooting only the masters.

Re-using the Address
To re-use the IP address, you must save the IP address of the old docker-registry service prior to deleting it, and arrange to replace the newly assigned IP address with the saved one in the new docker-registry service.
  1. Make a note of the clusterIP for the service:

    $ oc get svc/docker-registry -o yaml | grep clusterIP:
  2. Delete the service:

    $ oc delete svc/docker-registry dc/docker-registry
  3. Create the registry definition in registry.yaml, replacing <options> with, for example, those used in step 3 of the instructions in the Non-Production Use section:

    $ oadm registry <options> -o yaml > registry.yaml
  4. Edit registry.yaml, find the Service there, and change its clusterIP to the address noted in step 1.
  5. Create the registry using the modified registry.yaml:

    $ oc create -f registry.yaml
Rebooting the Masters

If you are unable to re-use the IP address, any operation that uses a pull specification that includes the old IP address will fail. To minimize cluster disruption, you must reboot the masters:

# systemctl restart atomic-openshift-master

This ensures that the old registry URL, which includes the old IP address, is cleared from the cache.

Note

We recommend against rebooting the entire cluster because that incurs unnecessary downtime for pods and does not actually clear the cache.

3.5.2. Whitelisting Docker Registries

You can specify a whitelist of docker registries, allowing you to curate a set of images and templates that are available for download by OpenShift Container Platform users. This curated set can be placed in one or more docker registries, and then added to the whitelist. When using a whitelist, only the specified registries are accessible within OpenShift Container Platform, and all other registries are denied access by default.

To configure a whitelist:

  1. Edit the /etc/sysconfig/docker file to block all registries:

    BLOCK_REGISTRY='--block-registry=all'

    You may need to uncomment the BLOCK_REGISTRY line.

  2. In the same file, add registries to which you want to allow access:

    ADD_REGISTRY='--add-registry=<registry1> --add-registry=<registry2>'

    Example 3.1. Allowing Access to Registries

    ADD_REGISTRY='--add-registry=registry.access.redhat.com'

    This example would restrict access to images available on the Red Hat Customer Portal.

Once the whitelist is configured, if a user tries to pull from a docker registry that is not on the whitelist, they will receive an error message stating that this registry is not allowed.

3.5.3. Overriding the Registry Configuration

You can override the integrated registry’s default configuration, found by default at /config.yml in a running registry’s container, with your own custom configuration.

Note

Upstream configuration options in this file may also be overridden using environment variables. The middleware section is an exception as there are just a few options that can be overridden using environment variables. Learn how to override specific configuration options.

To enable management of the registry configuration file directly and deploy an updated configuration, mount the configuration file as a secret volume:

  1. Deploy the registry.
  2. Edit the registry configuration file locally as needed. The initial YAML file deployed on the registry is provided below. Review supported options.

    Example 3.2. Registry Configuration File

    version: 0.1
    log:
      level: debug
    http:
      addr: :5000
    storage:
      cache:
        blobdescriptor: inmemory
      filesystem:
        rootdirectory: /registry
      delete:
        enabled: true
    auth:
      openshift:
        realm: openshift
    middleware:
      registry:
        - name: openshift
      repository:
        - name: openshift
          options:
            acceptschema2: false
            pullthrough: true
            enforcequota: false
            projectcachettl: 1m
            blobrepositorycachettl: 10m
      storage:
        - name: openshift
  3. Create a new secret called registry-config from your custom registry configuration file you edited locally:

    $ oc secrets new registry-config config.yml=</path/to/custom/registry/config.yml>
  4. Add the registry-config secret as a volume to the registry’s deployment configuration to mount the custom configuration file at /etc/docker/registry/:

    $ oc volume dc/docker-registry --add --type=secret \
        --secret-name=registry-config -m /etc/docker/registry/
  5. Update the registry to reference the configuration path from the previous step by adding the following environment variable to the registry’s deployment configuration:

    $ oc set env dc/docker-registry \
        REGISTRY_CONFIGURATION_PATH=/etc/docker/registry/config.yml

This may be performed as an iterative process to achieve the desired configuration. For example, during troubleshooting, the configuration may be temporarily updated to put it in debug mode.

To update an existing configuration:

Warning

This procedure will overwrite the currently deployed registry configuration.

  1. Edit the local registry configuration file, config.yml.
  2. Delete the registry-config secret:

    $ oc delete secret registry-config
  3. Recreate the secret to reference the updated configuration file:

    $ oc secrets new registry-config config.yml=</path/to/custom/registry/config.yml>
  4. Redeploy the registry to read the updated configuration:

    $ oc rollout latest docker-registry
Tip

Maintain configuration files in a source control repository.

3.5.4. Registry Configuration Reference

There are many configuration options available in the upstream docker distribution library. Not all configuration options are supported or enabled. Use this section as a reference when overriding the registry configuration.

Note

Upstream configuration options in this file may also be overridden using environment variables. However, the middleware section may not be overridden using environment variables. Learn how to override specific configuration options.

3.5.4.1. Log

Upstream options are supported.

Example:

log:
  level: debug
  formatter: text
  fields:
    service: registry
    environment: staging

3.5.4.2. Hooks

Mail hooks are not supported.

3.5.4.3. Storage

The following storage drivers are supported:

General registry storage configuration options are supported.

Example 3.3. General Storage Configuration Options

storage:
  delete:
    enabled: true 1
  redirect:
    disable: false
  cache:
    blobdescriptor: inmemory
  maintenance:
    uploadpurging:
      enabled: true
      age: 168h
      interval: 24h
      dryrun: false
    readonly:
      enabled: false
1
This entry is mandatory for image pruning to work properly.

3.5.4.4. Auth

Auth options should not be altered. The openshift extension is the only supported option.

auth:
  openshift:
    realm: openshift

3.5.4.5. Middleware

The repository middleware extension allows to configure OpenShift Container Platform middleware responsible for interaction with OpenShift Container Platform and image proxying.

middleware:
  registry:
    - name: openshift 1
  repository:
    - name: openshift 2
      options:
        acceptschema2: false 3
        pullthrough: true 4
        enforcequota: false 5
        projectcachettl: 1m 6
        blobrepositorycachettl: 10m 7
  storage:
    - name: openshift 8
1 2 8
These entries are mandatory. Their presence ensures required components get loaded. These values shouldn’t be changed.
3
Allow to store manifest schema v2 during a push to the registry. See below for more details.
4
Let the registry act as a proxy for remote blobs. See below for more details.
5
Prevent blob uploads exceeding size limit defined in targeted project.
6
An expiration timeout for limits cached in the registry. The lower the value, the less time will it take for the limit changes to propagate to the registry. However, the registry will query limits from the server more frequently and, as a consequence, pushes will be slower.
7
An expiration timeout for remembered associations between blob and repository. The higher the value, the higher probability of fast lookup and more efficient registry operation. On the other hand, memory usage will raise as well as a risk of serving image layer to user, who is no longer authorized to access it.
3.5.4.5.1. CloudFront Middleware

The CloudFront middleware extension can be added to support AWS, CloudFront CDN storage provider. CloudFront middleware speeds up distribution of image content internationally. The blobs are distributed to several edge locations around the world. The client is always directed to the edge with the lowest latency.

Note

The CloudFront middleware extension can be only used with S3 storage. It is utilized only during blob serving. Therefore, only blob downloads can be speeded up, not uploads.

The following is an example of minimal configuration of S3 storage driver with a CloudFront middleware:

version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  cache:
    blobdescriptor: inmemory
  delete:
    enabled: true
  s3: 1
    accesskey: BJKMSZBRESWJQXRWMAEQ
    secretkey: 5ah5I91SNXbeoUXXDasFtadRqOdy62JzlnOW1goS
    region: us-east-1
    bucket: docker.myregistry.com
auth:
  openshift:
    realm: openshift
middleware:
  registry:
    - name: openshift
  repository:
    - name: openshift
   storage:
    - name: cloudfront 2
      options:
        baseurl: https://jrpbyn0k5k88bi.cloudfront.net/ 3
        privatekey: /etc/docker/cloudfront-ABCEDFGHIJKLMNOPQRST.pem 4
        keypairid: ABCEDFGHIJKLMNOPQRST 5
    - name: openshift
1
The S3 storage must be configured the same way regardless of CloudFront middleware.
2
The CloudFront storage middleware needs to be listed before OpenShift middleware.
3
The CloudFront base URL. In the AWS management console, this is listed as Domain Name of CloudFront distribution.
4
The location of your AWS private key on the filesystem. This must be not confused with Amazon EC2 key pair. Please refer to AWS documentation on creating CloudFront key pairs for your trusted signers. The file needs to be mounted as a secret secret into the registry pod.
5
The ID of your Cloudfront key pair.
3.5.4.5.2. Overriding Middleware Configuration Options

The middleware section cannot be overridden using environment variables. There are a few exceptions, however. For example:

middleware:
  repository:
    - name: openshift
      options:
        acceptschema2: false 1
        enforcequota: false 2
        projectcachettl: 1m 3
        blobrepositorycachettl: 10m 4
1
A configuration option that can be overridden by the boolean environment variable REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ACCEPTSCHEMA2, which allows for the ability to accept manifest schema v2 on manifest put requests.
2
A configuration option that can be overridden by the boolean environment variable REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA, which allows the ability to turn quota enforcement on or off. By default, quota enforcement is off. It overrides OpenShift Container Platform middleware configuration option. Recognized values are true and false.
3
A configuration option that can be overridden by the environment variable REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_PROJECTCACHETTL, specifying an eviction timeout for project quota objects. It takes a valid time duration string (for example, 2m). If empty, you get the default timeout. If zero (0m), caching is disabled.
4
A configuration option that can be overriden by the environment variable REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_BLOBREPOSITORYCACHETTL, specifying an eviction timeout for associations between blob and containing repository. The format of the value is the same as in projectcachettl case.
3.5.4.5.3. Image Pullthrough

If enabled, the registry will attempt to fetch requested blob from a remote registry unless the blob exists locally. The remote candidates are calculated from DockerImage entries stored in status of the image stream, a client pulls from. All the unique remote registry references in such entries will be tried in turn until the blob is found. The blob, served this way, will not be stored in the registry.

This feature is on by default. However, it can be disabled using a configuration option.

3.5.4.5.4. Manifest Schema v2 Support

Each image has a manifest describing its blobs, instructions for running it and additional metadata. The manifest is versioned which have different structure and fields as it evolves over time. The same image can be represented by multiple manifest versions. Each version will have different digest though.

The registry currently supports manifest v2 schema 1 (schema1) and manifest v2 schema 2 (schema2). The former is being obsoleted but will be supported for an extended amount of time.

You should be wary of compatibility issues with various Docker clients:

  • Docker clients of version 1.9 or older support only schema1. Any manifest this client pulls or pushes will be of this legacy schema.
  • Docker clients of version 1.10 support both schema1 and schema2. And by default, it will push the latter to the registry if it supports newer schema.

The registry, storing an image with schema1 will always return it unchanged to the client. Schema2 will be transferred unchanged only to newer Docker client. For the older one, it will be converted on-the-fly to schema1.

This has significant consequences. For example an image pushed to the registry by a newer Docker client cannot be pulled by the older Docker by its digest. That’s because the stored image’s manifest is of schema2 and its digest can be used to pull only this version of manifest.

For this reason, the registry is configured by default not to store schema2. This ensures that any docker client will be able to pull from the registry any image pushed there regardless of client’s version.

Once you’re confident that all the registry clients support schema2, you’ll be safe to enable its support in the registry. See the middleware configuration reference above for particular option.

3.5.4.6. Reporting

Reporting is unsupported.

3.5.4.7. HTTP

Upstream options are supported. Learn how to alter these settings via environment variables. Only the tls section should be altered. For example:

http:
  addr: :5000
  tls:
    certificate: /etc/secrets/registry.crt
    key: /etc/secrets/registry.key

3.5.4.8. Notifications

Upstream options are supported. The REST API Reference provides more comprehensive integration options.

Example:

notifications:
  endpoints:
    - name: registry
      disabled: false
      url: https://url:port/path
      headers:
        Accept:
          - text/plain
      timeout: 500
      threshold: 5
      backoff: 1000

3.5.4.9. Redis

Redis is not supported.

3.5.4.10. Health

Upstream options are supported. The registry deployment configuration provides an integrated health check at /healthz.

3.5.4.11. Proxy

Proxy configuration should not be enabled. This functionality is provided by the OpenShift Container Platform repository middleware extension, pullthrough: true.

3.6. Known Issues

3.6.1. Overview

The following are the known issues when deploying or using the integrated registry.

3.6.2. Image Push Errors with Scaled Registry Using Shared NFS Volume

When using a scaled registry with a shared NFS volume, you may see one of the following errors during the push of an image:

  • digest invalid: provided digest did not match uploaded content
  • blob upload unknown
  • blob upload invalid

These errors are returned by an internal registry service when Docker attempts to push the image. Its cause originates in the synchronization of file attributes across nodes. Factors such as NFS client side caching, network latency, and layer size can all contribute to potential errors that might occur when pushing an image using the default round-robin load balancing configuration.

You can perform the following steps to minimize the probability of such a failure:

  1. Ensure that the sessionAffinity of your docker-registry service is set to ClientIP:

    $ oc get svc/docker-registry --template='{{.spec.sessionAffinity}}'

    This should return ClientIP, which is the default in recent OpenShift Container Platform versions. If not, change it:

    $ oc get -o yaml svc/docker-registry | \
          sed 's/\(sessionAffinity:\s*\).*/\1ClientIP/' | \
          oc replace -f -
  2. Ensure that the NFS export line of your registry volume on your NFS server has the no_wdelay options listed. See Export Settings in the Persistent Storage Using NFS topic for details.
Note

The guidelines for NFS are recommended to help you get started. You may switch off from NFS when moving to production.

3.6.3. Pull of Internally Managed Image Fails with "not found" Error

This error occurs when the pulled image is pushed to an image stream different from the one it is being pulled from. This is caused by re-tagging a built image into an arbitrary image stream:

$ oc tag srcimagestream:latest anyproject/pullimagestream:latest

And subsequently pulling from it, using an image reference such as:

internal.registry.url:5000/anyproject/pullimagestream:latest

During a manual Docker pull, this will produce a similar error:

Error: image anyproject/pullimagestream:latest not found

To prevent this, avoid the tagging of internally managed images completely, or re-push the built image to the desired namespace manually.

3.6.4. Image Push Fails with "500 Internal Server Error" on S3 Storage

There are problems reported happening when the registry runs on S3 storage back-end. Pushing to a Docker registry occasionally fails with the following error:

Received unexpected HTTP status: 500 Internal Server Error

To debug this, you need to view the registry logs. In there, look for similar error messages occurring at the time of the failed push:

time="2016-03-30T15:01:21.22287816-04:00" level=error msg="unknown error completing upload: driver.Error{DriverName:\"s3\", Enclosed:(*url.Error)(0xc20901cea0)}" http.request.method=PUT
...
time="2016-03-30T15:01:21.493067808-04:00" level=error msg="response completed with error" err.code=UNKNOWN err.detail="s3: Put https://s3.amazonaws.com/oso-tsi-docker/registry/docker/registry/v2/blobs/sha256/ab/abe5af443833d60cf672e2ac57589410dddec060ed725d3e676f1865af63d2e2/data: EOF" err.message="unknown error" http.request.method=PUT
...
time="2016-04-02T07:01:46.056520049-04:00" level=error msg="error putting into main store: s3: The request signature we calculated does not match the signature you provided. Check your key and signing method." http.request.method=PUT
atest

If you see such errors, contact your Amazon S3 support. There may be a problem in your region or with your particular bucket.

3.6.5. Image Pruning Fails

If you encounter the following error when pruning images:

BLOB sha256:49638d540b2b62f3b01c388e9d8134c55493b1fa659ed84e97cb59b87a6b8e6c error deleting blob

And your registry log contains the following information:

error deleting blob \"sha256:49638d540b2b62f3b01c388e9d8134c55493b1fa659ed84e97cb59b87a6b8e6c\": operation unsupported

It means that your custom configuration file lacks mandatory entries in the storage section, namely storage:delete:enabled set to true. Add them, re-deploy the registry, and repeat your image pruning operation.

Chapter 4. Setting up a Router

4.1. Router Overview

4.1.1. About Routers

The OpenShift Container Platform router is the ingress point for all external traffic destined for services in your OpenShift Container Platform installation. OpenShift Container Platform provides and supports the following two router plug-ins:

  • The HAProxy template router is the default plug-in. It uses the openshift3/ose-haproxy-router image to run an HAProxy instance alongside the template router plug-in inside a container on OpenShift Container Platform. It currently supports HTTP(S) traffic and TLS-enabled traffic via SNI. The router’s container listens on the host network interface, unlike most containers that listen only on private IPs. The router proxies external requests for route names to the IPs of actual pods identified by the service associated with the route.
  • The F5 router integrates with an existing F5 BIG-IP® system in your environment to synchronize routes. F5 BIG-IP® version 11.4 or newer is required in order to have the F5 iControl REST API.
Note

The F5 router plug-in is available starting in OpenShift Container Platform 3.0.2.

4.1.2. Router Service Account

Before deploying an OpenShift Container Platform cluster, you must have a service account for the router. Starting in OpenShift Container Platform 3.1, a router service account is automatically created during a quick or advanced installation (previously, this required manual creation). This service account has permissions to a security context constraint (SCC) that allows it to specify host ports.

Use of labels (e.g., to define router shards) requires cluster-reader permission.

$ oadm policy add-cluster-role-to-user \
    cluster-reader \
    system:serviceaccount:default:router

4.2. Using the Default HAProxy Router

4.2.1. Overview

The oadm router command is provided with the administrator CLI to simplify the tasks of setting up routers in a new installation. If you followed the quick installation, then a default router was automatically created for you. The oadm router command creates the service and deployment configuration objects. Just about every form of communication between OpenShift Container Platform components is secured by TLS and uses various certificates and authentication methods. Use the --service-account option to specify the service account the router will use to contact the master.

Important

Routers directly attach to port 80 and 443 on all interfaces on a host. Restrict routers to hosts where port 80/443 is available and not being consumed by another service, and set this using node selectors and the scheduler configuration. As an example, you can achieve this by dedicating infrastructure nodes to run services such as routers.

Important

It is recommended to use separate distinct openshift-router service account with your router. This can be provided using the --service-account flag to the oadm router command.

$ oadm router --dry-run --service-account=router 1
1
--service-account is the name of a service account for the openshift-router.
Important

Router pods created using oadm router have default resource requests that a node must satisfy for the router pod to be deployed. In an effort to increase the reliability of infrastructure components, the default resource requests are used to increase the QoS tier of the router pods above pods without resource requests. The default values represent the observed minimum resources required for a basic router to be deployed and can be edited in the routers deployment configuration and you may want to increase them based on the load of the router.

4.2.2. Creating a Router

The quick installation process automatically creates a default router. If the router does not exist, run the following to create a router:

$ oadm router <router_name> --replicas=<number> --service-account=router

You can also use router shards to ensure that the router is filtered to specific namespaces or routes, or set any environment variables after router creation.

4.2.3. Other Basic Router Commands

Checking the Default Router
The default router service account, named router, is automatically created during quick and advanced installations. To verify that this account already exists:
$ oadm router --dry-run --service-account=router
Viewing the Default Router
To see what the default router would look like if created:
$ oadm router -o yaml --service-account=router
Deploying the Router to a Labeled Node
To deploy the router to any node(s) that match a specified node label:
$ oadm router <router_name> --replicas=<number> --selector=<label> \
    --service-account=router

For example, if you want to create a router named router and have it placed on a node labeled with region=infra:

$ oadm router router --replicas=1 --selector='region=infra' \
  --service-account=router

During advanced installation, the openshift_hosted_router_selector and openshift_registry_selector Ansible settings are set to region=infra by default. The default router and registry will only be automatically deployed if a node exists that matches the region=infra label.

Multiple instances are created on different hosts according to the scheduler policy.

Using a Different Router Image
To use a different router image and view the router configuration that would be used:
$ oadm router <router_name> -o <format> --images=<image> \
    --service-account=router

For example:

$ oadm router region-west -o yaml --images=myrepo/somerouter:mytag \
    --service-account=router

4.2.4. Filtering Routes to Specific Routers

Using the ROUTE_LABELS environment variable, you can filter routes so that they are used only by specific routers.

For example, if you have multiple routers, and 100 routes, you can attach labels to the routes so that a portion of them are handled by one router, whereas the rest are handled by another.

  1. After creating a router, use the ROUTE_LABELS environment variable to tag the router:

    $ oc env dc/<router=name>  ROUTE_LABELS="key=value"
  2. Add the label to the desired routes:

    oc label route <route=name> key=value
  3. To verify that the label has been attached to the route, check the route configuration:

    $ oc describe dc/<route_name>

4.2.5. Highly-Available Routers

You can set up a highly-available router on your OpenShift Container Platform cluster using IP failover.

4.2.6. Customizing the Router Service Ports

You can customize the service ports that a template router binds to by setting the environment variables ROUTER_SERVICE_HTTP_PORT and ROUTER_SERVICE_HTTPS_PORT. This can be done by creating a template router, then editing its deployment configuration.

The following example creates a router deployment with 0 replicas and customizes the router service HTTP and HTTPS ports, then scales it appropriately (to 1 replica).

$ oadm router --replicas=0 --ports='10080:10080,10443:10443' 1
$ oc set env dc/router ROUTER_SERVICE_HTTP_PORT=10080  \
                   ROUTER_SERVICE_HTTPS_PORT=10443
$ oc scale dc/router --replicas=1
1
Ensures exposed ports are appropriately set for routers that use the container networking mode --host-network=false.
Important

If you do customize the template router service ports, you will also need to ensure that the nodes where the router pods run have those custom ports opened in the firewall (either via Ansible or iptables, or any other custom method that you use via firewall-cmd).

The following is an example using iptables to open the custom router service ports.

$ iptables -A INPUT -p tcp --dport 10080 -j ACCEPT
$ iptables -A INPUT -p tcp --dport 10443 -j ACCEPT

4.2.7. Working With Multiple Routers

An administrator can create multiple routers with the same definition to serve the same set of routes. By having different groups of routers with different namespace or route selectors, they can vary the routes that the router serves.

Multiple routers can be grouped to distribute routing load in the cluster and separate tenants to different routers or shards. Each router or shard in the group handles routes based on the selectors in the router. An administrator can create shards over the whole cluster using ROUTE_LABELS. A user can create shards over a namespace (project) by using NAMESPACE_LABELS.

4.2.8. Adding a Node Selector to a Deployment Configuration

Making specific routers deploy on specific nodes requires two steps:

  1. Add a label to the desired node:

    $ oc label node 10.254.254.28 "router=first"
  2. Add a node selector to the router deployment configuration:

    $ oc edit dc <deploymentConfigName>

    Add the template.spec.nodeSelector field with a key and value corresponding to the label:

    ...
      template:
        metadata:
          creationTimestamp: null
          labels:
            router: router1
        spec:
          nodeSelector:      1
            router: "first"
    ...
    1
    The key and value are router and first, respectively, corresponding to the router=first label.

4.2.9. Using Router Shards

The access controls are based on the service account that the router is run with.

Using NAMESPACE_LABELS and/or ROUTE_LABELS, a router can filter out the namespaces and/or routes that it should service. This enables you to partition routes amongst multiple router deployments effectively distributing the set of routes.

Example: A router deployment finops-router is run with route selector NAMESPACE_LABELS="name in (finance, ops)" and a router deployment dev-router is run with route selector NAMESPACE_LABELS="name=dev".

If all routes are in the three namespaces finance, ops or dev, then this could effectively distribute your routes across two router deployments.

In the above scenario, sharding becomes a special case of partitioning with no overlapping sets. Routes are divided amongst multiple router shards.

The criteria for route selection governs how the routes are distributed. It is possible to have routes that overlap across multiple router deployments.

Example: In addition to the finops-router and dev-router in the example above, you also have devops-router, which is run with a route selector NAMESPACE_LABELS="name in (dev, ops)".

The routes in namespaces dev or ops now are serviced by two different router deployments. This becomes a case in which you have partitioned the routes with an overlapping set.

In addition, this enables you to create more complex routing rules, allowing the diversion of high priority traffic to the dedicated finops-router, but sending the lower priority ones to the devops-router.

NAMESPACE_LABELS allows filtering of the projects to service and selecting all the routes from those projects, but you may want to partition routes based on other criteria in the routes themselves. The ROUTE_LABELS selector allows you to slice-and-dice the routes themselves.

Example: A router deployment prod-router is run with route selector ROUTE_LABELS="mydeployment=prod" and a router deployment devtest-router is run with route selector ROUTE_LABELS="mydeployment in (dev, test)".

The example assumes you have all the routes you want to be serviced tagged with a label "mydeployment=<tag>".

4.2.9.1. Creating Router Shards

Router sharding lets you select how routes are distributed among a set of routers.

Router sharding is based on labels; you set labels on the routes in the pool, and express the desired subset of those routes for the router to serve with a selection expression via the oc set env command.

First, ensure that service account associated with the router has the cluster reader permission.

The rest of this section describes an extended example. Suppose there are 26 routes, named a — z, in the pool, with various labels:

Possible labels on routes in the pool

sla=high       geo=east     hw=modest     dept=finance
sla=medium     geo=west     hw=strong     dept=dev
sla=low                                   dept=ops

These labels express the concepts: service level agreement, geographical location, hardware requirements, and department. The routes in the pool can have at most one label from each column. Some routes may have other labels, entirely, or none at all.

Name(s)SLAGeoHWDeptOther Labels

a

high

east

modest

finance

type=static

b

 

west

strong

 

type=dynamic

c, d, e

low

 

modest

 

type=static

g — k

medium

 

strong

dev

 

l — s

high

 

modest

ops

 

t — z

 

west

  

type=dynamic

Here is a convenience script mkshard that ilustrates how oadm router, oc set env, and oc scale work together to make a router shard.

#!/bin/bash
# Usage: mkshard ID SELECTION-EXPRESSION
id=$1
sel="$2"
router=router-shard-$id           1
oadm router $router --replicas=0  2
dc=dc/router-shard-$id            3
oc set env   $dc ROUTE_LABELS="$sel"  4
oc scale $dc --replicas=3         5
1
The created router has name router-shard-<id>.
2
Specify no scaling for now.
3
The deployment configuration for the router.
4
Set the selection expression using oc set env. The selection expression is the value of the ROUTE_LABELS environment variable.
5
Scale it up.

Running mkshard several times creates several routers:

RouterSelection ExpressionRoutes

router-shard-1

sla=high

a, l — s

router-shard-2

geo=west

b, t — z

router-shard-3

dept=dev

g — k

4.2.9.2. Modifying Router Shards

Because a router shard is a construct based on labels, you can modify either the labels (via oc label) or the selection expression.

This section extends the example started in the Creating Router Shards section, demonstrating how to change the selection expression.

Here is a convenience script modshard that modifies an existing router to use a new selection expression:

#!/bin/bash
# Usage: modshard ID SELECTION-EXPRESSION...
id=$1
shift
router=router-shard-$id       1
dc=dc/$router                 2
oc scale $dc --replicas=0     3
oc set env   $dc "$@"             4
oc scale $dc --replicas=3     5
1
The modified router has name router-shard-<id>.
2
The deployment configuration where the modifications occur.
3
Scale it down.
4
Set the new selection expression using oc set env. Unlike mkshard from the Creating Router Shards section, the selection expression specified as the non-ID arguments to modshard must include the environment variable name as well as its value.
5
Scale it back up.
Note

In modshard, the oc scale commands are not necessary if the deployment strategy for router-dhsard-<id> is Rolling.

For example, to expand the department for router-shard-3 to include ops as well as dev:

$ modshard 3 ROUTE_LABELS='dept in (dev, ops)'

The result is that router-shard-3 now selects routes g — s (the combined sets of g — k and l — s).

This example takes into account that there are only three departments in this example scenario, and specifies a department to leave out of the shard, thus achieving the same result as the preceding example:

$ modshard 3 ROUTE_LABELS='dept != finanace'

This example specifies shows three comma-separated qualities, and results in only route b being selected:

$ modshard 3 ROUTE_LABELS='hw=strong,type=dynamic,geo=west'

Similarly to ROUTE_LABELS, which involve a route’s labels, you can select routes based on the labels of the route’s namespace labels, with the NAMESPACE_LABELS environment variable. This example modifies router-shard-3 to serve routes whose namespace has the label frequency=weekly:

$ modshard 3 NAMESPACE_LABELS='frequency=weekly'

The last example combines ROUTE_LABELS and NAMESPACE_LABELS to select routes with label sla=low and whose namespace has the label frequency=weekly:

$ modshard 3 \
    NAMESPACE_LABELS='frequency=weekly' \
    ROUTE_LABELS='sla=low'

4.2.9.3. Using Namespace Router Shards

The routes for a project can be handled by a selected router by using NAMESPACE_LABELS. The router is given a selector for a NAMESPACE_LABELS label and the project that wants to use the router applies the NAMESPACE_LABELS label to its namespace.

First, ensure that service account associated with the router has the cluster reader permission. This permits the router to read the labels that are applied to the namespaces.

Now create and label the router:

$ oadm router ...  --service-account=router
$ oc set env dc/router NAMESPACE_LABELS="router=r1"

Because the router has a selector for a namespace, the router will handle routes for that namespace. So, for example:

$ oc label namespace default "router=r1"

Now create routes in the default namespace, and the route is available in the default router:

$ oc create -f route1.yaml

Now create a new project (namespace) and create a route, route2.

$ oc new-project p1
$ oc create -f route2.yaml

And notice the route is not available in your router. Now label namespace p1 with "router=r1"

$ oc label namespace p1 "router=r1"

Which makes the route available to the router.

Note that removing the label from the namespace won’t have immediate effect (as we don’t see the updates in the router), so if you redeploy/start a new router pod, you should see the unlabelled effects.

$ oc scale dc/router --replicas=0 && oc scale dc/router --replicas=1

4.2.10. Customizing the Default Routing Subdomain

You can customize the suffix used as the default routing subdomain for your environment by modifying the master configuration file (the /etc/origin/master/master-config.yaml file by default). Routes that do not specify a host name would have one generated using this default routing subdomain.

The following example shows how you can set the configured suffix to v3.openshift.test:

routingConfig:
  subdomain: v3.openshift.test
Note

This change requires a restart of the master if it is running.

With the OpenShift Container Platform master(s) running the above configuration, the generated host name for the example of a route named no-route-hostname without a host name added to a namespace mynamespace would be:

no-route-hostname-mynamespace.v3.openshift.test

4.2.11. Forcing Route Host Names to a Custom Routing Subdomain

If an administrator wants to restrict all routes to a specific routing subdomain, they can pass the --force-subdomain option to the oadm router command. This forces the router to override any host names specified in a route and generate one based on the template provided to the --force-subdomain option.

The following example runs a router, which overrides the route host names using a custom subdomain template ${name}-${namespace}.apps.example.com.

$ oadm router --force-subdomain='${name}-${namespace}.apps.example.com'

4.2.12. Using Wildcard Certificates

A TLS-enabled route that does not include a certificate uses the router’s default certificate instead. In most cases, this certificate should be provided by a trusted certificate authority, but for convenience you can use the OpenShift Container Platform CA to create the certificate. For example:

$ CA=/etc/origin/master
$ oadm ca create-server-cert --signer-cert=$CA/ca.crt \
      --signer-key=$CA/ca.key --signer-serial=$CA/ca.serial.txt \
      --hostnames='*.cloudapps.example.com' \
      --cert=cloudapps.crt --key=cloudapps.key

The router expects the certificate and key to be in PEM format in a single file:

$ cat cloudapps.crt cloudapps.key $CA/ca.crt > cloudapps.router.pem

From there you can use the --default-cert flag:

$ oadm router --default-cert=cloudapps.router.pem --service-account=router
Note

Browsers only consider wildcards valid for subdomains one level deep. So in this example, the certificate would be valid for a.cloudapps.example.com but not for a.b.cloudapps.example.com.

4.2.13. Using Secured Routes

Currently, password protected key files are not supported. HAProxy prompts for a password upon starting and does not have a way to automate this process. To remove a passphrase from a keyfile, you can run:

# openssl rsa -in <passwordProtectedKey.key> -out <new.key>

Here is an example of how to use a secure edge terminated route with TLS termination occurring on the router before traffic is proxied to the destination. The secure edge terminated route specifies the TLS certificate and key information. The TLS certificate is served by the router front end.

First, start up a router instance:

# oadm router --replicas=1 --service-account=router

Next, create a private key, csr and certificate for our edge secured route. The instructions on how to do that would be specific to your certificate authority and provider. For a simple self-signed certificate for a domain named www.example.test, see the example shown below:

# sudo openssl genrsa -out example-test.key 2048
#
# sudo openssl req -new -key example-test.key -out example-test.csr  \
  -subj "/C=US/ST=CA/L=Mountain View/O=OS3/OU=Eng/CN=www.example.test"
#
# sudo openssl x509 -req -days 366 -in example-test.csr  \
      -signkey example-test.key -out example-test.crt

Generate a route using the above certificate and key.

$ oc create route edge --service=my-service \
    --hostname=www.example.test \
    --key=example-test.key --cert=example-test.crt
route "my-service" created

Look at its definition.

$ oc get route/my-service -o yaml
apiVersion: v1
kind: Route
metadata:
  name:  my-service
spec:
  host: www.example.test
  to:
    kind: Service
    name: my-service
  tls:
    termination: edge
    key: |
      -----BEGIN PRIVATE KEY-----
      [...]
      -----END PRIVATE KEY-----
    certificate: |
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----

Make sure your DNS entry for www.example.test points to your router instance(s) and the route to your domain should be available. The example below uses curl along with a local resolver to simulate the DNS lookup:

# routerip="4.1.1.1"  #  replace with IP address of one of your router instances.
# curl -k --resolve www.example.test:443:$routerip https://www.example.test/

4.2.14. Using Wildcard Routes (for a Subdomain)

The HAProxy router has support for wildcard routes, which are enabled by setting the ROUTER_ALLOW_WILDCARD_ROUTES environment variable to true. Any routes with a wildcard policy of Subdomain that pass the router admission checks will be serviced by the HAProxy router. Then, the HAProxy router exposes the associated service (for the route) per the route’s wildcard policy.

$ oadm router --replicas=0 ...
$ oc set env dc/router ROUTER_ALLOW_WILDCARD_ROUTES=true
$ oc scale dc/router --replicas=1

Using a Secure Wildcard Edge Terminated Route

This example reflects TLS termination occurring on the router before traffic is proxied to the destination. Traffic sent to any hosts in the subdomain example.org (*.example.org) is proxied to the exposed service.

The secure edge terminated route specifies the TLS certificate and key information. The TLS certificate is served by the router front end for all hosts that match the subdomain (*.example.org).

  1. Start up a router instance:

    $ oadm router --replicas=0 --service-account=router
    $ oc set env dc/router ROUTER_ALLOW_WILDCARD_ROUTES=true
    $ oc scale dc/router --replicas=1
  2. Create a private key, certificate signing request (CSR), and certificate for the edge secured route.

    The instructions on how to do this are specific to your certificate authority and provider. For a simple self-signed certificate for a domain named *.example.test, see this example:

    # sudo openssl genrsa -out example-test.key 2048
    #
    # sudo openssl req -new -key example-test.key -out example-test.csr  \
      -subj "/C=US/ST=CA/L=Mountain View/O=OS3/OU=Eng/CN=*.example.test"
    #
    # sudo openssl x509 -req -days 366 -in example-test.csr  \
          -signkey example-test.key -out example-test.crt
  3. Generate a wildcard route using the above certificate and key:

    $ cat > route.yaml  <<REOF
    apiVersion: v1
    kind: Route
    metadata:
      name:  my-service
    spec:
      host: www.example.test
      wildcardPolicy: Subdomain
      to:
        kind: Service
        name: my-service
      tls:
        termination: edge
        key: "$(perl -pe 's/\n/\\n/' example-test.key)"
        certificate: "$(perl -pe 's/\n/\\n/' example-test.cert)"
    REOF
    $ oc create -f route.yaml

    Ensure your DNS entry for *.example.test points to your router instance(s) and the route to your domain is available.

    This example uses curl with a local resolver to simulate the DNS lookup:

    # routerip="4.1.1.1"  #  replace with IP address of one of your router instances.
    # curl -k --resolve www.example.test:443:$routerip https://www.example.test/
    # curl -k --resolve abc.example.test:443:$routerip https://abc.example.test/
    # curl -k --resolve anyname.example.test:443:$routerip https://anyname.example.test/

For routers that allow wildcard routes (ROUTER_ALLOW_WILDCARD_ROUTES set to true), there are some caveats to the ownership of a subdomain associated with a wildcard route.

Prior to wildcard routes, ownership was based on the claims made for a host name with the namespace with the oldest route winning against any other claimants. For example, route r1 in namespace ns1 with a claim for one.example.test would win over another route r2 in namespace ns2 for the same host name one.example.test if route r1 was older than route r2.

In addition, routes in other namespaces were allowed to claim non-overlapping hostnames. For example, route rone in namespace ns1 could claim www.example.test and another route rtwo in namespace d2 could claim c3po.example.test.

This is still the case if there are no wildcard routes claiming that same subdomain (example.test in the above example).

However, a wildcard route needs to claim all of the host names within a subdomain (host names of the form \*.example.test). A wildcard route’s claim is allowed or denied based on whether or not the oldest route for that subdomain (example.test) is in the same namespace as the wildcard route. The oldest route can be either a regular route or a wildcard route.

For example, if there is already a route eldest that exists in the ns1 namespace that claimed a host named owner.example.test and, if at a later point in time, a new wildcard route wildthing requesting for routes in that subdomain (example.test) is added, the claim by the wildcard route will only be allowed if it is the same namespace (ns1) as the owning route.

The following examples illustrate various scenarios in which claims for wildcard routes will succeed or fail.

In the example below, a router that allows wildcard routes will allow non-overlapping claims for hosts in the subdomain example.test as long as a wildcard route has not claimed a subdomain.

$ oadm router ...
$ oc set env dc/router
$ oc project ns1 ROUTER_ALLOW_WILDCARD_ROUTES=true

$ oc project ns1
$ oc expose service myservice --hostname=owner.example.test
$ oc expose service myservice --hostname=aname.example.test
$ oc expose service myservice --hostname=bname.example.test

$ oc project ns2
$ oc expose service anotherservice --hostname=second.example.test
$ oc expose service anotherservice --hostname=cname.example.test

$ oc project otherns
$ oc expose service thirdservice --hostname=emmy.example.test
$ oc expose service thirdservice --hostname=webby.example.test

In the example below, a router that allows wildcard routes will not allow the claim for owner.example.test or aname.example.test to succeed since the owning namespace is ns1.

$ oadm router ...
$ oc set env dc/router ROUTER_ALLOW_WILDCARD_ROUTES=true

$ oc project ns1
$ oc expose service myservice --hostname=owner.example.test
$ oc expose service myservice --hostname=aname.example.test

$ oc project ns2
$ oc expose service secondservice --hostname=bname.example.test
$ oc expose service secondservice --hostname=cname.example.test

$ # Router will not allow this claim with a different path name `/p1` as
$ # namespace `ns1` has an older route claiming host `aname.example.test`.
$ oc expose service secondservice --hostname=aname.example.test --path="/p1"

$ # Router will not allow this claim as namespace `ns1` has an older route
$ # claiming host name `owner.example.test`.
$ oc expose service secondservice --hostname=owner.example.test

$ oc project otherns

$ # Router will not allow this claim as namespace `ns1` has an older route
$ # claiming host name `aname.example.test`.
$ oc expose service thirdservice --hostname=aname.example.test

In the example below, a router that allows wildcard routes will allow the claim for `\*.example.test to succeed since the owning namespace is ns1 and the wildcard route belongs to that same namespace.

$ oadm router ...
$ oc set env dc/router ROUTER_ALLOW_WILDCARD_ROUTES=true

$ oc project ns1
$ oc expose service myservice --hostname=owner.example.test

$ # Reusing the route.yaml from the previous example.
$ # spec:
$ #   host: www.example.test
$ #   wildcardPolicy: Subdomain

$ oc create -f route.yaml   #  router will allow this claim.

In the example below, a router that allows wildcard routes will not allow the claim for `\*.example.test to succeed since the owning namespace is ns1 and the wildcard route belongs to another namespace cyclone.

$ oadm router ...
$ oc set env dc/router
$ oc project ns1 ROUTER_ALLOW_WILDCARD_ROUTES=true

$ oc project ns1
$ oc expose service myservice --hostname=owner.example.test

$ # Switch to a different namespace/project.
$ oc project cyclone

$ # Reusing the route.yaml from a prior example.
$ # spec:
$ #   host: www.example.test
$ #   wildcardPolicy: Subdomain

$ oc create -f route.yaml   #  router will deny (_NOT_ allow) this claim.

Similarly, once a namespace with a wildcard route claims a subdomain, only routes within that namespace can claim any hosts in that same subdomain.

In the example below, once a route in namespace ns1 with a wildcard route claims subdomain example.test, only routes in the namespace ns1 are allowed to claim any hosts in that same subdomain.

$ oadm router ...
$ oc set env dc/router
$ oc project ns1 ROUTER_ALLOW_WILDCARD_ROUTES=true

$ oc project ns1
$ oc expose service myservice --hostname=owner.example.test

$ oc project otherns

$ # namespace `otherns` is allowed to claim for other.example.test
$ oc expose service otherservice --hostname=other.example.test

$ oc project ns1

$ # Reusing the route.yaml from the previous example.
$ # spec:
$ #   host: www.example.test
$ #   wildcardPolicy: Subdomain

$ oc create -f route.yaml   #  Router will allow this claim.

$ #  In addition, route in namespace otherns will lose its claim to host
$ #  `other.example.test` due to the wildcard route claiming the subdomain.

$ # namespace `ns1` is allowed to claim for deux.example.test
$ oc expose service mysecondservice --hostname=deux.example.test

$ # namespace `ns1` is allowed to claim for deux.example.test with path /p1
$ oc expose service mythirdservice --hostname=deux.example.test --path="/p1"

$ oc project otherns

$ # namespace `otherns` is not allowed to claim for deux.example.test
$ # with a different path '/otherpath'
$ oc expose service otherservice --hostname=deux.example.test --path="/otherpath"

$ # namespace `otherns` is not allowed to claim for owner.example.test
$ oc expose service yetanotherservice --hostname=owner.example.test

$ # namespace `otherns` is not allowed to claim for unclaimed.example.test
$ oc expose service yetanotherservice --hostname=unclaimed.example.test

In the example below, different scenarios are shown, in which the owner routes are deleted and ownership is passed within and across namespaces. While a route claiming host eldest.example.test in the namespace ns1 exists, wildcard routes in that namespace can claim subdomain example.test. When the route for host eldest.example.test is deleted, the next oldest route senior.example.test would become the oldest route and would not affect any other routes. Once the route for host senior.example.test is deleted, the next oldest route junior.example.test becomes the oldest route and block the wildcard route claimant.

$ oadm router ...
$ oc set env dc/router
$ oc project ns1 ROUTER_ALLOW_WILDCARD_ROUTES=true

$ oc project ns1
$ oc expose service myservice --hostname=eldest.example.test
$ oc expose service seniorservice --hostname=senior.example.test

$ oc project otherns

$ # namespace `otherns` is allowed to claim for other.example.test
$ oc expose service juniorservice --hostname=junior.example.test

$ oc project ns1

$ # Reusing the route.yaml from the previous example.
$ # spec:
$ #   host: www.example.test
$ #   wildcardPolicy: Subdomain

$ oc create -f route.yaml   #  Router will allow this claim.

$ #  In addition, route in namespace otherns will lose its claim to host
$ #  `junior.example.test` due to the wildcard route claiming the subdomain.

$ # namespace `ns1` is allowed to claim for dos.example.test
$ oc expose service mysecondservice --hostname=dos.example.test

$ # Delete route for host `eldest.example.test`, the next oldest route is
$ # the one claiming `senior.example.test`, so route claims are unaffacted.
$ oc delete route myservice

$ # Delete route for host `senior.example.test`, the next oldest route is
$ # the one claiming `junior.example.test` in another namespace, so claims
$ # for a wildcard route would be affected. The route for the host
$ # `dos.example.test` would be unaffected as there are no other wildcard
$ # claimants blocking it.
$ oc delete route seniorservice

4.2.15. Using the Container Network Stack

The OpenShift Container Platform router runs inside a container and the default behavior is to use the network stack of the host (i.e., the node where the router container runs). This default behavior benefits performance because network traffic from remote clients does not need to take multiple hops through user space to reach the target service and container.

Additionally, this default behavior enables the router to get the actual source IP address of the remote connection rather than getting the node’s IP address. This is useful for defining ingress rules based on the originating IP, supporting sticky sessions, and monitoring traffic, among other uses.

This host network behavior is controlled by the --host-network router command line option, and the default behaviour is the equivalent of using --host-network=true. If you wish to run the router with the container network stack, use the --host-network=false option when creating the router. For example:

$ oadm router --service-account=router --host-network=false

Internally, this means the router container must publish the 80 and 443 ports in order for the external network to communicate with the router.

Note

Running with the container network stack means that the router sees the source IP address of a connection to be the NATed IP address of the node, rather than the actual remote IP address.

Note

On OpenShift Container Platform clusters using multi-tenant network isolation, routers on a non-default namespace with the --host-network=false option will load all routes in the cluster, but routes across the namespaces will not be reachable due to network isolation. With the --host-network=true option, routes bypass the container network and it can access any pod in the cluster. If isolation is needed in this case, then do not add routes across the namespaces.

4.2.16. Exposing Router Metrics

Using the --metrics-image and --expose-metrics options, you can configure the OpenShift Container Platform router to run a sidecar container that exposes or publishes router metrics for consumption by external metrics collection and aggregation systems (e.g. Prometheus, statsd).

Depending on your router implementation, the image is appropriately set up and the metrics sidecar container is started when the router is deployed. For example, the HAProxy-based router implementation defaults to using the prom/haproxy-exporter image to run as a sidecar container, which can then be used as a metrics datasource by the Prometheus server.

Note

The --metrics-image option overrides the defaults for HAProxy-based router implementations and, in the case of custom implementations, enables the image to use for a custom metrics exporter or publisher.

  1. Grab the HAProxy Prometheus exporter image from the Docker registry:

    $ sudo docker pull prom/haproxy-exporter
  2. Create the OpenShift Container Platform router:

    $ oadm router --service-account=router --expose-metrics

    Or, optionally, use the --metrics-image option to override the HAProxy defaults:

    $ oadm router --service-account=router --expose-metrics \
        --metrics-image=prom/haproxy-exporter
  3. Once the haproxy-exporter containers (and your HAProxy router) have started, point Prometheus to the sidecar container on port 9101 on the node where the haproxy-exporter container is running:

    $ haproxy_exporter_ip="<enter-ip-address-or-hostname>"
    $ cat > haproxy-scraper.yml  <<CFGEOF
    ---
    global:
      scrape_interval: "60s"
      scrape_timeout:  "10s"
      # external_labels:
        # source: openshift-router
    
    scrape_configs:
      - job_name:  "haproxy"
        target_groups:
          - targets:
            - "${haproxy_exporter_ip}:9101"
    CFGEOF
    
    $ #  And start prometheus as you would normally using the above config file.
    $ echo "  - Example:  prometheus -config.file=haproxy-scraper.yml "
    $ echo "              or you can start it as a container on {product-title}!!
    
    $ echo "  - Once the prometheus server is up, view the {product-title} HAProxy "
    $ echo "    router metrics at: http://<ip>:9090/consoles/haproxy.html "

4.2.17. Preventing Connection Failures During Restarts

If you connect to the router while the proxy is reloading, there is a small chance that your connection will end up in the wrong network queue and be dropped. The issue is being addressed. In the meantime, it is possible to work around the problem by installing iptables rules to prevent connections during the reload window. However, doing so means that the router needs to run with elevated privilege so that it can manipulate iptables on the host. It also means that connections that happen during the reload are temporarily ignored and must retransmit their connection start, lengthening the time it takes to connect, but preventing connection failure.

To prevent this, configure the router to use iptables by changing the service account, and setting an environment variable on the router.

Use a Privileged SCC

When creating the router, allow it to use the privileged SCC. This gives the router user the ability to create containers with root privileges on the nodes:

$ oadm policy add-scc-to-user privileged -z router

Patch the Router Deployment Configuration to Create a Privileged Container

You can now create privileged containers. Next, configure the router deployment configuration to use the privilege so that the router can set the iptables rules it needs. This patch changes the router deployment configuration so that the container that is created runs as privileged (and therefore gets correct capabilities) and run as root:

$ oc patch dc router -p '{"spec":{"template":{"spec":{"containers":[{"name":"router","securityContext":{"privileged":true}}],"securityContext":{"runAsUser": 0}}}}}'

Configure the Router to Use iptables

Set the option on the router deployment configuration:

$ oc set env dc/router -c router DROP_SYN_DURING_RESTART=true

If you used a non-default name for the router, you must change dc/router accordingly.

4.2.18. ARP Cache Tuning for Large-scale Clusters

In OpenShift Container Platform clusters with large numbers of routes (greater than the value of net.ipv4.neigh.default.gc_thresh3, which is 1024 by default), you must increase the default values of sysctl variables to allow more entries in the ARP cache.

The kernel messages would be similar to the following:

[ 1738.811139] net_ratelimit: 1045 callbacks suppressed
[ 1743.823136] net_ratelimit: 293 callbacks suppressed

When this issue occurs, the oc commands might start to fail with the following error:

Unable to connect to the server: dial tcp: lookup <hostname> on <ip>:<port>: write udp <ip>:<port>-><ip>:<port>: write: invalid argument

To verify the actual amount of ARP entries for IPv4, run the following:

# ip -4 neigh show nud all | wc -l

If the number begins to approach the net.ipv4.neigh.default.gc_thresh3 threshold, increase the values. Run the following on the nodes running the router pods. The following sysctl values are suggested for clusters with large numbers of routes:

net.ipv4.neigh.default.gc_thresh1 = 8192
net.ipv4.neigh.default.gc_thresh2 = 32768
net.ipv4.neigh.default.gc_thresh3 = 65536

To make these settings permanent across reboots, create a custom tuned profile.

4.2.19. Protecting Against DDoS Attacks

Add timeout http-request to the default HAProxy router image to protect the deployment against distributed denial-of-service (DDoS) attacks (for example, slowloris):

# and the haproxy stats socket is available at /var/run/haproxy.stats
global
  stats socket ./haproxy.stats level admin

defaults
  option http-server-close
  mode http
  timeout http-request 5s
  timeout connect 5s 1
  timeout server 10s
  timeout client 30s
1
timeout http-request is set up to 5 seconds. HAProxy gives a client 5 seconds *to send its whole HTTP request. Otherwise, HAProxy shuts the connection with *an error.

Also, when the environment variable ROUTER_SLOWLORIS_TIMEOUT is set, it limits the amount of time a client has to send the whole HTTP request. Otherwise, HAProxy will shut down the connection.

Setting the environment variable allows information to be captured as part of the router’s deployment configuration and does not require manual modification of the template, whereas manually adding the HAProxy setting requires you to rebuild the router pod and maintain your router template file.

Using annotations implements basic DDoS protections in the HAProxy template router, including the ability to limit the:

  • number of concurrent TCP connections
  • rate at which a client can request TCP connections
  • rate at which HTTP requests can be made

These are enabled on a per route basis because applications can have extremely different traffic patterns.

Table 4.1. HAProxy Template Router Settings

SettingDescription

haproxy.router.openshift.io/rate-limit-connections

Enables the settings be configured (when set to true, for example).

haproxy.router.openshift.io/rate-limit-connections.concurrent-tcp

The number of concurrent TCP connections that can be made by the same IP address on this route.

haproxy.router.openshift.io/rate-limit-connections.rate-tcp

The number of TCP connections that can be opened by a client IP.

haproxy.router.openshift.io/rate-limit-connections.rate-http

The number of HTTP requests that a client IP can make in a 3-second period.

4.3. Deploying a Customized HAProxy Router

4.3.1. Overview

The HAProxy router is based on a golang template that generates the HAProxy configuration file from a list of routes. If you want a customized template router to meet your needs, you can customize the template file, build a new container image, and run a customized router. Alternatively you can use a ConfigMap.

One common case for this might be implementing new features within the application back ends. For example, it might be desirable in a highly-available setup to use stick-tables that synchronizes between peers. The router plug-in provides all the facilities necessary to make this customization.

4.3.2. Obtaining the Router Configuration Template

You can obtain a new haproxy-config.template file from the latest router image by running:

# docker run --rm --interactive=true --tty --entrypoint=cat \
    registry.access.redhat.com/openshift3/ose-haproxy-router:v3.0.2.0 haproxy-config.template

Save this content to a file for use as the basis of your customized template.

4.3.3. Using a ConfigMap to Replace the Router Configuration Template

You can use ConfigMap to customize the router instance without rebuilding the router image. The haproxy-config.template, reload-haproxy, and other scripts can be modified as well as creating and modifying router environment variables.

  1. Copy the haproxy-config.template that you want to modify as described above. Modify it as desired.
  2. Create a ConfigMap:

    $ oc create configmap customrouter --from-file=haproxy-config.template

    The customrouter ConfigMap now contains a copy of the modified haproxy-config.template file.

  3. Modify the router deployment configuration to mount the ConfigMap as a file and point the TEMPLATE_FILE environment variable to it. This can be done via oc set env and oc volume commands, or alternatively by editing the router deployment configuration.

    Using oc commands
    $ oc set env dc/router \
        TEMPLATE_FILE=/var/lib/haproxy/conf/custom/haproxy-config.template
    $ oc volume dc/router --add --overwrite \
        --name=config-volume \
        --mount-path=/var/lib/haproxy/conf/custom \
        --source='{"configMap": { "name": "customrouter"}}'
    Editing the Router Deployment Configuration

    Use oc edit dc router to edit the router deployment configuration with a text editor.

    ...
            - name: STATS_USERNAME
              value: admin
            - name: TEMPLATE_FILE  1
              value: /var/lib/haproxy/conf/custom/haproxy-config.template
            image: openshift/origin-haproxy-routerp
    ...
            terminationMessagePath: /dev/termination-log
            volumeMounts: 2
            - mountPath: /var/lib/haproxy/conf/custom
              name: config-volume
          dnsPolicy: ClusterFirst
    ...
          terminationGracePeriodSeconds: 30
          volumes: 3
          - configMap:
              name: customrouter
            name: config-volume
      test: false
    ...
    1
    In the spec.container.env field, add the TEMPLATE_FILE environment variable to point to the mounted haproxy-config.template file.
    2
    Add the spec.container.volumeMounts field to create the mount point.
    3
    Add a new spec.volumes field to mention the ConfigMap.

    Save the changes and exit the editor. This restarts the router.

4.3.4. Using Stick Tables

The following example customization can be used in a highly-available routing setup to use stick-tables that synchronize between peers.

Adding a Peer Section

In order to synchronize stick-tables amongst peers you must a define a peers section in your HAProxy configuration. This section determines how HAProxy will identify and connect to peers. The plug-in provides data to the template under the .PeerEndpoints variable to allow you to easily identify members of the router service. You may add a peer section to the haproxy-config.template file inside the router image by adding:

{{ if (len .PeerEndpoints) gt 0 }}
peers openshift_peers
  {{ range $endpointID, $endpoint := .PeerEndpoints }}
    peer {{$endpoint.TargetName}} {{$endpoint.IP}}:1937
  {{ end }}
{{ end }}

Changing the Reload Script

When using stick-tables, you have the option of telling HAProxy what it should consider the name of the local host in the peer section. When creating endpoints, the plug-in attempts to set the TargetName to the value of the endpoint’s TargetRef.Name. If TargetRef is not set, it will set the TargetName to the IP address. The TargetRef.Name corresponds with the Kubernetes host name, therefore you can add the -L option to the reload-haproxy script to identify the local host in the peer section.

peer_name=$HOSTNAME 1

if [ -n "$old_pid" ]; then
  /usr/sbin/haproxy -f $config_file -p $pid_file -L $peer_name -sf $old_pid
else
  /usr/sbin/haproxy -f $config_file -p $pid_file -L $peer_name
fi
1
Must match an endpoint target name that is used in the peer section.

Modifying Back Ends

Finally, to use the stick-tables within back ends, you can modify the HAProxy configuration to use the stick-tables and peer set. The following is an example of changing the existing back end for TCP connections to use stick-tables:

            {{ if eq $cfg.TLSTermination "passthrough" }}
backend be_tcp_{{$cfgIdx}}
  balance leastconn
  timeout check 5000ms
  stick-table type ip size 1m expire 5m{{ if (len $.PeerEndpoints) gt 0 }} peers openshift_peers {{ end }}
  stick on src
                {{ range $endpointID, $endpoint := $serviceUnit.EndpointTable }}
  server {{$endpointID}} {{$endpoint.IP}}:{{$endpoint.Port}} check inter 5000ms
                {{ end }}
            {{ end }}

After this modification, you can rebuild your router.

4.3.5. Rebuilding Your Router

After you have made any desired modifications to the template, such as the example stick tables customization, you must rebuild your router for your changes to go in effect:

  1. Rebuild the container image to include your customized template.
  2. Push the resulting image to your repository.
  3. Create the router specifying your new image, either:

    1. in the pod’s object definition directly, or
    2. by adding the --images=<repo>/<image>:<tag> flag to the oadm router command when creating a highly-available routing service.

4.4. Using the F5 Router Plug-in

4.4.1. Overview

Note

The F5 router plug-in is available starting in OpenShift Container Platform 3.0.2.

The F5 router plug-in is provided as a container image and run as a pod, just like the default HAProxy router.

4.4.2. Prerequisites

When deploying the F5 router plug-in, ensure you meet the following requirements:

  1. An F5 host IP and credentials
  2. The name of the virtual servers (for both HTTP and HTTPS)
  3. The private key to the F5 instance
  4. The host-internal IP and VXLAN gateway

4.4.3. Deploying the F5 Router

Important

The F5 router must be run in privileged mode, because route certificates are copied using the scp command:

$ oadm policy remove-scc-from-user hostnetwork -z router
$ oadm policy add-scc-to-user privileged -z router

Deploy the F5 router with the oadm router command, but provide additional flags (or environment variables) specifying the following parameters for the F5 BIG-IP® host:

FlagDescription

--type=f5-router

Specifies that an F5 router should be launched (the default --type is haproxy-router).

--external-host

Specifies the F5 BIG-IP® host’s management interface’s host name or IP address.

--external-host-username

Specifies the F5 BIG-IP® user name (typically admin).

--external-host-password

Specifies the F5 BIG-IP® password.

--external-host-http-vserver

Specifies the name of the F5 virtual server for HTTP connections.

--external-host-https-vserver

Specifies the name of the F5 virtual server for HTTPS connections.

--external-host-private-key

Specifies the path to the SSH private key file for the F5 BIG-IP® host. Required to upload and delete key and certificate files for routes.

--external-host-insecure

A Boolean flag that indicates that the F5 router should skip strict certificate verification with the F5 BIG-IP® host.

--external-host-partition-path

Specifies the F5 BIG-IP® partition path (the default is /Common).

For example:

$ oadm router \
    --type=f5-router \
    --external-host=10.0.0.2 \
    --external-host-username=admin \
    --external-host-password=mypassword \
    --external-host-http-vserver=ose-vserver \
    --external-host-https-vserver=https-ose-vserver \
    --external-host-private-key=/path/to/key \
    --host-network=false \
    --service-account=router

As with the HAProxy router, the oadm router command creates the service and deployment configuration objects, and thus the replication controllers and pod(s) in which the F5 router itself runs. The replication controller restarts the F5 router in case of crashes. Because the F5 router is watching routes, endpoints, and nodes and configuring F5 BIG-IP® accordingly, running the F5 router in this way, along with an appropriately configured F5 BIG-IP® deployment, should satisfy high-availability requirements.

4.4.4. F5 Router Partition Paths

Partition paths allow you to store your OpenShift Container Platform routing configuration in a custom F5 BIG-IP® administrative partition, instead of the default /Common partition. You can use custom administrative partitions to secure F5 BIG-IP® environments. This means that an OpenShift Container Platform-specific configuration stored in F5 BIG-IP® system objects reside within a logical container, allowing administrators to define access control policies on that specific administrative partition.

See the F5 BIG-IP® documentation for more information about administrative partitions.

Use the --external-host-partition-path flag when deploying the F5 router to specify a partition path:

$ oadm router --external-host-partition-path=/OpenShift/zone1 ...

4.4.5. Setting Up F5 Native Integration

Note

This section reviews how to set up F5 native integration with OpenShift Container Platform. The concepts of F5 appliance and OpenShift Container Platform connection and data flow of F5 native integration are discussed in the F5 Native Integration section of the Routes topic.

As of OpenShift Container Platform version 3.4, using native integration of F5 with OpenShift Container Platform does not require configuring a ramp node for F5 to be able to reach the pods on the overlay network as created by OpenShift SDN.

The F5 controller pod needs to be launched with enough information so that it can successfully directly connect to pods.

  1. Create a ghost hostsubnet on the OpenShift Container Platform cluster:

    $ cat > f5-hostsubnet.yaml << EOF
    {
        "kind": "HostSubnet",
        "apiVersion": "v1",
        "metadata": {
            "name": "openshift-f5-node",
            "annotations": {
            "pod.network.openshift.io/assign-subnet": "true",
    	"pod.network.openshift.io/fixed-vnid-host": "0"  1
            }
        },
        "host": "openshift-f5-node",
        "hostIP": "10.3.89.213"  2
    } EOF
    $ oc create -f f5-hostsubnet.yaml
    1
    Make F5 global.
    2
    The internal IP of the F5 appliance.
  2. Determine the subnet allocated for the ghost hostsubnet just created:

    $ oc get hostsubnets
    NAME                    HOST                    HOST IP       SUBNET
    openshift-f5-node       openshift-f5-node       10.3.89.213   10.131.0.0/23
    openshift-master-node   openshift-master-node   172.17.0.2    10.129.0.0/23
    openshift-node-1        openshift-node-1        172.17.0.3    10.128.0.0/23
    openshift-node-2        openshift-node-2        172.17.0.4    10.130.0.0/23
  3. Check the SUBNET for the newly created hostsubnet. In this example, 10.131.0.0/23.
  4. Get the entire pod network’s CIDR:

    $ oc get clusternetwork

    This value will be something like 10.128.0.0/14, noting the mask (14 in this example).

  5. To construct the gateway address, pick any IP address from the hostsubnet (for example, 10.131.0.5). Use the mask of the pod network (14). The gateway address becomes: 10.131.0.5/14.
  6. Launch the F5 controller pod, following these instructions. Additionally, allow the access to 'node' cluster resource for the service account and use the two new additional options for VXLAN native integration.

    $ # Add policy to allow router to access nodes using the sdn-reader role
    $ oadm policy add-cluster-role-to-user system:sdn-reader system:serviceaccount:default:router
    $ # Launch the router pod with vxlan-gw and F5's internal IP as extra arguments
    $ #--external-host-internal-ip=10.3.89.213
    $ #--external-host-vxlan-gw=10.131.0.5/14
    $ oadm router \
        --type=f5-router \
        --external-host=10.3.89.90 \
        --external-host-username=admin \
        --external-host-password=mypassword \
        --external-host-http-vserver=ose-vserver \
        --external-host-https-vserver=https-ose-vserver \
        --external-host-private-key=/path/to/key \
        --credentials='/etc/openshift/master/openshift-router.kubeconfig' \
        --service-account=router \
        --host-network=false \
        --external-host-internal-ip=10.3.89.213 \
        --external-host-vxlan-gw=10.131.0.5/14

    The F5 setup is now ready, without the need to set up the ramp node.

Chapter 5. Upgrading a Cluster

5.1. Overview

When new versions of OpenShift Container Platform are released, you can upgrade your existing cluster to apply the latest enhancements and bug fixes. This includes upgrading from previous minor versions, such as release 3.2 to 3.3, and applying asynchronous errata updates within a minor version (3.3.z releases). See the OpenShift Container Platform 3.4 Release Notes to review the latest changes.

Note

Due to the core architectural changes between the major versions, OpenShift Enterprise 2 environments cannot be upgraded to OpenShift Container Platform 3 and require a fresh installation.

Unless noted otherwise, node and masters within a major version are forward and backward compatible, so upgrading your cluster should go smoothly. However, you should not run mismatched versions longer than necessary to upgrade the entire cluster.

5.1.1. In-place or Blue-Green Upgrades

There are two methods for performing OpenShift Container Platform cluster upgrades. You can either do in-place upgrades (automated or manual), or upgrade using a blue-green deployment method.

In-place Upgrades

With in-place upgrades, the cluster upgrade is performed on all hosts in a single, running cluster: first masters and then nodes. Pods are evacuated off of nodes and recreated on other running nodes before a node upgrade begins; this helps reduce downtime of user applications.

If you installed using the quick or advanced installation and the ~/.config/openshift/installer.cfg.yml or inventory file that was used is available, you can perform an automated in-place upgrade. Alternatively, you can upgrade in-place manually.

Blue-green Deployments

The blue-green deployment upgrade method follows a similar flow to the in-place method: masters and etcd servers are still upgraded first, however a parallel environment is created for new nodes instead of upgrading them in-place.

This method allows administrators to switch traffic from the old set of nodes (e.g., the "blue" deployment) to the new set (e.g., the "green" deployment) after the new deployment has been verified. If a problem is detected, it is also then easy to rollback to the old deployment quickly.

5.2. Performing Automated In-place Cluster Upgrades

5.2.1. Overview

Important

An etcd performance issue has been discovered on new and upgraded OpenShift Container Platform 3.4 clusters. See the following Knowledgebase Solution for further details:

https://access.redhat.com/solutions/2916381
(BZ#1415839)

If you installed using the advanced installation and the inventory file that was used is available, you can use the upgrade playbook to automate the OpenShift cluster upgrade process. If you installed using the quick installation method and a ~/.config/openshift/installer.cfg.yml file is available, you can use the installer to perform the automated upgrade.

The automated upgrade performs the following steps for you:

  • Applies the latest configuration.
  • Upgrades master and etcd components and restarts services.
  • Upgrades node components and restarts services.
  • Applies the latest cluster policies.
  • Updates the default router if one exists.
  • Updates the default registry if one exists.
  • Updates default image streams and InstantApp templates.
Important

Ensure that you have met all prerequisites before proceeding with an upgrade. Failure to do so can result in a failed upgrade.

5.2.2. Preparing for an Automated Upgrade

Important

Before upgrading your cluster to OpenShift Container Platform 3.4, the cluster must be already upgraded to the latest asynchronous release of version 3.3. Cluster upgrades cannot span more than one minor version at a time, so if your cluster is at a version earlier than 3.3, you must first upgrade incrementally (e.g., 3.1 to 3.2, then 3.2 to 3.3).

To prepare for an automated upgrade:

  1. If you are upgrading from OpenShift Container Platform 3.3 to 3.4, manually disable the 3.3 channel and enable the 3.4 channel on each master and node host:

    # subscription-manager repos --disable="rhel-7-server-ose-3.3-rpms" \
        --enable="rhel-7-server-ose-3.4-rpms"\
        --enable="rhel-7-server-extras-rpms"
    # yum clean all
  2. For any upgrade path, always ensure that you have the latest version of the atomic-openshift-utils package on each RHEL 7 system, which also updates the openshift-ansible-* packages:

    # yum update atomic-openshift-utils
  3. Install or update to the following latest available atomic-openshift-excluder package on each RHEL 7 system, which helps ensure your systems stay on the correct versions of atomic-openshift packages when you are not trying to upgrade, according to the OpenShift Container Platform version:

    # yum install atomic-openshift-excluder

    These packages add entries to the exclude directive in the host’s /etc/yum.conf file.

  4. Due to a bug (BZ#1430929), if you previously installed the atomic-openshift-docker-excluder package as instructed in OpenShift Container Platform 3.3, upgrading this package to the 3.4 version does not remove docker*1.12* from the exclude directive as intended.

    As a workaround until the bug is fixed, remove the installed version of atomic-openshift-docker-excluder package, then install the 3.4 version:

    # yum remove atomic-openshift-docker-excluder
    # yum install atomic-openshift-docker-excluder
  5. You must be logged in as a cluster administrative user on the master host for the upgrade to succeed:

    $ oc login

After satisfying these steps, there are two methods for running the automated upgrade:

Choose and follow one of these methods.

5.2.3. Using the Installer to Upgrade

If you installed OpenShift Container Platform using the quick installation method, you should have an installation configuration file located at ~/.config/openshift/installer.cfg.yml. The installer requires this file to start an upgrade.

The installer supports upgrading between minor versions of OpenShift Container Platform (one minor version at a time, e.g., 3.3 to 3.4) as well as between asynchronous errata updates within a minor version (e.g., 3.4.z).

If you have an older format installation configuration file in ~/.config/openshift/installer.cfg.yml from an installation of a previous cluster version, the installer will attempt to upgrade the file to the new supported format. If you do not have an installation configuration file of any format, you can create one manually.

To start an upgrade with the quick installer:

  1. Satisfy the steps in Preparing for an Automated Upgrade to ensure you are using the latest upgrade playbooks.
  2. Run the following command on each host to remove the atomic-openshift packages from the list of yum excludes on the host:

    # atomic-openshift-excluder unexclude
  3. Run the installer with the upgrade subcommand:

    # atomic-openshift-installer upgrade
  4. Then, follow the on-screen instructions to upgrade to the latest release.
  5. Run the following command on each host to add the atomic-openshift packages back to the list of yum excludes on the host:

    # atomic-openshift-excluder exclude
  6. After all master and node upgrades have completed, a recommendation will be printed to reboot all hosts. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features have you previously enabled.

    FeatureNext Step

    Aggregated Logging

    Upgrade the EFK logging stack.

    Cluster Metrics

    Upgrade cluster metrics.

5.2.4. Running Upgrade Playbooks Directly

You can run automated upgrade playbooks using Ansible directly, similar to the advanced installation method, if you have an inventory file. Playbooks can be run using the ansible-playbook command.

The same v3_4 upgrade playbooks can be used for either of the following scenarios:

  • Upgrading existing OpenShift Container Platform 3.3 clusters to 3.4
  • Upgrading existing OpenShift Container Platform 3.4 clusters to the latest asynchronous errata updates

5.2.4.1. Upgrading the Control Plane and Nodes in Separate Phases

An OpenShift Container Platform cluster can be upgraded in one or more phases. You can choose whether to upgrade all hosts in one phase by running a single Ansible playbook, or upgrade the control plane (master components) and nodes in multiple phases using separate playbooks.

Note

Instructions on the full upgrade process and when to call these playbooks are described in Upgrading to the Latest OpenShift Container Platform 3.4 Release.

When upgrading in separate phases, the control plane phase includes upgrading:

  • etcd
  • master components
  • Docker on any stand-alone etcd hosts

It does does not include:

  • node services running on masters
  • Docker running on masters
  • node services running on stand-alone nodes

When upgrading only the nodes, the control plane must already be upgraded. The node phase includes upgrading:

  • node services running on masters and stand-alone nodes
  • Docker running on masters and nodes

5.2.4.2. Customizing Node Upgrades

Whether upgrading in a single or multiple phases, you can customize how the node portion of the upgrade progresses by passing certain Ansible variables to an upgrade playbook using the -e option.

Note

Instructions on the full upgrade process and when to call these playbooks are described in Upgrading to the Latest OpenShift Container Platform 3.4 Release.

The openshift_upgrade_nodes_serial variable can be set to an integer or percentage to control how many node hosts are upgraded at the same time. The default is 1, upgrading nodes one at a time.

For example, to upgrade 20 percent of the total number of detected nodes at a time:

$ ansible-playbook -i <path/to/inventory/file> \
    </path/to/upgrade/playbook> \
    -e openshift_upgrade_nodes_serial="20%"

The openshift_upgrade_nodes_label variable allows you to specify that only nodes with a certain label are upgraded. This can also be combined with the openshift_upgrade_nodes_serial variable.

For example, to only upgrade nodes in the group1 region, two at a time:

$ ansible-playbook -i <path/to/inventory/file> \
    </path/to/upgrade/playbook> \
    -e openshift_upgrade_nodes_serial="2" \
    -e openshift_upgrade_nodes_label="region=group1"

See Manging Nodes for more on node labels.

5.2.4.3. Upgrading to the Latest OpenShift Container Platform 3.4 Release

To upgrade an existing OpenShift Container Platform 3.3 or 3.4 cluster to the latest 3.4 release:

  1. Satisfy the steps in Preparing for an Automated Upgrade to ensure you are using the latest upgrade playbooks.
  2. Run the following command on each host to remove the atomic-openshift packages from the list of yum excludes on the host:

    # atomic-openshift-excluder unexclude
  3. Ensure the deployment_type parameter in your inventory file is set to openshift-enterprise.
  4. If you have multiple masters configured and want to enable rolling, full system restarts of the hosts, you can set the openshift_rolling_restart_mode parameter in your inventory file to system. Otherwise, the default value services performs rolling service restarts on HA masters, but does not reboot the systems. See Configuring Cluster Variables for details.
  5. At this point, you can choose to run the upgrade in a single or multiple phases. See Upgrading the Control Plane and Nodes in Separate Phases for more details which components are upgraded in each phase.

    If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify its location. If you previously used the atomic-openshift-installer command to run your installation, you can check ~/.config/openshift/hosts for the last inventory file that was used, if needed.

    Note

    You can add --tags pre_upgrade to the following ansible-playbook commands to run the pre-upgrade checks for the playbook. This is a dry-run option that preforms all pre-upgrade checks without actually upgrading any hosts, and reports any problems found.

    Option A) Upgrade control plane and nodes in a single phase.

    Run the upgrade.yml playbook to upgrade the cluster in a single phase using one playbook; the control plane is still upgraded first, then nodes in-place:

    # ansible-playbook -i </path/to/inventory/file> \
        /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_4/upgrade.yml \
        [-e <customized_node_upgrade_variables>] 1
    1
    See Customizing Node Upgrades for any desired <customized_node_upgrade_variables>.

    Option B) Upgrade the control plane and nodes in separate phases.

    1. To upgrade only the control plane, run the upgrade_control_plane.yaml playbook:

      # ansible-playbook -i </path/to/inventory/file> \
          /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_4/upgrade_control_plane.yml
    2. To upgrade only the nodes, run the upgrade_nodes.yaml playbook:

      # ansible-playbook -i </path/to/inventory/file> \
          /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_4/upgrade_nodes.yml \
          [-e <customized_node_upgrade_variables>] 1
      1
      See Customizing Node Upgrades for any desired <customized_node_upgrade_variables>.

      If you are upgrading the nodes in groups as described in Customizing Node Upgrades, continue invoking the upgrade_nodes.yml playbook until all nodes have been successfully upgraded.

  6. Run the following command on each host to add the atomic-openshift packages back to the list of yum excludes on the host:

    # atomic-openshift-excluder exclude
  7. After all master and node upgrades have completed, a recommendation will be printed to reboot all hosts. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features have you previously enabled.

    FeatureNext Step

    Aggregated Logging

    Upgrade the EFK logging stack.

    Cluster Metrics

    Upgrade cluster metrics.

5.2.5. Upgrading the EFK Logging Stack

If you have previously deployed the EFK logging stack and want to upgrade to the latest logging component images, the steps must be performed manually as shown in Manual Upgrades.

5.2.6. Upgrading Cluster Metrics

If you have previously deployed cluster metrics, you must manually update to the latest metric components.

5.2.7. Verifying the Upgrade

To verify the upgrade:

  1. First check that all nodes are marked as Ready:

    # oc get nodes
    NAME                        STATUS                     AGE
    master.example.com          Ready,SchedulingDisabled   165d
    node1.example.com           Ready                      165d
    node2.example.com           Ready                      165d
  2. Then, verify that you are running the expected versions of the docker-registry and router images, if deployed. Replace <tag> with v3.4.1.10 for the latest version.

    # oc get -n default dc/docker-registry -o json | grep \"image\"
        "image": "openshift3/ose-docker-registry:<tag>",
    # oc get -n default dc/router -o json | grep \"image\"
        "image": "openshift3/ose-haproxy-router:<tag>",
  3. After upgrading, you can use the diagnostics tool on the master to look for common issues:

    # oc adm diagnostics
    ...
    [Note] Summary of diagnostics execution:
    [Note] Completed with no errors or warnings seen.

5.3. Performing Manual In-place Cluster Upgrades

5.3.1. Overview

Important

An etcd performance issue has been discovered on new and upgraded OpenShift Container Platform 3.4 clusters. See the following Knowledgebase Solution for further details:

https://access.redhat.com/solutions/2916381
(BZ#1415839)

As an alternative to performing an automated upgrade, you can manually upgrade your OpenShift cluster. To manually upgrade without disruption, it is important to upgrade each component as documented in this topic.

Before you begin your upgrade, familiarize yourself now with the entire procedure. Specific releases may require additional steps to be performed at key points before or during the standard upgrade process.

Important

Ensure that you have met all prerequisites before proceeding with an upgrade. Failure to do so can result in a failed upgrade.

5.3.2. Preparing for a Manual Upgrade

Note

Before upgrading your cluster to OpenShift Container Platform 3.4, the cluster must be already upgraded to the latest asynchronous release of version 3.3. Cluster upgrades cannot span more than one minor version at a time, so if your cluster is at a version earlier than 3.3, you must first upgrade incrementally (e.g., 3.1 to 3.2, then 3.2 to 3.3).

To prepare for a manual upgrade, follow these steps:

  1. If you are upgrading from OpenShift Container Platform 3.3 to 3.4, manually disable the 3.3 channel and enable the 3.4 channel on each host:

    # subscription-manager repos --disable="rhel-7-server-ose-3.3-rpms" \
        --enable="rhel-7-server-ose-3.4-rpms" \
        --enable="rhel-7-server-extras-rpms"

    On RHEL 7 systems, also clear the yum cache:

    # yum clean all
  2. Install or update to the latest available version of the atomic-openshift-utils package on each RHEL 7 system, which provides files that will be used in later sections:

    # yum install atomic-openshift-utils
  3. Install or update to the following latest available atomic-openshift-excluder package on each RHEL 7 system, which helps ensure your systems stay on the correct versions of atomic-openshift packages when you are not trying to upgrade, according to the OpenShift Container Platform version:

    # yum install atomic-openshift-excluder

    These packages add entries to the exclude directive in the host’s /etc/yum.conf file.

  4. Due to a bug (BZ#1430929), if you previously installed the atomic-openshift-docker-excluder package as instructed in OpenShift Container Platform 3.3, upgrading this package to the 3.4 version does not remove docker*1.12* from the exclude directive as intended.

    As a workaround until the bug is fixed, remove the installed version of atomic-openshift-docker-excluder package, then install the 3.4 version:

    # yum remove atomic-openshift-docker-excluder
    # yum install atomic-openshift-docker-excluder
  5. Create an etcd backup on each master. The etcd package is required, even if using embedded etcd, for access to the etcdctl command to make the backup.

    Note

    The etcd package is installed by default for RHEL Atomic Host 7 systems. If the master is a RHEL 7 system and etcd is not already installed, install it now:

    # yum install etcd

    To create the backup, run:

    # ETCD_DATA_DIR=/var/lib/origin/openshift.local.etcd 1
    # etcdctl backup \
        --data-dir $ETCD_DATA_DIR \
        --backup-dir $ETCD_DATA_DIR.bak.<date> 2
    1
    This directory is for embedded etcd. For external etcd, use /var/lib/etcd instead.
    2
    Use the date of the backup, or some unique identifier, for <date>. The command will not make a backup if the --backup-dir location already exists.
  6. For any upgrade path, ensure that you are running the latest kernel on each RHEL 7 system:

    # yum update kernel

5.3.3. Upgrading Master Components

Before upgrading any stand-alone nodes, upgrade the master components (which provide the control plane for the cluster).

  1. Run the following command on each master to remove the atomic-openshift packages from the list of yum excludes on the host:

    # atomic-openshift-excluder unexclude
  2. Upgrade etcd on all master hosts and any external etcd hosts.

    1. For RHEL 7 systems using the RPM-based method:

      1. Upgrade the etcd package:

        # yum update etcd
      2. Restart the etcd service and review the logs to ensure it restarts successfully:

        # systemctl restart etcd
        # journalctl -r -u etcd
    2. For RHEL Atomic Host 7 systems and RHEL 7 systems using the containerized method:

      1. Pull the latest rhel7/etcd image:

        # docker pull registry.access.redhat.com/rhel7/etcd
      2. Restart the etcd_container service and review the logs to ensure it restarts successfully:

        # systemctl restart etcd_container
        # journalctl -r -u etcd_container
  3. On each master host, upgrade the atomic-openshift packages or related images.

    1. For masters using the RPM-based method on a RHEL 7 system, upgrade all installed atomic-openshift packages:

      # yum upgrade atomic-openshift\*
    2. For masters using the containerized method on a RHEL 7 or RHEL Atomic Host 7 system, set the IMAGE_VERSION parameter to the version you are upgrading to in the following files:

      • /etc/sysconfig/atomic-openshift-master (single master clusters only)
      • /etc/sysconfig/atomic-openshift-master-controllers (multi-master clusters only)
      • /etc/sysconfig/atomic-openshift-master-api (multi-master clusters only)
      • /etc/sysconfig/atomic-openshift-node
      • /etc/sysconfig/atomic-openshift-openvswitch

      For example:

      IMAGE_VERSION=<tag>

      Replace <tag> with v3.4.1.10 for the latest version.

  4. Restart the master service(s) on each master and review logs to ensure they restart successfully.

    For single master clusters:

    # systemctl restart atomic-openshift-master
    # journalctl -r -u atomic-openshift-master

    For multi-master clusters:

    # systemctl restart atomic-openshift-master-controllers
    # systemctl restart atomic-openshift-master-api
    # journalctl -r -u atomic-openshift-master-controllers
    # journalctl -r -u atomic-openshift-master-api
  5. Because masters also have node components running on them in order to be configured as part of the OpenShift SDN, restart the atomic-openshift-node and openvswitch services:

    # systemctl restart atomic-openshift-node
    # systemctl restart openvswitch
    # journalctl -r -u openvswitch
    # journalctl -r -u atomic-openshift-node
  6. If you are performing a cluster upgrade that requires updating Docker to version 1.12, you must also perform the following steps if you are not already on Docker 1.12:

    Important

    The node component on masters is set by default to unschedulable status during initial installation, so that pods are not deployed to them. However, it is possible to set them schedulable during the initial installation or manually thereafter. If any of your masters are also configured as a schedulable node, skip the following Docker upgrade steps for those masters and instead run all steps described in Upgrading Nodes when you get to that section for those hosts as well.

    1. Upgrade the docker package.

      1. For RHEL 7 systems:

        # yum update docker

        Then, restart the docker service and review the logs to ensure it restarts successfully:

        # systemctl restart docker
        # journalctl -r -u docker
      2. For RHEL Atomic Host 7 systems, upgrade to the latest Atomic tree if one is available:

        Note

        If upgrading to RHEL Atomic Host 7.3.2, this upgrades Docker to version 1.12.

        # atomic host upgrade
    2. After the upgrade is completed and prepared for the next boot, reboot the host and ensure the docker service starts successfully:

      # systemctl reboot
      # journalctl -r -u docker
    3. Remove the following file, which is no longer required:

      # rm /etc/systemd/system/docker.service.d/docker-sdn-ovs.conf
  7. Run the following command on each master to add the atomic-openshift packages back to the list of yum excludes on the host:

    # atomic-openshift-excluder exclude
Note

During the cluster upgrade, it can sometimes be useful to take a master out of rotation since some DNS client libraries will not properly to the other masters for cluster DNS. In addition to stopping the master and controller services, you can remove the EndPoint from the Kubernetes service’s subsets.addresses.

$ oc edit ep/kubernetes -n default

When the master is restarted, the Kubernetes service will be automatically updated.

5.3.4. Updating Policy Definitions

After a cluster upgrade, the recommended default cluster roles may be updated. To check if an update is recommended for your environment, you can run:

# oadm policy reconcile-cluster-roles
Warning

If you have customized default cluster roles and want to ensure a role reconciliation does not modify those customized roles, annotate them with openshift.io/reconcile-protect set to true. Doing so means you are responsible for manually updating those roles with any new or required permissions during upgrades.

This command outputs a list of roles that are out of date and their new proposed values. For example:

# oadm policy reconcile-cluster-roles
apiVersion: v1
items:
- apiVersion: v1
  kind: ClusterRole
  metadata:
    creationTimestamp: null
    name: admin
  rules:
  - attributeRestrictions: null
    resources:
    - builds/custom
...
Note

Your output will vary based on the OpenShift version and any local customizations you have made. Review the proposed policy carefully.

You can either modify this output to re-apply any local policy changes you have made, or you can automatically apply the new policy using the following process:

  1. Reconcile the cluster roles:

    # oadm policy reconcile-cluster-roles \
        --additive-only=true \
        --confirm
  2. Reconcile the cluster role bindings:

    # oadm policy reconcile-cluster-role-bindings \
        --exclude-groups=system:authenticated \
        --exclude-groups=system:authenticated:oauth \
        --exclude-groups=system:unauthenticated \
        --exclude-users=system:anonymous \
        --additive-only=true \
        --confirm
  3. Reconcile security context constraints:

    # oadm policy reconcile-sccs \
        --additive-only=true \
        --confirm

5.3.5. Upgrading Nodes

After upgrading your masters, you can upgrade your nodes. When restarting the atomic-openshift-node service, there will be a brief disruption of outbound network connectivity from running pods to services while the service proxy is restarted. The length of this disruption should be very short and scales based on the number of services in the entire cluster.

Note

You can alternatively use the blue-green deployment method at this point to create a parallel environment for new nodes instead of upgrading them in place.

One at at time for each node that is not also a master, you must disable scheduling and evacuate its pods to other nodes, then upgrade packages and restart services.

  1. Run the following command on each node to remove the atomic-openshift packages from the list of yum excludes on the host:

    # atomic-openshift-excluder unexclude
  2. As a user with cluster-admin privileges, disable scheduling for the node:

    # oadm manage-node <node> --schedulable=false
  3. Evacuate pods on the node to other nodes:

    Important

    The --force option deletes any pods that are not backed by a replication controller.

    # oadm manage-node <node> --evacuate --force
  4. Upgrade the node component packages or related images.

    1. For nodes using the RPM-based method on a RHEL 7 system, upgrade all installed atomic-openshift packages:

      # yum upgrade atomic-openshift\*
    2. For nodes using the containerized method on a RHEL 7 or RHEL Atomic Host 7 system, set the IMAGE_VERSION parameter in the /etc/sysconfig/atomic-openshift-node and /etc/sysconfig/openvswitch files to the version you are upgrading to. For example:

      IMAGE_VERSION=<tag>

      Replace <tag> with v3.4.1.10 for the latest version.

  5. Restart the atomic-openshift-node and openvswitch services and review the logs to ensure they restart successfully:

    # systemctl restart atomic-openshift-node
    # systemctl restart openvswitch
    # journalctl -r -u atomic-openshift-node
    # journalctl -r -u openvswitch
  6. If you are performing a cluster upgrade that requires updating Docker to version 1.12, you must also perform the following steps if you are not already on Docker 1.12:

    1. Upgrade the docker package.

      1. For RHEL 7 systems:

        # yum update docker

        Then, restart the docker service and review the logs to ensure it restarts successfully:

        # systemctl restart docker
        # journalctl -r -u docker

        After Docker is restarted, restart the atomic-openshift-node service again and review the logs to ensure it restarts successfully:

        # systemctl restart atomic-openshift-node
        # journalctl -r -u atomic-openshift-node
      2. For RHEL Atomic Host 7 systems, upgrade to the latest Atomic tree if one is available:

        Note

        If upgrading to RHEL Atomic Host 7.3.2, this upgrades Docker to version 1.10.

        # atomic host upgrade

        After the upgrade is completed and prepared for the next boot, reboot the host and ensure the docker service starts successfully:

        # systemctl reboot
        # journalctl -r -u docker
    2. Remove the following file, which is no longer required:

      # rm /etc/systemd/system/docker.service.d/docker-sdn-ovs.conf
  7. Re-enable scheduling for the node:

    # oadm manage-node <node> --schedulable
  8. Run the following command on each node to add the atomic-openshift packages back to the list of yum excludes on the host:

    # atomic-openshift-excluder exclude
  9. Repeat the previous steps on the next node, and continue repeating these steps until all nodes have been upgraded.
  10. After all nodes have been upgraded, as a user with cluster-admin privileges, verify that all nodes are showing as Ready:

    # oc get nodes
    NAME                        STATUS                     AGE
    master.example.com          Ready,SchedulingDisabled   165d
    node1.example.com           Ready                      165d
    node2.example.com           Ready                      165d

5.3.6. Upgrading the Router

If you have previously deployed a router, the router deployment configuration must be upgraded to apply updates contained in the router image. To upgrade your router without disrupting services, you must have previously deployed a highly-available routing service.

Edit your router’s deployment configuration. For example, if it has the default router name:

# oc edit dc/router

Apply the following changes:

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: registry.access.redhat.com/openshift3/ose-haproxy-router:<tag> 1
        imagePullPolicy: IfNotPresent
        ...
1
Adjust <tag> to match the version you are upgrading to (use v3.4.1.10 for the latest version).

You should see one router pod updated and then the next.

5.3.7. Upgrading the Registry

The registry must also be upgraded for changes to take effect in the registry image. If you have used a PersistentVolumeClaim or a host mount point, you may restart the registry without losing the contents of your registry. Storage for the Registry details how to configure persistent storage for the registry.

Edit your registry’s deployment configuration:

# oc edit dc/docker-registry

Apply the following changes:

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: registry.access.redhat.com/openshift3/ose-docker-registry:<tag> 1
        imagePullPolicy: IfNotPresent
        ...
1
Adjust <tag> to match the version you are upgrading to (use v3.4.1.10 for the latest version).
Important

Images that are being pushed or pulled from the internal registry at the time of upgrade will fail and should be restarted automatically. This will not disrupt pods that are already running.

5.3.8. Updating the Default Image Streams and Templates

By default, the quick and advanced installation methods automatically create default image streams, InstantApp templates, and database service templates in the openshift project, which is a default project to which all users have view access. These objects were created during installation from the JSON files located under the /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/ directory.

Note

Because RHEL Atomic Host 7 cannot use yum to update packages, the following steps must take place on a RHEL 7 system.

  1. Update the packages that provide the example JSON files. On a subscribed Red Hat Enterprise Linux 7 system where you can run the CLI as a user with cluster-admin permissions, install or update to the latest version of the atomic-openshift-utils package, which should also update the openshift-ansible- packages:

    # yum update atomic-openshift-utils

    The openshift-ansible-roles package provides the latest example JSON files.

  2. After a manual upgrade, get the latest templates from openshift-ansible-roles:

    rpm -ql openshift-ansible-roles | grep examples | grep v1.4

    In this example, /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/image-streams/image-streams-rhel7.json is the latest file that you want in the latest openshift-ansible-roles package.

    /usr/share/openshift/examples/image-streams/image-streams-rhel7.json is not owned by a package, but is updated by Ansible. If you are upgrading outside of Ansible. you need to get the latest .json files on the system where you are running oc, which can run anywhere that has access to the master.

  3. Install atomic-openshift-utils and its dependencies to install the new content into /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/.:

    $ oc create -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/image-streams/image-streams-rhel7.json
    $ oc create -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/image-streams/dotnet_imagestreams.json
    $ oc replace -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/image-streams/image-streams-rhel7.json
    $ oc replace -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/image-streams/dotnet_imagestreams.json
  4. Update the templates:

    $ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/quickstart-templates/
    $ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/db-templates/
    $ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/infrastructure-templates/
    $ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/xpaas-templates/
    $ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/xpaas-streams/
    $ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/quickstart-templates/
    $ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/db-templates/
    $ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/infrastructure-templates/
    $ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/xpaas-templates/
    $ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/xpaas-streams/

    Errors are generated for items that already exist. This is expected behavior:

    # oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/quickstart-templates/
    Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/quickstart-templates/cakephp-mysql.json": templates "cakephp-mysql-example" already exists
    Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/quickstart-templates/cakephp.json": templates "cakephp-example" already exists
    Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/quickstart-templates/dancer-mysql.json": templates "dancer-mysql-example" already exists
    Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/quickstart-templates/dancer.json": templates "dancer-example" already exists
    Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.4/quickstart-templates/django-postgresql.json": templates "django-psql-example" already exists

Now, content can be updated. Without running the automated upgrade playbooks, the content is not updated in /usr/share/openshift/.

5.3.9. Importing the Latest Images

After updating the default image streams, you may also want to ensure that the images within those streams are updated. For each image stream in the default openshift project, you can run:

# oc import-image -n openshift <imagestream>

For example, get the list of all image streams in the default openshift project:

# oc get is -n openshift
NAME     DOCKER REPO                                                      TAGS                   UPDATED
mongodb  registry.access.redhat.com/openshift3/mongodb-24-rhel7           2.4,latest,v3.1.1.6    16 hours ago
mysql    registry.access.redhat.com/openshift3/mysql-55-rhel7             5.5,latest,v3.1.1.6    16 hours ago
nodejs   registry.access.redhat.com/openshift3/nodejs-010-rhel7           0.10,latest,v3.1.1.6   16 hours ago
...

Update each image stream one at a time:

# oc import-image -n openshift nodejs
The import completed successfully.

Name:			nodejs
Created:		10 seconds ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2016-07-05T19:20:30Z
Docker Pull Spec:	172.30.204.22:5000/openshift/nodejs

Tag	Spec								Created		PullSpec						Image
latest	4								9 seconds ago	registry.access.redhat.com/rhscl/nodejs-4-rhel7:latest	570ad8ed927fd5c2c9554ef4d9534cef808dfa05df31ec491c0969c3bd372b05
4	registry.access.redhat.com/rhscl/nodejs-4-rhel7:latest		9 seconds ago	<same>							570ad8ed927fd5c2c9554ef4d9534cef808dfa05df31ec491c0969c3bd372b05
0.10	registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest	9 seconds ago	<same>							a1ef33be788a28ec2bdd48a9a5d174ebcfbe11c8e986d2996b77f5bccaaa4774
Important

In order to update your S2I-based applications, you must manually trigger a new build of those applications after importing the new images using oc start-build <app-name>.

5.3.10. Upgrading the EFK Logging Stack

Use the following to upgrade an already-deployed EFK logging stack.

Note

The following steps apply when upgrading to OpenShift Container Platform 3.4+.

  1. Ensure you are working in the project where the EFK stack was previously deployed. For example, if the project is named logging:

    $ oc project logging
  2. Recreate the deployer templates for service accounts and running the deployer:

    $ oc apply -n openshift -f \
        /usr/share/ansible/openshift-ansible/roles/openshift_hosted_templates/files/v1.4/enterprise/logging-deployer.yaml
  3. Generate any missing service accounts and roles:

    $ oc process logging-deployer-account-template | oc apply -f -
  4. Ensure that the cluster role oauth-editor is assigned to the logging-deployer service account:

    $ oadm policy add-cluster-role-to-user oauth-editor \
           system:serviceaccount:logging:logging-deployer
  5. Ensure that the cluster role rolebinding-reader is assigned to the aggregated-logging-elasticsearch service account where logging is the namespace with aggregated logging installed:

    $ oadm policy add-cluster-role-to-user rolebinding-reader \
         system:serviceaccount:logging:aggregated-logging-elasticsearch
  6. If you are upgrading from OpenShift Container Platform 3.3 to 3.4, add the privileged SCC to the aggregated-logging-fluentd service account:

    $ oadm policy add-scc-to-user privileged \
        system:serviceaccount:logging:aggregated-logging-fluentd
  7. In preparation for running the deployer, ensure that you have the configurations for your current deployment in the logging-deployer ConfigMap.

    Important

    Ensure that your image version is the latest version, not the currently installed version.

  8. Run the deployer with the parameter in upgrade mode:

    $ oc new-app logging-deployer-template -p MODE=upgrade

    Running the deployer in this mode handles scaling down the components to minimize loss of logs, patching configurations, generating missing secrets and keys, and scaling the components back up to their previous replica count.

    Important

    Due to the privileges needed to label and unlabel a node for controlling the deployment of Fluentd pods, the deployer does delete the logging-fluentd Daemonset and recreates it from the logging-fluentd-template template.

5.3.11. Upgrading Cluster Metrics

After upgrading an already-deployed Cluster Metrics install, you must update to a newer version of the metrics components.

  • The update process stops all the metrics containers, updates the metrics configuration files, and redeploys the newer components.
  • It does not change the metrics route.
  • It does not delete the metrics persistent volume claim. Metrics stored to persistent volumes before the update are available after the update completes.
Important

The update deletes all non-persisted metric values and overwrites local changes to the metrics configurations. For example, the number of instances in a replica set is not saved.

To upgrade cluster metrics:

  1. If you are upgrading from OpenShift Container Platform 3.3 to 3.4, first add the view permission to the hawkular service account:

    $ oadm policy add-role-to-user view \
            system:serviceaccount:openshift-infra:hawkular \
            -n openshift-infra
  2. Then, follow the same steps as when the metrics components were first deployed, using the correct template, except this time, specify the MODE=refresh option:

    $ oc new-app --as=system:serviceaccount:openshift-infra:metrics-deployer \
        -f metrics-deployer.yaml \
        -p HAWKULAR_METRICS_HOSTNAME=hm.example.com \
        -p MODE=refresh 1
    1
    In the original deployment command, there was no MODE=refresh.
Note

During the update, the metrics components do not run. Because of this, they cannot collect data and a gap normally appears in the graphs.

5.3.12. Additional Manual Steps Per Release

Some OpenShift Container Platform releases may have additional instructions specific to that release that must be performed to fully apply the updates across the cluster. This section will be updated over time as new asynchronous updates are released for OpenShift Container Platform 3.4.

See the OpenShift Container Platform 3.4 Release Notes to review the latest release notes.

5.3.12.1. OpenShift Container Platform 3.4.0.40

If you had previously already upgraded to 3.4.0.39 (the GA release of OpenShift Container Platform 3.4), after upgrading to the 3.4.0.40 release you must also then perform a data migration using a data migration tool. See the following Knowledgebase Solution for further details on this tool:

https://access.redhat.com/solutions/2887651

5.3.12.2. OpenShift Container Platform 3.4.1.2

There are no additional manual steps for the upgrade to OpenShift Container Platform 3.4.1.2 that are not already mentioned inline during the standard manual upgrade process.

5.3.12.3. OpenShift Container Platform 3.4.1.5

OpenShift Container Platform 3.4.1.5 delivers the migration tool mentioned in the above OpenShift Container Platform 3.4.0.40 steps. See the following Knowledgebase Solution for instructions on running the script:

https://access.redhat.com/solutions/2887651

5.3.12.4. OpenShift Container Platform 3.4.1.7

There are no additional manual steps for the upgrade to OpenShift Container Platform 3.4.1.7 that are not already mentioned inline during the standard manual upgrade process.

5.3.12.5. OpenShift Container Platform 3.4.1.10

If you want to free up space in etcd, see OpenShift Container Platform 3.4.1.10 in Release Notes for an optional image manifest migration script.

5.3.13. Verifying the Upgrade

To verify the upgrade, first check that all nodes are marked as Ready:

# oc get nodes
NAME                        STATUS                     AGE
master.example.com          Ready,SchedulingDisabled   165d
node1.example.com           Ready                      165d
node2.example.com           Ready                      165d

Then, verify that you are running the expected versions of the docker-registry and router images, if deployed. Replace <tag> with v3.4.1.10 for the latest version.

# oc get -n default dc/docker-registry -o json | grep \"image\"
    "image": "openshift3/ose-docker-registry:<tag>",
# oc get -n default dc/router -o json | grep \"image\"
    "image": "openshift3/ose-haproxy-router:<tag>",

After upgrading, you can use the diagnostics tool on the master to look for common issues:

# oadm diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.

5.4. Blue-Green Deployments

5.4.1. Overview

Important

An etcd performance issue has been discovered on new and upgraded OpenShift Container Platform 3.4 clusters. See the following Knowledgebase Solution for further details:

https://access.redhat.com/solutions/2916381
(BZ#1415839)

Note

This topic serves as an alternative approach for node upgrades to the in-place upgrade method.

The blue-green deployment upgrade method follows a similar flow to the in-place method: masters and etcd servers are still upgraded first, however a parallel environment is created for new nodes instead of upgrading them in-place.

This method allows administrators to switch traffic from the old set of nodes (e.g., the "blue" deployment) to the new set (e.g., the "green" deployment) after the new deployment has been verified. If a problem is detected, it is also then easy to rollback to the old deployment quickly.

While blue-green is a proven and valid strategy for deploying just about any software, there are always trade-offs. Not all environments have the same uptime requirements or the resources to properly perform blue-green deployments. In an OpenShift Container Platform environment, the most suitable candidate for blue-green deployments are the nodes. All user processes run on these systems and even critical pieces of OpenShift Container Platform infrastructure are self-hosted there. Uptime is most important for these workloads and the additional complexity of blue-green deployments can be justified.

The exact implementation of this approach varies based on your requirements. Often the main challenge is having the excess capacity to facilitate such an approach.

Another lesser challenge is that the administrator must temporarily share the Red Hat software entitlements between the blue-green deployments or provide access to the installation content by means of a system such as Red Hat Satellite. This can be accomplished by sharing the consumer ID from the previous host.

5.4.2. Preparing for Upgrade

  1. On the old host:

    # subscription-manager identity | grep system
    system identity: 6699375b-06db-48c4-941e-689efd6ce3aa
  2. On the new host:

    # subscription-manager register --consumerid=6699375b-06db-48c4-941e-689efd6ce3aa
    Important

    After a successful deployment, remember to unregister the old host with subscription-manager clean to prevent the environment from being out of compliance.

  3. After the master and etcd servers have been upgraded, you must ensure that your current production nodes are labeled either blue or green. In this example, the current installation will be blue and the new environment will be green. On each production node in your current installation:

    $ oc label --all nodes color=blue

    In the case of nodes requiring the uptime guarantees of a blue-green deployment, the -l flag can be used to match a subset of the environment using a selector.

  4. Create the new green environment for any nodes that are to be replaced by adding an equal number of new nodes to the existing cluster. Ansible can apply the color=green label using the openshift_node_labels variable for each node.
  5. In order to delay workload scheduling until the nodes are healthy, be sure to set the openshift_schedulable=false variable. After the green nodes are in Ready state, they can be made schedulable.
  6. Blue nodes are disabled so that no new pods are run on them:

    # oadm manage-node --schedulable=true --selector=color=green
    # oadm manage-node --schedulable=false --selector=color=blue

A common practice is to scale the registry and router pods until they are migrated to the green nodes. For these pods, a canary deployment approach is commonly used. Scaling them up will make them immediately active on the new nodes. Pointing the deployment configuration to the new image initiates a rolling update. However, because of node anti-affinity, and the fact that the blue nodes are still unschedulable, the deployments to the old nodes will fail. At this point, the registry and router deployments can be scaled down to the original number of pods. At any given point, the original number of pods is still available so no capacity is lost.

5.4.3. Warming the New Nodes

In order for pods to be migrated from the blue environment to the green, the images must be pulled. Network latency and load on the registry can cause delays if there is not sufficient capacity built in to the environment. Often, the best way to minimize impact to the running system is to trigger new pod deployments that will land on the new nodes. Accomplish this by importing new image streams.

A major release of OpenShift Container Platform is the motivation for a blue-green deployment. At that time, new image streams become available for users of Source-to-Image (S2I). Upon import, any builds or deployments configured with ImageChangeTriggers are automatically created.

Note

It is important to realize that this process can trigger a large number of builds. The good news is that the builds are performed on the green nodes and, therefore, do not impact any traffic on the blue deployment.

To monitor build progress across all namespaces (projects) in the cluster:

$ oc get events -w --all-namespaces

In large environments, builds rarely completely stop. However, you should see a large increase and decrease caused by the administrative import.

Another benefit of triggering the builds is that it does a fairly good job of fetching the majority of the ancillary images to all nodes such as the various build images, the pod infrastructure image, and deployers. Everything else can be moved over using node evacuation and will proceed more quickly as a result.

5.4.4. Node Evacuation

For larger deployments, it is possible to have other labels that help determine how evacuation can be coordinated. The most conservative approach for avoiding downtime is to evacuate one node at a time. If services are composed of pods using zone anti-affinity, then an entire zone can be evacuated at once. It is important to ensure that the storage volumes used are available in the new zone as this detail can vary among cloud providers.

In OpenShift Enterprise 3.2 and later, a node evacuation is triggered whenever the service is stopped. Achieve manual evacuation and deletion of all blue nodes at once by:

# oadm manage-node --selector=color=blue --evacuate
# oc delete node --selector=color=blue

5.5. Operating System Updates and Upgrades

5.5.1. Updating and Upgrading the Operating System

Updating or upgrading your operating system (OS), by either changing OS versions or updating the system software, can impact the OpenShift Container Platform software running on those machines. In particular, these updates can affect the iptables rules or ovs flows that OpenShift Container Platform requires to operate.

Use the following to safely upgrade the OS on a host:

  1. Ensure the host is unschedulable, meaning that no new pods will be placed onto the host:

    $ oadm manage-node <node_name> --schedulable=false
  2. Migrate the pods from the host:

    $ oadm drain <node_name> --force --delete-local-data --ignore-daemonsets
  3. Install or update to the following latest available atomic-openshift-excluder package on each RHEL 7 system, which helps ensure your systems stay on the correct versions of atomic-openshift packages when you are not trying to upgrade, according to the OpenShift Container Platform version:

    # yum install atomic-openshift-excluder

    These packages add entries to the exclude directive in the host’s /etc/yum.conf file.

  4. Due to a bug (BZ#1430929), if you previously installed the atomic-openshift-docker-excluder package as instructed in OpenShift Container Platform 3.3, upgrading this package to the 3.4 version does not remove docker*1.12* from the exclude directive as intended.

    As a workaround until the bug is fixed, remove the installed version of atomic-openshift-docker-excluder package, then install the 3.4 version:

    # yum remove atomic-openshift-docker-excluder
    # yum install atomic-openshift-docker-excluder
  5. Update or upgrade the host packages, and reboot the host. A reboot ensures that the host is running the newest versions, and means that the docker and OpenShift Container Platform processes have been restarted, which will force them to check that all of the rules in other services are correct.

    However, instead of rebooting a node host, you can restart the services that are affected, or preserve the iptables state. Both processes are described in the OpenShift Container Platform IPtables topic. The ovs flow rules do not need to be saved, but restarting the OpenShift Container Platform node software will fix the flow rules.

  6. Configure the host to be schedulable again:

    $ oadm manage-node <node_name> --schedulable=true

Chapter 6. Downgrading OpenShift

6.1. Overview

Following an OpenShift Container Platform upgrade, it may be desirable in extreme cases to downgrade your cluster to a previous version. The following sections outline the required steps for each system in a cluster to perform such a downgrade for the OpenShift Container Platform 3.2 to 3.1 downgrade path.

Warning

These steps are currently only supported for RPM-based installations of OpenShift Container Platform and assumes downtime of the entire cluster.

Important

For the OpenShift Container Platform 3.1 to 3.0 downgrade path, see the OpenShift Enterprise 3.1 documentation, which has modified steps.

6.2. Verifying Backups

The Ansible playbook used during the upgrade process should have created a backup of the master-config.yaml file and the etcd data directory. Ensure these exist on your masters and etcd members:

/etc/origin/master/master-config.yaml.<timestamp>
/var/lib/origin/etcd-backup-<timestamp>

Also, back up the node-config.yaml file on each node (including masters, which have the node component on them) with a timestamp:

/etc/origin/node/node-config.yaml.<timestamp>

If you are using an external etcd cluster (versus the single embedded etcd), the backup is likely created on all etcd members, though only one is required for the recovery process.

The RPM downgrade process in a later step should create .rpmsave backups of the following files, but it may be a good idea to keep a separate copy regardless:

/etc/sysconfig/atomic-openshift-master
/etc/etcd/etcd.conf 1
1
Only required if using external etcd.

6.3. Shutting Down the Cluster

On all masters, nodes, and etcd members (if using an external etcd cluster), ensure the relevant services are stopped.

On the master in a single master cluster:

# systemctl stop atomic-openshift-master

On each master in a multi-master cluster:

# systemctl stop atomic-openshift-master-api
# systemctl stop atomic-openshift-master-controllers

On all master and node hosts:

# systemctl stop atomic-openshift-node

On any external etcd hosts:

# systemctl stop etcd

6.4. Removing RPMs

On all masters, nodes, and etcd members (if using an external etcd cluster), remove the following packages:

# yum remove atomic-openshift \
    atomic-openshift-clients \
    atomic-openshift-node \
    atomic-openshift-master \
    openvswitch \
    atomic-openshift-sdn-ovs \
    tuned-profiles-atomic-openshift-node

If you are using external etcd, also remove the etcd package:

# yum remove etcd

For embedded etcd, you can leave the etcd package installed, as the package is only required so that the etcdctl command is available to issue operations in later steps.

6.5. Downgrading Docker

OpenShift Container Platform 3.2 requires Docker 1.9.1 and also supports Docker 1.10.3, however OpenShift Container Platform 3.1 requires Docker 1.8.2.

Downgrade to Docker 1.8.2 on each host using the following steps:

  1. Remove all local containers and images on the host. Any pods backed by a replication controller will be recreated.

    Warning

    The following commands are destructive and should be used with caution.

    Delete all containers:

    # docker rm $(docker ps -a -q)

    Delete all images:

    # docker rmi $(docker images -q)
  2. Use yum swap (instead of yum downgrade) to install Docker 1.8.2:

    # yum swap docker-* docker-*1.8.2
    # sed -i 's/--storage-opt dm.use_deferred_deletion=true//' /etc/sysconfig/docker-storage
    # systemctl restart docker
  3. You should now have Docker 1.8.2 installed and running on the host. Verify with the following:

    # docker version
    Client:
     Version:      1.8.2-el7
     API version:  1.20
     Package Version: docker-1.8.2-10.el7.x86_64
    [...]
    
    # systemctl status docker
    ● docker.service - Docker Application Container Engine
       Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
       Active: active (running) since Mon 2016-06-27 15:44:20 EDT; 33min ago
    [...]

6.6. Reinstalling RPMs

Disable the OpenShift Container Platform 3.3 repositories, and re-enable the 3.2 repositories:

# subscription-manager repos \
    --disable=rhel-7-server-ose-3.3-rpms \
    --enable=rhel-7-server-ose-3.2-rpms

On each master, install the following packages:

# yum install atomic-openshift \
    atomic-openshift-clients \
    atomic-openshift-node \
    atomic-openshift-master \
    openvswitch \
    atomic-openshift-sdn-ovs \
    tuned-profiles-atomic-openshift-node

On each node, install the following packages:

# yum install atomic-openshift \
    atomic-openshift-node \
    openvswitch \
    atomic-openshift-sdn-ovs \
    tuned-profiles-atomic-openshift-node

If using an external etcd cluster, install the following package on each etcd member:

# yum install etcd

6.7. Restoring etcd

Whether using embedded or external etcd, you must first restore the etcd backup by creating a new, single node etcd cluster. If using external etcd with multiple members, you must then also add any additional etcd members to the cluster one by one.

However, the details of the restoration process differ between embedded and external etcd. See the following section that matches your etcd configuration and follow the relevant steps before continuing to Bringing OpenShift Container Platform Services Back Online.

Follow the Cluster Restore procedure to restore single-member etcd clusters.

6.7.1. Embedded etcd

Restore your etcd backup and configuration:

  1. Run the following on the master with the embedded etcd:

    # ETCD_DIR=/var/lib/origin/openshift.local.etcd
    # mv $ETCD_DIR /var/lib/etcd.orig
    # cp -Rp /var/lib/origin/etcd-backup-<timestamp>/ $ETCD_DIR
    # chcon -R --reference /var/lib/etcd.orig/ $ETCD_DIR
    # chown -R etcd:etcd $ETCD_DIR
    Warning

    The $ETCD_DIR location differs between external and embedded etcd.

  2. Create the new, single node etcd cluster:

    # etcd -data-dir=/var/lib/origin/openshift.local.etcd \
        -force-new-cluster

    Verify etcd has started successfully by checking the output from the above command, which should look similar to the following near the end:

    [...]
    2016-06-24 12:14:45.644073 I | etcdserver: starting server... [version: 2.2.5, cluster version: 2.2]
    [...]
    2016-06-24 12:14:46.834394 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379 http://localhost:4001]} to cluster 5580663a6e0002
  3. Shut down the process by running the following from a separate terminal:

    # pkill etcd
  4. Continue to Bringing OpenShift Container Platform Services Back Online.

6.7.2. External etcd

Choose a system to be the initial etcd member, and restore its etcd backup and configuration:

  1. Run the following on the etcd host:

    # ETCD_DIR=/var/lib/etcd/
    # mv $ETCD_DIR /var/lib/etcd.orig
    # cp -Rp /var/lib/origin/etcd-backup-<timestamp>/ $ETCD_DIR
    # chcon -R --reference /var/lib/etcd.orig/ $ETCD_DIR
    # chown -R etcd:etcd $ETCD_DIR
    Warning

    The $ETCD_DIR location differs between external and embedded etcd.

  2. Restore your /etc/etcd/etcd.conf file from backup or .rpmsave.
  3. Create the new single node cluster using etcd’s --force-new-cluster option. You can do this with a long complex command using the values from /etc/etcd/etcd.conf, or you can temporarily modify the systemd unit file and start the service normally.

    To do so, edit the /usr/lib/systemd/system/etcd.service and add --force-new-cluster:

    # sed -i '/ExecStart/s/"$/  --force-new-cluster"/' /usr/lib/systemd/system/etcd.service
    # cat /usr/lib/systemd/system/etcd.service  | grep ExecStart
    
    ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --force-new-cluster"

    Then restart the etcd service:

    # systemctl daemon-reload
    # systemctl start etcd
  4. Verify the etcd service started correctly, then re-edit the /usr/lib/systemd/system/etcd.service file and remove the --force-new-cluster option:

    # sed -i '/ExecStart/s/ --force-new-cluster//' /usr/lib/systemd/system/etcd.service
    # cat /usr/lib/systemd/system/etcd.service  | grep ExecStart
    
    ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd"
  5. Restart the etcd service, then verify the etcd cluster is running correctly and displays OpenShift Container Platform’s configuration:

    # systemctl daemon-reload
    # systemctl restart etcd
    # etcdctl --cert-file=/etc/etcd/peer.crt \
        --key-file=/etc/etcd/peer.key \
        --ca-file=/etc/etcd/ca.crt \
        --peers="https://172.16.4.18:2379,https://172.16.4.27:2379" \
        ls /
  6. If you have additional etcd members to add to your cluster, continue to Adding Additional etcd Members. Otherwise, if you only want a single node external etcd, continue to Bringing OpenShift Container Platform Services Back Online.

6.7.2.1. Adding Additional etcd Members

To add additional etcd members to the cluster, you must first adjust the default localhost peer in the peerURLs value for the first member:

  1. Get the member ID for the first member using the member list command:

    # etcdctl --cert-file=/etc/etcd/peer.crt \
        --key-file=/etc/etcd/peer.key \
        --ca-file=/etc/etcd/ca.crt \
        --peers="https://172.18.1.18:2379,https://172.18.9.202:2379,https://172.18.0.75:2379" \
        member list
  2. Update the value of peerURLs using the etcdctl member update command by passing the member ID obtained from the previous step:

    # etcdctl --cert-file=/etc/etcd/peer.crt \
        --key-file=/etc/etcd/peer.key \
        --ca-file=/etc/etcd/ca.crt \
        --peers="https://172.18.1.18:2379,https://172.18.9.202:2379,https://172.18.0.75:2379" \
        member update 511b7fb6cc0001 https://172.18.1.18:2380

    Alternatively, you can use curl:

    # curl --cacert /etc/etcd/ca.crt \
        --cert /etc/etcd/peer.crt \
        --key /etc/etcd/peer.key \
        https://172.18.1.18:2379/v2/members/511b7fb6cc0001 \
        -XPUT -H "Content-Type: application/json" \
        -d '{"peerURLs":["https://172.18.1.18:2380"]}'
  3. Re-run the member list command and ensure the peer URLs no longer include localhost.
  4. Now add each additional member to the cluster, one at a time.

    Warning

    Each member must be fully added and brought online one at a time. When adding each additional member to the cluster, the peerURLs list must be correct for that point in time, so it will grow by one for each member added. The etcdctl member add command will output the values that need to be set in the etcd.conf file as you add each member, as described in the following instructions.

    1. For each member, add it to the cluster using the values that can be found in that system’s etcd.conf file:

      # etcdctl --cert-file=/etc/etcd/peer.crt \
          --key-file=/etc/etcd/peer.key \
          --ca-file=/etc/etcd/ca.crt \
          --peers="https://172.16.4.18:2379,https://172.16.4.27:2379" \
          member add 10.3.9.222 https://172.16.4.27:2380
      
      Added member named 10.3.9.222 with ID 4e1db163a21d7651 to cluster
      
      ETCD_NAME="10.3.9.222"
      ETCD_INITIAL_CLUSTER="10.3.9.221=https://172.16.4.18:2380,10.3.9.222=https://172.16.4.27:2380"
      ETCD_INITIAL_CLUSTER_STATE="existing"
    2. Using the environment variables provided in the output of the above etcdctl member add command, edit the /etc/etcd/etcd.conf file on the member system itself and ensure these settings match.
    3. Now start etcd on the new member:

      # rm -rf /var/lib/etcd/member
      # systemctl enable etcd
      # systemctl start etcd
    4. Ensure the service starts correctly and the etcd cluster is now healthy:

      # etcdctl --cert-file=/etc/etcd/peer.crt \
          --key-file=/etc/etcd/peer.key \
          --ca-file=/etc/etcd/ca.crt \
          --peers="https://172.16.4.18:2379,https://172.16.4.27:2379" \
          member list
      
      51251b34b80001: name=10.3.9.221 peerURLs=https://172.16.4.18:2380 clientURLs=https://172.16.4.18:2379
      d266df286a41a8a4: name=10.3.9.222 peerURLs=https://172.16.4.27:2380 clientURLs=https://172.16.4.27:2379
      
      # etcdctl --cert-file=/etc/etcd/peer.crt \
          --key-file=/etc/etcd/peer.key \
          --ca-file=/etc/etcd/ca.crt \
          --peers="https://172.16.4.18:2379,https://172.16.4.27:2379" \
          cluster-health
      
      cluster is healthy
      member 51251b34b80001 is healthy
      member d266df286a41a8a4 is healthy
    5. Now repeat this process for the next member to add to the cluster.
  5. After all additional etcd members have been added, continue to Bringing OpenShift Container Platform Services Back Online.

6.8. Bringing OpenShift Container Platform Services Back Online

On each OpenShift Container Platform master, restore your master and node configuration from backup and enable and restart all relevant services.

On the master in a single master cluster:

# cp /etc/sysconfig/atomic-openshift-master.rpmsave /etc/sysconfig/atomic-openshift-master
# cp /etc/origin/master/master-config.yaml.<timestamp> /etc/origin/master/master-config.yaml
# cp /etc/origin/node/node-config.yaml.<timestamp> /etc/origin/node/node-config.yaml
# systemctl enable atomic-openshift-master
# systemctl enable atomic-openshift-node
# systemctl start atomic-openshift-master
# systemctl start atomic-openshift-node

On each master in a multi-master cluster:

# cp /etc/sysconfig/atomic-openshift-master-api.rpmsave /etc/sysconfig/atomic-openshift-master-api
# cp /etc/sysconfig/atomic-openshift-master-controllers.rpmsave /etc/sysconfig/atomic-openshift-master-controllers
# cp /etc/origin/master/master-config.yaml.<timestamp> /etc/origin/master/master-config.yaml
# cp /etc/origin/node/node-config.yaml.<timestamp> /etc/origin/node/node-config.yaml
# systemctl enable atomic-openshift-master-api
# systemctl enable atomic-openshift-master-controllers
# systemctl enable atomic-openshift-node
# systemctl start atomic-openshift-master-api
# systemctl start atomic-openshift-master-controllers
# systemctl start atomic-openshift-node

On each OpenShift Container Platform node, restore your node-config.yaml file from backup and enable and restart the atomic-openshift-node service:

# cp /etc/origin/node/node-config.yaml.<timestamp> /etc/origin/node/node-config.yaml
# systemctl enable atomic-openshift-node
# systemctl start atomic-openshift-node

Your OpenShift Container Platform cluster should now be back online.

6.9. Verifying the Downgrade

To verify the downgrade, first check that all nodes are marked as Ready:

# oc get nodes
NAME                        STATUS                     AGE
master.example.com          Ready,SchedulingDisabled   165d
node1.example.com           Ready                      165d
node2.example.com           Ready                      165d

Then, verify that you are running the expected versions of the docker-registry and router images, if deployed:

# oc get -n default dc/docker-registry -o json | grep \"image\"
    "image": "openshift3/ose-docker-registry:v3.1.1.6",
# oc get -n default dc/router -o json | grep \"image\"
    "image": "openshift3/ose-haproxy-router:v3.1.1.6",

You can use the diagnostics tool on the master to look for common issues and provide suggestions. In OpenShift Container Platform 3.1, the oc adm diagnostics tool is available as openshift ex diagnostics:

# openshift ex diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.

Chapter 7. Master and Node Configuration

7.1. Overview

The openshift start command is used to launch OpenShift Container Platform servers. The command and its subcommands (master to launch a master server and node to launch a node server) all take a limited set of arguments that are sufficient for launching servers in a development or experimental environment.

However, these arguments are insufficient to describe and control the full set of configuration and security options that are necessary in a production environment. To provide those options, it is necessary to use the dedicated master and node configuration files.

Master configuration files and node configuration files are fully specified with no default values. Therefore, any empty value indicates that you want to start up with an empty value for that parameter. This makes it easy to reason about exactly what your configuration is, but it also makes it difficult to remember all of the options to specify. To make this easier, the configuration files can be created with the --write-config option and then used with the --config option.

7.2. Master Configuration Files

This section reviews parameters mentioned in the master-config.yaml file.

You can create a new master configuration file to see the valid options for your installed version of OpenShift Container Platform.

7.2.1. Admission Control Configuration

Table 7.1. Admission Control Configuration Parameters

Parameter NameDescription

AdmissionConfig

Contains admission control plug-in configuration.

APIServerArguments

Key-value pairs that will be passed directly to the Kube API server that match the API servers' command line arguments. These are not migrated, but if you reference a value that does not exist the server will not start. These values may override other settings in KubernetesMasterConfig, which may cause invalid configurations.

ControllerArguments

Key-value pairs that will be passed directly to the Kube controller manager that match the controller manager’s command line arguments. These are not migrated, but if you reference a value that does not exist the server will not start. These values may override other settings in KubernetesMasterConfig, which may cause invalid configurations.

DefaultAdmissionConfig

Used to enable or disable various admission plug-ins. When this type is present as the configuration object under pluginConfig and if the admission plug-in supports it, this will cause an off by default admission plug-in to be enabled.

PluginConfig

Allows specifying a configuration file per admission control plug-in.

PluginOrderOverride

A list of admission control plug-in names that will be installed on the master. Order is significant. If empty, a default list of plug-ins is used.

SchedulerArguments

Key-value pairs that will be passed directly to the Kube scheduler that match the scheduler’s command line arguments. These are not migrated, but if you reference a value that does not exist the server will not start. These values may override other settings in KubernetesMasterConfig, which may cause invalid configurations.

7.2.2. Asset Configuration

Table 7.2. Asset Configuration Parameters

Parameter NameDescription

AssetConfig

Holds the necessary configuration options for serving assets.

DisabledFeatures

A list of features that should not be started. You will likely want to set this as null. It is very unlikely that anyone will want to manually disable features and that is not encouraged.

Extensions

Files to serve from the asset server file system under a subcontext.

ExtensionDevelopment

When set to true, tells the asset server to reload extension scripts and stylesheets for every request rather than only at startup. It lets you develop extensions without having to restart the server for every change.

ExtensionProperties

Key- (string) and value- (string) pairs that will be injected into the console under the global variable OPENSHIFT_EXTENSION_PROPERTIES.

ExtensionScripts

File paths on the asset server files to load as scripts when the web console loads.

ExtensionStylesheets

File paths on the asset server files to load as style sheets when the web console loads.

LoggingPublicURL

The public endpoint for logging (optional).

LogoutURL

An optional, absolute URL to redirect web browsers to after logging out of the web console. If not specified, the built-in logout page is shown.

MasterPublicURL

How the web console can access the OpenShift Container Platform server.

MetricsPublicURL

The public endpoint for metrics (optional).

PublicURL

URL of the the asset server.

7.2.3. Authentication and Authorization Configuration

Table 7.3. Authentication and Authorization Parameters

Parameter NameDescription

authConfig

Holds authentication and authorization configuration options.

AuthenticationCacheSize

Indicates how many authentication results should be cached. If 0, the default cache size is used.

AuthorizationCacheTTL

Indicates how long an authorization result should be cached. It takes a valid time duration string (e.g. "5m"). If empty, you get the default timeout. If zero (e.g. "0m"), caching is disabled.

7.2.4. Controller Configuration

Table 7.4. Controller Configuration Parameters

Parameter NameDescription

Controllers

List of the controllers that should be started. If set to none, no controllers will start automatically. The default value is * which will start all controllers. When using *, you may exclude controllers by prepending a - in front of their name. No other values are recognized at this time.

ControllerLeaseTTL

Enables controller election, instructing the master to attempt to acquire a lease before controllers start and renewing it within a number of seconds defined by this value. Setting this value non-negative forces pauseControllers=true. This value defaults off (0, or omitted) and controller election can be disabled with -1.

PauseControllers

Instructs the master to not automatically start controllers, but instead to wait until a notification to the server is received before launching them.

7.2.5. etcd Configuration

Table 7.5. etcd Configuration Parameters

Parameter NameDescription

Address

The advertised host:port for client connections to etcd.

etcdClientInfo

Contains information about how to connect to etcd.

etcdConfig

Holds the necessary configuration options for connecting with an etcd database.

etcdStorageConfig

Contains information about how API resources are stored in etcd. These values are only relevant when etcd is the backing store for the cluster.

KubernetesStoragePrefix

The path within etcd that the Kubernetes resources will be rooted under. This value, if changed, will mean existing objects in etcd will no longer be located. The default value is kubernetes.io.

KubernetesStorageVersion

The API version that Kubernetes resources in etcd should be serialized to. This value should not be advanced until all clients in the cluster that read from etcd have code that allows them to read the new version.

OpenShiftStoragePrefix

The path within etcd that the OpenShift Container Platform resources will be rooted under. This value, if changed, will mean existing objects in etcd will no longer be located. The default value is openshift.io.

OpenShiftStorageVersion

API version that OS resources in etcd should be serialized to. This value should not be advanced until all clients in the cluster that read from etcd have code that allows them to read the new version.

PeerAddress

The advertised host:port for peer connections to etcd.

PeerServingInfo

Describes how to start serving the etcd peer.

ServingInfo

Describes how to start serving the etcd master.

StorageDir

The path to the etcd storage directory.

7.2.6. Grant Configuration

Table 7.6. Grant Configuration Parameters

Parameter NameDescription

GrantConfig

Describes how to handle grants.

GrantHandlerAuto

Auto-approves client authorization grant requests.

GrantHandlerDeny

Auto-denies client authorization grant requests.

GrantHandlerPrompt

Prompts the user to approve new client authorization grant requests.

Method

Determines the default strategy to use when an OAuth client requests a grant.This method will be used only if the specific OAuth client does not provide a strategy of their own. Valid grant handling methods are:

  • auto: always approves grant requests, useful for trusted clients
  • prompt: prompts the end user for approval of grant requests, useful for third-party clients
  • deny: always denies grant requests, useful for black-listed clients

7.2.7. Image Configuration

Table 7.7. Image Configuration Parameters

Parameter NameDescription

DisableScheduledImport

Allows scheduled background import of images to be disabled.

Format

The format of the name to be built for the system component.

ImageConfig

Holds options that describe how to build image names for system components.

ImagePolicyConfig

Controls limits and behavior for importing images.

Latest

Determines if the latest tag will be pulled from the registry.

MaxImagesBulkImportedPerRepository

Controls the number of images that are imported when a user does a bulk import of a Docker repository. This number defaults to 5 to prevent users from importing large numbers of images accidentally. Set -1 for no limit.

MaxScheduledImageImportsPerMinute

The maximum number of scheduled image streams that will be imported in the background per minute. The default value is 60. Set to -1 for unlimited.

ScheduledImageImportMinimumIntervalSeconds

The minimum number of seconds that can elapse between when image streams scheduled for background import are checked against the upstream repository. The default value is 15 minutes.

7.2.8. Kubernetes Master Configuration

Table 7.8. Kubernetes Master Configuration Parameters

Parameter NameDescription

APILevels

A list of API levels that should be enabled on startup, v1 as examples.

DisabledAPIGroupVersions

A map of groups to the versions (or *) that should be disabled.

KubeletClientInfo

Contains information about how to connect to kubelets.

KubernetesMasterConfig

Holds the necessary configuration options for the Kubernetes master.

MasterCount

The number of expected masters that should be running. This value defaults to 1 and may be set to a positive integer, or if set to -1, indicates this is part of a cluster.

MasterIP

The public IP address of Kubernetes resources. If empty, the first result from net.InterfaceAddrs will be used.

MasterKubeConfig

File name for the .kubeconfig file that describes how to connect this node to the master.

ServicesNodePortRange

The range to use for assigning service public ports on a host.

ServicesSubnet

The subnet to use for assigning service IPs.

StaticNodeNames

The list of nodes that are statically known.

7.2.9. Network Configuration

Table 7.9. Network Configuration Parameters

Parameter NameDescription

ClusterNetworkCIDR

The CIDR string to specify the global overlay network’s L3 space.

ExternalIPNetworkCIDRs

Controls what values are acceptable for the service external IP field. If empty, no externalIP may be set. It may contain a list of CIDRs which are checked for access. If a CIDR is prefixed with !, IPs in that CIDR will be rejected. Rejections will be applied first, then the IP checked against one of the allowed CIDRs. You hould ensure this range does not overlap with your nodes, pods, or service CIDRs for security reasons.

HostSubnetLength

The number of bits to allocate to each host’s subnet. For example, 8 would mean a /24 network on the host.

IngressIPNetworkCIDR

Controls the range to assign ingress IPs from for services of type LoadBalancer on bare metal. If empty, ingress IPs will not be assigned. It may contain a single CIDR that will be allocated from. For security reasons, you should ensure that this range does not overlap with the CIDRs reserved for external IPs, nodes, pods, or services.

NetworkConfig

Provides network options for the node.

NetworkPluginName

The name of the network plug-in to use.

ServiceNetwork

The CIDR string to specify the service networks.

7.2.10. OAuth Authentication Configuration

Table 7.10. OAuth Configuration Parameters

Parameter NameDescription

AlwaysShowProviderSelection

Forces the provider selection page to render even when there is only a single provider.

AssetPublicURL

Used for building valid client redirect URLs for external access.

Error

A path to a file containing a go template used to render error pages during the authentication or grant flow If unspecified, the default error page is used.

IdentityProviders

Ordered list of ways for a user to identify themselves.

Login

A path to a file containing a go template used to render the login page. If unspecified, the default login page is used.

MasterCA

CA for verifying the TLS connection back to the MasterURL.

MasterPublicURL

Used for building valid client redirect URLs for external access.

MasterURL

Used for making server-to-server calls to exchange authorization codes for access tokens.

OAuthConfig

Holds the necessary configuration options for OAuth authentication.

OAuthTemplates

Allows for customization of pages like the login page.

ProviderSelection

A path to a file containing a go template used to render the provider selection page. If unspecified, the default provider selection page is used.

SessionConfig

Holds information about configuring sessions.

Templates

Allows you to customize pages like the login page.

TokenConfig

Contains options for authorization and access tokens.

7.2.11. Project Configuration

Table 7.11. Project Configuration Parameters

Parameter NameDescription

DefaultNodeSelector

Holds default project node label selector.

ProjectConfig

Holds information about project creation and defaults.

ProjectRequestMessage

The string presented to a user if they are unable to request a project via the project request API endpoint.

ProjectRequestTemplate

The template to use for creating projects in response to projectrequest. It is in the format namespace/template and it is optional. If it is not specified, a default template is used.

7.2.12. Scheduler Configuration

Table 7.12. Scheduler Configuration Parameters

Parameter NameDescription

SchedulerConfigFile

Points to a file that describes how to set up the scheduler. If empty, you get the default scheduling rules

7.2.13. Security Allocator Configuration

Table 7.13. Security Allocator Parameters

Parameter NameDescription

MCSAllocatorRange

Defines the range of MCS categories that will be assigned to namespaces. The format is <prefix>/<numberOfLabels>[,<maxCategory>]. The default is s0/2 and will allocate from c0 to c1023, which means a total of 535k labels are available (1024 choose 2 ~ 535k). If this value is changed after startup, new projects may receive labels that are already allocated to other projects. Prefix may be any valid SELinux set of terms (including user, role, and type), although leaving them as the default will allow the server to set them automatically.

SecurityAllocator

Controls the automatic allocation of UIDs and MCS labels to a project. If nil, allocation is disabled.

UIDAllocatorRange

Defines the total set of Unix user IDs (UIDs) that will be allocated to projects automatically, and the size of the block each namespace gets. For example, 1000-1999/10 will allocate ten UIDs per namespace, and will be able to allocate up to 100 blocks before running out of space. The default is to allocate from 1 billion to 2 billion in 10k blocks (which is the expected size of the ranges container images will use once user namespaces are started).

7.2.14. Service Account Configuration

Table 7.14. Service Account Configuration Parameters

Parameter NameDescription

LimitSecretReferences

Controls whether or not to allow a service account to reference any secret in a namespace without explicitly referencing them.

ManagedNames

A list of service account names that will be auto-created in every namespace. If no names are specified, the ServiceAccountsController will not be started.

MasterCA

The CA for verifying the TLS connection back to the master. The service account controller will automatically inject the contents of this file into pods so they can verify connections to the master.

PrivateKeyFile

A file containing a PEM-encoded private RSA key, used to sign service account tokens. If no private key is specified, the service account TokensController will not be started.

PublicKeyFiles

A list of files, each containing a PEM-encoded public RSA key. If any file contains a private key, the public portion of the key is used. The list of public keys is used to verify presented service account tokens. Each key is tried in order until the list is exhausted or verification succeeds. If no keys are specified, no service account authentication will be available.

ServiceAccountConfig

Holds the necessary configuration options for a service account.

7.2.15. Serving Information Configuration

Table 7.15. Serving Information Configuration Parameters

Parameter NameDescription

AllowRecursiveQueries

Allows the DNS server on the master to answer queries recursively. Note that open resolvers can be used for DNS amplification attacks and the master DNS should not be made accessible to public networks.

BindAddress

The ip:port to serve on.

BindNetwork

Controls limits and behavior for importing images.

CertFile

A file containing a PEM-encoded certificate.

CertInfo

TLS cert information for serving secure traffic.

ClientCA

The certificate bundle for all the signers that you recognize for incoming client certificates.

dnsConfig

Holds the necessary configuration options for DNS.

DNSDomain

Holds the domain suffix.

DNSIP

Holds the IP.

KeyFile

A file containing a PEM-encoded private key for the certificate specified by CertFile.

MasterClientConnectionOverrides

Provides overrides to the client connection used to connect to the master.

MaxRequestsInFlight

The number of concurrent requests allowed to the server. If zero, no limit.

NamedCertificates

A list of certificates to use to secure requests to specific host names.

RequestTimeoutSecond

The number of seconds before requests are timed out. The default is 60 minutes. If -1, there is no limit on requests.

ServingInfo

The HTTP serving information for the assets.

7.2.16. Volume Configuration

Table 7.16. Volume Configuration Parameters

Parameter NameDescription

DynamicProvisioningEnabled

A boolean to enable or disable dynamic provisioning. Default is true.

FSGroup

Can be specified to enable a quota on local storage use per unique FSGroup ID. At present this is only implemented for emptyDir volumes, and if the underlying volumeDirectory is on an XFS filesystem.

LocalQuota

Contains options for controlling local volume quota on the node.

MasterVolumeConfig

Contains options for configuring volume plug-ins in the master node.

NodeVolumeConfig

Contains options for configuring volumes on the node.

VolumeConfig

Contains options for configuring volumes on the node.

VolumeDirectory

The directory that volumes are stored under.

7.2.17. Audit Configuration

Audit provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators, or other components of the system.

Audit works at the API server level, logging all requests coming to the server. Each audit log contains two entries:

  1. The request line containing:

    1. A Unique ID allowing to match the response line (see #2)
    2. The source IP of the request
    3. The HTTP method being invoked
    4. The original user invoking the operation
    5. The impersonated user for the operation (self meaning himself)
    6. The impersonated group for the operation (lookup meaning user’s group)
    7. The namespace of the request or <none>
    8. The URI as requested
  2. The response line containing:

    1. The the unique ID from #1
    2. The response code

Example output for user admin asking for a list of pods:

AUDIT: id="5c3b8227-4af9-4322-8a71-542231c3887b" ip="127.0.0.1" method="GET" user="admin" as="<self>" asgroups="<lookup>" namespace="default" uri="/api/v1/namespaces/default/pods"
AUDIT: id="5c3b8227-4af9-4322-8a71-542231c3887b" response="200"

Table 7.17. Audit Configuration Parameters

Parameter NameDescription

Enabled

A boolean to enable or disable audit logs. Default is false.

AuditFilePath

File path where the requests should be logged to. If not set, logs are printed to master logs.

MaximumFileRetentionDays

Specifies maximum number of days to retain old audit log files based on the time stamp encoded in their filename.

MaximumRetainedFiles

Specifies the maximum number of old audit log files to retain.

MaximumFileSizeMegabytes

Specifies maximum size in megabytes of the log file before it gets rotated. Defaults to 100MB.

7.3. Node Configuration Files

The following node-config.yaml file is a sample node configuration file that was generated with the default values as of writing. You can create a new node configuration file to see the valid options for your installed version of OpenShift Container Platform.

Example 7.1. Sample Node Configuration File

allowDisabledDocker: false
apiVersion: v1
authConfig:
  authenticationCacheSize: 1000
  authenticationCacheTTL: 5m
  authorizationCacheSize: 1000
  authorizationCacheTTL: 5m
dnsDomain: cluster.local
dnsIP: 10.0.2.15 1
dockerConfig:
  execHandlerName: native
imageConfig:
  format: openshift/origin-${component}:${version}
  latest: false
iptablesSyncPeriod: 5s
kind: NodeConfig
masterKubeConfig: node.kubeconfig
networkConfig:
  mtu: 1450
  networkPluginName: ""
nodeIP: ""
nodeName: node1.example.com
podManifestConfig: 2
  path: "/path/to/pod-manifest-file" 3
  fileCheckIntervalSeconds: 30 4
proxyArguments:
  proxy-mode:
  - iptables 5
volumeConfig:
  localQuota:
   perFSGroup: null6
servingInfo:
  bindAddress: 0.0.0.0:10250
  bindNetwork: tcp4
  certFile: server.crt
  clientCA: node-client-ca.crt
  keyFile: server.key
  namedCertificates: null
volumeDirectory: /root/openshift.local.volumes
1
Configures an IP address to be prepended to a pod’s /etc/resolv.conf by adding the address here.
2
Allows pods to be placed directly on certain set of nodes, or on all nodes without going through the scheduler. You can then use pods to perform the same administrative tasks and support the same services on each node.
3
Specifies the path for the pod manifest file or directory. If it is a directory, then it is expected to contain one or more manifest files. This is used by the Kubelet to create pods on the node.
4
This is the interval (in seconds) for checking the manifest file for new data. The interval must be a positive value.
5
6
Preliminary support for local emptyDir volume quotas, set this value to a resource quantity representing the desired quota per FSGroup, per node. (i.e. 1Gi, 512Mi, etc) Currently requires that the volumeDirectory be on an XFS filesystem mounted with the 'gquota' option, and the matching security context contraint’s fsGroup type set to 'MustRunAs'.

7.3.1. Pod and Node Configuration

Table 7.18. Pod and Node Configuration Parameters

Parameter NameDescription

NodeConfig

The fully specified configuration starting an OpenShift Container Platform node.

NodeIP

Node may have multiple IPs, so this specifies the IP to use for pod traffic routing. If not specified, network parse/lookup on the nodeName is performed and the first non-loopback address is used.

NodeName

The value used to identify this particular node in the cluster. If possible, this should be your fully qualified hostname. If you are describing a set of static nodes to the master, this value must match one of the values in the list.

PodEvictionTimeout

Controls grace period for deleting pods on failed nodes. It takes valid time duration string. If empty, you get the default pod eviction timeout.

ProxyClientInfo

Specifies the client cert/key to use when proxying to pods.

7.3.2. Docker Configuration

Table 7.19. Docker Configuration Parameters

Parameter NameDescription

AllowDisabledDocker

If true, the kubelet will ignore errors from Docker. This means that a node can start on a machine that does not have docker started.

DockerConfig

Holds Docker related configuration options

ExecHandlerName

The handler to use for executing commands in Docker containers.

7.3.3. Parallel Image Pulls with Docker 1.9+

If you are using Docker 1.9+, you may want to consider enabling parallel image pulling, as the default is to pull images one at a time.

Note

There is a potential issue with data corruption prior to Docker 1.9. However, starting with 1.9, the corruption issue is resolved and it is safe to switch to parallel pulls.

kubeletArguments:
  serialize-image-pulls:
  - "false" 1
1
Change to true to disable parallel pulls. (This is the default config)

7.4. Passwords and Other Sensitive Data

For some authentication configurations, an LDAP bindPassword or OAuth clientSecret value is required. Instead of specifying these values directly in the master configuration file, these values may be provided as environment variables, external files, or in encrypted files.

Environment Variable Example

  ...
  bindPassword:
    env: BIND_PASSWORD_ENV_VAR_NAME

External File Example

  ...
  bindPassword:
    file: bindPassword.txt

Encrypted External File Example

  ...
  bindPassword:
    file: bindPassword.encrypted
    keyFile: bindPassword.key

To create the encrypted file and key file for the above example:

$ oadm ca encrypt --genkey=bindPassword.key --out=bindPassword.encrypted
> Data to encrypt: B1ndPass0rd!
Warning

Encrypted data is only as secure as the decrypting key. Care should be taken to limit filesystem permissions and access to the key file.

7.5. Creating New Configuration Files

When defining an OpenShift Container Platform configuration from scratch, start by creating new configuration files.

For master host configuration files, use the openshift start command with the --write-config option to write the configuration files. For node hosts, use the oadm create-node-config command to write the configuration files.

The following commands write the relevant launch configuration file(s), certificate files, and any other necessary files to the specified --write-config or --node-dir directory.

To create configuration files for an all-in-one server (a master and a node on the same host) in the specified directory:

$ openshift start --write-config=/openshift.local.config

To create a master configuration file and other required files in the specified directory:

$ openshift start master --write-config=/openshift.local.config/master

To create a node configuration file and other related files in the specified directory:

$ oadm create-node-config \
    --node-dir=/openshift.local.config/node-<node_hostname> \
    --node=<node_hostname> \
    --hostnames=<node_hostname>,<ip_address> \
    --certificate-authority="/path/to/ca.crt" \
    --signer-cert="/path/to/ca.crt" \
    --signer-key="/path/to/ca.key"
    --signer-serial="/path/to/ca.serial.txt"
    --node-client-certificate-authority="/path/to/ca.crt"

When creating node configuration files, the --hostnames option accepts a comma-delimited list of every host name or IP address you want server certificates to be valid for.

7.6. Launching Servers Using Configuration Files

Once you have modified the master and/or node configuration files to your specifications, you can use them when launching servers by specifying them as an argument. Keep in mind that if you specify a configuration file, none of the other command line options you pass are respected.

To launch an all-in-one server using a master configuration and a node configuration file:

$ openshift start --master-config=/openshift.local.config/master/master-config.yaml --node-config=/openshift.local.config/node-<node_hostname>/node-config.yaml

To launch a master server using a master configuration file:

$ openshift start master --config=/openshift.local.config/master/master-config.yaml

To launch a node server using a node configuration file:

$ openshift start node --config=/openshift.local.config/node-<node_hostname>/node-config.yaml

Chapter 8. Adding Hosts to an Existing Cluster

8.1. Overview

Depending on how your OpenShift Container Platform cluster was installed, you can add new hosts (either nodes or masters) to your installation by using the install tool for quick installations, or by using the scaleup.yml playbook for advanced installations.

8.2. Adding Hosts Using the Quick Installer Tool

If you used the quick install tool to install your OpenShift Container Platform cluster, you can use the quick install tool to add a new node host to your existing cluster.

Note

Currently, you can not use the quick installer tool to add new master hosts. You must use the advanced installation method to do so.

If you used the installer in either interactive or unattended mode, you can re-run the installation as long as you have an installation configuration file at ~/.config/openshift/installer.cfg.yml (or specify a different location with the -c option).

Important

The recommended maximum number of nodes is 300.

To add nodes to your installation:

  1. Ensure you have the latest installer and playbooks by updating the atomic-openshift-utils package:

    # yum update atomic-openshift-utils
  2. Run the installer with the scaleup subcommand in interactive or unattended mode:

    # atomic-openshift-installer [-u] [-c </path/to/file>] scaleup
  3. The installer detects your current environment and allows you to add additional nodes:

    *** Installation Summary ***
    
    Hosts:
    - 100.100.1.1
      - OpenShift master
      - OpenShift node
      - Etcd (Embedded)
      - Storage
    
    Total OpenShift masters: 1
    Total OpenShift nodes: 1
    
    
    ---
    
    We have detected this previously installed OpenShift environment.
    
    This tool will guide you through the process of adding additional
    nodes to your cluster.
    
    Are you ready to continue? [y/N]:

    Choose (y) and follow the on-screen instructions to complete your desired task.

8.3. Adding Hosts Using the Advanced Install

If you installed using the advanced install, you can add new hosts to your cluster by running the scaleup.yml playbook. This playbook queries the master, generates and distributes new certificates for the new hosts, then runs the configuration playbooks on the new hosts only. Before running the scaleup.yml playbook, complete all prerequisite host preparation steps.

This process is similar to re-running the installer in the quick installation method to add nodes, however you have more configuration options available when using the advanced method and when running the playbooks directly.

You must have an existing inventory file (for example, /etc/ansible/hosts) that is representative of your current cluster configuration in order to run the scaleup.yml playbook. If you previously used the atomic-openshift-installer command to run your installation, you can check ~/.config/openshift/hosts (previously located at ~/.config/openshift/.ansible/hosts) for the last inventory file that the installer generated, and use or modify that as needed as your inventory file. You must then specify the file location with -i when calling ansible-playbook later.

Important

The recommended maximum number of nodes is 300.

To add a host to an existing cluster:

  1. Ensure you have the latest playbooks by updating the atomic-openshift-utils package:

    # yum update atomic-openshift-utils
  2. Edit your /etc/ansible/hosts file and add new_<host_type> to the [OSEv3:children] section:

    For example, to add a new node host, add new_nodes:

    [OSEv3:children]
    masters
    nodes
    new_nodes

    To add new master hosts, add new_masters.

  3. Create a [new_<host_type>] section much like an existing section, specifying host information for any new hosts you want to add. For example, when adding a new node:

    [nodes]
    master[1:3].example.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
    node1.example.com openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
    node2.example.com openshift_node_labels="{'region': 'primary', 'zone': 'west'}"
    
    [new_nodes]
    node3.example.com openshift_node_labels="{'region': 'primary', 'zone': 'west'}"

    See Configuring Host Variables for more options.

    When adding new masters, hosts added to the [new_masters] section must also be added to the [new_nodes] section with the openshift_schedulable=false variable. This ensures the new master host is part of the OpenShift SDN and that pods are not scheduled for placement on them. For example:

    [masters]
    master[1:2].example.com
    
    [new_masters]
    master3.example.com
    
    [nodes]
    node[1:3].example.com openshift_node_labels="{'region': 'infra'}"
    master[1:2].example.com openshift_schedulable=false
    
    [new_nodes]
    master3.example.com openshift_schedulable=false
  4. Run the scaleup.yml playbook. If your inventory file is located somewhere other than the default of /etc/ansible/hosts, specify the location with the -i option.

    For additional nodes:

    # ansible-playbook [-i /path/to/file] \
        /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-node/scaleup.yml

    For additional masters:

    # ansible-playbook [-i /path/to/file] \
        /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-master/scaleup.yml
  5. After the playbook completes successfully, verify the installation.
  6. Finally, move any hosts you had defined in the [new_<host_type>] section into their appropriate section (but leave the [new_<host_type>] section definition itself in place) so that subsequent runs using this inventory file are aware of the nodes but do not handle them as new nodes. For example, when adding new nodes:

    [nodes]
    master[1:3].example.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
    node1.example.com openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
    node2.example.com openshift_node_labels="{'region': 'primary', 'zone': 'west'}"
    node3.example.com openshift_node_labels="{'region': 'primary', 'zone': 'west'}"
    
    [new_nodes]

Chapter 9. Loading the Default Image Streams and Templates

9.1. Overview

Your OpenShift Container Platform installation includes useful sets of Red Hat-provided image streams and templates to make it easy for developers to create new applications. By default, the quick and advanced installation methods automatically create these sets in the openshift project, which is a default global project to which all users have view access.

9.2. Offerings by Subscription Type

Depending on the active subscriptions on your Red Hat account, the following sets of image streams and templates are provided and supported by Red Hat. Contact your Red Hat sales representative for further subscription details.

9.2.1. OpenShift Container Platform Subscription

The core set of image streams and templates are provided and supported with an active OpenShift Container Platform subscription. This includes the following technologies:

TypeTechnology

Languages & Frameworks

Databases

Middleware Services

Other Services

9.2.2. xPaaS Middleware Add-on Subscriptions

Support for xPaaS middleware images are provided by xPaaS Middleware add-on subscriptions, which are separate subscriptions for each xPaaS product. If the relevant subscription is active on your account, image streams and templates are provided and supported for the following technologies:

9.3. Before You Begin

Before you consider performing the tasks in this topic, confirm if these image streams and templates are already registered in your OpenShift Container Platform cluster by doing one of the following:

  • Log into the web console and click Add to Project.
  • List them for the openshift project using the CLI:

    $ oc get is -n openshift
    $ oc get templates -n openshift

If the default image streams and templates are ever removed or changed, you can follow this topic to create the default objects yourself. Otherwise, the following instructions are not necessary.

9.4. Prerequisites

Before you can create the default image streams and templates:

  • The integrated Docker registry service must be deployed in your OpenShift Container Platform installation.
  • You must be able to run the oc create command with cluster-admin privileges, because they operate on the default openshiftproject.
  • You must have installed the atomic-openshift-utils RPM package. See Software Prerequisites for instructions.
  • Define shell variables for the directories containing image streams and templates. This significantly shortens the commands in the following sections. To do this:

    $ IMAGESTREAMDIR="/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.1/image-streams"; \
        XPAASSTREAMDIR="/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.1/xpaas-streams"; \
        XPAASTEMPLATES="/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.1/xpaas-templates"; \
        DBTEMPLATES="/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.1/db-templates"; \
        QSTEMPLATES="/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.1/quickstart-templates"

9.5. Creating Image Streams for OpenShift Container Platform Images

If your node hosts are subscribed using Red Hat Subscription Manager and you want to use the core set of image streams that used Red Hat Enterprise Linux (RHEL) 7 based images:

$ oc create -f $IMAGESTREAMDIR/image-streams-rhel7.json -n openshift

Alternatively, to create the core set of image streams that use the CentOS 7 based images:

$ oc create -f $IMAGESTREAMDIR/image-streams-centos7.json -n openshift

Creating both the CentOS and RHEL sets of image streams is not possible, because they use the same names. To have both sets of image streams available to users, either create one set in a different project, or edit one of the files and modify the image stream names to make them unique.

9.6. Creating Image Streams for xPaaS Middleware Images

The xPaaS Middleware image streams provide images for JBoss EAP, JBoss JWS, JBoss A-MQ, JBoss Fuse Integration Services, Decision Server, and JBoss Data Grid. They can be used to build applications for those platforms using the provided templates.

To create the xPaaS Middleware set of image streams:

$ oc create -f $XPAASSTREAMDIR/jboss-image-streams.json -n openshift
Note

Access to the images referenced by these image streams requires the relevant xPaaS Middleware subscriptions.

9.7. Creating Database Service Templates

The database service templates make it easy to run a database image which can be utilized by other components. For each database (MongoDB, MySQL, and PostgreSQL), two templates are defined.

One template uses ephemeral storage in the container which means data stored will be lost if the container is restarted, for example if the pod moves. This template should be used for demonstration purposes only.

The other template defines a persistent volume for storage, however it requires your OpenShift Container Platform installation to have persistent volumes configured.

To create the core set of database templates:

$ oc create -f $DBTEMPLATES -n openshift

After creating the templates, users are able to easily instantiate the various templates, giving them quick access to a database deployment.

9.8. Creating Instant App and Quickstart Templates

The Instant App and Quickstart templates define a full set of objects for a running application. These include:

Some of the templates also define a database deployment and service so the application can perform database operations.

Note

The templates which define a database use ephemeral storage for the database content. These templates should be used for demonstration purposes only as all database data will be lost if the database pod restarts for any reason.

Using these templates, users are able to easily instantiate full applications using the various language images provided with OpenShift Container Platform. They can also customize the template parameters during instantiation so that it builds source from their own repository rather than the sample repository, so this provides a simple starting point for building new applications.

To create the core Instant App and Quickstart templates:

$ oc create -f $QSTEMPLATES -n openshift

There is also a set of templates for creating applications using various xPaaS Middleware products (JBoss EAP, JBoss JWS, JBoss A-MQ, JBoss Fuse Integration Services, Decision Server, and JBoss Data Grid), which can be registered by running:

$ oc create -f $XPAASTEMPLATES -n openshift
Note

The xPaaS Middleware templates require the xPaaS Middleware image streams, which in turn require the relevant xPaaS Middleware subscriptions.

Note

The templates which define a database use ephemeral storage for the database content. These templates should be used for demonstration purposes only as all database data will be lost if the database pod restarts for any reason.

9.9. What’s Next?

With these artifacts created, developers can now log into the web console and follow the flow for creating from a template. Any of the database or application templates can be selected to create a running database service or application in the current project. Note that some of the application templates define their own database services as well.

The example applications are all built out of GitHub repositories which are referenced in the templates by default, as seen in the SOURCE_REPOSITORY_URL parameter value. Those repositories can be forked, and the fork can be provided as the SOURCE_REPOSITORY_URL parameter value when creating from the templates. This allows developers to experiment with creating their own applications.

You can direct your developers to the Using the Instant App and Quickstart Templates section in the Developer Guide for these instructions.

Chapter 10. Configuring Custom Certificates

10.1. Overview

Administrators can configure custom serving certificates for the public host names of the OpenShift Container Platform API and web console. This can be done during an advanced installation or configured after installation.

10.2. Configuring Custom Certificates with Ansible

During advanced installations, custom certificates can be configured using the openshift_master_named_certificates and openshift_master_overwrite_named_certificates parameters, which are configurable in the inventory file. More details are available about configuring custom certificates with Ansible.

Example 10.1. Example Custom Certificate Configuration with Ansible

# Configure custom named certificates
# NOTE: openshift_master_named_certificates is cached on masters and is an
# additive fact, meaning that each run with a different set of certificates
# will add the newly provided certificates to the cached set of certificates.
#
# An optional CA may be specified for each named certificate. CAs will
# be added to the OpenShift CA bundle which allows for the named
# certificate to be served for internal cluster communication.
#
# If you would like openshift_master_named_certificates to be overwritten with
# the provided value, specify openshift_master_overwrite_named_certificates.
openshift_master_overwrite_named_certificates=true
#
# Provide local certificate paths which will be deployed to masters
openshift_master_named_certificates=[{"certfile": "/path/on/host/to/custom1.crt", "keyfile": "/path/on/host/to/custom1.key", "cafile": "/path/on/host/to/custom-ca1.crt"}]
#
# Detected names may be overridden by specifying the "names" key
#openshift_master_named_certificates=[{"certfile": "/path/on/host/to/custom1.crt", "keyfile": "/path/on/host/to/custom1.key", "names": ["public-master-host.com"], "cafile": "/path/on/host/to/custom-ca1.crt"}]

10.3. Configuring Custom Certificates

In the master configuration file you can list the namedCertificates section in the assetConfig.servingInfo section so the custom certificate serves up for the web console, and in the servingInfo section so the custom certificate serves up for the CLI and other API calls. Multiple certificates can be configured this way and each certificate may be associated with multiple host names or wildcards.

A default certificate must be configured in the servingInfo.certFile and servingInfo.keyFile configuration sections in addition to namedCertificates.

Note

The namedCertificates section should only be configured for the host name associated with the masterPublicURL, assetConfig.publicURL, and oauthConfig.assetPublicURL settings. Using a custom serving certificate for the host name associated with the masterURL will result in TLS errors as infrastructure components will attempt to contact the master API using the internal masterURL host.

Example 10.2. Custom Certificates Configuration

servingInfo:
  ...
  namedCertificates:
  - certFile: custom.crt
    keyFile: custom.key
    names:
    - "customhost.com"
    - "api.customhost.com"
    - "console.customhost.com"
  - certFile: wildcard.crt
    keyFile: wildcard.key
    names:
    - "*.wildcardhost.com"
  ...

Relative paths are resolved relative to the master configuration file. Restart the server to pick up the configuration changes.

Chapter 11. Redeploying Certificates

11.1. Overview

This topic reviews how to check cluster certificate expiration dates, back up the certificates, and redeploy them using Ansible playbooks. This method fixes common certificate errors. Possible use cases include:

  • The installer detected the wrong host names and the issue was identified too late.
  • The certificates are expired and you need to update them.
  • You have a new CA and would like to create certificates using it instead

11.2. Checking Certificate Expirations

You can use the installer to warn you about any certificates expiring within a configurable window of days and notify you about any certificates that have already expired. Certificate expiry playbooks use the Ansible role openshift_certificate_expiry.

Certificates examined by the role include:

  • Master and node service certificates
  • Router and registry service certificates from etcd secrets
  • Master, node, router, registry, and kubeconfig files for cluster-admin users
  • etcd certificates (including embedded)

11.2.1. Role Variables

The openshift_certificate_expiry role uses the following variables:

Table 11.1. Core Variables

Variable NameDefault ValueDescription

openshift_certificate_expiry_config_base

/etc/origin

Base OpenShift Container Platform configuration directory.

openshift_certificate_expiry_warning_days

30

Flag certificates that will expire in this many days from now.

openshift_certificate_expiry_show_all

no

Include healthy (non-expired and non-warning) certificates in results.

Table 11.2. Optional Variables

Variable NameDefault ValueDescription

openshift_certificate_expiry_generate_html_report

no

Generate an HTML report of the expiry check results.

openshift_certificate_expiry_html_report_path

/tmp/cert-expiry-report.html

The full path for saving the HTML report.

openshift_certificate_expiry_save_json_results

no

Save expiry check results as a JSON file.

openshift_certificate_expiry_json_results_path

/tmp/cert-expiry-report.json

The full path for saving the JSON report.

11.2.2. Running Certificate Expiration Playbooks

The OpenShift Container Platform installer provides a set of example certificate expiration playbooks, using different sets of configuration for the openshift_certificate_expiry role.

Just like the redeploying certificates playbook, these playbooks must be used with an inventory that is representative of the cluster. For best results, run ansible-playbook with the -v option.

Using the easy-mode.yaml example playbook, you can try the role out before tweaking it to your specifications as needed. This playbook:

  • Produces JSON and stylized HTML reports in /tmp/.
  • Sets the warning window very large, so you will almost always get results back.
  • Includes all certificates (healthy or not) in the results.

easy-mode.yaml Playbook

- name: Check cert expirys
  hosts: nodes:masters:etcd
  become: yes
  gather_facts: no
  vars:
    openshift_certificate_expiry_warning_days: 1500
    openshift_certificate_expiry_save_json_results: yes
    openshift_certificate_expiry_generate_html_report: yes
    openshift_certificate_expiry_show_all: yes
  roles:
    - role: openshift_certificate_expiry

To run the easy-mode.yaml playbook:

$ ansible-playbook -v -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/certificate_expiry/easy-mode.yaml
Other Example Playbooks

The other example playbooks are also available to run directly out of the /usr/share/ansible/openshift-ansible/playbooks/certificate_expiry/ directory.

Table 11.3. Other Example Playbooks

File NameUsage

default.yaml

Produces the default behavior of the openshift_certificate_expiry role.

html_and_json_default_paths.yaml

Generates HTML and JSON artifacts in their default paths.

longer_warning_period.yaml

Changes the expiration warning window to 1500 days.

longer-warning-period-json-results.yaml

Changes the expiration warning window to 1500 days and saves the results as a JSON file.

To run any of these example playbooks:

$ ansible-playbook -v -i <inventory_file> \
    /usr/share/ansible/openshift-ansible/playbooks/certificate_expiry/<playbook>

11.2.3. Output Formats

As noted above, there are two ways to format your check report. In JSON format for machine parsing, or as a stylized HTML page for easy skimming.

HTML Report

An example of an HTML report is provided with the installer. You can open the following file in your browser to view it:

/usr/share/ansible/openshift-ansible/roles/openshift_certificate_expiry/examples/cert-expiry-report.html

JSON Report

There are two top-level keys in the saved JSON results: data and summary.

The data key is a hash where the keys are the names of each host examined and the values are the check results for the certificates identified on each respective host.

The summary key is a hash that summarizes the total number of certificates:

  • examined on the entire cluster
  • that are OK
  • expiring within the configured warning window
  • already expired

For an example of the full JSON report, see /usr/share/ansible/openshift-ansible/roles/openshift_certificate_expiry/examples/cert-expiry-report.json.

The summary from the JSON data can be easily checked for warnings or expirations using a variety of command-line tools. For example, using grep you can look for the word summary and print out the two lines after the match (-A2):

$ grep -A2 summary /tmp/cert-expiry-report.json
    "summary": {
        "warning": 16,
        "expired": 0

If available, the jq tool can also be used to pick out specific values. The first two examples below show how to select just one value, either warning or expired. The third example shows how to select both values at once:

$ jq '.summary.warning' /tmp/cert-expiry-report.json
16

$ jq '.summary.expired' /tmp/cert-expiry-report.json
0

$ jq '.summary.warning,.summary.expired' /tmp/cert-expiry-report.json
16
0

11.3. Running the Certificate Redeploy Playbook

Running the certificate redeploy playbook will redeploy OpenShift Container Platform certificates that exist on systems (master, node, etcd):

$ ansible-playbook -i <inventory> playbooks/byo/openshift-cluster/redeploy-certificates.yml
Warning

This playbook must be run with an inventory that is representative of the cluster. The inventory must specify or override all host names and IP addresses set via openshift_hostname, openshift_public_hostname, openshift_ip, openshift_public_ip, openshift_master_cluster_hostname, or openshift_master_cluster_public_hostname such that they match the current cluster configuration.

By default, the redeploy playbook does not redeploy the OpenShift Container Platform CA. New certificates are created using the original OpenShift Container Platform CA.

To redeploy all certificates including the OpenShift Container Platform CA, specify openshift_certificates_redeploy_ca=true.

For example:

$ ansible-playbook -i <inventory> playbooks/byo/openshift-cluster/redeploy-certificates.yml \
--extra-vars "openshift_certificates_redeploy_ca=true"

This also adds a custom CA certificate:

# Configure custom ca certificate
# NOTE: CA certificate will not be replaced with existing clusters.
# This option may only be specified when creating a new cluster or
# when redeploying cluster certificates with the redeploy-certificates
# playbook.
#openshift_master_ca_certificate={'certfile': '/path/to/ca.crt', 'keyfile': '/path/to/ca.key'}

All pods using service accounts to communicate with the OpenShift Container Platform API must be redeployed when the OpenShift Container Platform CA is replaced so the certificate redeploy playbook will serially evacuate all nodes in the cluster when this variable is set.

Chapter 12. Configuring Authentication and User Agent

12.1. Overview

The OpenShift Container Platform master includes a built-in OAuth server. Developers and administrators obtain OAuth access tokens to authenticate themselves to the API.

As an administrator, you can configure OAuth using the master configuration file to specify an identity provider. This can be done during an advanced installation or configured after installation.

If you installed OpenShift Container Platform using the Quick Installation or Advanced Installation method, the Deny All identity provider is used by default, which denies access for all user names and passwords. To allow access, you must choose a different identity provider and configure the master configuration file appropriately (located at /etc/origin/master/master-config.yaml by default).

When running a master without a configuration file, the Allow All identity provider is used by default, which allows any non-empty user name and password to log in. This is useful for testing purposes. To use other identity providers, or to modify any token, grant, or session options, you must run the master from a configuration file.

Note

Roles need to be assigned to administer the setup with an external user.

12.2. Configuring Identity Providers with Ansible

For initial advanced installations, the Deny All identity provider is configured by default, though it can be overridden during installation using the openshift_master_identity_providers parameter, which is configurable in the inventory file. Session options in the OAuth configuration are also configurable in the inventory file.

Example 12.1. Example Identity Provider Configuration with Ansible

# htpasswd auth
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
# Defining htpasswd users
#openshift_master_htpasswd_users={'user1': '<pre-hashed password>', 'user2': '<pre-hashed password>'
# or
#openshift_master_htpasswd_file=<path to local pre-generated htpasswd file>

# Allow all auth
#openshift_master_identity_providers=[{'name': 'allow_all', 'login': 'true', 'challenge': 'true', 'kind': 'AllowAllPasswordIdentityProvider'}]

# LDAP auth
#openshift_master_identity_providers=[{'name': 'my_ldap_provider', 'challenge': 'true', 'login': 'true', 'kind': 'LDAPPasswordIdentityProvider', 'attributes': {'id': ['dn'], 'email': ['mail'], 'name': ['cn'], 'preferredUsername': ['uid']}, 'bindDN': '', 'bindPassword': '', 'ca': '', 'insecure': 'false', 'url': 'ldap://ldap.example.com:389/ou=users,dc=example,dc=com?uid'}]
# Configuring the ldap ca certificate
#openshift_master_ldap_ca=<ca text>
# or
#openshift_master_ldap_ca_file=<path to local ca file to use>

# Available variables for configuring certificates for other identity providers:
#openshift_master_openid_ca
#openshift_master_openid_ca_file
#openshift_master_request_header_ca
#openshift_master_request_header_ca_file

12.3. Identity Providers

You can configure the master host for authentication using your desired identity provider by modifying the master configuration file. The following sections detail the identity providers supported by OpenShift Container Platform.

There are four parameters common to all identity providers:

ParameterDescription

name

The provider name is prefixed to provider user names to form an identity name.

challenge

When true, unauthenticated token requests from non-web clients (like the CLI) are sent a WWW-Authenticate challenge header. Not supported by all identity providers.

To prevent cross-site request forgery (CSRF) attacks against browser clients Basic authentication challenges are only sent if a X-CSRF-Token header is present on the request. Clients that expect to receive Basic WWW-Authenticate challenges should set this header to a non-empty value.

login

When true, unauthenticated token requests from web clients (like the web console) are redirected to a login page backed by this provider. Not supported by all identity providers.

If you want users to be sent to a branded page before being redirected to the identity provider’s login, then set oauthConfig → alwaysShowProviderSelection: true in the master configuration file. This provider selection page can be customized.

mappingMethod

Defines how new identities are mapped to users when they login. See Mapping Identities to Users for more information.

Note

When adding or changing identity providers, you can map identities from the new provider to existing users by setting the mappingMethod parameter to add.

12.3.1. Mapping Identities to Users

Setting the mappingMethod parameter in a master configuration file determines how identities are mapped to users:

...
oauthConfig:
  identityProviders:
  - name: htpasswd_auth
    challenge: true
    login: false
    mappingMethod: "claim"
...

When set to the default claim value, OAuth will fail if the identity is mapped to a previously-existing user name. The following table outlines the use cases for the available mappingMethod parameter values:

ParameterDescription

claim

The default value. Provisions a user with the identity’s preferred user name. Fails if a user with that user name is already mapped to another identity.

lookup

Looks up an existing identity, user identity mapping, and user, but does not automatically provision users or identities. This allows cluster administrators to set up identities and users manually, or using an external process.

generate

Provisions a user with the identity’s preferred user name. If a user with the preferred user name is already mapped to an existing identity, a unique user name is generated. For example, myuser2. This method should not be used in combination with external processes that require exact matches between OpenShift Container Platform user names and identity provider user names, such as LDAP group sync.

add

Provisions a user with the identity’s preferred user name. If a user with that user name already exists, the identity is mapped to the existing user, adding to any existing identity mappings for the user. Required when multiple identity providers are configured that identify the same set of users and map to the same user names.

12.3.2. Allow All

Set AllowAllPasswordIdentityProvider in the identityProviders stanza to allow any non-empty user name and password to log in. This is the default identity provider when running OpenShift Container Platform without a master configuration file.

Example 12.2. Master Configuration Using AllowAllPasswordIdentityProvider

oauthConfig:
  ...
  identityProviders:
  - name: my_allow_provider 1
    challenge: true 2
    login: true 3
    mappingMethod: claim 4
    provider:
      apiVersion: v1
      kind: AllowAllPasswordIdentityProvider
1
This provider name is prefixed to provider user names to form an identity name.
2
When true, unauthenticated token requests from non-web clients (like the CLI) are sent a WWW-Authenticate challenge header for this provider.
3
When true, unauthenticated token requests from web clients (like the web console) are redirected to a login page backed by this provider.
4
Controls how mappings are established between this provider’s identities and user objects, as described above.

12.3.3. Deny All

Set DenyAllPasswordIdentityProvider in the identityProviders stanza to deny access for all user names and passwords.

Example 12.3. Master Configuration Using DenyAllPasswordIdentityProvider

oauthConfig:
  ...
  identityProviders:
  - name: my_deny_provider 1
    challenge: true 2
    login: true 3
    mappingMethod: claim 4
    provider:
      apiVersion: v1
      kind: DenyAllPasswordIdentityProvider
1
This provider name is prefixed to provider user names to form an identity name.
2
When true, unauthenticated token requests from non-web clients (like the CLI) are sent a WWW-Authenticate challenge header for this provider.
3
When true, unauthenticated token requests from web clients (like the web console) are redirected to a login page backed by this provider.
4
Controls how mappings are established between this provider’s identities and user objects, as described above.

12.3.4. HTPasswd

Set HTPasswdPasswordIdentityProvider in the identityProviders stanza to validate user names and passwords against a flat file generated using htpasswd.

Note

The htpasswd utility is in the httpd-tools package:

# yum install httpd-tools

OpenShift Container Platform supports the Bcrypt, SHA-1, and MD5 cryptographic hash functions, and MD5 is the default for htpasswd. Plaintext, encrypted text, and other hash functions are not currently supported.

The flat file is reread if its modification time changes, without requiring a server restart.

To use the htpasswd command:

  • To create a flat file with a user name and hashed password, run:

    $ htpasswd -c </path/to/users.htpasswd> <user_name>

    Then, enter and confirm a clear-text password for the user. The command generates a hashed version of the password.

    For example:

    htpasswd -c users.htpasswd user1
    New password:
    Re-type new password:
    Adding password for user user1
    Note

    You can include the -b option to supply the password on the command line:

    $ htpasswd -c -b <user_name> <password>

    For example:

    $ htpasswd -c -b file user1 MyPassword!
    Adding password for user user1
  • To add or update a login to the file, run:

    $ htpasswd </path/to/users.htpasswd> <user_name>
  • To remove a login from the file, run:

    $ htpasswd -D </path/to/users.htpasswd> <user_name>

Example 12.4. Master Configuration Using HTPasswdPasswordIdentityProvider

oauthConfig:
  ...
  identityProviders:
  - name: my_htpasswd_provider 1
    challenge: true 2
    login: true 3
    mappingMethod: claim 4
    provider:
      apiVersion: v1
      kind: HTPasswdPasswordIdentityProvider
      file: /path/to/users.htpasswd 5
1
This provider name is prefixed to provider user names to form an identity name.
2
When true, unauthenticated token requests from non-web clients (like the CLI) are sent a WWW-Authenticate challenge header for this provider.
3
When true, unauthenticated token requests from web clients (like the web console) are redirected to a login page backed by this provider.
4
Controls how mappings are established between this provider’s identities and user objects, as described above.
5
File generated using htpasswd.

12.3.5. Keystone

Set KeystonePasswordIdentityProvider in the identityProviders stanza to validate user names and passwords against an OpenStack Keystone v3 server. This enables shared authentication with an OpenStack server configured to store users in an internal Keystone database.

Example 12.5. Master Configuration Using KeystonePasswordIdentityProvider

oauthConfig:
  ...
  identityProviders:
  - name: my_keystone_provider 1
    challenge: true 2
    login: true 3
    mappingMethod: claim 4
    provider:
      apiVersion: v1
      kind: KeystonePasswordIdentityProvider
      domainName: default 5
      url: http://keystone.example.com:5000 6
      ca: ca.pem 7
      certFile: keystone.pem 8
      keyFile: keystonekey.pem 9
1
This provider name is prefixed to provider user names to form an identity name.
2
When true, unauthenticated token requests from non-web clients (like the CLI) are sent a WWW-Authenticate challenge header for this provider.
3
When true, unauthenticated token requests from web clients (like the web console) are redirected to a login page backed by this provider.
4
Controls how mappings are established between this provider’s identities and user objects, as described above.
5
Keystone domain name. In Keystone, usernames are domain-specific. Only a single domain is supported.
6
The URL to use to connect to the Keystone server (required).
7
Optional: Certificate bundle to use to validate server certificates for the configured URL.
8
Optional: Client certificate to present when making requests to the configured URL.
9
Key for the client certificate. Required if certFile is specified.

12.3.6. LDAP Authentication

Set LDAPPasswordIdentityProvider in the identityProviders stanza to validate user names and passwords against an LDAPv3 server, using simple bind authentication.

During authentication, the LDAP directory is searched for an entry that matches the provided user name. If a single unique match is found, a simple bind is attempted using the distinguished name (DN) of the entry plus the provided password.

These are the steps taken:

  1. Generate a search filter by combining the attribute and filter in the configured url with the user-provided user name.
  2. Search the directory using the generated filter. If the search does not return exactly one entry, deny access.
  3. Attempt to bind to the LDAP server using the DN of the entry retrieved from the search, and the user-provided password.
  4. If the bind is unsuccessful, deny access.
  5. If the bind is successful, build an identity using the configured attributes as the identity, email address, display name, and preferred user name.

The configured url is an RFC 2255 URL, which specifies the LDAP host and search parameters to use. The syntax of the URL is:

ldap://host:port/basedn?attribute?scope?filter

For the above example:

URL ComponentDescription

ldap

For regular LDAP, use the string ldap. For secure LDAP (LDAPS), use ldaps instead.

host:port

The name and port of the LDAP server. Defaults to localhost:389 for ldap and localhost:636 for LDAPS.

basedn

The DN of the branch of the directory where all searches should start from. At the very least, this must be the top of your directory tree, but it could also specify a subtree in the directory.

attribute

The attribute to search for. Although RFC 2255 allows a comma-separated list of attributes, only the first attribute will be used, no matter how many are provided. If no attributes are provided, the default is to use uid. It is recommended to choose an attribute that will be unique across all entries in the subtree you will be using.

scope

The scope of the search. Can be either either one or sub. If the scope is not provided, the default is to use a scope of sub.

filter

A valid LDAP search filter. If not provided, defaults to (objectClass=*)

When doing searches, the attribute, filter, and provided user name are combined to create a search filter that looks like:

(&(<filter>)(<attribute>=<username>))

For example, consider a URL of:

ldap://ldap.example.com/o=Acme?cn?sub?(enabled=true)

When a client attempts to connect using a user name of bob, the resulting search filter will be (&(enabled=true)(cn=bob)).

If the LDAP directory requires authentication to search, specify a bindDN and bindPassword to use to perform the entry search.

Example 12.6. Master Configuration Using LDAPPasswordIdentityProvider

oauthConfig:
  ...
  identityProviders:
  - name: "my_ldap_provider" 1
    challenge: true 2
    login: true 3
    mappingMethod: claim 4
    provider:
      apiVersion: v1
      kind: LDAPPasswordIdentityProvider
      attributes:
        id: 5
        - dn
        email: 6
        - mail
        name: 7
        - cn
        preferredUsername: 8
        - uid
      bindDN: "" 9
      bindPassword: "" 10
      ca: my-ldap-ca-bundle.crt 11
      insecure: false 12
      url: "ldap://ldap.example.com/ou=users,dc=acme,dc=com?uid" 13
1
This provider name is prefixed to the returned user ID to form an identity name.
2
When true, unauthenticated token requests from non-web clients (like the CLI) are sent a WWW-Authenticate challenge header for this provider.
3
When true, unauthenticated token requests from web clients (like the web console) are redirected to a login page backed by this provider.
4
Controls how mappings are established between this provider’s identities and user objects, as described above.
5
List of attributes to use as the identity. First non-empty attribute is used. At least one attribute is required. If none of the listed attribute have a value, authentication fails.
6
List of attributes to use as the email address. First non-empty attribute is used.
7
List of attributes to use as the display name. First non-empty attribute is used.
8
List of attributes to use as the preferred user name when provisioning a user for this identity. First non-empty attribute is used.
9
Optional DN to use to bind during the search phase.
10
Optional password to use to bind during the search phase. This value may also be provided in an environment variable, external file, or encrypted file.
11
Certificate bundle to use to validate server certificates for the configured URL. If empty, system trusted roots are used. Only applies if insecure: false.
12
When true, no TLS connection is made to the server. When false, ldaps:// URLs connect using TLS, and ldap:// URLs are upgraded to TLS.
13
An RFC 2255 URL which specifies the LDAP host and search parameters to use, as described above.

12.3.7. Basic Authentication (Remote)

Set BasicAuthPasswordIdentityProvider in the identityProviders stanza to validate user names and passwords against a remote server using a server-to-server Basic authentication request. User names and passwords are validated against a remote URL that is protected by Basic authentication and returns JSON.

A 401 response indicates failed authentication.

A non-200 status, or the presence of a non-empty "error" key, indicates an error:

{"error":"Error message"}

A 200 status with a sub (subject) key indicates success:

{"sub":"userid"} 1
1
The subject must be unique to the authenticated user and must not be able to be modified.

A successful response may optionally provide additional data, such as:

  • A display name using the name key. For example:

    {"sub":"userid", "name": "User Name", ...}
  • An email address using the email key. For example:

    {"sub":"userid", "email":"user@example.com", ...}
  • A preferred user name using the preferred_username key. This is useful when the unique, unchangeable subject is a database key or UID, and a more human-readable name exists. This is used as a hint when provisioning the OpenShift Container Platform user for the authenticated identity. For example:

    {"sub":"014fbff9a07c", "preferred_username":"bob", ...}

Example 12.7. Master Configuration Using BasicAuthPasswordIdentityProvider

oauthConfig:
  ...
  identityProviders:
  - name: my_remote_basic_auth_provider 1
    challenge: true 2
    login: true 3
    mappingMethod: claim 4
    provider:
      apiVersion: v1
      kind: BasicAuthPasswordIdentityProvider
      url: https://www.example.com/remote-idp 5
      ca: /path/to/ca.file 6
      certFile: /path/to/client.crt 7
      keyFile: /path/to/client.key 8
1
This provider name is prefixed to the returned user ID to form an identity name.
2
When true, unauthenticated token requests from non-web clients (like the CLI) are sent a WWW-Authenticate challenge header for this provider.
3
When true, unauthenticated token requests from web clients (like the web console) are redirected to a login page backed by this provider.
4
Controls how mappings are established between this provider’s identities and user objects, as described above.
5
URL accepting credentials in Basic authentication headers.
6
Optional: Certificate bundle to use to validate server certificates for the configured URL.
7
Optional: Client certificate to present when making requests to the configured URL.
8
Key for the client certificate. Required if certFile is specified.

12.3.8. Request Header

Set RequestHeaderIdentityProvider in the identityProviders stanza to identify users from request header values, such as X-Remote-User. It is typically used in combination with an authenticating proxy, which sets the request header value. This is similar to how the remote user plug-in in OpenShift Enterprise 2 allowed administrators to provide Kerberos, LDAP, and many other forms of enterprise authentication.

For users to authenticate using this identity provider, they must access https://<master>/oauth/authorize via an authenticating proxy. You can either proxy the entire master API server so that all access goes through the proxy, or you can configure the OAuth server to redirect unauthenticated requests to the proxy.

To redirect unauthenticated requests from clients expecting login flows:

  1. Set the login parameter to true.
  2. Set the provider.loginURL parameter to the proxy URL to send those clients to.

To redirect unauthenticated requests from clients expecting WWW-Authenticate challenges:

  1. Set the challenge parameter to true.
  2. Set the provider.challengeURL parameter to the proxy URL to send those clients to.

The provider.challengeURL and provider.loginURL parameters can include the following tokens in the query portion of the URL:

  • ${url} is replaced with the current URL, escaped to be safe in a query parameter.

    For example: https://www.example.com/sso-login?then=${url}

  • ${query} is replaced with the current query string, unescaped.

    For example: https://www.example.com/auth-proxy/oauth/authorize?${query}

Warning

If you expect unauthenticated requests to reach the OAuth server, a clientCA parameter should be set for this identity provider, so that incoming requests are checked for a valid client certificate before the request’s headers are checked for a user name. Otherwise, any direct request to the OAuth server can impersonate any identity from this provider, merely by setting a request header.

Example 12.8. Master Configuration Using RequestHeaderIdentityProvider

oauthConfig:
  ...
  identityProviders:
  - name: my_request_header_provider 1
    challenge: true 2
    login: true 3
    mappingMethod: claim 4
    provider:
      apiVersion: v1
      kind: RequestHeaderIdentityProvider
      challengeURL: "https://www.example.com/challenging-proxy/oauth/authorize?${query}" 5
      loginURL: "https://www.example.com/login-proxy/oauth/authorize?${query}" 6
      clientCA: /path/to/client-ca.file 7
      clientCommonNames: 8
      - my-auth-proxy
      headers: 9
      - X-Remote-User
      - SSO-User
      emailHeaders: 10
      - X-Remote-User-Email
      nameHeaders: 11
      - X-Remote-User-Display-Name
      preferredUsernameHeaders: 12
      - X-Remote-User-Login
1
This provider name is prefixed to the user name in the request header to form an identity name.
2
RequestHeaderIdentityProvider can only respond to clients that request WWW-Authenticate challenges by redirecting to a configured challengeURL. The configured URL should respond with a WWW-Authenticate challenge.
3
RequestHeaderIdentityProvider can only respond to clients requesting a login flow by redirecting to a configured loginURL. The configured URL should respond with a login flow.
4
Controls how mappings are established between this provider’s identities and user objects, as described above.
5
Optional: URL to redirect unauthenticated /oauth/authorize requests to, for clients which expect interactive logins. ${url} is replaced with the current URL, escaped to be safe in a query parameter. ${query} is replaced with the current query string.
6
Optional: URL to redirect unauthenticated /oauth/authorize requests to, for clients which expect WWW-Authenticate challenges. ${url} is replaced with the current URL, escaped to be safe in a query parameter. ${query} is replaced with the current query string.
7
Optional: PEM-encoded certificate bundle. If set, a valid client certificate must be presented and validated against the certificate authorities in the specified file before the request headers are checked for user names.
8
Optional: list of common names (cn). If set, a valid client certificate with a Common Name (cn) in the specified list must be presented before the request headers are checked for user names. If empty, any Common Name is allowed. Can only be used in combination with clientCA.
9
Header names to check, in order, for the user identity. The first header containing a value is used as the identity. Required, case-insensitive.
10
Header names to check, in order, for an email address. The first header containing a value is used as the email address. Optional, case-insensitive.
11
Header names to check, in order, for a display name. The first header containing a value is used as the display name. Optional, case-insensitive.
12
Header names to check, in order, for a preferred user name, if different than the immutable identity determined from the headers specified in headers. The first header containing a value is used as the preferred user name when provisioning. Optional, case-insensitive.

Example 12.9. Apache Authentication Using RequestHeaderIdentityProvider

This example configures an authentication proxy on the same host as the master. Having the proxy and master on the same host is merely a convenience and may not be suitable for your environment. For example, if you were already running a router on the master, port 443 would not be available.

It is also important to note that while this reference configuration uses Apache’s mod_auth_form, it is by no means required and other proxies can easily be used if the following requirements are met:

  1. Block the X-Remote-User header from client requests to prevent spoofing.
  2. Enforce client certificate authentication in the RequestHeaderIdentityProvider configuration.
  3. Require the X-Csrf-Token header be set for all authentication request using the challenge flow.
  4. Only the /oauth/authorize endpoint should be proxied, and redirects should not be rewritten to allow the backend server to send the client to the correct location.

Installing the Prerequisites

The mod_auth_form module is shipped as part of the mod_session package that is found in the Optional channel:

# yum install -y httpd mod_ssl mod_session apr-util-openssl

Generate a CA for validating requests that submit the trusted header. This CA should be used as the file name for clientCA in the master’s identity provider configuration.

# oadm ca create-signer-cert \
  --cert='/etc/origin/master/proxyca.crt' \
  --key='/etc/origin/master/proxyca.key' \
  --name='openshift-proxy-signer@1432232228' \
  --serial='/etc/origin/master/proxyca.serial.txt'

Generate a client certificate for the proxy. This can be done using any x509 certificate tooling. For convenience, the oadm CLI can be used:

# oadm create-api-client-config \
  --certificate-authority='/etc/origin/master/proxyca.crt' \
  --client-dir='/etc/origin/master/proxy' \
  --signer-cert='/etc/origin/master/proxyca.crt' \
  --signer-key='/etc/origin/master/proxyca.key' \
  --signer-serial='/etc/origin/master/proxyca.serial.txt' \
  --user='system:proxy' 1

# pushd /etc/origin/master
# cp master.server.crt /etc/pki/tls/certs/localhost.crt 2
# cp master.server.key /etc/pki/tls/private/localhost.key
# cp ca.crt /etc/pki/CA/certs/ca.crt
# cat proxy/system\:proxy.crt \
  proxy/system\:proxy.key > \
  /etc/pki/tls/certs/authproxy.pem
# popd
1
The user name can be anything, however it is useful to give it a descriptive name as it will appear in logs.
2
When running the authentication proxy on a different host name than the master, it is important to generate a certificate that matches the host name instead of using the default master certificate as shown above. The value for masterPublicURL in the /etc/origin/master/master-config.yaml file must be included in the X509v3 Subject Alternative Name in the certificate that is specified for SSLCertificateFile. If a new certificate needs to be created, the oadm ca create-server-cert command can be used.

Configuring Apache

Unlike OpenShift Enterprise 2, this proxy does not need to reside on the same host as the master. It uses a client certificate to connect to the master, which is configured to trust the X-Remote-User header.

Configure Apache per the following:

LoadModule auth_form_module modules/mod_auth_form.so
LoadModule session_module modules/mod_session.so
LoadModule request_module modules/mod_request.so

# Nothing needs to be served over HTTP.  This virtual host simply redirects to
# HTTPS.
<VirtualHost *:80>
  DocumentRoot /var/www/html
  RewriteEngine              On
  RewriteRule     ^(.*)$     https://%{HTTP_HOST}$1 [R,L]
</VirtualHost>

<VirtualHost *:443>
  # This needs to match the certificates you generated.  See the CN and X509v3
  # Subject Alternative Name in the output of:
  # openssl x509 -text -in /etc/pki/tls/certs/localhost.crt
  ServerName www.example.com

  DocumentRoot /var/www/html
  SSLEngine on
  SSLCertificateFile /etc/pki/tls/certs/localhost.crt
  SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
  SSLCACertificateFile /etc/pki/CA/certs/ca.crt

  SSLProxyEngine on
  SSLProxyCACertificateFile /etc/pki/CA/certs/ca.crt
  # It's critical to enforce client certificates on the Master.  Otherwise
  # requests could spoof the X-Remote-User header by accessing the Master's
  # /oauth/authorize endpoint directly.
  SSLProxyMachineCertificateFile /etc/pki/tls/certs/authproxy.pem

  # Send all requests to the console
  RewriteEngine              On
  RewriteRule     ^/console(.*)$     https://%{HTTP_HOST}:8443/console$1 [R,L]

  # In order to using the challenging-proxy an X-Csrf-Token must be present.
  RewriteCond %{REQUEST_URI} ^/challenging-proxy
  RewriteCond %{HTTP:X-Csrf-Token} ^$ [NC]
  RewriteRule ^.* - [F,L]

  <Location /challenging-proxy/oauth/authorize>
    # Insert your backend server name/ip here.
    ProxyPass https://[MASTER]:8443/oauth/authorize
    AuthType basic
  </Location>

  <Location /login-proxy/oauth/authorize>
    # Insert your backend server name/ip here.
    ProxyPass https://[MASTER]:8443/oauth/authorize

    # mod_auth_form providers are implemented by mod_authn_dbm, mod_authn_file,
    # mod_authn_dbd, mod_authnz_ldap and mod_authn_socache.
    AuthFormProvider file
    AuthType form
    AuthName openshift
    ErrorDocument 401 /login.html
  </Location>

  <ProxyMatch /oauth/authorize>
    AuthUserFile /etc/origin/master/htpasswd
    AuthName openshift
    Require valid-user
    RequestHeader set X-Remote-User %{REMOTE_USER}s env=REMOTE_USER

    # For ldap:
    # AuthBasicProvider ldap
    # AuthLDAPURL "ldap://ldap.example.com:389/ou=People,dc=my-domain,dc=com?uid?sub?(objectClass=*)"

    # It's possible to remove the mod_auth_form usage and replace it with
    # something like mod_auth_kerb, mod_auth_gsspai or even mod_auth_mellon.
    # The former would be able to support both the login and challenge flows
    # from the Master.  Mellon would likely only support the login flow.

    # For Kerberos
    # yum install mod_auth_gssapi
    # AuthType GSSAPI
    # GssapiCredStore keytab:/etc/httpd.keytab
  </ProxyMatch>

</VirtualHost>

RequestHeader unset X-Remote-User

Additional mod_auth_form Requirements

A sample login page is available from the openshift_extras repository. This file should be placed in the DocumentRoot location (/var/www/html by default).

Creating Users

At this point, you can create the users in the system Apache is using to store accounts information. In this example, file-backed authentication is used:

# yum -y install httpd-tools
# touch /etc/origin/master/htpasswd
# htpasswd /etc/origin/master/htpasswd <user_name>

Configuring the Master

The identityProviders stanza in the /etc/origin/master/master-config.yaml file must be updated as well:

  identityProviders:
  - name: requestheader
    challenge: true
    login: true
    provider:
      apiVersion: v1
      kind: RequestHeaderIdentityProvider
      challengeURL: "https://[MASTER]/challenging-proxy/oauth/authorize?${query}"
      loginURL: "https://[MASTER]/login-proxy/oauth/authorize?${query}"
      clientCA: /etc/origin/master/proxyca.crt
      headers:
      - X-Remote-User

Restarting Services

Finally, restart the following services:

# systemctl restart httpd
# systemctl restart atomic-openshift-master

Verifying the Configuration

  1. Test by bypassing the proxy. You should be able to request a token if you supply the correct client certificate and header:

    # curl -L -k -H "X-Remote-User: joe" \
       --cert /etc/pki/tls/certs/authproxy.pem \
       https://[MASTER]:8443/oauth/token/request
  2. If you do not supply the client certificate, the request should be denied:

    # curl -L -k -H "X-Remote-User: joe" \
       https://[MASTER]:8443/oauth/token/request
  3. This should show a redirect to the configured challengeURL (with additional query parameters):

    # curl -k -v -H 'X-Csrf-Token: 1' \
       '<masterPublicURL>/oauth/authorize?client_id=openshift-challenging-client&response_type=token'
  4. This should show a 401 response with a WWW-Authenticate basic challenge:

    #  curl -k -v -H 'X-Csrf-Token: 1' \
        '<redirected challengeURL from step 3 +query>'
  5. This should show a redirect with an access token:

    #  curl -k -v -u <your_user>:<your_password> \
        -H 'X-Csrf-Token: 1' '<redirected_challengeURL_from_step_3 +query>'

12.3.9. GitHub

Set GitHubIdentityProvider in the identityProviders stanza to use GitHub as an identity provider, using the OAuth integration.

Note

Using GitHub as an identity provider requires users to get a token using <master>/oauth/token/request to use with command-line tools.

Warning

Using GitHub as an identity provider allows any GitHub user to authenticate to your server. You can limit authentication to members of specific GitHub organizations with the organizations configuration attribute, as shown below.

Example 12.10. Master Configuration Using GitHubIdentityProvider

oauthConfig:
  ...
  identityProviders:
  - name: github 1
    challenge: false 2
    login: true 3
    mappingMethod: claim 4
    provider:
      apiVersion: v1
      kind: GitHubIdentityProvider
      clientID: ... 5
      clientSecret: ... 6
      organizations: 7
      - myorganization1
      - myorganization2
1
This provider name is prefixed to the GitHub numeric user ID to form an identity name. It is also used to build the callback URL.
2
GitHubIdentityProvider cannot be used to send WWW-Authenticate challenges.
3
When true, unauthenticated token requests from web clients (like the web console) are redirected to GitHub to log in.
4
Controls how mappings are established between this provider’s identities and user objects, as described above.
5
The client ID of a registered GitHub OAuth application. The application must be configured with a callback URL of <master>/oauth2callback/<identityProviderName>.
6
The client secret issued by GitHub. This value may also be provided in an environment variable, external file, or encrypted file.
7
Optional list of organizations. If specified, only GitHub users that are members of at least one of the listed organizations will be allowed to log in. If the GitHub OAuth application configured in clientID is not owned by the organization, an organization owner must grant third-party access in order to use this option. This can be done during the first GitHub login by the organization’s administrator, or from the GitHub organization settings.

12.3.10. GitLab

Set GitLabIdentityProvider in the identityProviders stanza to use GitLab.com or any other GitLab instance as an identity provider, using the OAuth integration. The OAuth provider feature requires GitLab version 7.7.0 or higher.

Note

Using GitLab as an identity provider requires users to get a token using <master>/oauth/token/request to use with command-line tools.

Example 12.11. Master Configuration Using GitLabIdentityProvider

oauthConfig:
  ...
  identityProviders:
  - name: gitlab 1
    challenge: true 2
    login: true 3
    mappingMethod: claim 4
    provider:
      apiVersion: v1
      kind: GitLabIdentityProvider
      url: ... 5
      clientID: ... 6
      clientSecret: ... 7
      ca: ... 8
1
This provider name is prefixed to the GitLab numeric user ID to form an identity name. It is also used to build the callback URL.
2
When true, unauthenticated token requests from non-web clients (like the CLI) are sent a WWW-Authenticate challenge header for this provider. This uses the Resource Owner Password Credentials grant flow to obtain an access token from GitLab.
3
When true, unauthenticated token requests from web clients (like the web console) are redirected to GitLab to log in.
4
Controls how mappings are established between this provider’s identities and user objects, as described above.
5
The host URL of a GitLab OAuth provider. This could either be https://gitlab.com/ or any other self hosted instance of GitLab.
6
The client ID of a registered GitLab OAuth application. The application must be configured with a callback URL of <master>/oauth2callback/<identityProviderName>.
7
The client secret issued by GitLab. This value may also be provided in an environment variable, external file, or encrypted file.
8
CA is an optional trusted certificate authority bundle to use when making requests to the GitLab instance. If empty, the default system roots are used.

12.3.11. Google

Set GoogleIdentityProvider in the identityProviders stanza to use Google as an identity provider, using Google’s OpenID Connect integration.

Note

Using Google as an identity provider requires users to get a token using <master>/oauth/token/request to use with command-line tools.

Warning

Using Google as an identity provider allows any Google user to authenticate to your server. You can limit authentication to members of a specific hosted domain with the hostedDomain configuration attribute, as shown below.

Example 12.12. Master Configuration Using GoogleIdentityProvider

oauthConfig:
  ...
  identityProviders:
  - name: google 1
    challenge: false 2
    login: true 3
    mappingMethod: claim 4
    provider:
      apiVersion: v1
      kind: GoogleIdentityProvider
      clientID: ... 5
      clientSecret: ... 6
      hostedDomain: "" 7
1
This provider name is prefixed to the Google numeric user ID to form an identity name. It is also used to build the redirect URL.
2
GoogleIdentityProvider cannot be used to send WWW-Authenticate challenges.
3
When true, unauthenticated token requests from web clients (like the web console) are redirected to Google to log in.
4
Controls how mappings are established between this provider’s identities and user objects, as described above.
5
The client ID of a registered Google project. The project must be configured with a redirect URI of <master>/oauth2callback/<identityProviderName>.
6
The client secret issued by Google. This value may also be provided in an environment variable, external file, or encrypted file.
7
Optional hosted domain to restrict sign-in accounts to. If empty, any Google account is allowed to authenticate.

12.3.12. OpenID Connect

Set OpenIDIdentityProvider in the identityProviders stanza to integrate with an OpenID Connect identity provider using an Authorization Code Flow.

Note

ID Token and UserInfo decryptions are not supported.

By default, the openid scope is requested. If required, extra scopes can be specified in the extraScopes field.

Claims are read from the JWT id_token returned from the OpenID identity provider and, if specified, from the JSON returned by the UserInfo URL.

At least one claim must be configured to use as the user’s identity. The standard identity claim is sub.

You can also indicate which claims to use as the user’s preferred user name, display name, and email address. If multiple claims are specified, the first one with a non-empty value is used. The standard claims are:

sub

The user identity.

preferred_username

The preferred user name when provisioning a user.

email

Email address.

name

Display name.

Note

Using an OpenID Connect identity provider requires users to get a token using <master>/oauth/token/request to use with command-line tools.

Example 12.13. Standard Master Configuration Using OpenIDIdentityProvider

oauthConfig:
  ...
  identityProviders:
  - name: my_openid_connect 1
    challenge: true 2
    login: true 3
    mappingMethod: claim 4
    provider:
      apiVersion: v1
      kind: OpenIDIdentityProvider
      clientID: ... 5
      clientSecret: ... 6
      claims:
        id:
        - sub 7
        preferredUsername:
        - preferred_username
        name:
        - name
        email:
        - email
      urls:
        authorize: https://myidp.example.com/oauth2/authorize 8
        token: https://myidp.example.com/oauth2/token 9
1
This provider name is prefixed to the value of the identity claim to form an identity name. It is also used to build the redirect URL.
2
When true, unauthenticated token requests from non-web clients (like the CLI) are sent a WWW-Authenticate challenge header for this provider. This requires the OpenID provider to support the Resource Owner Password Credentials grant flow.
3
When true, unauthenticated token requests from web clients (like the web console) are redirected to the authorize URL to log in.
4
Controls how mappings are established between this provider’s identities and user objects, as described above.
5
The client ID of a client registered with the OpenID provider. The client must be allowed to redirect to <master>/oauth2callback/<identityProviderName>.
6
The client secret. This value may also be provided in an environment variable, external file, or encrypted file.
7
Use the value of the sub claim in the returned id_token as the user’s identity.
8
Authorization Endpoint described in the OpenID spec. Must use https.
9
Token Endpoint described in the OpenID spec. Must use https.

A custom certificate bundle, extra scopes, extra authorization request parameters, and userInfo URL can also be specified:

Example 12.14. Full Master Configuration Using OpenIDIdentityProvider

oauthConfig:
  ...
  identityProviders:
  - name: my_openid_connect
    challenge: false
    login: true
    mappingMethod: claim
    provider:
      apiVersion: v1
      kind: OpenIDIdentityProvider
      clientID: ...
      clientSecret: ...
      ca: my-openid-ca-bundle.crt 1
      extraScopes: 2
      - email
      - profile
      extraAuthorizeParameters: 3
        include_granted_scopes: "true"
      claims:
        id: 4
        - custom_id_claim
        - sub
        preferredUsername: 5
        - preferred_username
        - email
        name: 6
        - nickname
        - given_name
        - name
        email: 7
        - custom_email_claim
        - email
      urls:
        authorize: https://myidp.example.com/oauth2/authorize
        token: https://myidp.example.com/oauth2/token
        userInfo: https://myidp.example.com/oauth2/userinfo 8
1
Certificate bundle to use to validate server certificates for the configured URLs. If empty, system trusted roots are used.
2
Optional list of scopes to request, in addition to the openid scope, during the authorization token request.
3
Optional map of extra parameters to add to the authorization token request.
4
List of claims to use as the identity. First non-empty claim is used. At least one claim is required. If none of the listed claims have a value, authentication fails.
5
List of claims to use as the preferred user name when provisioning a user for this identity. First non-empty claim is used.
6
List of claims to use as the display name. First non-empty claim is used.
7
List of claims to use as the email address. First non-empty claim is used.
8
UserInfo Endpoint described in the OpenID spec. Must use https.

12.4. Token Options

The OAuth server generates two kinds of tokens:

Access tokens

Longer-lived tokens that grant access to the API.

Authorize codes

Short-lived tokens whose only use is to be exchanged for an access token.

Use the tokenConfig stanza to set token options:

Example 12.15. Master Configuration Token Options

oauthConfig:
  ...
  tokenConfig:
    accessTokenMaxAgeSeconds: 86400 1
    authorizeTokenMaxAgeSeconds: 300 2
1
Set accessTokenMaxAgeSeconds to control the lifetime of access tokens. The default lifetime is 24 hours.
2
Set authorizeTokenMaxAgeSeconds to control the lifetime of authorize codes. The default lifetime is five minutes.

12.5. Grant Options

When the OAuth server receives token requests for a client to which the user has not previously granted permission, the action that the OAuth server takes is dependent on the OAuth client’s grant strategy.

When the OAuth client requesting token does not provide its own grant strategy, the server-wide default strategy is used. To configure the default strategy, set the method value in the grantConfig stanza. Valid values for method are:

auto

Auto-approve the grant and retry the request.

prompt

Prompt the user to approve or deny the grant.

deny

Auto-deny the grant and return a failure error to the client.

Example 12.16. Master Configuration Grant Options

oauthConfig:
  ...
  grantConfig:
    method: auto

12.6. Session Options

The OAuth server uses a signed and encrypted cookie-based session during login and redirect flows.

Use the sessionConfig stanza to set session options:

Example 12.17. Master Configuration Session Options

oauthConfig:
  ...
  sessionConfig:
    sessionMaxAgeSeconds: 300 1
    sessionName: ssn 2
    sessionSecretsFile: "..." 3
1
Controls the maximum age of a session; sessions auto-expire once a token request is complete. If auto-grant is not enabled, sessions must last as long as the user is expected to take to approve or reject a client authorization request.
2
Name of the cookie used to store the session.
3
File name containing serialized SessionSecrets object. If empty, a random signing and encryption secret is generated at each server start.

If no sessionSecretsFile is specified, a random signing and encryption secret is generated at each start of the master server. This means that any logins in progress will have their sessions invalidated if the master is restarted. It also means that if multiple masters are configured, they will not be able to decode sessions generated by one of the other masters.

To specify the signing and encryption secret to use, specify a sessionSecretsFile. This allows you separate secret values from the configuration file and keep the configuration file distributable, for example for debugging purposes.

Multiple secrets can be specified in the sessionSecretsFile to enable rotation. New sessions are signed and encrypted using the first secret in the list. Existing sessions are decrypted and authenticated by each secret until one succeeds.

Example 12.18. Session Secret Configuration:

apiVersion: v1
kind: SessionSecrets
secrets: 1
- authentication: "..." 2
  encryption: "..." 3
- authentication: "..."
  encryption: "..."
...
1
List of secrets used to authenticate and encrypt cookie sessions. At least one secret must be specified. Each secret must set an authentication and encryption secret.
2
Signing secret, used to authenticate sessions using HMAC. Recommended to use a secret with 32 or 64 bytes.
3
Encrypting secret, used to encrypt sessions. Must be 16, 24, or 32 characters long, to select AES-128, AES-192, or AES-256.

12.7. Preventing CLI Version Mismatch With User Agent

OpenShift Container Platform implements a user agent that can be used to prevent an application developer’s CLI accessing the OpenShift Container Platform API.

User agents for the OpenShift Container Platform CLI are constructed from a set of values within OpenShift Container Platform:

<command>/<version> (<platform>/<architecture>) <client>/<git_commit>

So, for example, when:

  • <command> = oc
  • <version> = The client version. For example, v3.3.0. Requests made against the Kubernetes API at /api receive the Kubernetes version, while requests made against the OpenShift Container Platform API at /oapi receive the OpenShift Container Platform version (as specified by oc version)
  • <platform> = linux
  • <architecture> = amd64
  • <client> = openshift, or kubernetes depending on if the request is made against the Kubernetes API at /api, or the OpenShift Container Platform API at /oapi
  • <git_commit> = The Git commit of the client version (for example, f034127)

the user agent will be:

oc/v3.3.0 (linux/amd64) openshift/f034127

As an OpenShift Container Platform administrator, you can prevent clients from accessing the API with the userAgentMatching configuration setting of a master configuration. So, if a client is using a particular library or binary, they will be prevented from accessing the API.

The following user agent example denies the Kubernetes 1.2 client binary, OpenShift Origin 1.1.3 binary, and the POST and PUT httpVerbs:

policyConfig:
  userAgentMatchingConfig:
    defaultRejectionMessage: "Your client is too old.  Go to https://example.org to update it."
    deniedClients:
    - regex: '\w+/v(?:(?:1\.1\.1)|(?:1\.0\.1)) \(.+/.+\) openshift/\w{7}'
    - regex: '\w+/v(?:1\.1\.3) \(.+/.+\) openshift/\w{7}'
      httpVerbs:
      - POST
      - PUT
    - regex: '\w+/v1\.2\.0 \(.+/.+\) kubernetes/\w{7}'
      httpVerbs:
      - POST
      - PUT
    requiredClients: null

Administrators can also deny clients that do not exactly match the expected clients:

policyConfig:
  userAgentMatchingConfig:
    defaultRejectionMessage: "Your client is too old.  Go to https://example.org to update it."
    deniedClients: []
    requiredClients:
    - regex: '\w+/v1\.1\.3 \(.+/.+\) openshift/\w{7}'
    - regex: '\w+/v1\.2\.0 \(.+/.+\) kubernetes/\w{7}'
      httpVerbs:
      - POST
      - PUT
Note

When the client’s user agent mismatches the configuration, errors occur. To ensure that mutating requests match, enforce a whitelist. Rules are mapped to specific verbs, so you can ban mutating requests while allowing non-mutating requests.

Chapter 13. Syncing Groups With LDAP

13.1. Overview

As an OpenShift Container Platform administrator, you can use groups to manage users, change their permissions, and enhance collaboration. Your organization may have already created user groups and stored them in an LDAP server. OpenShift Container Platform can sync those LDAP records with internal OpenShift Container Platform records, enabling you to manage your groups in one place. OpenShift Container Platform currently supports group sync with LDAP servers using three common schemas for defining group membership: RFC 2307, Active Directory, and augmented Active Directory.

Note

You must have cluster-admin privileges to sync groups.

13.2. Configuring LDAP Sync

Before you can run LDAP sync, you need a sync configuration file. This file contains LDAP client configuration details:

  • Configuration for connecting to your LDAP server.
  • Sync configuration options that are dependent on the schema used in your LDAP server.

A sync configuration file can also contain an administrator-defined list of name mappings that maps OpenShift Container Platform Group names to groups in your LDAP server.

13.2.1. LDAP Client Configuration

Example 13.1. LDAP Client Configuration

url: ldap://10.0.0.0:389 1
bindDN: cn=admin,dc=example,dc=com 2
bindPassword: password 3
insecure: true 4
ca: my-ldap-ca-bundle.crt 5
1
The connection protocol, IP address of the LDAP server hosting your database, and the port to connect to, formatted as scheme://host:port.
2
Optional distinguished name (DN) to use as the Bind DN. OpenShift Container Platform uses this if elevated privilege is required to retrieve entries for the sync operation.
3
Optional password to use to bind. OpenShift Container Platform uses this if elevated privilege is necessary to retrieve entries for the sync operation. This value may also be provided in an environment variable, external file, or encrypted file.
4
When true, no TLS connection is made to the server. When false, secure LDAP (ldaps://) URLs connect using TLS, and insecure LDAP (ldap://) URLs are upgraded to TLS.
5
The certificate bundle to use for validating server certificates for the configured URL. If empty, OpenShift Container Platform uses system-trusted roots. This only applies if insecure is set to false.

13.2.2. LDAP Query Definition

Sync configurations consist of LDAP query definitions for the entries that are required for synchronization. The specific definition of an LDAP query depends on the schema used to store membership information in the LDAP server.

Example 13.2. LDAP Query Definition

baseDN: ou=users,dc=example,dc=com 1
scope: sub 2
derefAliases: never 3
timeout: 0 4
filter: (objectClass=inetOrgPerson) 5
pageSize: 0 6
1
The distinguished name (DN) of the branch of the directory where all searches will start from. It is required that you specify the top of your directory tree, but you can also specify a subtree in the directory.
2
The scope of the search. Valid values are base, one, or sub. If this is left undefined, then a scope of sub is assumed. Descriptions of the scope options can be found in the table below.
3
The behavior of the search with respect to aliases in the LDAP tree. Valid values are never, search, base, or always. If this is left undefined, then the default is to always dereference aliases. Descriptions of the dereferencing behaviors can be found in the table below.
4
The time limit allowed for the search by the client, in seconds. A value of 0 imposes no client-side limit.
5
A valid LDAP search filter. If this is left undefined, then the default is (objectClass=*).
6
The optional maximum size of response pages from the server, measured in LDAP entries. If set to 0, no size restrictions will be made on pages of responses. Setting paging sizes is necessary when queries return more entries than the client or server allow by default.

Table 13.2. LDAP Dereferencing Behaviors

Dereferencing BehaviorDescription

never

Never dereference any aliases found in the LDAP tree.

search

Only dereference aliases found while searching.

base

Only dereference aliases while finding the base object.

always

Always dereference all aliases found in the LDAP tree.

13.2.3. User-Defined Name Mapping

A user-defined name mapping explicitly maps the names of OpenShift Container Platform Groups to unique identifiers that find groups on your LDAP server. The mapping uses normal YAML syntax. A user-defined mapping can contain an entry for every group in your LDAP server or only a subset of those groups. If there are groups on the LDAP server that do not have a user-defined name mapping, the default behavior during sync is to use the attribute specified as the Group’s name.

Example 13.3. User-Defined Name Mapping

groupUIDNameMapping:
  "cn=group1,ou=groups,dc=example,dc=com": firstgroup
  "cn=group2,ou=groups,dc=example,dc=com": secondgroup
  "cn=group3,ou=groups,dc=example,dc=com": thirdgroup

13.3. Running LDAP Sync

Once you have created a sync configuration file, then sync can begin. OpenShift Container Platform allows administrators to perform a number of different sync types with the same server.

Note

By default, all group synchronization or pruning operations are dry-run, so you must set the --confirm flag on the sync-groups command in order to make changes to OpenShift Container Platform Group records.

To sync all groups from the LDAP server with OpenShift Container Platform:

$ oadm groups sync --sync-config=config.yaml --confirm

To sync all Groups already in OpenShift Container Platform that correspond to groups in the LDAP server specified in the configuration file:

$ oadm groups sync --type=openshift --sync-config=config.yaml --confirm

To sync a subset of LDAP groups with OpenShift Container Platform, you can use whitelist files, blacklist files, or both:

Note

Any combination of blacklist files, whitelist files, or whitelist literals will work; whitelist literals can be included directly in the command itself. This applies to groups found on LDAP servers, as well as Groups already present in OpenShift Container Platform. Your files must contain one unique group identifier per line.

$ oadm groups sync --whitelist=<whitelist_file> \
                   --sync-config=config.yaml    \
                   --confirm
$ oadm groups sync --blacklist=<blacklist_file> \
                   --sync-config=config.yaml    \
                   --confirm
$ oadm groups sync <group_unique_identifier>    \
                   --sync-config=config.yaml    \
                   --confirm
$ oadm groups sync <group_unique_identifier>    \
                   --whitelist=<whitelist_file> \
                   --blacklist=<blacklist_file> \
                   --sync-config=config.yaml    \
                   --confirm
$ oadm groups sync --type=openshift             \
                   --whitelist=<whitelist_file> \
                   --sync-config=config.yaml    \
                   --confirm

13.4. Running a Group Pruning Job

An administrator can also choose to remove groups from OpenShift Container Platform records if the records on the LDAP server that created them are no longer present. The prune job will accept the same sync configuration file and white- or black-lists as used for the sync job.

For example, if groups had previously been synchronized from LDAP using some config.yaml file, and some of those groups no longer existed on the LDAP server, the following command would determine which Groups in OpenShift Container Platform corresponded to the deleted groups in LDAP and then remove them from OpenShift Container Platform:

$ oadm groups prune --sync-config=config.yaml --confirm

13.5. Sync Examples

This section contains examples for the RFC 2307, Active Directory, and augmented Active Directory schemas. All of the following examples synchronize a group named admins that has two members: Jane and Jim. Each example explains:

  • How the group and users are added to the LDAP server.
  • What the LDAP sync configuration file looks like.
  • What the resulting Group record in OpenShift Container Platform will be after synchronization.

13.5.1. RFC 2307

In the RFC 2307 schema, both users (Jane and Jim) and groups exist on the LDAP server as first-class entries, and group membership is stored in attributes on the group. The following snippet of ldif defines the users and group for this schema:

Example 13.4. LDAP Entries Using RFC 2307 Schema: rfc2307.ldif

  dn: ou=users,dc=example,dc=com
  objectClass: organizationalUnit
  ou: users

  dn: cn=Jane,ou=users,dc=example,dc=com
  objectClass: person
  objectClass: organizationalPerson
  objectClass: inetOrgPerson
  cn: Jane
  sn: Smith
  displayName: Jane Smith
  mail: jane.smith@example.com

  dn: cn=Jim,ou=users,dc=example,dc=com
  objectClass: person
  objectClass: organizationalPerson
  objectClass: inetOrgPerson
  cn: Jim
  sn: Adams
  displayName: Jim Adams
  mail: jim.adams@example.com

  dn: ou=groups,dc=example,dc=com
  objectClass: organizationalUnit
  ou: groups

  dn: cn=admins,ou=groups,dc=example,dc=com 1
  objectClass: groupOfNames
  cn: admins
  owner: cn=admin,dc=example,dc=com
  description: System Administrators
  member: cn=Jane,ou=users,dc=example,dc=com 2
  member: cn=Jim,ou=users,dc=example,dc=com
1
The group is a first-class entry in the LDAP server.
2
Members of a group are listed with an identifying reference as attributes on the group.

To sync this group, you must first create the configuration file. The RFC 2307 schema requires you to provide an LDAP query definition for both user and group entries, as well as the attributes with which to represent them in the internal OpenShift Container Platform records.

For clarity, the Group you create in OpenShift Container Platform should use attributes other than the distinguished name whenever possible for user- or administrator-facing fields. For example, identify the users of a Group by their e-mail, and use the name of the group as the common name. The following configuration file creates these relationships:

Note

If using user-defined name mappings, your configuration file will differ.

Example 13.5. LDAP Sync Configuration Using RFC 2307 Schema: rfc2307_config.yaml

kind: LDAPSyncConfig
apiVersion: v1
url: ldap://LDAP_SERVICE_IP:389 1
insecure: true 2
rfc2307:
    groupsQuery:
        baseDN: "ou=groups,dc=example,dc=com"
        scope: sub
        derefAliases: never
        pageSize: 0
    groupUIDAttribute: dn 3
    groupNameAttributes: [ cn ] 4
    groupMembershipAttributes: [ member ] 5
    usersQuery:
        baseDN: "ou=users,dc=example,dc=com"
        scope: sub
        derefAliases: never
        pageSize: 0
    userUIDAttribute: dn 6
    userNameAttributes: [ mail ] 7
    tolerateMemberNotFoundErrors: false
    tolerateMemberOutOfScopeErrors: false
1
The IP address and host of the LDAP server where this group’s record is stored.
2
When true, no TLS connection is made to the server. When false, secure LDAP (ldaps://) URLs connect using TLS, and insecure LDAP (ldap://) URLs are upgraded to TLS.
3
The attribute that uniquely identifies a group on the LDAP server. You cannot specify groupsQuery filters when using DN for groupUIDAttribute. For fine-grained filtering, use the whitelist / blacklist method.
4
The attribute to use as the name of the Group.
5
The attribute on the group that stores the membership information.
6
The attribute that uniquely identifies a user on the LDAP server. You cannot specify usersQuery filters when using DN for userUIDAttribute. For fine-grained filtering, use the whitelist / blacklist method.
7
The attribute to use as the name of the user in the OpenShift Container Platform Group record.

To run sync with the rfc2307_config.yaml file:

$ oadm groups sync --sync-config=rfc2307_config.yaml --confirm

OpenShift Container Platform creates the following Group record as a result of the above sync operation:

Example 13.6. OpenShift Container Platform Group Created Using rfc2307_config.yaml

apiVersion: v1
kind: Group
metadata:
  annotations:
    openshift.io/ldap.sync-time: 2015-10-13T10:08:38-0400 1
    openshift.io/ldap.uid: cn=admins,ou=groups,dc=example,dc=com 2
    openshift.io/ldap.url: LDAP_SERVER_IP:389 3
  creationTimestamp:
  name: admins 4
users: 5
- jane.smith@example.com
- jim.adams@example.com
1
The last time this Group was synchronized with the LDAP server, in ISO 6801 format.
2
The unique identifier for the group on the LDAP server.
3
The IP address and host of the LDAP server where this Group’s record is stored.
4
The name of the Group as specified by the sync file.
5
The users that are members of the Group, named as specified by the sync file.

13.5.1.1. RFC2307 with User-Defined Name Mappings

When syncing groups with user-defined name mappings, the configuration file changes to contain these mappings as shown below.

Example 13.7. LDAP Sync Configuration Using RFC 2307 Schema With User-Defined Name Mappings: rfc2307_config_user_defined.yaml

kind: LDAPSyncConfig
apiVersion: v1
groupUIDNameMapping:
  "cn=admins,ou=groups,dc=example,dc=com": Administrators 1
rfc2307:
    groupsQuery:
        baseDN: "ou=groups,dc=example,dc=com"
        scope: sub
        derefAliases: never
        pageSize: 0
    groupUIDAttribute: dn 2
    groupNameAttributes: [ cn ] 3
    groupMembershipAttributes: [ member ]
    usersQuery:
        baseDN: "ou=users,dc=example,dc=com"
        scope: sub
        derefAliases: never
        pageSize: 0
    userUIDAttribute: dn 4
    userNameAttributes: [ mail ]
    tolerateMemberNotFoundErrors: false
    tolerateMemberOutOfScopeErrors: false
1
The user-defined name mapping.
2
The unique identifier attribute that is used for the keys in the user-defined name mapping. You cannot specify groupsQuery filters when using DN for groupUIDAttribute. For fine-grained filtering, use the whitelist / blacklist method.
3
The attribute to name OpenShift Container Platform Groups with if their unique identifier is not in the user-defined name mapping.
4
The attribute that uniquely identifies a user on the LDAP server. You cannot specify usersQuery filters when using DN for userUIDAttribute. For fine-grained filtering, use the whitelist / blacklist method.

To run sync with the rfc2307_config_user_defined.yaml file:

$ oadm groups sync --sync-config=rfc2307_config_user_defined.yaml --confirm

OpenShift Container Platform creates the following Group record as a result of the above sync operation:

Example 13.8. OpenShift Container Platform Group Created Using rfc2307_config_user_defined.yaml

apiVersion: v1
kind: Group
metadata:
  annotations:
    openshift.io/ldap.sync-time: 2015-10-13T10:08:38-0400
    openshift.io/ldap.uid: cn=admins,ou=groups,dc=example,dc=com
    openshift.io/ldap.url: LDAP_SERVER_IP:389
  creationTimestamp:
  name: Administrators 1
users:
- jane.smith@example.com
- jim.adams@example.com
1
The name of the Group as specified by the user-defined name mapping.

13.5.2. RFC 2307 with User-Defined Error Tolerances

By default, if the groups being synced contain members whose entries are outside of the scope defined in the member query, the group sync fails with an error:

Error determining LDAP group membership for "<group>": membership lookup for user "<user>" in group "<group>" failed because of "search for entry with dn="<user-dn>" would search outside of the base dn specified (dn="<base-dn>")".

This often indicates a mis-configured baseDN in the usersQuery field. However, in cases where the baseDN intentionally does not contain some of the members of the group, setting tolerateMemberOutOfScopeErrors: true allows the group sync to continue. Out of scope members will be ignored.

Similarly, when the group sync process fails to locate a member for a group, it fails outright with errors:

Error determining LDAP group membership for "<group>": membership lookup for user "<user>" in group "<group>" failed because of "search for entry with base dn="<user-dn>" refers to a non-existent entry".

Error determining LDAP group membership for "<group>": membership lookup for user "<user>" in group "<group>" failed because of "search for entry with base dn="<user-dn>" and filter "<filter>" did not return any results".

This often indicates a mis-configured usersQuery field. However, in cases where the group contains member entries that are known to be missing, setting tolerateMemberNotFoundErrors: true allows the group sync to continue. Problematic members will be ignored.

Warning

Enabling error tolerances for the LDAP group sync causes the sync process to ignore problematic member entries. If the LDAP group sync is not configured correctly, this could result in synced OpenShift Container Platform groups missing members.

Example 13.9. LDAP Entries Using RFC 2307 Schema With Problematic Group Membership: rfc2307_problematic_users.ldif

  dn: ou=users,dc=example,dc=com
  objectClass: organizationalUnit
  ou: users

  dn: cn=Jane,ou=users,dc=example,dc=com
  objectClass: person
  objectClass: organizationalPerson
  objectClass: inetOrgPerson
  cn: Jane
  sn: Smith
  displayName: Jane Smith
  mail: jane.smith@example.com

  dn: cn=Jim,ou=users,dc=example,dc=com
  objectClass: person
  objectClass: organizationalPerson
  objectClass: inetOrgPerson
  cn: Jim
  sn: Adams
  displayName: Jim Adams
  mail: jim.adams@example.com

  dn: ou=groups,dc=example,dc=com
  objectClass: organizationalUnit
  ou: groups

  dn: cn=admins,ou=groups,dc=example,dc=com
  objectClass: groupOfNames
  cn: admins
  owner: cn=admin,dc=example,dc=com
  description: System Administrators
  member: cn=Jane,ou=users,dc=example,dc=com
  member: cn=Jim,ou=users,dc=example,dc=com
  member: cn=INVALID,ou=users,dc=example,dc=com 1
  member: cn=Jim,ou=OUTOFSCOPE,dc=example,dc=com 2
1
A member that does not exist on the LDAP server.
2
A member that may exist, but is not under the baseDN in the user query for the sync job.

In order to tolerate the errors in the above example, the following additions to your sync configuration file must be made:

Example 13.10. LDAP Sync Configuration Using RFC 2307 Schema Tolerating Errors: rfc2307_config_tolerating.yaml

kind: LDAPSyncConfig
apiVersion: v1
url: ldap://LDAP_SERVICE_IP:389
insecure: true
rfc2307:
    groupsQuery:
        baseDN: "ou=groups,dc=example,dc=com"
        scope: sub
        derefAliases: never
    groupUIDAttribute: dn
    groupNameAttributes: [ cn ]
    groupMembershipAttributes: [ member ]
    usersQuery:
        baseDN: "ou=users,dc=example,dc=com"
        scope: sub
        derefAliases: never
    userUIDAttribute: dn 1
    userNameAttributes: [ mail ]
    tolerateMemberNotFoundErrors: true 2
    tolerateMemberOutOfScopeErrors: true 3
2
When true, the sync job tolerates groups for which some members were not found, and members whose LDAP entries are not found are ignored. The default behavior for the sync job is to fail if a member of a group is not found.
3
When true, the sync job tolerates groups for which some members are outside the user scope given in the usersQuery base DN, and members outside the member query scope are ignored. The default behavior for the sync job is to fail if a member of a group is out of scope.
1
The attribute that uniquely identifies a user on the LDAP server. You cannot specify usersQuery filters when using DN for userUIDAttribute. For fine-grained filtering, use the whitelist / blacklist method.

To run sync with the rfc2307_config_tolerating.yaml file:

$ oadm groups sync --sync-config=rfc2307_config_tolerating.yaml --confirm

OpenShift Container Platform creates the following group record as a result of the above sync operation:

Example 13.11. OpenShift Container Platform Group Created Using rfc2307_config.yaml

apiVersion: v1
kind: Group
metadata:
  annotations:
    openshift.io/ldap.sync-time: 2015-10-13T10:08:38-0400
    openshift.io/ldap.uid: cn=admins,ou=groups,dc=example,dc=com
    openshift.io/ldap.url: LDAP_SERVER_IP:389
  creationTimestamp:
  name: admins
users: 1
- jane.smith@example.com
- jim.adams@example.com
1
The users that are members of the group, as specified by the sync file. Members for which lookup encountered tolerated errors are absent.

13.5.3. Active Directory

In the Active Directory schema, both users (Jane and Jim) exist in the LDAP server as first-class entries, and group membership is stored in attributes on the user. The following snippet of ldif defines the users and group for this schema:

Example 13.12. LDAP Entries Using Active Directory Schema: active_directory.ldif

dn: ou=users,dc=example,dc=com
objectClass: organizationalUnit
ou: users

dn: cn=Jane,ou=users,dc=example,dc=com
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: testPerson
cn: Jane
sn: Smith
displayName: Jane Smith
mail: jane.smith@example.com
testMemberOf: admins 1

dn: cn=Jim,ou=users,dc=example,dc=com
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: testPerson
cn: Jim
sn: Adams
displayName: Jim Adams
mail: jim.adams@example.com
testMemberOf: admins
1
The user’s group memberships are listed as attributes on the user, and the group does not exist as an entry on the server. The testMemberOf attribute cannot be a literal attribute on the user; it can be created during search and returned to the client, but not committed to the database.

To sync this group, you must first create the configuration file. The Active Directory schema requires you to provide an LDAP query definition for user entries, as well as the attributes to represent them with in the internal OpenShift Container Platform Group records.

For clarity, the Group you create in OpenShift Container Platform should use attributes other than the distinguished name whenever possible for user- or administrator-facing fields. For example, identify the users of a Group by their e-mail, but define the name of the Group by the name of the group on the LDAP server. The following configuration file creates these relationships:

Example 13.13. LDAP Sync Configuration Using Active Directory Schema: active_directory_config.yaml

kind: LDAPSyncConfi