RHOCP establishes a network for communication between containers on separate nodes. In the Architecture Diagram this is designated by the tenant network. However, the tenant network is actually two networks; one network encapsulated within another. The outer network (tenant network) is the neutron network that carries traffic between nodes. The second network carries container to container traffic.

RHOCP offers two mechanisms to operate the internal network; openshift_sdn and flannel. This reference environment uses the flannel network for the reasons detailed below.

The default mechanism is the openshift_sdn. This is a custom Open vSwitch (OVS) SDN that offers fine-grained dynamic routing and traffic isolation. On a bare-metal installation, openshift_sdn offers both control and high performance.

If the default mechanism (openshift_sdn) is used to operate the RHOCP internal network, all RHOCP internode traffic carried over OpenStack neutron undergoes double encapsulation, creating significant network performance degradation.

flannel replaces the default openshift_sdn mechanism thus eliminating the second OVS instance. flannel is used in host-gateway mode that routes packets using a lookup table for the destination instance and then forwards them directly to that instance encapsulated only with a single UDP header. This provides a performance benefit over OVS double-encapsulation. It, however, does come with a cost.

flannel uses a single IP network space for all of the containers, allocating a contiguous subset of the space to each instance. Consequently, nothing prevents a container from attempting to contact any IP address in the same network space. This hinders multi-tenancy because the network cannot be used to isolate containers in one application from another.

In this reference environment, performance is given precedence over tenant isolation.

RHOCP is a software container service. When fully deployed in a cloud environment, it has three operational views for the different classes of users. These users work with RHOCP in different ways, performing distinct tasks. They see different views of RHOCP.

When deploying RHOCP, it is important to understand these views and how the service presents itself to different users.

Operations Users use a bastion host to reach the service internals. The bastion host serves as a stable trusted platform for service monitoring and managing updates.

Application Developers use the CLI or web interfaces of the RHOCP service to create and manage their applications.

Application Users see RHOCP applications as a series of web services published under a blanket DNS domain name. These all forward to the OpenShift Router an internal service that creates the connections between external users and internal containers.

Figure 2.2. RHOCP User Views

When the deployment process is complete, each type of user is able to reach and use their resources. The RHOCP service makes the best possible use of the OpenStack resources and where appropriate, making them available to the users.

Each of these views corresponds to a host or virtual host (via a load-balancer) that the user sees as their gateway to the RHOCP service.

An external DNS server is used to make the service views visible. Most installations contain an existing DNS infrastructure capable of dynamic updates.

The developer interface, provided by the OpenShift master servers, can be spread across multiple instances to provide both load-balancing and high-availablity properties. This guide uses an external load-balancer running haproxy to offer a single entry point for developers.

Application traffic passes through the OpenShift Router on its way to the container processes inside the service. The OpenShift Router is really a reverse proxy service that multiplexes the traffic to multiple containers making up a scaled application running inside RHOCP. It itself can accept inbound traffic on multiple hosts as well. The load-balancer used for developer HA acts as the public view for the RHOCP applications, forwarding to the OpenShift Router endpoints.

Since the load-balancer is an external component and the hostnames for the OpenShift master service and application domain are known beforehand, it can be configured into the DNS prior to starting the deployment procedure.

The load-balancer can handle both the master and application traffic. The master web and REST interfaces are on port 8443/TCP. The applications use ports 80/TCP and 443/TCP. A single load-balancer can forward both sets of traffic to different destinations.

With a load-balancer host address of 10.19.x.y, the two DNS records can be added as follows:

The second record is called a wildcard record. It allows all hostnames under that subdomain to have the same IP address without needing to create a separate record for each name.

This allows RHOCP to add applications with arbitrary names as long as they are under that subdomain. For example DNS name lookups for tax.apps.ocp3.example.com and home-goods.apps.ocp3.example.com would both get the same IP address: 10.19.x.y. All of the traffic is forwarded to the OpenShift Routers that examines the HTTP headers of the queries and forward them to the correct destination.

The destination for the master and application traffic is set in the load-balancer configuration after each instance is created and the floating IP address is assigned.

This document describes two ways of deploying RHOCP. The first is a manual process to create and prepare the run-time environment in RHOSP and then install RHOCP on the instances. The manual process is presented first to provide a detailed explanation of all the steps involved in the deployment process and to provide a better understanding of all the components and their interworkings.

The second method uses Heat orchestration to deploy the RHOCP service. Heat is the OpenStack service that automates complex infrastructure and service deployments. With Heat, the user describes the structure of a set of OpenStack resources in a template, and the Heat Engine creates a stack from the template, configuring all of the networks, instances, volumes in a single step.

The RHOCP on RHOSP Heat templates, along with some customization parameters, define the RHOSP service and the Heat engine executes the deployment process.

Heat orchestration offers serveral benefits including the removal of the operator from the process thus providing a consistent repeatable result.

The high level steps include:

The following sections describe the steps in detail.

This procedure was developed and tested on Red Hat OpenStack Platform 10. While it may apply to other releases, no claims are made for installations on versions other than RHOSP 10.

This procedure makes use of the following RHOSP service components:

All of the RHOSP commands in this reference environment are executed using the CLI tools. The user needs a RHOSP account, and a project to contain the service.

For the manual procedure the user needs only the member role on the project. Running the installation using Heat requires the heat_stack_owner role.

The RHOSP credentials are commonly stored in a source file that sets the values of their shell environment.

A script fragement from the RHOSP keystone web console is available as shown below.

A simpler format can be used and is provided in the following code snippet:

unset OS_AUTH_URL OS_USERNAME OS_PASSWORD OS_PROJECT_NAME OS_REGION
# replace the IP address with the RHOSP10 keystone server IP
export OS_AUTH_URL=http://10.1.0.100:5000/v2.0
export OS_USERNAME=ocp3ops
export OS_PASSWORD=password
export OS_TENANT_NAME=ocp3
export OS_REGION=RegionOne

To use these values for RHOSP CLI commands, source the file into the environment.

source ./ks_ocp3

This procedure expects the instances to be created from the stock Red Hat Enterprise Linux 7.3 cloud image.

The Red Hat Enterprise Linux 7.3 cloud image is in the rhel-guest-image-7 package that is located in the rhel-7-server-rh-common-rpms repository. Install the package on any Red Hat Enterprise Linux system and load the image file into glance. For the purposes of this procedure, the image is named rhel7.

sudo subscription-manager repos --enable rhel-7-server-rh-common-rpms
sudo yum -y install rhel-guest-image-7
sudo cp /usr/share/rhel-guest-image7/*.qcow2 ~/images
cd ~/images
qemu-img convert -f qcow2 -O raw rhel-guest-image-<ver>.x86_64.qcow2 ./rhel7.raw
openstack image create rhel7 \
--disk-format=raw \
--container-format=bare
--public < rhel7.raw

Confirm the image has been created as follows:

openstack image list

RHOSP uses cloud-init to place an ssh public key on each instance as it is created to allow ssh access to the instance. RHOSP expects the user to hold the private key.

openstack keypair create ocp3 > ~/ocp3.pem
chmod 600 ~/ocp3.pem

When creating a heat stack, the heat engine gets a keystone token to continue taking actions on behalf of the user. By default the token is only good for 1 hour (3600 seconds). The token expires after 1 hour, and the heat engine can no longer continue.

The RHOCP 3 heat stack deployment may take more than one hour. To ensure deployment completion, increase the expiration time on the RHOSP controllers and restart the httpd service.

# On the RHOSP controller:
sudo sed -i -e 's/#expiration = .*/expiration = 7200/' /etc/keystone/keystone.conf
sudo systemctl restart httpd

The RHOCP service must be connected to an external public network so that users can reach it. In general, this requires a physical network or at least a network that is otherwise out of the control of the RHOSP service.

The control network is connected to the public network via a router during the network setup.

Each instance is attached to the control network when it is created. Instances that accept direct inbound connections (the bastion, masters and infrastructure nodes) are given floating IP addresses from a pool on the public network.

The provisioner must have permission necessary to create and attach the router to the public network and to create floating IPs from the pool on that network.

The floating IP pool must be large enough to accomodate all of the accessable hosts.

Before beginning the provisioning process, the hostname for developer access and the wild card DNS entry for applications must be in place. Both of these should point to the IP address of the load balancer.

Both the IP address and the FQDN for the OpenShift master name and the application wild card entry are known before the provisioning process begins.

Each host in the RHOCP service must have a DNS entry for the interfaces on the control and tenant networks. These names are only used internally. Additionally, each host with a floating IP must have a DNS entry with a public name in order to be found by the load-balancer host.

Since the instances are created and addressed dynamically, the name/IP pairs cannot be known before the provisioning process begins. The entries are added using dynamic DNS after each instance is created.

Dynamic DNS updates use a tool called nsupdate found in the bind-utils RPM. The update requests are authorized using a symmetric key found on the DNS host in /etc/rndc.key.

The dynamic domains must be created before the provisioning process begins and must be configured to allow dynamic updates. Configuring DNS zones and configuring them to allow dynamic updates is outside the scope of this paper.

Appendix B contains a procedure to create a heat stack running a compliant DNS service.

A load-balancer provides a single view of the OpenShift master services for the applications. The IP address of the load-balancer must be known before beginning. The master services and the applications use different TCP ports so a single TCP load balancer can handle all of the inbound connections.

The load-balanced DNS name that developers use must be in a DNS A record pointing to the haproxy server before installation. For applications, a wildcard DNS entry must point to the haproxy host.

The configuration of the load-balancer is created after all of the RHOCP instances have been created and floating IP addresses assigned.

When installing RHOCP using Heat, the load-balancer can be created and configured as part of the heat stack.

RHOCP can use an external LDAP or Active Directory service to manage developers from a single user base.

The hostname or IP address of the LDAP service must be known before commencing. This procees requires the base_dn and a small set of parameters to allow LDAP authentication by the RHOCP service.

In summary, this table lists all of the information and settings that an operator requires to begin a RHOCP deployment on RHOSP.

The manual deployment process starts with a series of steps that are executed using the RHOSP CLI tools. These CLI tools are available in the rhel-7-server-openstack-10-rpms repository via the package python-openstackclient. This package installs all the underlying clients that include nova, neutron, cinder and glance. The minimum package version for python-openstackclient recommended is version 3.2.1-1. This can be verified via:

# rpm -q python-openstackclient
python-openstackclient-3.2.1-1.el7ost.noarch

Using the openstack command, one can create and configure the infrastructure that hosts the RHOCP service.

The procedures in this section consists of a set of RHOSP CLI script fragments. These fragments are to be executed on an operator’s workstation that has the ability to install the python-openstackclient package. These script fragments allow the operator to create the infrastructure and the instances within the RHOSP environment as required for a successful RHOCP installation. Once the operator can create RHOSP instances, the remainder of the script fragments are run on the bastion host to complete the RHOCP deployment using Ansible.

This section describes the RHOSP network components required by RHOCP and how to create them.

RHOCP depends on three networks.

The public network is a prerequisite requirement prior to continuing with the RHOCP installation. Creation of a public network requires administrative access to the RHOSP environment. Consult the local RHOSP administrator if a public network is not currently available.

Once the public network is accessible, a pool of floating IP addresses is required for RHOSP instances that are to be created, specifically the bastion host, master nodes and infrastructure nodes. Consult the local RHOSP administrator if a pool of floating IP addresses is not currently available.

The control network and the tenant network can be created by a RHOSP user that within their respective RHOSP project thus not requiring any additional RHOSP administrator privileges.

This reference environment refers to the public network name as public_network

The following two tables describe the internal RHOCP networks.

Create the control network as indicated below. Update the default values as required.

#!/bin/sh
OCP3_DNS_NAMESERVER=${OCP3_DNS_NAMESERVER:-8.8.8.8}
PUBLIC_NETWORK=${PUBLIC_NETWORK:-public_network}
CONTROL_SUBNET_CIDR=${CONTROL_SUBNET_CIDR:-172.18.10.0/24}

openstack network create control-network
openstack subnet create --network control-network --subnet-range ${CONTROL_SUBNET_CIDR} \
--dns-nameserver ${OCP3_DNS_NAMESERVER} control-subnet
openstack router create control-router
openstack router add subnet control-router control-subnet
neutron router-gateway-set control-router ${PUBLIC_NETWORK}

In a similar fashion, create the tenant network. This network is for internal use only and does not have a gateway to connect to other networks.

#!/bin/sh
TENANT_SUBNET_CIDR=${TENANT_SUBNET_CIDR:-172.18.20.0/24}

openstack network create tenant-network
openstack subnet create --network tenant-network \
  --subnet-range ${TENANT_SUBNET_CIDR} --gateway none tenant-subnet

At this point, three networks exist. Two of those networks created by the operator (control and tenant network), while the public network is provided by the RHOSP administrator. The control network is bound to a router that provides external access.

openstack network list
+--------------------------------------+-----------------+--------------------------------------+
| ID                                   | Name            | Subnets                              |
+--------------------------------------+-----------------+--------------------------------------+
| c913bfd7-94e9-4caa-abe5-df9dcd0c0eab | control-network | e804600d-67fc-40bb-a7a8-2f1ce49a81e6 |
| 53377111-4f85-401e-af9d-5166a4be7f99 | tenant-network  | 830845ca-f35c-44d9-be90-abe532a32cbe |
| 957dae95-5b93-44aa-a2c3-37eb16c1bd2f | public_network  | 4d07867c-34d8-4a7c-84e3-fdc485de66be |
+--------------------------------------+-----------------+--------------------------------------+

RHOSP networking allows the user to define inbound and outbound traffic filters that can be applied to each instance on a network. This allows the user to limit network traffic to each instance based on the function of the instance services and not depend on host based filtering.

This section describes the ports and services required for each type of host and how to create the security groups in RHOSP.

The details of the communication ports and their purposes are defined in the Prerequisites section of the OpenShift Container Platform Installation and Configuration Guide.

#!/bin/sh
openstack security group create bastion-sg
openstack security group rule create --ingress --protocol icmp bastion-sg
openstack security group rule create --protocol tcp \
--dst-port 22 bastion-sg
#Verification of security group
openstack security group show bastion-sg

#!/bin/sh
openstack security group create master-sg
openstack security group rule create --protocol icmp master-sg
neutron security-group-rule-create master-sg \
 --protocol tcp --port-range-min 22 --port-range-max 22 \
 --remote-group-id bastion-sg


neutron security-group-rule-create master-sg \
 --protocol tcp --port-range-min 2380 --port-range-max 2380 \
 --remote-group-id master-sg

for PORT in 53 2379 2380 8053 8443 10250 24224
do
  openstack security group rule create --protocol tcp --dst-port $PORT master-sg
done

for PORT in 53 4789 8053 24224
do
  openstack security group rule create --protocol udp --dst-port $PORT master-sg
done

#!/bin/sh
openstack security group create infra-node-sg
openstack security group rule create --protocol icmp infra-node-sg
neutron security-group-rule-create infra-node-sg \
  --protocol tcp --port-range-min 22 --port-range-max 22 \
  --remote-group-id bastion-sg

for PORT in 80 443 10250 4789
do
 openstack security group rule create --protocol tcp --dst-port $PORT infra-node-sg
done

#!/bin/sh
openstack security group create app-node-sg
openstack security group rule create --protocol icmp app-node-sg
neutron security-group-rule-create app-node-sg \
     --protocol tcp --port-range-min 22 --port-range-max 22 \
     --remote-group-id bastion-sg
openstack security group rule create --protocol tcp --dst-port 10250 app-node-sg
openstack security group rule create --protocol udp --dst-port 4789 app-node-sg

With the networks established, begin building the RHOSP instances to build the RHOCP environment.

This reference environment consists of:

With three exceptions, the specifications all of the instances in this service are identical. Each instance has a unique name and each of the node hosts have external storage. Each one is assigned a security group based on its function. The common elements are presented first followed by the differences.

mkdir -p ~/user-data

#!/bin/sh

#set DOMAIN and SUBDOMAIN to override
OCP3_DOMAIN=${OCP3_DOMAIN:-ocp3.example.com}
CONTROL_DOMAIN=${CONTROL_DOMAIN:-control.${OCP3_DOMAIN}}

BASTION="bastion"
MASTERS=$(for M in $(seq 0 $(($MASTER_COUNT-1))) ; do echo master-$M ; done)
INFRA_NODES=$(for I in $(seq 0 $(($INFRA_NODE_COUNT-1))) ; do echo infra-node-$I ; done)
APP_NODES=$(for A in $(seq 0 $(($APP_NODE_COUNT-1))) ; do echo app-node-$A ; done)
ALL_NODES="$INFRA_NODES $APP_NODES"
ALL_HOSTS="$BASTION $MASTERS $ALL_NODES"

function generate_userdata_mime() {
  cat <<EOF
From nobody Fri Oct  7 17:05:36 2016
Content-Type: multipart/mixed; boundary="===============6355019966770068462=="
MIME-Version: 1.0

--===============6355019966770068462==
MIME-Version: 1.0
Content-Type: text/cloud-config; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="$1.yaml"

#cloud-config
hostname: $1
fqdn: $1.$2

write_files:
  - path: "/etc/sysconfig/network-scripts/ifcfg-eth1"
    permissions: "0644"
    owner: "root"
    content: |
      DEVICE=eth1
      TYPE=Ethernet
      BOOTPROTO=dhcp
      ONBOOT=yes
      DEFTROUTE=no
      PEERDNS=no

--===============6355019966770068462==
MIME-Version: 1.0
Content-Type: text/x-shellscript; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="allow-sudo-ssh.sh"

#!/bin/sh
sed -i "/requiretty/s/^/#/" /etc/sudoers

--===============6355019966770068462==--

EOF
}

[ -d user-data ] || mkdir -p user-data
for HOST in $ALL_HOSTS
do
  generate_userdata_mime ${HOST} ${CONTROL_DOMAIN} > user-data/${HOST}.yaml
done

$ openstack image list
$ openstack keypair list
$ openstack network list

#!/bin/sh
OCP3_DOMAIN=${OCP3_DOMAIN:-ocp3.example.com}
OCP3_CONTROL_DOMAIN=${OCP3_CONTROL_DOMAIN:-control.${OCP3_DOMAIN}}
IMAGE=${IMAGE:-rhel7}
FLAVOR=${FLAVOR:-m1.small}
OCP3_KEY_NAME=${OCP3_KEY_NAME:-ocp3}
netid1=$(openstack network list | awk "/control-network/ { print \$2 }")
netid2=$(openstack network list | awk "/tenant-network/ { print \$2 }")
openstack server create --flavor ${FLAVOR} --image ${IMAGE} \
--key-name ${OCP3_KEY_NAME} \
--nic net-id=$netid1 \
--nic net-id=$netid2 \
--security-group bastion-sg --user-data=user-data/bastion.yaml \
bastion.${OCP3_CONTROL_DOMAIN}

#!/bin/sh
OCP3_DOMAIN=${OCP3_DOMAIN:-ocp3.example.com}
OCP3_CONTROL_DOMAIN=${OCP3_CONTROL_DOMAIN:-control.${OCP3_DOMAIN}}
OCP3_KEY_NAME=${OCP3_KEY_NAME:-ocp3}
IMAGE=${IMAGE:-rhel7}
FLAVOR=${FLAVOR:-m1.small}
MASTER_COUNT=${MASTER_COUNT:-3}
netid1=$(openstack network list | awk "/control-network/ { print \$2 }")
netid2=$(openstack network list | awk "/tenant-network/ { print \$2 }")
for HOSTNUM in $(seq 0 $(($MASTER_COUNT-1))) ; do
    openstack server create --flavor ${FLAVOR} --image ${IMAGE} \
      --key-name ${OCP3_KEY_NAME} \
      --nic net-id=$netid1 --nic net-id=$netid2 \
      --security-group master-sg --user-data=user-data/master-${HOSTNUM}.yaml \
      master-${HOSTNUM}.${OCP3_CONTROL_DOMAIN}
done

#!/bin/sh
VOLUME_SIZE=${VOLUME_SIZE:-8}
BASTION="bastion"
INFRA_NODE_COUNT=${INFRA_NODE_COUNT:-2}
APP_NODE_COUNT=${APP_NODE_COUNT:-3}

INFRA_NODES=$(for I in $(seq 0 $(($INFRA_NODE_COUNT-1))) ; do echo infra-node-$I ; done)
APP_NODES=$(for I in $(seq 0 $(($APP_NODE_COUNT-1))) ; do echo app-node-$I ; done)
ALL_NODES="$INFRA_NODES $APP_NODES"

for NODE in $ALL_NODES ; do
    cinder create --name ${NODE}-docker ${VOLUME_SIZE}
done

openstack volume list

#!/bin/sh
OCP3_DOMAIN=${OCP3_DOMAIN:-ocp3.example.com}
OCP3_CONTROL_DOMAIN=${OCP3_CONTROL_DOMAIN:-control.${OCP3_DOMAIN}}
OCP3_KEY_NAME=${OCP3_KEY_NAME:-ocp3}
IMAGE=${IMAGE:-rhel7}
FLAVOR=${FLAVOR:-m1.small}
netid1=$(openstack network list | awk "/control-network/ { print \$2 }")
netid2=$(openstack network list | awk "/tenant-network/ { print \$2 }")
INFRA_NODE_COUNT=${INFRA_NODE_COUNT:-2}
for HOSTNUM in $(seq 0 $(($INFRA_NODE_COUNT-1))) ; do
    HOSTNAME=infra-node-$HOSTNUM
    VOLUMEID=$(cinder list | awk "/${HOSTNAME}-docker/ { print \$2 }")
    openstack server create --flavor ${FLAVOR} --image ${IMAGE} \
       --key-name ${OCP3_KEY_NAME} \
       --nic net-id=$netid1 --nic net-id=$netid2 \
       --security-group infra-node-sg --user-data=user-data/${HOSTNAME}.yaml \
       --block-device-mapping vdb=${VOLUMEID} \
       ${HOSTNAME}.${OCP3_CONTROL_DOMAIN}
done

#!/bin/sh
OCP3_DOMAIN=${OCP3_DOMAIN:-ocp3.example.com}
OCP3_CONTROL_DOMAIN=${OCP3_CONTROL_DOMAIN:-control.${OCP3_DOMAIN}}
OCP3_KEY_NAME=${OCP3_KEY_NAME:-ocp3}
IMAGE=${IMAGE:-rhel7}
FLAVOR=${FLAVOR:-m1.small}
netid1=$(openstack network list | awk "/control-network/ { print \$2 }")
netid2=$(openstack network list | awk "/tenant-network/ { print \$2 }")
APP_NODE_COUNT=${APP_NODE_COUNT:-3}
for HOSTNUM in $(seq 0 $(($APP_NODE_COUNT-1))) ; do
    HOSTNAME=app-node-${HOSTNUM}
    VOLUMEID=$(cinder list | awk "/${HOSTNAME}-docker/ { print \$2 }")
    openstack server create --flavor ${FLAVOR} --image ${IMAGE} \
        --key-name ${OCP3_KEY_NAME} \
        --nic net-id=$netid1 --nic net-id=$netid2 \
        --security-group app-node-sg --user-data=user-data/${HOSTNAME}.yaml \
        --block-device-mapping vdb=${VOLUMEID} \
        ${HOSTNAME}.${OCP3_CONTROL_DOMAIN}
done

#!/bin/sh
OCP3_DOMAIN=${OCP3_DOMAIN:-ocp3.example.com}
OCP3_CONTROL_DOMAIN=${OCP3_CONTROL_DOMAIN:-control.${OCP3_DOMAIN}}
TENANT_NETWORK=${TENANT_NETWORK:-tenant-network}

MASTER_COUNT=${MASTER_COUNT:-3}
INFRA_NODE_COUNT=${INFRA_NODE_COUNT:-2}
APP_NODE_COUNT=${APP_NODE_COUNT:-3}

MASTERS=$(for M in $(seq 0 $(($MASTER_COUNT-1))) ; do echo master-$M ; done)
INFRA_NODES=$(for I in $(seq 0 $(($INFRA_NODE_COUNT-1))) ; do echo infra-node-$I ; done)
APP_NODES=$(for A in $(seq 0 $(($APP_NODE_COUNT-1))) ; do echo app-node-$A ; done)

function tenant_ip() {
  # HOSTNAME=$1
  nova show ${1} | grep ${TENANT_NETWORK} | cut -d\| -f3 | cut -d, -f1 | tr -d ' '
}

function port_id_by_ip() {
  # IP=$1
  neutron port-list --field id --field fixed_ips | grep $1 | cut -d' ' -f2
}

for NAME in $MASTERS $INFRA_NODES $APP_NODES
do
  TENANT_IP=$(tenant_ip ${NAME}.${OCP3_CONTROL_DOMAIN})
  PORT_ID=$(port_id_by_ip $TENANT_IP)
  neutron port-update $PORT_ID --no-security-groups --port-security-enabled=False
done

#!/bin/sh
OCP3_DOMAIN=${OCP3_DOMAIN:-ocp3.example.com}
OCP3_CONTROL_DOMAIN=${OCP3_CONTROL_DOMAIN:-control.${OCP3_DOMAIN}}
PUBLIC_NETWORK=${PUBLIC_NETWORK:-public_network}
MASTER_COUNT=${MASTER_COUNT:-3}
INFRA_NODE_COUNT=${INFRA_NODE_COUNT:-2}
APP_NODE_COUNT=${APP_NODE_COUNT:3}

BASTION="bastion"
MASTERS=$(for M in $(seq 0 $(($MASTER_COUNT-1))) ; do echo master-$M ; done)
INFRA_NODES=$(for I in $(seq 0 $(($INFRA_NODE_COUNT-1))) ; do echo infra-node-$I ; done)
for HOST in $BASTION $MASTERS $INFRA_NODES
do
  openstack floating ip create ${PUBLIC_NETWORK}
  FLOATING_IP=$(openstack floating ip list | awk "/None/ { print \$4 }")
  openstack server add floating ip ${HOST}.${OCP3_CONTROL_DOMAIN} ${FLOATING_IP}
done

While end users only interact with the RHOCP services through the public interfaces, it is important to assign a name to each interface to ensure each RHOCP component is able to identify and communicate across the environment. The simplest way to accomplish this is to update the DNS entries for each interface within each RHOSP instance.

All of the RHOSP instances connect to the control-network and the tenant-network. The bastion, master and infra nodes contain floating IPs that are assigned on the public_network and must be published in a zone. This reference environment uses the ocp3.example.com zone. The IP addresses on the control and tenant networks reside in a subdomain named for these networks. For example, DNS updates require the IP address of the primary DNS server for the zone and a key to enable updates.

When using BIND (Berkeley Internet Name Domain), the DNS server included in Red Hat Enterprise Linux, the key service it provides is the named program. When using the named service, the key is located in the /etc/rndc.key file. Create a copy of the contents of the file to a local file such as ocp3_dns.key. In order to implement updates, the bind-utils RPM must be installed. This contains the /usr/bin/nsupdate binary that implements dynamic updates.

Prior to making DNS updates, acquire the hostname, FQDN and floating IP for each instance.

On a server that has access to the key, i.e. /etc/rndc.key, perform the following for all nodes:

nsupdate -k ocp3_dns.key <<EOF
server <dns-server-ip>
zone ocp3.example.com
update add <FQDN> 3600 A <ip-address>
.
.
.
update add <FQDN> 3600 A <ip-address>
send
quit
EOF

An example of adding the public, tenant and control networks of particular node.

nsupdate -k ocp3_dns.key <<EOF
server <dns-server-ip>
zone ocp3.example.com
update add master-0.control.ocp3.example.com 3600 A 172.x.10.7
update add master-0.ocp3.example.com 3600 A 10.x.0.166
send
quit
EOF

10.19.114.153 bastion.ocp3.example.com bastion
172.18.10.6 bastion.control.example.com

10.19.114.114 master-0.ocp3.example.com
172.18.10.7 master-0.control.example.com

10.19.114.115 master-1.ocp3.example.com
172.18.10.8 master-1.control.example.com

10.19.114.116 master-2.ocp3.example.com
172.18.10.9 master-2.control.example.com

10.19.114.120 infra-node-0.ocp3.example.com
172.18.10.10 infra-node-0.control.example.com

10.19.114.121 infra-node-1.ocp3.example.com
172.18.10.11 infra-node-1.control.example.com

172.18.10.17 app-node-0.control.example.com
172.18.10.18 app-node-1.control.example.com
172.18.10.19 app-node-2.control.example.com

This reference environment consists of a standalone server running a HAProxy instance with the following configuration. Please include the floating IPs for the master and infra nodes to the load balancer.

The file that follows is a sample configuration for HAProxy.

global
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

defaults
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
listen  stats :9000
        stats enable
        stats realm Haproxy\ Statistics
        stats uri /haproxy_stats
        stats auth admin:password
        stats refresh 30
        mode http

frontend  main *:80
    default_backend             router80

backend router80
    balance source
    mode tcp
    server infra-node-0.ocp3.example.com 10.19.114.120:80 check
    server infra-node-1.ocp3.example.com 10.19.114.121:80 check
    server infra-node-2.ocp3.example.com 10.19.114.122:80 check

frontend  main *:443
    default_backend             router443

backend router443
    balance source
    mode tcp
    server infra-node-0.ocp3.example.com 10.19.114.120:443 check
    server infra-node-1.ocp3.example.com 10.19.114.121:443 check
    server infra-node-2.ocp3.example.com 10.19.114.122:443 check


frontend  main *:8443
    default_backend             mgmt8443

backend mgmt8443
    balance source
    mode tcp
    server master-0.ocp3.example.com 10.19.114.114:8443 check
    server master-1.ocp3.example.com 10.19.114.115:8443 check
    server master-2.ocp3.example.com 10.19.114.116:8443 check

With access to the RHOSP instances, this section turns its focus to running the openshift-ansible playbooks. For a successful deployment of RHOCP, the RHOSP instances require additional configuration.

The additional configuration on the RHOCP nodes is all done via the bastion host. The bastion host acts as a jump server to access all RHOCP nodes via an ssh session using the control network.

THE RHOCP master nodes require software updates and accessibility to install ansible. The infra and app nodes require access to multiple software repositories, and a cinder volume i.e. /dev/vdb, to store the Docker container files.

ssh -i ocp3.pem  cloud-user@bastion.control.ocp3.example.com

#!/bin/sh
# Set RHN_USERNAME, RHN_PASSWORD RHN_POOL_ID for your environment
sudo subscription-manager register \
  --username $RHN_USERNAME \
  --password $RHN_PASSWORD
sudo subscription-manager subscribe --pool $RHN_POOL_ID

#!/bin/sh
OCP3_VERSION=${OCP3_VERSION:-3.4}
sudo subscription-manager repos --disable="*"
sudo subscription-manager repos \
  --enable=rhel-7-server-rpms \
  --enable=rhel-7-server-extras-rpms \
  --enable=rhel-7-server-optional-rpms \
  --enable=rhel-7-server-ose-${OCP3_VERSION}-rpms

#!/bin/sh
sudo yum install -y openshift-ansible-playbooks

scp -i ocp3.pem ocp3.pem cloud-user@bastion.control.ocp3.example.com:.ssh/id_rsa

chmod 600 ~/.ssh/id_rsa

MASTERS="master-0 master-1 master-2"
INFRA_NODES="infra-node-0 infra-node-1"
APP_NODES="app-node-0 app-node-1 app-node-2"
ALL_HOSTS="$MASTERS $INFRA_NODES $APP_NODES"

ssh master-0

ssh <node-name>
sudo cp /etc/sudoers{,.orig}
sudo sed -i "/requiretty/s/^/#/" /etc/sudoers

ssh master-0 sudo id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

#!/bin/sh
RHN_USERNAME=${RHN_USERNAME:-changeme}
RHN_PASSWORD=${RHN_PASSWORD:-changeme}
RHN_POOL_ID=${RHN_POOL_ID:-changeme}

for H in $ALL_HOSTS
do
  ssh $H sudo subscription-manager register \
      --username ${RHN_USERNAME} \
      --password ${RHN_PASSWORD}
  ssh $H sudo subscription-manager attach \
      --pool ${RHN_POOL_ID}
done

#!/bin/sh
OCP3_VERSION=${OCP3_VERSION:-3.4}

for H in $ALL_HOSTS
do
  ssh $H sudo subscription-manager repos --disable="*"
  ssh $H sudo subscription-manager repos \
      --enable="rhel-7-server-rpms" \
      --enable="rhel-7-server-extras-rpms" \
      --enable="rhel-7-server-optional-rpms" \
      --enable="rhel-7-server-ose-${OCP3_VERSION}-rpms"
done

#!/bin/sh
for H in $ALL_HOSTS
do
  ssh $H sudo yum install -y \
    wget git net-tools bind-utils iptables-services \
    bridge-utils bash-completion atomic-openshift-excluder \
    atomic-openshift-docker-excluder
done

# /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
BOOTPROTO="dhcp"
BOOTPROTOv6="dhcp"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="yes"
PEERDNS="no"
IPV6INIT="yes"
PERSISTENT_DHCLIENT="1"

#!/bin/sh
for H in $ALL_HOSTS ; do
    scp ifcfg-eth1 $H:
    ssh $H sudo cp ifcfg-eth1 /etc/sysconfig/network-scripts
    ssh $H sudo ifup eth1
done

#!/bin/sh
for H in $INFRA_NODES $APP_NODES
do
  ssh $H sudo yum install -y docker
done

#!/bin/sh
for H in $INFRA_NODES $APP_NODES
do
  ssh $H sudo systemctl enable lvm2-lvmetad
  ssh $H sudo systemctl start lvm2-lvmetad
done

DEVS=/dev/vdb
VG=docker-vg

#!/bin/sh
for H in $INFRA_NODES $APP_NODES
do
  scp docker-storage-setup $H:
  ssh $H sudo cp ./docker-storage-setup /etc/sysconfig
  ssh $H sudo /usr/bin/docker-storage-setup
done

#!/bin/sh
for H in $INFRA_NODES $APP_NODES
do
  ssh $H sudo systemctl enable docker
  ssh $H sudo systemctl start docker
done

The OpenShift Ansible playbooks are separated into three input files.

The inventory is an ini formatted set of host names, and groupings of hosts with common configuration attributes. It defines the configuration variables for each group and individual host. The user defines a set of groups at the top level and fills out a section for each group defined. Each group section list the hosts that are members of that group.

The OSEv3.yml file is a yaml formatted data file. It defines a structured set of key/value pairs that are used to configure all of the RHOSP instances.

The ansible.cfg file is an ini formatted file that defines Ansible run-time customizations.

[OSEv3:children]
masters
nodes
etcd

[masters]
master-[0:2].control.ocp3.example.com openshift_public_hostname=master-[0:2].ocp3.example.com

[masters:vars]
openshift_schedulable=false
openshift_router_selector="region=infra"
openshift_registry_selector="region=infra"

[etcd]
master-[0:2].control.ocp3.example.com openshift_public_hostname=master-[0:2].ocp3.example.com

[infra-nodes]
infra-node-[0:1].control.ocp3.example.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_public_hostname=infra-node-[0:1].ocp3.example.com

[app-nodes]
app-node-[0:2].control.ocp3.example.com openshift_node_labels="{'region': 'primary', 'zone': 'default'}"

[nodes]
master-[0:2].control.ocp3.example.com openshift_node_labels="{'region': 'primary', 'zone': 'default'}" openshift_schedulable=false openshift_public_hostname=master-[0:2].ocp3.example.com

[nodes:children]
infra-nodes
app-nodes

deployment_type: openshift-enterprise
openshift_master_default_subdomain: apps.ocp3.example.com 1

# Developer access to WebUI and API
openshift_master_cluster_hostname: devs.ocp3.example.com 2
openshift_master_cluster_public_hostname: devs.ocp3.example.com 3
openshift_master_cluster_method: native

openshift_override_hostname_check: true
openshift_set_node_ip: true
openshift_use_dnsmasq: false
osm_default_node_selector: 'region=primary'

# Enable Flannel and set interface
openshift_use_openshift_sdn: false
openshift_use_flannel: true
flannel_interface: eth1

openshift_cloudprovider_kind: openstack
openshift_cloudprovider_openstack_auth_url: http://10.0.0.1:5000/v2.0 4
openshift_cloudprovider_openstack_username: <username> 5
openshift_cloudprovider_openstack_password: <password> 6
openshift_cloudprovider_openstack_tenant_name: <tenant name> 7

#If enabling a certificate, uncomment the following line.
#openshift_master_ldap_ca_file: /path/to/certificate.crt

# NOTE: Ensure to include the certificate name i.e certificate.crt within the
# ca field under openshift_master_identity_providers
# Change insecure value from true to false if using a certificate in the
# openshift_master_identity_providers section

openshift_master_identity_providers:
  - name: ldap_auth
    kind: LDAPPasswordIdentityProvider
    challenge: true
    login: true
    bindDN: cn=openshift,cn=users,dc=example,dc=com 8
    bindPassword: password 9
    url: ldap://ad1.example.com:389/cn=users,dc=example,dc=com?sAMAccountName 10
    attributes:
      id: ['dn']
      email: ['mail']
      name: ['cn']
      preferredUsername: ['sAMAccountName']
    ca: ''
    insecure: True

# config file for ansible -- http://ansible.com/
# ==============================================
[defaults]
remote_user = cloud-user
forks = 50
host_key_checking = False
gathering = smart
retry_files_enabled = false
nocows = true

[privilege_escalation]
become = True

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=900s -o GSSAPIAuthentication=no
control_path = /var/tmp/%%h-%%r

#!/bin/sh
ansible nodes -i ~/ansible_files/inventory -b -m yum -a "name=* state=latest"
ansible nodes -i ~/ansible_files/inventory -b -m yum -a "name=atomic-openshift-utils state=latest"

#!/bin/sh
export OCP_ANSIBLE_ROOT=${OCP_ANSIBLE_ROOT:-/usr/share/ansible}
export ANSIBLE_ROLES_PATH=${ANSIBLE_ROLES_PATH:-${OCP_ANSIBLE_ROOT}/openshift-ansible/roles}
export ANSIBLE_HOST_KEY_CHECKING=False

ansible-playbook -i ~/ansible_files/inventory \
  ${OCP_ANSIBLE_ROOT}/openshift-ansible/playbooks/byo/config.yml

#!/bin/sh
mkdir ~/.kube/
ssh master-0 sudo cat /etc/origin/master/admin.kubeconfig > ~/.kube/config
sudo yum -y install atomic-openshift-clients
oc whoami

#!/bin/sh
oc create -f - <<API
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: standard
provisioner: kubernetes.io/cinder
parameters:
  availability: nova
API

Perform the following steps from the first master node.

To run diagnostics, SSH into the the first master node. Direct access is provided to the first master node because of the configuration of the local ~/.ssh/config file.

ssh master-0

Connectivity to the master00 host as the root user should have been established. Run the diagnostics that are included as part of the install.

sudo oadm diagnostics
[Note] Determining if client configuration exists for client/cluster diagnostics
Info:  Successfully read a client config file at '/root/.kube/config'
Info:  Using context for cluster-admin access: 'default/devs-ocp3-example-com:8443/system:admin'
[Note] Performing systemd discovery

[Note] Running diagnostic: ConfigContexts[default/devs-ocp3-example-com:8443/system:admin]
       Description: Validate client config context is complete and has connectivity

Info:  The current client config context is 'default/devs-ocp3-example-com:8443/system:admin':
       The server URL is 'https://devs.ocp3.example.com:8443'
       The user authentication is 'system:admin/devs-ocp3-example-com:8443'
       The current project is 'default'
       Successfully requested project list; has access to project(s):
         [logging management-infra markllama openshift openshift-infra default]

[Note] Running diagnostic: DiagnosticPod
       Description: Create a pod to run diagnostics from the application standpoint


[Note] Running diagnostic: ClusterRegistry
       Description: Check that there is a working Docker registry

[Note] Running diagnostic: ClusterRoleBindings
       Description: Check that the default ClusterRoleBindings are present and contain the expected subjects

Info:  clusterrolebinding/cluster-readers has more subjects than expected.

       Use the oadm policy reconcile-cluster-role-bindings command to update the role binding to remove extra subjects.

Info:  clusterrolebinding/cluster-readers has extra subject {ServiceAccount management-infra management-admin    }.

[Note] Running diagnostic: ClusterRoles
       Description: Check that the default ClusterRoles are present and contain the expected permissions

[Note] Running diagnostic: ClusterRouterName
       Description: Check there is a working router

[Note] Skipping diagnostic: MasterNode
       Description: Check if master is also running node (for Open vSwitch)
       Because: Network plugin does not require master to also run node:

[Note] Running diagnostic: NodeDefinitions
       Description: Check node records on master

[Note] Running diagnostic: AnalyzeLogs
       Description: Check for recent problems in systemd service logs

Info:  Checking journalctl logs for 'docker' service

[Note] Running diagnostic: MasterConfigCheck
       Description: Check the master config file

Info:  Found a master config file: /etc/origin/master/master-config.yaml

... output abbreviated ...

[Note] Running diagnostic: UnitStatus
       Description: Check status for related systemd units

[Note] Summary of diagnostics execution (version v3.2.1.15):
[Note] Warnings seen: 3

Based on the results of the diagnostics, actions can be taken to alleviate any issues.

Perform the following steps from a local workstation.

This section focuses on the ETCD cluster. It describes the different commands to ensure the cluster is healthy. The internal DNS names of the nodes running ETCD must be used.

Issue the etcdctl command to confirm that the cluster is healthy.

sudo etcdctl \ --ca-file /etc/etcd/ca.crt \ --cert-file=/etc/origin/master/master.etcd-client.crt \ --key-file=/etc/origin/master/master.etcd-client.key \ --endpoints https://master-0.ocp3.example.com:2379 \ --endpoints https://master-1.ocp3.example.com:2379 \ --endpoints https://master-2.ocp3.example.com:2379 \ cluster-health
member 9bd1d7731aa447e is healthy: got healthy result from https://172.18.10.4:2379
member 2663a31f4ce5756b is healthy: got healthy result from https://172.18.10.5:2379
member 3e8001b17125a44e is healthy: got healthy result from https://172.18.10.6:2379
cluster is healthy

The role docker-storage-setup tells the Docker service to use /dev/vdb and create the volume group of docker-vol. The extra Docker storage options ensures that a container can grow no larger than 3G. Docker storage setup is performed on all master, infrastructure, and application nodes.

# vi /etc/sysconfig/docker-storage-setup
DEVS=/dev/vdb
VG=docker-vg

The specfic repositories for a successful OpenShift installation are define inn section Register for Software Updates. All systems except for the bastion host must have the same subscriptions. To verify those subscriptions, match those defined earlier, perfom the following.

sudo yum repolist
Loaded plugins: search-disabled-repos, subscription-manager
repo id                                        repo name                  status
rhel-7-server-extras-rpms/x86_64               Red Hat Enterprise Linux 7    536
rhel-7-server-openstack-10-rpms/7Server/x86_64 Red Hat OpenStack Platform  1,179
rhel-7-server-optional-rpms/7Server/x86_64     Red Hat Enterprise Linux 7 11,098
rhel-7-server-ose-3.2-rpms/x86_64              Red Hat OpenShift Enterpri    847
rhel-7-server-rpms/7Server/x86_64              Red Hat Enterprise Linux 7 14,619
repolist: 28,279

This section covers logging into the OpenShift Container Platform management console via the GUI and the CLI. After logging in via one of these methods applications can then be deployed and managed.

oc login --username openshift
Authentication required for https://devs.ocp3.example.com:8443 (openshift)
Username: openshift
Password:
Login successful.

You don't have any projects. You can try to create a new project, by running

    $ oc new-project <projectname>

$ oc new-project test-app

$ oc new-app https://github.com/openshift/cakephp-ex.git --name=php
--> Found image 2997627 (7 days old) in image stream "php" in project "openshift" under tag "5.6" for "php"

    Apache 2.4 with PHP 5.6
    -----------------------
    Platform for building and running PHP 5.6 applications

    Tags: builder, php, php56, rh-php56

    * The source repository appears to match: php
    * A source build using source code from https://github.com/openshift/cakephp-ex.git is created
      * The resulting image is pushed to image stream "php:latest"
    * This image is deployed in deployment config "php"
    * Port 8080/tcp is load balanced by service "php"
      * Other containers access this service through the hostname "php"

--> Creating resources with label app=php ...
    imagestream "php" created
    buildconfig "php" created
    deploymentconfig "php" created
    service "php" created
--> Success
    Build scheduled, use 'oc logs -f bc/php' to track its progress.
    Run 'oc status' to view apps.


$ oc expose service php
route "php" exposed

$ oc status
In project test-app on server https://openshift-master.sysdeseng.com:8443

http://test-app.apps.sysdeseng.com to pod port 8080-tcp (svc/php)
  dc/php deploys istag/php:latest <- bc/php builds https://github.com/openshift/cakephp-ex.git with openshift/php:5.6
    deployment #1 deployed about a minute ago - 1 pod

1 warning identified, use 'oc status -v' to see details.

In this section, reactions to failure are explored. After a sucessful install and some of the smoke tests noted above have been completed, failure testing is executed.

nova stop master-0.control.ocp3.example.com
Request to stop server master-0.control.ocp3.example.com has been accepted.
nova list --field name,status,power_state | grep master
| 4565505c-e48b-43e7-8c77-da6c1fc3d7d8 | master-0.control.ocp3.example.com     | SHUTOFF | Shutdown     |
| 12692288-013b-4891-a8a0-71e6967c656d | master-1.control.ocp3.example.com     | ACTIVE | Running     |
| 3cc0c6f0-59d8-4833-b294-a3a47c37d268 | master-2.control.ocp3.example.com     | ACTIVE | Running     |

# etcdctl -C https://master-0.control.ocp3.example.com:2379,https://master-1.control.ocp3.example.com:2379,https://master-2.control.ocp3.example.com:2379 --ca-file /etc/etcd/ca.crt --cert-file=/etc/origin/master/master.etcd-client.crt --key-file=/etc/origin/master/master.etcd-client.key cluster-health
failed to check the health of member 82c895b7b0de4330 on https://10.30.1.251:2379: Get https://10.30.1.251:2379/health: dial tcp 10.30.1.251:2379: i/o timeout
member 82c895b7b0de4330 is unreachable: [https://10.30.1.251:2379] are all unreachable
member c8e7ac98bb93fe8c is healthy: got healthy result from https://10.30.3.74:2379
member f7bbfc4285f239ba is healthy: got healthy result from https://10.30.2.157:2379
cluster is healthy

nova start master-0.control.ocp3.example.com
Request to start server master-0.control.example.com has been accepted.
nova list --field name,status,power_state | grep master
| 4565505c-e48b-43e7-8c77-da6c1fc3d7d8 | master-0.control.ocp3.example.com     | ACTIVE | Running     |
| 12692288-013b-4891-a8a0-71e6967c656d | master-1.control.ocp3.example.com     | ACTIVE | Running     |
| 3cc0c6f0-59d8-4833-b294-a3a47c37d268 | master-2.control.ocp3.example.com     | ACTIVE | Running     |

# *etcdctl -C https://master-0.control.ocp3.example.com:2379,https://master-1.control.ocp3.example.com:2379,https://master-2.control.ocp3.example.com:2379 --ca-file /etc/etcd/ca.crt --cert-file=/etc/origin/master/master.etcd-client.crt --key-file=/etc/origin/master/master.etcd-client.key cluster-health*
member 82c895b7b0de4330 is healthy: got healthy result from https://10.30.1.251:2379
member c8e7ac98bb93fe8c is healthy: got healthy result from https://10.30.3.74:2379
member f7bbfc4285f239ba is healthy: got healthy result from https://10.30.2.157:2379
cluster is healthy

Persistent volumes (pv) are OpenShift objects that allow for storage to be defined and then claimed by pods to allow for data persistence. Mounting of persistent volumes is done by using a persistent volume claim (pvc). This claim mounts the persistent storage to a specific directory within a pod referred to as the mountpath.

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: <name>
provisioner: kubernetes.io/cinder
parameters:
  availability: nova

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: <name>
provisioner: kubernetes.io/cinder
parameters:
  type: <volumetypename>
  availability: nova

$ oc new-app --docker-image registry.access.redhat.com/openshift3/mysql-55-rhel7 --name=db -e 'MYSQL_USER=myuser' -e 'MYSQL_PASSWORD=d0nth@x' -e 'MYSQL_DATABASE=persistent'

$ oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
db-1-6kn6c                 1/1       Running   0          5m

$ oc describe pod db-1-6kn6c | grep -i volumes -A3
Volumes:
  db-volume-1:
    Type:   EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:

$ oc get storageclass
NAME      TYPE
gold      kubernetes.io/cinder

$ cat db-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: db
 annotations:
   volume.beta.kubernetes.io/storage-class: gold
spec:
 accessModes:
  - ReadWriteMany
 resources:
   requests:
     storage: 10Gi

$ oc create -f db-claim.yaml
persistentvolumeclaim "db" created

$ oc get pvc
NAME      STATUS    VOLUME                                     CAPACITY
ACCESSMODES   AGE
db        Bound     pvc-cd8f0e34-02b6-11e7-b807-fa163e5c3cb8   10Gi       RWX
10s

$ oc volume dc/db --add --overwrite --name=db-volume-1 --type=persistentVolumeClaim --claim-name=db
deploymentconfig "db" updated

$ oc describe dc/db | grep -i volumes -A3
  Volumes:
   db-volume-1:
    Type:   PersistentVolumeClaim (a reference to a PersistentVolumeClaim
in the same namespace)
    ClaimName:  db
    ReadOnly:   false

Each project in OSP has a set of resource quotas which are set by default. Several of these values should be increased to allow the OCP stack to fit.

These numbers are for a basic general-purpose installation with low to moderate use. It allows for scaling up to 10 more application nodes. The correct values for a specific installation depends on the expected use. They are calculated using a detailed analysis of the actual resources needed and available.

Heat orchestration and the heat engine offer on-demand detection that can respond to automatic adding and removing nodes based upon the running workloads. The Heat templates integrate the RHOCP service with the Ceilometer monitoring service in RHOSP. While Ceilometer monitors the environment, it can signal the Heat stack to appropriate increase or decrease nodes upon workload requirements.

The installation of the Heat Templates is best approached using the RHOSP heat CLI tool in conjunction with the OpenShift Heat templates. A successful installation depends on the enablement of the rhel-7-server-openstack-10-rpms repository. Once enabled, the heat stack installation process can be initiated from any host that can access the RHOSP service using an RHOSP CLI client.

sudo subscription-manager repos --enable rhel-7-server-openstack-10-rpms
sudo yum -y install python-heatclient openshift-heat-templates

The template files are installed at /usr/share/openshift/heat/templates.

Heat uses one or more YAML input files to customize the stacks it creates.

This configuration uses an external DNS service as detailed in Appendix B and a dedicated loadbalancer created as part of the Heat stack. DNS records are created to direct inbound traffic for the masters and the OpenShift router through this loadbalancer. This configuration uses the flannel SDN. The Docker storage is set to a low value as this reference environment is for demonstration purposes only. When deploying in production environments, ensure to tune these values to accomodate for appropriate container density.

# Invoke:
# heat stack-create ocp3-heat \
#      -e openshift_parameters.yaml \
#      -e /usr/share/openshift-heat-templates/env_loadbalancer_dedicated.yaml \
#      -t /usr/share/openshift-heat-templates/openshift.yaml
#
parameters:
  # OpenShift service characteristics 1
  deployment_type: openshift-enterprise
  domain_name: "ocp3.example.com"
  app_subdomain: "apps.ocp3.example.com"
  lb_hostname: "devs"
  loadbalancer_type: dedicated
  openshift_sdn: flannel
  deploy_router: true
  deploy_registry: true

  # Number of each server type 2
  master_count: 3
  infra_count: 3
  node_count: 2

  # OpenStack network characteristics 3
  external_network: public_network
  internal_subnet: "172.22.10.0/24"
  container_subnet: "172.22.20.0/24"

  # DNS resolver and updates 4
  dns_nameserver: 10.x.x.130,10.x.x.29,10.x.x.19
  dns_update_key: <HMAC:MD5 string>

  # Instance access 5
  ssh_key_name: ocp3
  ssh_user: cloud-user

  # Image selection 6
  bastion_image: rhel7
  master_image: rhel7
  infra_image: rhel7
  node_image: rhel7
  loadbalancer_image: rhel7

  # Docker Storage controls 7
  master_docker_volume_size_gb: 10
  infra_docker_volume_size_gb: 20
  node_docker_volume_size_gb: 100

  # OpenStack user credentials 8
  os_auth_url: http://10.x.x.62:5000/v2.0
  os_username: <username>
  os_password: <password>
  os_region_name: <region name>
  os_tenant_name: <project name>

  # Red Hat Subscription information 9
  rhn_username: "<username>"
  rhn_password: "<password>"
  rhn_pool: '<pool id containing openshift>'

parameter_defaults:
  # Authentication service information 10
  ldap_url: "ldap://ad.example.com.com:389/cn=users,dc=example,dc=com?sAMAccountName"
  ldap_preferred_username: "sAMAccountName"
  ldap_bind_dn: "cn=openshift,cn=users,dc=example,dc=com"
  ldap_bind_password: "password"
  ldap_insecure: true

resource_registry: 11
  # Adjust path for each entry
  OOShift::LoadBalancer: /usr/share/openshift-heat-templates/loadbalancer_dedicated.yaml
  OOShift::ContainerPort: /usr/share/openshift-heat-templates/sdn_flannel.yaml
  OOShift::IPFailover: /usr/share/openshift-heat-templates/ipfailover_keepalived.yaml
  OOShift::DockerVolume: /usr/share/openshift-heat-templates/volume_docker.yaml
  OOShift::DockerVolumeAttachment: /usr/share/openshift-heat-templates/volume_attachment_docker.yaml
  OOShift::RegistryVolume: /usr/share/openshift-heat-templates/registry_ephemeral.yaml

Two of the hostnames generated by the heat stack installation are significant for users who need to find the service. The hostnames are generated from the domain_name and the lb_hostname parameters in the YAML file and from the heat stack name given on the CLI when the stack is created. This is to avoid naming conflicts if multiple stacks are created.

The instances that make up the Heat deployment get names composed from the stack name and domain. The master and infrastructure nodes are distinguished from each other by simple integer serial numbers. The nodes are handled differently as they can be created on demand when a load trigger event occurs. The heat stack assigns each node a random string to distingush them.

A listing of all the instance names can be found using nova list --field name.

These are the names developers and application users use to reach the OCP service. These two domain names must be registered in DNS and must point to the load-balancer. The load-balancer must be configured with the floating IP addresses of the master instances for port 8443 and with the addresses of the infrastructure nodes on ports 80 and 443 for access to the OCP service.