Container technology is rapidly changing the way organizations develop, deploy, and manage applications. Red Hat OpenShift Container Platform provides organizations with a reliable platform for deploying and scaling container based applications. This reference architecture will provide an example of how to deploy Red Hat OpenShift Container Platform on HPE ProLiant DL baremetal servers using Ansible Tower, Red Hat Satellite, and HPE OneView.

Intended Audience

This document is intended for systems engineers, systems administrators, architects, and installers responsible for installing and maintaining Red Hat OpenShift Container Platform. The reader of this document should be familiar with Red Hat OpenShift Container Platform, Red Hat Satellite, Ansible Tower, Red Hat Enterprise Linux®, HPE ProLiant servers, and networking equipment.

Ansible is a powerful configuration management and orchestration tool, which can be used to automate complicated infrastructure or application deployment tasks.

At it’s core, ansible ships with hundreds of modules to manage different host types and software. These hosts can be traditional servers running Red Hat Entperise Linux and other operating systems, as well as an impressive count of network switches and storage arrays. Modules are also included to interact with APIs (application program interface) directly, or wrapped up in a separate module to present a standard set of arguments. For example, the openstack (os_) modules are provided to orchestrate an OpenStack cloud.

An ansible playbook is built by defining tasks. A given task will call a module with the desired set of arguments. The playbook is then run against an inventory which specifies the hosts to run those tasks.

Ansible Tower takes those ansible constructs, and presents them in a clean, modern web GUI. This reference architecture used Ansible Tower by Red Hat version 3.1.1 to orchestrate the provisioning of HPE baremetal ProLiant DL servers, integration with HPE OneView 3.1, installation of Red Hat Enterprise Linux 7.3, and deployment of Red Hat OpenShift Container Platform 3.5 with Container-native storage. This was accomplished using Ansible Tower inventory, templates, roles, tasks,and workflows, to execute the multiple ansible playbooks required for this reference architecture. The installation of Ansible Tower is beyond the scope of this reference architecture. Refer to the Ansible Tower by Red Hat documentation at http://docs.ansible.com/ansible-tower/index.html for complete instructions on the installation and configuration of Ansible Tower by Red Hat.

Dashboard

Figure 3: Ansible Tower Dashboard

Inventory

Figure 4: Ansible Tower Inventory

An inventory is a list of targets to run a given playbook against. Inventories can be statically defined, or leverage dynamic inventories. A dynamic inventory allows the operator to point Ansible Tower at a source which can change over time, such as a public or private cloud, to generate a list of targets.

Projects

A project refers to a software repository (typically git) where playbooks are stored.

Templates

Figure 5: Ansible Tower Templates

Templates tie inventory and projects together. A template will specific which playbook to run from a given project, and execute it against a specific set of hosts from an inventory.

Workflows

Figure 6: Ansible Tower Workflow

Workflows are a way to chain multiple ansible playbooks together. Different actions can be taken whether a given playbook has succeeded or failed.

A collection of ansible playbooks were created for this reference architecture and synced to Ansible Tower. These playbooks are detailed in a later chapter and available in a github repo.

In addition, the Red Hat OpenShift Container Platform installer is based on Ansible, and an advanced install exposes this to a systems administrator. Inventory and variables can be configured and the playbook run using the standard ansible-playbook command. The use of these playbooks is well documented on the linked web site. This reference architecture only covers integrating and running those playbooks in Tower.

OneView is HPE’s tool to manage and control many aspects of physical hardware. This includes firmware, BIOS settings, on-board RAID configuration and boot order. OneView supports both blade and rackmount servers. As mentioned previously, this reference architecture is running HPE OneView 3.1 and the OneView management appliance is running on a Red Hat Enterprise Linux 7.3 KVM server. KVM is a newly supported platform for the OneView management appliance with the release of HPE OneView 3.1.

Dashboard

The OneView Dashboard illustrates a highlevel status of all servers, storage, and network equipment registered with OneView. from the Dashboard, an administrator can drill down to find detailed information on alerts.

Figure 7: HPE OneView Dashboard

Servers

The servers view shows the servers that have been registered with OneView. Once a server is registered, OneView provides a management interface for the server and reports on the status and health of the server.

Figure 8: HPE OneView Servers

Server Profiles

Server Profiles are used by OneView to configure a server. When a profile is applied to a server the configuration options specified in the server profile are applied to the server, this includes Firmware levels, BIOS options, boot order, and disk configuration.

Figure 9: HPE OneView Server Profiles

Firmware Management

OneView provides the ability to store firmware bundles that can then be applied to servers via a server profile. Firmware bundles can be standard bundles that are available through the Service Pack for ProLiant or an administrator can create a custom firmware bundle for specifc configurations. In this reference architecture the firmware bundle used was from the Service Pack for ProLiant version 2017.04.0.

Figure 10: HPE OneView Firmware

OneView Ansible Playbooks

HPE has created a collection of ansible modules to interact with a OneView server. They can be found on github here. There are module for various actions that can be performed via the OneView GUI.

In particular, this reference architecture leverages the modules for the following tasks:

Red Hat Satellite is the best way to manage your Red Hat infrastructure.

Red Hat Satellite manages the life cycle of an operating system, from initial deployment to ongoing updates. It provides a local mirror of all available Red Hat packages for faster software delivery inside the data center.

A built in DNS server provides automated DNS entry when a host is created, and deletion when the host is decommissioned.

Satellite 6.2 was used in this reference architecture to provision the operating systems, provide access to the required repositories, and to provide DHCP and DNS services. The following sections provide an overview of the Satellite server configuration to support the deployment of Red Hat OpenShift Container Platform 3.5 on baremetal HPE ProLiant DL servers. Installation of the Satellite server is beyond the scope of this document, please refer to the official Satellite document for installation and configuration of Satellite servers.

Configuration to Support OpenShift Container Platform

Repositories

The following repositories are required to be available from the Red Hat Satellite server to deploy OpenShift with Container-native storage:

Figure 11: Satellite Products

Content Views

Red Hat Satellite Content views are used to manage the selection of content available to the hosts registered in Satellite. The Content views provide lifecycle management by maintaining repositories with specific software versions available for deployment. In the figure below, titled Satellite Content View, the required yum repositories for OpenShift have been added to the _rhel7-ocp3-5 content view:

Figure 12: Satellite Content View

Lifecycle Environment

To manage the promotion of content views between development and production, a lifecycle environment is used. A content view is published to a specific environment. When that view has been tested and vetted in an environment, it can then be promoted to the next level e.g. production.

A lifecycle environment named ocp-dev was created and associated with the rhel7-ocp-3-5 content view.

Figure 13: Lifecycle Environment

Subnets

Subnets are defined in Red Hat Satellite so when a host is created, it is allocated an IP from a set range along with the proper netmask and gateway. In this environment, a subnet with the name hpecloud_ext was created for the 10.19.20.128/25 network.

Figure 14: Subnet

Activation Keys

When a new host is deployed for the first time, an activation key is typically supplied. This key is an object in Red Hat Satellite that allows a host to automatically register and attach to any required subscriptions and software channels. An activation called hpe-ocp was created so that hosts would have access to the OpenShift Container Platform RPMs.

Figure 15: Activation Key Details

Figure 16: Activation Key Product Content

Kickstart Template

Red Hat Satellite dynamically generates a kickstart configuration for every host that it builds. This allows for operating system installation via PXE boot, using tftp to serve the required kernel and disk image files.

Satellite ships with a number of initial templates. The Satellite Kickstart Default template is suitable for most standard installations. To allow Ansible Tower to manage a deployed host, the template must be modified slightly to inject an SSH key.

The Satellite Kickstart Default was cloned as HPE-Kickstart and modified to install an authorized key for the root user. Below is the relevant portion that was modified:

#update local time
echo "updating system time"
/usr/sbin/ntpdate -sub <%= @host.params['ntp-server'] || '0.fedora.pool.ntp.org' %>
/usr/sbin/hwclock --systohc

# deploy ansible tower key
mkdir /root/.ssh/
chmod 700 /root/.ssh
echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRcFhowHWK8ZfzTRFZcqs5BbFqryBXWwXKj1HI4dDipPTNka0GD6+qeMZiLgNHQ2bn24HXoWSzWRyKSU+cDD5LWpPq9sPRLTO/japC5YQfeOMQbkSnV4GaglX50oqcI1whSovCXNL0JtxDg8YoWQrhqpM+r3nD+IATOFLeB/kk3Vuc1UHAZvO0Ww9bIw32tK4hOtB2CWsZr3T0xe/k5OZF5v9Y21aiLA//p655N0LrVF08EqOmPQi93EUWTLYvZXQyLFuu80PdCIDdhvU1mrQj5iBFDJrQiKSL02zRKr6JDKsvrPyb750R5HsOohEHQlD3KsONkJNnzphtVHM1dkf3 dcritch@sputnik.xana.du' >> /root/.ssh/authorized_keys
echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5/HWM5BBBG+Oj8teXxc0A7bYT7ke6qttnnAm7e90uaom5tZvjscQHUINo1gvZSv5vMgV4x3Rgm5+/W+3FZEkR9BlymjltWOw5KBu+XvDcJnhUGKA2gLmaEclQsvB4TVcTv1m3ALa7W0ReqHR64geu638cNLrXiWRXdyttLNq28qQsSIf/m7Pv5By/jOfmc/xKXbajDuhMa/NsJ1XlHHEljhb2cl/mtkt2TcWht/2nZF1ozAeNDnhHzDLrtmYNqX0qKyrjF+RHH2t6hBF8iaf/8IxLdgycAxpcY2IPmp2p8RuO4Fs1j4xw/gPwRotYQQ3i0zNdkdMfr5NTQoBjpRrf root@dc-dev.cloud.lab.eng.bos.redhat.com' >> /root/.ssh/authorized_keys

<%= snippet "subscription_manager_registration" %>

Partition Table

Kickstart templates may include a parition table, but Red Hat Satellite also allows for the creation of custom, dynamic partition tables. For the OpenShift nodes, the OS volume that is configured through HPE OneView is partitioned in Satellite to provide a dedicated docker volume group:

<%#
kind: ptable
name: Kickstart default
oses:
- CentOS 5
- CentOS 6
- CentOS 7
- Fedora 16
- Fedora 17
- Fedora 18
- Fedora 19
- Fedora 20
- RedHat 5
- RedHat 6
- RedHat 7
%>
zerombr
clearpart --all --initlabel
#autopart
#ignoredisk --only-use=sda
part pv.192 --fstype="lvmpv" --ondisk=sda --size=131072
part /boot --fstype="xfs" --ondisk=sda --size=500
volgroup vgos --pesize=4096 pv.192
logvol /  --fstype="xfs" --size=16128 --name=lvroot --vgname=vgos
logvol swap  --fstype="swap" --size=16128 --name=swap --vgname=vgos
logvol /var  --fstype="xfs" --size=65536 --name=lvvar --vgname=vgos
part pv.193 --fstype="lvmpv" --ondisk=sda --size=131072 --grow
volgroup docker-vg --pesize=4096 pv.193

The docker-vg is later used in the predeploy Ansible playbook configure docker storage. More details can be found in the official OpenShift docs.

Host Groups

Common combinations of subnets, operating system and activation keys can be expressed as a Hostgroup in Red Hat Satellite. For hosts deployed in this reference architecture, a hostgroup called hpe-ocp was created to incorporate the various customizations outlined for host creation.

This section provides a detailed walk-through of the Ansible playbooks created for this reference architecture and how the Ansible Tower Workflow ties it all together.

## required for provisoning
ocp-master1.hpecloud.test ansible_host=192.168.1.173 ilo_ip=192.168.1.136
ocp-master2.hpecloud.test ansible_host=192.168.1.174 ilo_ip=192.168.1.137
ocp-master3.hpecloud.test ansible_host=192.168.1.175 ilo_ip=192.168.1.138
ocp-infra0.hpecloud.test ansible_host=192.168.1.172 ilo_ip=192.168.1.135
ocp-infra1.hpecloud.test ansible_host=192.168.1.170 ilo_ip=192.168.1.133
ocp-infra2.hpecloud.test ansible_host=192.168.1.171 ilo_ip=192.168.1.134
ocp-cns1.hpecloud.test	ansible_host=192.168.1.176 ilo_ip=192.168.1.140
ocp-cns2.hpecloud.test 	ansible_host=192.168.1.177 ilo_ip=192.168.1.141
ocp-cns3.hpecloud.test 	ansible_host=192.168.1.178 ilo_ip=192.168.1.142
[cns]
ocp-cns1.hpecloud.test
ocp-cns2.hpecloud.test
ocp-cns3.hpecloud.test

## standard openshift-ansible inventory
[OSEv3:children]
masters
nodes
etcd
lb
nfs

[masters]
ocp-master1.hpecloud.test
ocp-master2.hpecloud.test
ocp-master3.hpecloud.test

[etcd]
ocp-master1.hpecloud.test
ocp-master2.hpecloud.test
ocp-master3.hpecloud.test


[lb]
ocp-infra0.hpecloud.test
[nfs]
ocp-infra0.hpecloud.test

[nodes]
ocp-master1.hpecloud.test
ocp-master2.hpecloud.test
ocp-master3.hpecloud.test
ocp-infra1.hpecloud.test openshift_node_labels="{'region': 'infra', 'zone': 'west'}"
ocp-infra2.hpecloud.test openshift_node_labels="{'region': 'infra', 'zone': 'east'}"
ocp-cns1.hpecloud.test 	openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
ocp-cns2.hpecloud.test 	openshift_node_labels="{'region': 'primary', 'zone': 'west'}"
ocp-cns3.hpecloud.test 	openshift_node_labels="{'region': 'primary', 'zone': 'west'}"

---
## satellite vars (change to match target environment)
satellite_user: "admin"
satellite_url: "https://192.168.1.211"
location_id: 2
organization_id: 1
environment_id: 1
hostgroup_id: 1
ansible_domain: hpecloud.test
ntp_server: "clock.corp.redhat.com"
rndc_key: "r/24yYpTOcnIxvXul+xz3Q=="
ilo_username: "Administrator"
bios_settings:
- id: "CustomPostMessage"
  value: "!! automate the planet !!"
- id: "MinProcIdlePkgState"
  value: "NoState"
- id: "EnergyPerfBias"
  value: "MaxPerf"
- id: "PowerProfile"
  value: "MaxPerf"
- id: "IntelQpiPowerManagement"
  value: "Disabled"
- id: "PowerRegulator"
  value: "StaticHighPerf"
- id: "MinProcIdlePower"
  value: "NoCStates"
storage_cfg:
  controllers:
    - deviceSlot: "Embedded"
      importConfiguration: false
      initialize: true
      mode: "RAID"
      logicalDrives:
      - bootable: true
        driveNumber: 1
        driveTechnology: "SasHdd"
        name: "os"
        numPhysicalDrives: 2
        raidLevel: "RAID1"
## oneview vars
oneview_auth:
  ip: "192.168.1.233"
  username: "Administrator"
  api_version: 300

---
storage_cfg:
  controllers:
    - deviceSlot: "Embedded"
      importConfiguration: false
      initialize: true
      mode: "RAID"
      logicalDrives:
      - bootable: true
        driveNumber: 1
        driveTechnology: "SasSsd"
        name: "os"
        numPhysicalDrives: 2
        raidLevel: "RAID1"
      - bootable: false
        driveNumber: 2
        driveTechnology: "SasHdd"
        name: "cns"
        numPhysicalDrives: 12
        raidLevel: "RAID6"

ansible_ssh_user: root
deployment_type: openshift-enterprise
openshift_master_identity_providers: [{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
openshift_master_cluster_method: native
openshift_master_cluster_hostname: openshift-master.hpecloud.test
openshift_master_cluster_public_hostname: openshift-master.hpecloud.test
openshift_use_openshift_sdn: true
openshift_hosted_registry_storage_kind: nfs
openshift_hosted_registry_storage_access_modes: ['ReadWriteMany']
openshift_hosted_registry_storage_nfs_directory: /exports
openshift_hosted_registry_storage_nfs_options: '*(rw,root_squash)'
openshift_hosted_registry_storage_volume_name: registry
openshift_hosted_registry_storage_volume_size: 10Gi
openshift_hosted_metrics_deploy: false
openshift_hosted_logging_deploy: false
openshift_master_default_subdomain: paas.hpecloud.test
openshift_master_htpasswd_users: {'davidc': '$apr1$znPqASZl$WjOV1pFe4diJJPhjZgW2q1', 'kbell': '$apr1$.WBeJqbh$aic2L/5dxbnkdoEC0UWiT.', 'gluster': '$apr1$OMdwuyb6$bF2f3hSfwsE9XOyCaFEOP.'}

$ htpasswd -n gluster
New password:
Re-type new password:
gluster:$apr1$OMdwuyb6$bF2f3hSfwsE9XOyCaFEOP.

---
- name: set ilo password
  set_fact:
    ilo_pw: "P@ssw0rd"
- name: set oneview password
  set_fact:
    oneview_pw: "P@ssw0rd"
- name: set satellite password
  set_fact:
    satellite_pw: "P@ssw0rd"
- name: set root password
  set_fact:
    root_pw: "P@ssw0rd"
- name: set gluster admin password
  set_fact:
    gluster_pw: "n0tP@ssw0rd"
- name: set rndc_key
  set_fact:
    rndc_key: "r/42yYsCOcnIxvXul+xz3Q=="

$ ansible-vault encrypt passwords.yaml --output=roles/passwords/tasks/main.yaml

---
## bare metal provisioning

- name: get passwords
  hosts: all
  remote_user: root
  gather_facts: false
  roles:
    - ../roles/passwords

- name: baremetal provisioning
  hosts: all
  remote_user: root
  gather_facts: false
  roles:
    - ../roles/provisioning
  environment:
    PYTHONPATH: "$PYTHONPATH:/usr/share/ansible"
    ONEVIEWSDK_IP: "{{ oneview_auth.ip }}"
    ONEVIEWSDK_USERNAME: "{{ oneview_auth.username }}"
    ONEVIEWSDK_PASSWORD: "{{ oneview_pw }}"
    ONEVIEWSDK_API_VERSION: "{{ oneview_auth.api_version }}"

├── roles
│   └── provisioning
│       └── tasks
│           ├── addserver.yaml
│           ├── facts.yaml
│           ├── firmware.yaml
│           ├── main.yaml
│           ├── poweron.yaml
│           ├── profiles.yaml
│           └── satellite.yaml

---
# set some vars
- include: ./facts.yaml
# add the server to oneview
- include: ./addserver.yaml
# get firmware version
- include: ./firmware.yaml
# create profile templates -> profiles
- include: ./profiles.yaml
# create the host in satellite
- include: ./satellite.yaml
# power on server to kickstart over pxe
- include: ./poweron.yaml

---
- set_fact:
    ilo_name: "{{ inventory_hostname.split('.').0 }}-ilo"
  tags:
  - ilo

- name: get facts from ilo
  hpilo_facts:
    host: "{{ ilo_ip }}"
    login: "{{ ilo_username }}"
    password: "{{ ilo_pw }}"
  register: ilo_facts
  delegate_to: localhost
  tags:
  - ilo

- name: set PXE MAC address
  set_fact:
    pxe_mac: "{{ ilo_facts.ansible_facts.hw_eth0.macaddress }}"
  tags:
  - ilo

## IP + 20, disabled after kickstart
- name: set PXE IP of new host
  set_fact:
    pxe_ip: "{{ ansible_host.split('.').0 }}.{{ ansible_host.split('.').1 }}.{{ ansible_host.split('.').2 }}.{{ ansible_host.split('.').3|int + 20 }}"

- name: set slave0 MAC address
  set_fact:
    slave0_mac: "{{ ilo_facts.ansible_facts.hw_eth4.macaddress }}"
  tags:
  - ilo

- name: set slave1 MAC address
  set_fact:
    slave1_mac: "{{ ilo_facts.ansible_facts.hw_eth6.macaddress }}"
  tags:
  - ilo

---
- name: check if server is in oneview
  oneview_server_hardware_facts:
    name: "{{ ilo_name }}"
  register: server_facts_exists
  delegate_to: localhost
  tags:
  - mkserver

- set_fact:
    server_exists: false
  when:
  - server_facts_exists.ansible_facts.server_hardwares|length == 0
  tags:
  - mkserver
- set_fact:
    server_exists: true
  when:
  - server_facts_exists.ansible_facts.server_hardwares|length > 0
  tags:
  - mkserver

- name: add a server to oneview
  oneview_server_hardware:
    state: present
    data:
      hostname: "{{ ilo_ip }}"
      username: "{{ ilo_username }}"
      password: "{{ ilo_pw }}"
      force: false
      licensingIntent: "OneViewNoiLO"
      configurationState: "Managed"
  register: server_facts_new
  delegate_to: localhost
  tags:
  - mkserver
  when:
  - server_exists == false
- set_fact:
    server_facts: "{{ server_facts_new.ansible_facts.server_hardware }}"
  tags:
  - mkserver
  when:
  - server_exists == false
- set_fact:
    server_facts: "{{ server_facts_exists.ansible_facts.server_hardwares.0 }}"
  tags:
  - mkserver
  when:
  - server_exists == true

---
- name: get fw driver info
  oneview_firmware_driver_facts:
    params:
      sort: 'releaseDate:descending'
      start: 0
      count: 1
  register: fw_info
  delegate_to: localhost
  tags:
  - firmware
- name: set fw version
  set_fact:
    fw_baseline_uri: "{{ fw_info['ansible_facts']['firmware_drivers'].0['uri'] }}"
  tags:
  - firmware

---
- set_fact:
      model: "{{ server_facts.model }}"
      short_model: "{{ server_facts.shortModel }}"
      dl_model: "{{ server_facts.shortModel.split(' ').0 }}"
  tags:
  - model

- name: create a service profile and assign it to a node
  oneview_server_profile:
    state: present
    data:
      name: "{{ inventory_hostname.split('.').0 }}"
      server_hardware: "{{ ilo_name }}"
      description: "OpenShift Nodes - {{ short_model }}"
      serverHardwareTypeName: "{{ short_model }} 1"
      boot:
        manageBoot: true
        order: ["PXE", "CD", "USB", "HardDisk"]
      bootMode:
        manageMode: true
        mode: "BIOS"
        pxeBootPolicy: null
      bios:
        manageBios: true
        overriddenSettings: "{{ bios_settings }}"
      firmware:
        firmwareBaselineUri: "{{ fw_baseline_uri }}"
        #firmwareInstallType: "FirmwareOnly"
        firmwareInstallType: "FirmwareOnlyOfflineMode"
        forceInstallFirmware: false
        manageFirmware: true
      localStorage: "{{ storage_cfg }}"
  register: output
  delegate_to: localhost
  tags:
  - templates

- name: "Get Host ID from Satellite"
  uri:
    url: "{{ satellite_url }}/api/hosts/?search=name={{ inventory_hostname }}"
    user: "{{ satellite_user }}"
    password: "{{ satellite_pw }}"
    headers:
      Content-Type: "application/json"
      Accept: "application/json"
    force_basic_auth: yes
    validate_certs: False
    body_format: json
    return_content: yes
    status_code: 200
  ignore_errors: false
  delegate_to: localhost
  register: check_host_response
  tags:
  - satellite
- debug:
    var: check_host_response
    verbosity: 2

- set_fact:
    host_in_satellite: true
  when:
  - check_host_response.json.subtotal == 1
  tags:
  - satellite
- set_fact:
    host_in_satellite: false
  when:
  - check_host_response.json.subtotal == 0
  tags:
  - satellite

- name: "Add Host to Satellite"
  uri:
    url: "{{ satellite_url }}/api/hosts/"
    method: POST
    user: "{{ satellite_user }}"
    password: "{{ satellite_pw }}"
    headers:
      Content-Type: "application/json"
      Accept: "application/json"
    force_basic_auth: yes
    validate_certs: False
    return_content: yes
    body_format: json
    body:
      host:
        name: "{{ inventory_hostname }}"
        location_id: "{{ location_id }}"
        organization_id: "{{ organization_id }}"
        environment_id: "{{ environment_id }}"
        hostgroup_id: "{{ hostgroup_id }}"
        build: true
        enabled: true
        managed: true
        root_pass: "{{ root_pw }}"
        overwrite: true
        interfaces_attributes:
          -
            mac: "{{ pxe_mac }}"
            primary: false
            type: "interface"
            identifier: "pxe0"
            provision: true
            ip: "{{ pxe_ip }}"
            managed: true
            subnet_id: 1
            domain_id: 1
          -
            mac: "{{ slave0_mac }}"
            primary: true
            type: "bond"
            ip: "{{ ansible_host }}"
            mode: "active-backup"
            identifier: "bond0"
            attached_devices: "slave0,slave1"
          -
            mac: "{{ slave0_mac }}"
            primary: false
            type: "interface"
            identifier: "slave0"
          -
            mac: "{{ slave1_mac }}"
            primary: false
            type: "interface"
            identifier: "slave1"
    status_code: 201
  ignore_errors: false
  delegate_to: localhost
  register: add_host_response
  when:
  - host_in_satellite == false
  tags:
  - satellite

---
- name: power on server
  oneview_server_hardware:
    state: power_state_set
    data:
      name: "{{ ilo_name }}"
      powerStateData:
        powerState: "On"
        powerControl: "MomentaryPress"
  register: output
  delegate_to: localhost
  tags:
  - poweron
- name: wait for server to become available
  local_action: wait_for port=22 host='{{ ansible_host }}' delay=30 timeout=3600
  tags:
  - poweron

---
## common tasks for all servers

- name: common
  hosts: all
  remote_user: root
  roles:
    - ../roles/common

├── roles
│   ├── common
│   │   ├── handlers
│   │   │   └── main.yaml
│   │   ├── tasks
│   │   │   ├── main.yaml
│   │   │   ├── motd.yaml
│   │   │   ├── ntp.yaml
│   │   │   └── packages.yaml
│   │   └── templates
│   │       ├── chrony.j2
│   │       └── motd.j2

---
- include: packages.yaml
  tags:
  - packages
- include: ntp.yaml
  tags:
  - ntp
- include: motd.yaml
  tags:
  - motd

---
- name: install common packages
  yum:
    name: '{{ item }}'
    state: latest
  tags:
    - rhn
  with_items:
  - screen
  - tmux
  - nfs-utils
  - sg3_utils
  - policycoreutils-python
  - '@network-file-system-client'

---
- name: install chrony
  yum:
    name: chrony
    state: latest

- name: config chronyd
  template:
    src: chrony.j2
    dest: /etc/chrony.conf
  notify:
  - restart chronyd

server {{ ntp_server }} iburst
stratumweight 0
driftfile /var/lib/chrony/drift
rtcsync
makestep 10 3
bindcmdaddress 127.0.0.1
bindcmdaddress ::1
keyfile /etc/chrony.keys
commandkey 1
generatecommandkey
noclientlog
logchange 0.5
logdir /var/log/chrony

---
- name: restart chronyd
  service:
    name: chronyd
    state: restarted
    enabled: true

---
- name: get build date
  stat:
    path: /root/anaconda-ks.cfg
  register: build_stat
- name: convert to a nice string for the template
  command: date --date='@{{ build_stat.stat.ctime }}' +'%Y.%m.%d @ %H:%M %Z'
  register: pretty_date
- name: create motd file
  template:
    src: motd.j2
    dest: /etc/motd

welcome to {{ ansible_fqdn }} built on {{ pretty_date.stdout }}

---
- name: run predeployment tasks
  hosts: all
  gather_facts: true
  remote_user: root
  roles:
    - ../roles/passwords
    - ../roles/predeploy
  environment:
    PYTHONPATH: "$PYTHONPATH:/usr/share/ansible"

---
- include: dns.yaml
  tags:
  - dns
- include: packages.yaml
  tags:
  - packages
- include: docker.yaml
  tags:
  - docker

---
- name: get IP of load balancer
  set_fact:
    lb_ip: "{{ hostvars[groups['lb'][0]]['ansible_default_ipv4']['address'] }}"
  run_once: true
  delegate_to: localhost

- name: get IP of DNS server via the load balancer
  set_fact:
    dns_server: "{{ hostvars[groups['lb'][0]]['ansible_dns']['nameservers'].0 }}"
  run_once: true
  delegate_to: localhost

- name: create DNS entry for master
  nsupdate:
    state: present
    key_name: "rndc-key"
    key_secret: "{{ rndc_key }}"
    server: "{{ dns_server }}"
    zone: "{{ ansible_domain }}"
    record: "{{ openshift_master_cluster_public_hostname.split('.').0 }}"
    value: "{{ lb_ip }}"
  run_once: true
  delegate_to: localhost

- set_fact:
    region: "{{ openshift_node_labels.region }}"
  when:
  - openshift_node_labels is defined

- name: set subdomain record
  set_fact:
    subdomain: "{{ openshift_master_default_subdomain.split('.').0 }}"
  run_once: true
  delegate_to: localhost

- name: define router IP
  set_fact:
    router_ip: '{{ ansible_default_ipv4.address }}'
  delegate_to: localhost
  when:
  - region|default('none') == 'infra'

- name: blacklist non-router IPs
  set_fact:
    router_ip: '0.0.0.0'
  delegate_to: localhost
  when:
  - region|default('none') != 'infra'

- name: create router IP list
  set_fact:
    router_ips: "{{ groups['all']|map('extract', hostvars, 'router_ip')|list|unique|difference(['0.0.0.0']) }}"
  delegate_to: localhost
  run_once: true

- name: update wildcard DNS entry
  nsupdate:
    state: present
    key_name: "rndc-key"
    key_secret: "r/42yYsCOcnIxvXul+xz3Q=="
    server: "{{ dns_server }}"
    zone: "{{ ansible_domain }}"
    record: "*.{{ subdomain }}"
    value: "{{ router_ips }}"
  delegate_to: localhost
  run_once: true

---
- name: install ocp prereq packages
  package:
    name: '{{ item }}'
    state: latest
  with_items:
  - wget
  - git
  - net-tools
  - bind-utils
  - iptables-services
  - bridge-utils
  - bash-completion
  - kexec-tools
  - sos
  - psacct

---
- name: configure docker-storage-setup
  lineinfile:
    path: /etc/sysconfig/docker-storage-setup
    line: "VG=docker-vg"
    create: yes
  register: dss

---
## clean up!

- name: get passwords
  hosts: all
  remote_user: root
  gather_facts: false
  roles:
    - ../roles/passwords

- name: delete nodes from satellite and oneview
  hosts: all
  remote_user: root
  gather_facts: false
  roles:
    - ../roles/cleanup
  environment:
    PYTHONPATH: "$PYTHONPATH:/usr/share/ansible"
    ONEVIEWSDK_IP: "{{ oneview_auth.ip }}"
    ONEVIEWSDK_USERNAME: "{{ oneview_auth.username }}"
    ONEVIEWSDK_PASSWORD: "{{ oneview_pw }}"
    ONEVIEWSDK_API_VERSION: "{{ oneview_auth.api_version }}"

├── roles
│   ├── cleanup
│   │   └── tasks
│   │       ├── dns.yaml
│   │       ├── facts.yaml
│   │       ├── main.yaml
│   │       ├── poweroff.yaml
│   │       ├── profiles.yaml
│   │       ├── rmserver.yaml
│   │       └── satellite.yaml

---
- include: ./facts.yaml
- include: ./poweroff.yaml
- include: ./satellite.yaml
- include: ./dns.yaml
- include: ./profiles.yaml
- include: ./rmserver.yaml

---
- set_fact:
    ilo_name: "{{ inventory_hostname.split('.').0 }}-ilo"
  tags:
  - ilo
- name: check if server is in oneview
  oneview_server_hardware_facts:
    name: "{{ ilo_name }}"
  register: server_facts_exists
  delegate_to: localhost
  tags:
  - mkserver
- set_fact:
    server_exists: false
  when:
  - server_facts_exists.ansible_facts.server_hardwares|length == 0
  tags:
  - mkserver
- set_fact:
    server_exists: true
  when:
  - server_facts_exists.ansible_facts.server_hardwares|length > 0
  tags:
  - mkserver
- set_fact:
    server_facts: "{{ server_facts_exists.ansible_facts.server_hardwares.0 }}"
  tags:
  - mkserver
  when:
  - server_exists == true

- set_fact:
      model: "{{ server_facts.model }}"
      short_model: "{{ server_facts.shortModel }}"
      dl_model: "{{ server_facts.shortModel.split(' ').0 }}"
  when:
  - server_exists == true

- name: get uri of profile template
  oneview_server_profile_template_facts:
    params:
      filter: name='OCP-{{ dl_model }}'
  register: profile_output
  delegate_to: localhost
  when:
  - server_exists == true

- set_fact:
    template_uri: "{{ profile_output.ansible_facts.server_profile_templates.0.uri }}"
  when:
  - server_exists == true
  - profile_output.ansible_facts.server_profile_templates|length > 0

---
- name: power off server
  oneview_server_hardware:
    state: power_state_set
    data:
      name: "{{ ilo_name }}"
      powerStateData:
        powerState: "Off"
        powerControl: "PressAndHold"
  register: output
  delegate_to: localhost
  when:
  - server_exists == true

---
- name: "Get Host ID from Satellite"
  uri:
    url: "{{ satellite_url }}/api/hosts/?search=name={{ inventory_hostname }}"
    user: "{{ satellite_user }}"
    password: "{{ satellite_pw }}"
    headers:
      Content-Type: "application/json"
      Accept: "application/json"
    force_basic_auth: yes
    validate_certs: False
    return_content: yes
    status_code: 200
  ignore_errors: false
  register: check_host_response
  delegate_to: localhost

- name: set host ID
  set_fact:
    host_id: "{{ check_host_response.json.results.0.id }}"
  when:
  - check_host_response.json.subtotal == 1

- name: "delete host from satellite"
  uri:
    url: "{{ satellite_url }}/api/hosts/{{ host_id }}"
    user: "{{ satellite_user }}"
    password: "{{ satellite_pw }}"
    method: DELETE
    headers:
      Content-Type: "application/json"
      Accept: "application/json"
    force_basic_auth: yes
    validate_certs: False
    return_content: yes
    status_code: 200
  ignore_errors: false
  register: rm_host_response
  delegate_to: localhost
  when:
  - check_host_response.json.subtotal == 1

---
- name: get IP of DNS server via the load balancer
  set_fact:
    dns_server: "{{ satellite_url.split('//').1 }}"
  run_once: true
  delegate_to: localhost

- name: set subdomain record
  set_fact:
    subdomain: "{{ openshift_master_default_subdomain.split('.').0 }}"
  run_once: true
  delegate_to: localhost

- name: remove DNS entry for master
  nsupdate:
    state: absent
    key_name: "rndc-key"
    key_secret: "{{ rndc_key }}"
    server: "{{ dns_server }}"
    zone: "{{ ansible_domain }}"
    record: "{{ openshift_master_cluster_public_hostname.split('.').0 }}"
  run_once: true
  delegate_to: localhost


- name: delete wildcard DNS entries
  nsupdate:
    state: absent
    key_name: "rndc-key"
    key_secret: "{{ rndc_key }}"
    server: "{{ dns_server }}"
    zone: "{{ ansible_domain }}"
    record: "*.{{ subdomain }}"
  run_once: true
  delegate_to: localhost

---
- name: unassign server profile
  oneview_server_profile:
    state: absent
    data:
      name: "{{ inventory_hostname.split('.').0 }}"
      server_template: "OCP-{{ dl_model }}"
      server_hardware: "{{ ilo_name }}"
  register: output
  delegate_to: localhost
  when:
  - server_exists == true

---
- name: remove a server from oneview
  oneview_server_hardware:
    state: absent
    data:
      name: "{{ ilo_name }}"
      force: false
      licensingIntent: "OneView"
      configurationState: "Managed"
  register: output
  delegate_to: localhost
  tags:
  - mkserver
  when:
  - server_exists == true

---
- name: deploy container native storage w/ gluster
  hosts: all
  gather_facts: true
  remote_user: root
  roles:
    - ../roles/passwords
    - ../roles/cns
  environment:
    PYTHONPATH: "$PYTHONPATH:/usr/share/ansible"

├── roles
│   ├── cns
│   │   ├── tasks
│   │   │   ├── cnsdeploy.yaml
│   │   │   ├── disks.yaml
│   │   │   ├── heketitopo.yaml
│   │   │   ├── iptables.yaml
│   │   │   ├── main.yaml
│   │   │   ├── oc.yaml
│   │   │   └── packages.yaml
│   │   └── templates
│   │       └── topology-hpe.j2

---
- include: disks.yaml
  when:
  - "'cns' in group_names"
  tags:
  - disks

- include: iptables.yaml
  when:
  - "'cns' in group_names"
  tags:
  - iptables

- include: packages.yaml
  when:
  - "ansible_fqdn == groups['masters'][0]"

- include: oc.yaml
  when:
   - "ansible_fqdn == groups['masters'][0]"

- include: heketitopo.yaml
  when:
  - "ansible_fqdn == groups['masters'][0]"
  tags:
  - json

- include: cnsdeploy.yaml
  when:
  - "ansible_fqdn == groups['masters'][0]"
  tags:
  - cnsdeploy

---
- name: nuke sdb on cns nodes for gluster
  command: sgdisk -Z /dev/sdb

---
- name: required gluster ports
  lineinfile:
    dest: /etc/sysconfig/iptables
    state: present
    insertbefore: 'COMMIT'
    line: "{{ item }}"
  with_items:
  - '-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 24007 -j ACCEPT'
  - '-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 24008 -j ACCEPT'
  - '-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2222 -j ACCEPT'
  - '-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m multiport --dports 49152:49664 -j ACCEPT'

- name: reload iptables
  systemd:
    name: iptables
    state: reloaded

---

- name: install cns and heketi packages
  package:
    name: '{{ item }}'
    state: latest
  with_items:
  - cns-deploy
  - heketi-client

---
- name: loging as cns cluster admin
  command: oc login -u system:admin -n default

- name:  add cluster admin role for gluster
  command: oadm policy add-cluster-role-to-user cluster-admin gluster

- name: login as gluster user
  command: oc login -u gluster -p {{ gluster_pw }}

- name: create storage project
  command: oc new-project rhgs-cluster

- name: add scc policy default
  command: oadm policy add-scc-to-user privileged -z default

- name: add scc policy router
  command: oadm policy add-scc-to-user privileged -z router

---
- template:
   src: topology-hpe.j2
   dest: /usr/share/heketi/topology-hpe.json
   owner: bin
   group: wheel
   mode: 0644

{
  "clusters": [
    {
      "nodes": [
        {% for host in groups['cns'] %}
          {
          "node": {
            "hostnames": {
              "manage": [
                "{{ host }}"
              ],
              "storage": [
                "{{ hostvars[host]['ansible_default_ipv4']['address'] }}"
              ]
              },
              "zone": 1
              },
              "devices": [
                "/dev/sdb"
              ]
{% if loop.last %}
          }
{% else %}
          },
{% endif %}
{% endfor %}
        ]
    }
  ]
}

---
- name: loging as cns gluster
  command: oc login -u gluster -p {{ gluster_pw }}


- name: deploy cns
  command: cns-deploy -n rhgs-cluster -g /usr/share/heketi/topology-hpe.json -y  --admin-key {{ gluster_pw }} --user-key {{ gluster_pw }}

This section provides the installation and configuration details for installing OpenShift Container Platform using Ansible Tower.

curl https://bootstrap.pypa.io/get-pip.py
python get-pip.py
pip install hpOneView

git clone https://github.com/HewlettPackard/oneview-ansible.git
cp oneview-ansible/library/* /usr/share/ansible/

curl -o /usr/share/ansible/nsupdate.py https://raw.githubusercontent.com/dcritch/ansible/nsupdate_one2many/lib/ansible/modules/net_tools/nsupdate.py

7.4. Credentials

Ansible typically communicates to a target host via SSH, using a public key. Ansible Tower can store and encrypt these credentials to run playbooks.

An SSH key pair is generated:

ssh-keygen -f tower
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in tower.
Your public key has been saved in tower.pub.
The key fingerprint is:
...

And then stored in Ansible Tower by clicking the gear icon, then credentials and finally +ADD:

Figure 17 SSH credentials

The password used to encrypt the password playbook is also stored under 'Vault Password'.

The public SSH key is deployed via Red Hat Satellite when a host is provisioned.

In some cases, the git repository may require login details, which can also be stored safely in Ansible Tower:

Figure 18 git credentials

7.5. Projects

The locally maintained git repository is added to Ansible Tower:

Figure 19 HPE Project

As well as the publically accessibly openshift-ansible playbooks:

Figure 20 OpenShift Project

7.6. Inventories

The list of hosts and variables for this reference architecture can be entered via the web GUI, however for complex playbooks with a lot of variables, this can be cumbersome. Ansible Tower provides a handy CLI tool that can be used to import this data.

Before using the tool, the empty inventory must first be created in the web GUI:

Figure 21 Ansible Tower Inventory

The host file and variables are copied to a directory on the Tower server:

ocp
├── ansible-hosts
└── group_vars
    ├── all
    ├── cns
    └── OSEv3

And then imported using tower-manage:

tower-manage inventory_import --source=./ocp/ --inventory-name="hpe-ocp-3.5" --overwrite --overwrite-vars

Unfortunately there is one issue with this approach. Due to a bug, Tower can not parse certain inventory lines, for example:

ocp-cns1.hpecloud.test openshift_node_labels="{'region': 'primary', 'zone': 'east'}"

openshift_node_labels is interpreted as a string rather than a dictionary. After importing the inventory, the variable must be formatted slightly different to parse correctly:

Figure 22 OpenShift Node Labels

7.7. Templates

With credentials, projects and inventories in place, the job templates can be created in Ansible Tower. This is accomplished by clicking on Templates, then Add → Job Template in Ansible Tower:

Figure 23 Cleanup Template

Figure 24 Provisioning Template

Figure 25 common Template

Figure 26 Cleanup Template

Figure 27 OpenShift Template

Figure 28 Container-native storage Template

7.8. Workflow

The workflow to chain all the job templates together is then created by navigating to Templates, then Add → Workflow Job Template:

Figure 29 Ansible Tower Workflow

7.9. Launch Job

To launch the workflow, browse to Templates, find the newly created workflow and hit the rocket ship button:

Figure 30 Launch Tower Workflow

This section provides post deployment and validation information

Validate the Deployment Once the deployment is complete the Red Hat OpenShift Container Platform will resemble the deployment illustrated in the image below. The deployment will have three master nodes running the etcd cluster, three infrastructure nodes, and three Container-native storage Nodes. The registry for container applications is running on two of the infrastructure nodes. Moving the registry to persistent storage backed by Gluster Container-native storage is described later in this document. One of the infrastructure nodes is running HA Proxy to provide loadbalancing of external traffic among the master nodes. Routing services are running on two of the infrastructure nodes as well, the routers provide routing services between the external network and the container network. There are three nodes running Container-native storage to provide containers with persistent storage.

Figure 31 Red Hat OpenShift Container Platform Deployment

Verify the Red Hat OpenShift Platform ETCD cluster is running

To verify the Red Hat OpenShift Platfrom deployment was successful check the ectd cluster by logging into the first master node and running the following command:

etcdctl -C https://ocp-master1.hpecloud.test:2379,https://ocp-master2.hpecloud.test:2379,https://ocp-master3.hpecloud.test:2379\
     --ca-file=/etc/origin/master/master.etcd-ca.crt \
     --cert-file=/etc/origin/master/master.etcd-client.crt \
     --key-file=/etc/origin/master/master.etcd-client.key cluster-health
2017-06-15 14:44:06.566247 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-06-15 14:44:06.566839 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
member 70f2422ba978ec65 is healthy: got healthy result from https://10.19.20.173:2379
member b6d844383be04fb2 is healthy: got healthy result from https://10.19.20.175:2379
member fe30d0c37c03d494 is healthy: got healthy result from https://10.19.20.174:2379
cluster is healthy

Check the members of the etcd cluster by executing the command etcdctl -C command as shown below:

etcdctl -C https://ocp-master1.hpecloud.test:2379,https://ocp-master2.hpecloud.test:2379,https://ocp-master3.hpecloud.test:2379 \
     --ca-file=/etc/origin/master/master.etcd-ca.crt \
     --cert-file=/etc/origin/master/master.etcd-client.crt \
     --key-file=/etc/origin/master/master.etcd-client.key member list
2017-06-15 14:48:58.657439 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
70f2422ba978ec65: name=ocp-master1.hpecloud.test peerURLs=https://10.19.20.173:2380 clientURLs=https://10.19.20.173:2379 isLeader=false
b6d844383be04fb2: name=ocp-master3.hpecloud.test peerURLs=https://10.19.20.175:2380 clientURLs=https://10.19.20.175:2379 isLeader=false
fe30d0c37c03d494: name=ocp-master2.hpecloud.test peerURLs=https://10.19.20.174:2380 clientURLs=https://10.19.20.174:2379 isLeader=true

List nodes

Executing the oc get nodes command will display the OpenShift Container Platform nodes and their respective status.

oc get nodes
NAME                        STATUS                     AGE
ocp-cns1.hpecloud.test      Ready                      1h
ocp-cns2.hpecloud.test      Ready                      1h
ocp-cns3.hpecloud.test      Ready                      1h
ocp-infra1.hpecloud.test    Ready                      1h
ocp-infra2.hpecloud.test    Ready                      1h
ocp-master1.hpecloud.test   Ready,SchedulingDisabled   1h
ocp-master2.hpecloud.test   Ready,SchedulingDisabled   1h
ocp-master3.hpecloud.test   Ready,SchedulingDisabled   1h

List projects

Using the oc get projects command will display all projects that the user has permission to access.

oc get projects
NAME               DISPLAY NAME   STATUS
default                           Active
kube-system                       Active
logging                           Active
management-infra                  Active
openshift                         Active
openshift-infra                   Active
Using project "default" on server "https://openshift-master.hpecloud.test:8443".

List pods

The oc get pods command will list the running pods.

oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-sx2sh    1/1       Running   0          1h
docker-registry-1-tbz80    1/1       Running   0          1h
registry-console-1-sxm4d   1/1       Running   0          1h
router-1-ntql2             1/1       Running   0          1h
router-1-s6xqt             1/1       Running   0          1h

List Services

The oc get services command will list the available services, Cluster IP addresses and ports.

oc get services
NAME               CLUSTER-IP      EXTERNAL-IP   PORT(S)                   AGE
docker-registry    172.30.80.1     <none>        5000/TCP                  1h
kubernetes         172.30.0.1      <none>        443/TCP,53/UDP,53/TCP     1h
registry-console   172.30.105.3    <none>        9000/TCP                  1h
router             172.30.206.66   <none>        80/TCP,443/TCP,1936/TCP   1h

List endpoints

The oc get endpoints command will display a list of external endpoint IP addresses and ports.

oc get endpoints
NAME               ENDPOINTS                                                           AGE
docker-registry    10.128.2.4:5000,10.128.2.5:5000                                     1h
kubernetes         10.19.20.173:8443,10.19.20.174:8443,10.19.20.175:8443 + 6 more...   1h
registry-console   10.128.2.3:9090                                                     1h
router             10.19.20.170:443,10.19.20.171:443,10.19.20.170:1936 + 3 more...     1h

Verify the Red Hat OpenShift Container Platform User Interface

Log into the Red Hat OpenShift Container Platform user interface using the public url defined in the openshift_master_cluster_public_hostname variable as shown below.

openshift_master_cluster_public_hostname: openshift-master.hpecloud.test

Figure 32 Red Hat OpenShift Container Platform User Interface

Verify Container-native storage

Container-native storage is a hyper-converged solution that allows containerized applications and Gluster based storage to reside on the same nodes. The Container-native storage provides containers with persistent storage. Secrets are defined to enhance security for Container-native storage, these secrets are applied to the configuration as variables in the cnsdeploy.yaml ansible task. The secrets are stored in an encrypted format in ansible vault and initially defined in the passwords.yaml file. There is an admin secret and a user secret defined. Set the environment variables by exporting the heketi user and the heketi secret to be used with the heketi cli command as HEKETI_CLI_USER and HEKETI_CLI_KEY, for example export HEKETI_CLI_USER=admin and export HEKETI_CLI_KEY=n0tP@ssw0rd . Alternatively, the user and secret can be passed on the command line using the --secret and --user options as shown below.

heketi-cli topology info -s http://heketi-rhgs-cluster.paas.hpecloud.test --user admin --secret n0tP@ssw0rd

Use the oc get route heketi command to get the heketi server name.

oc get route heketi
NAME      HOST/PORT                                PATH      SERVICES   PORT      TERMINATION   WILDCARD
heketi    heketi-rhgs-cluster.paas.hpecloud.test             heketi     <all>                   None

Export the heketi server name as HEKETI_CLI_SERVER by executing export HEKETI_CLI_SERVER=http://heketi-rhgs-cluster.paas.hpecloud.test .

Using the heketi client create a 100 GB test volume on the cluster named test_volume

heketi-cli volume create --size=100 --name test_volume
Name: test_volume
Size: 100
Volume Id: b39a544984eb9848a1727564ea66cb6f
Cluster Id: 05f9e98b350c2341c1dffb746d0549e9
Mount: 10.19.20.176:test_volume
Mount Options: backup-volfile-servers=10.19.20.177,10.19.20.178
Durability Type: replicate
Distributed+Replica: 3

In our example the heketi server name is heketi-rhgs-cluster.paas.hpecloud.test, export the server name the to the HEKETI_CLI_SERVER variable. Use the heketi client, heketi-cli , to view the Gluster topology as shown below. Notice the 100 GB volume previously created test_volume. To delete the test volume use heketi-cli volume delete <volume ID> .

heketi-cli topology info
Cluster Id: 05f9e98b350c2341c1dffb746d0549e9
    Volumes:
	Name: heketidbstorage
	Size: 2
	Id: 51b91b87d77bff02cb5670cbab621d6d
	Cluster Id: 05f9e98b350c2341c1dffb746d0549e9
	Mount: 10.19.20.176:heketidbstorage
	Mount Options: backup-volfile-servers=10.19.20.177,10.19.20.178
	Durability Type: replicate
	Replica: 3
	Snapshot: Disabled
		Bricks:
			Id: 7f243adb17ab1dbf64f032bff7c47d81
			Path: /var/lib/heketi/mounts/vg_5e25001fb558bff388f3304897440f33/brick_7f243adb17ab1dbf64f032bff7c47d81/brick
			Size (GiB): 2
			Node: 0fd823a6c17500ffcfdf797f9eaa9d4a
			Device: 5e25001fb558bff388f3304897440f33
-
			Id: 8219d3b014d751bbb58fcfae57eb1cf0
			Path: /var/lib/heketi/mounts/vg_159c4b0cca850b7549b3957e93014ffe/brick_8219d3b014d751bbb58fcfae57eb1cf0/brick
			Size (GiB): 2
			Node: 8b1e2abf4076ef4dd93299f2be042b89
			Device: 159c4b0cca850b7549b3957e93014ffe
-
			Id: 843b35abc2ab505ed7591a2873cdd29d
			Path: /var/lib/heketi/mounts/vg_01423a48fb095139ce7b229a73eac064/brick_843b35abc2ab505ed7591a2873cdd29d/brick
			Size (GiB): 2
			Node: a5923b6b35fef3131bd077a33f658239
			Device: 01423a48fb095139ce7b229a73eac064
	Name: test_volume
	Size: 100
	Id: b39a544984eb9848a1727564ea66cb6f
	Cluster Id: 05f9e98b350c2341c1dffb746d0549e9
	Mount: 10.19.20.176:test_volume
	Mount Options: backup-volfile-servers=10.19.20.177,10.19.20.178
	Durability Type: replicate
	Replica: 3
	Snapshot: Disabled
		Bricks:
			Id: 45cc45e08b6215ee206093489fa12216
			Path: /var/lib/heketi/mounts/vg_01423a48fb095139ce7b229a73eac064/brick_45cc45e08b6215ee206093489fa12216/brick
			Size (GiB): 100
			Node: a5923b6b35fef3131bd077a33f658239
			Device: 01423a48fb095139ce7b229a73eac064
-
			Id: 47722ac826e6a263c842fe6a87f30d84
			Path: /var/lib/heketi/mounts/vg_159c4b0cca850b7549b3957e93014ffe/brick_47722ac826e6a263c842fe6a87f30d84/brick
			Size (GiB): 100
			Node: 8b1e2abf4076ef4dd93299f2be042b89
			Device: 159c4b0cca850b7549b3957e93014ffe
-
			Id: a257cdad1725fcfa22de954e59e70b64
			Path: /var/lib/heketi/mounts/vg_5e25001fb558bff388f3304897440f33/brick_a257cdad1725fcfa22de954e59e70b64/brick
			Size (GiB): 100
			Node: 0fd823a6c17500ffcfdf797f9eaa9d4a
			Device: 5e25001fb558bff388f3304897440f33
    Nodes:
	Node Id: 0fd823a6c17500ffcfdf797f9eaa9d4a
	State: online
	Cluster Id: 05f9e98b350c2341c1dffb746d0549e9
	Zone: 1
	Management Hostname: ocp-cns3.hpecloud.test
	Storage Hostname: 10.19.20.178
	Devices:
		Id:5e25001fb558bff388f3304897440f33   Name:/dev/sdb            State:online    Size (GiB):11177   Used (GiB):102     Free (GiB):11075
			Bricks:
				Id:7f243adb17ab1dbf64f032bff7c47d81   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_5e25001fb558bff388f3304897440f33/brick_7f243adb17ab1dbf64f032bff7c47d81/brick
				Id:a257cdad1725fcfa22de954e59e70b64   Size (GiB):100     Path: /var/lib/heketi/mounts/vg_5e25001fb558bff388f3304897440f33/brick_a257cdad1725fcfa22de954e59e70b64/brick
	Node Id: 8b1e2abf4076ef4dd93299f2be042b89
	State: online
	Cluster Id: 05f9e98b350c2341c1dffb746d0549e9
	Zone: 1
	Management Hostname: ocp-cns2.hpecloud.test
	Storage Hostname: 10.19.20.177
	Devices:
		Id:159c4b0cca850b7549b3957e93014ffe   Name:/dev/sdb            State:online    Size (GiB):11177   Used (GiB):102     Free (GiB):11075
			Bricks:
				Id:47722ac826e6a263c842fe6a87f30d84   Size (GiB):100     Path: /var/lib/heketi/mounts/vg_159c4b0cca850b7549b3957e93014ffe/brick_47722ac826e6a263c842fe6a87f30d84/brick
				Id:8219d3b014d751bbb58fcfae57eb1cf0   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_159c4b0cca850b7549b3957e93014ffe/brick_8219d3b014d751bbb58fcfae57eb1cf0/brick
	Node Id: a5923b6b35fef3131bd077a33f658239
	State: online
	Cluster Id: 05f9e98b350c2341c1dffb746d0549e9
	Zone: 1
	Management Hostname: ocp-cns1.hpecloud.test
	Storage Hostname: 10.19.20.176
	Devices:
		Id:01423a48fb095139ce7b229a73eac064   Name:/dev/sdb            State:online    Size (GiB):11177   Used (GiB):102     Free (GiB):11075
			Bricks:
				Id:45cc45e08b6215ee206093489fa12216   Size (GiB):100     Path: /var/lib/heketi/mounts/vg_01423a48fb095139ce7b229a73eac064/brick_45cc45e08b6215ee206093489fa12216/brick
				Id:843b35abc2ab505ed7591a2873cdd29d   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_01423a48fb095139ce7b229a73eac064/brick_843b35abc2ab505ed7591a2873cdd29d/brick

Backing the Docker Registry with Gluster Container-native storage

In a production environment the local Docker registry should be backed with persistent storage. This section will describe how to create a Persistent Volume Claim (PVC) and Persistent Volume (PV) for the Docker registry.

In this example a user account named gluster was created and assigned the cluster admin role. Log in under this account to perform the oc commands described in this section.

The following files are required to create the gluster service, endpoints, persistent volume and persistent volume claim. In this example the files were created on the master nodes, ocp-master1.hpecloud.test, and the oc commands were executed from this node.

Gluster Service File

The contents of the glusterservice.yaml file are shown below. This file will create an OpenShift service named glusterfs-cluster.

apiVersion: v1
kind: Service
metadata:
  name: glusterfs-cluster
spec:
  ports:
  - port: 1

Gluster Endpoint File

The contents of the glusterendpoints.yaml file are shown below, this file defines the gluster endpoints. The IP addresses in this file are the IP addresses of the Gluster nodes.

apiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs-cluster
subsets:
  - addresses:
      - ip: 10.19.20.176
    ports:
      - port: 1
  - addresses:
      - ip: 10.19.20.177
    ports:
      - port: 1
  - addresses:
      - ip: 10.19.20.178
    ports:
      - port: 1

Gluster PV File

The contents of the glusterpv.yaml file is shown below, this file describes the persistent volume that will be used for the Docker registry persistent storage. The path specified must match the name of the volume that will be used for the Docker registry on the Gluster Container-native storage.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: gluster-default-volume
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: glusterfs-cluster
    path: regVol1
    readOnly: false
  persistentVolumeReclaimPolicy: Retain

Gluster PVC File

The contents of the glusterpvc.yaml file are shown below. This file describes the persistent volume claim that will be used by the Docker registry pods to mount the persistent volume and provide the Docker registry with persistent storage.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gluster-claim
spec:
  accessModes:
  - ReadWriteMany
  resources:
     requests:
       storage: 1Gi

Creating a volume for the Docker registry

Use the heketi-cli volume create command to create a volume on the Container-native storage for the Docker registry persistent storage. The name used when creating the volume must match the path specified in the glusterpv.yaml file.

heketi-cli volume create --size=20 --name=regVol1

The code snipet below shows the commands along with thier respective outputs that are used to create the glusterfs service, the gluster endpoints, the persistent volume and the persistent volume claim that will be used to provide persistent storage for the Docker registry.

[root@ocp-master1 ~]# oc whoami
gluster
[root@ocp-master1 ~]# oc create -f glusterservice.yaml
service "glusterfs-cluster" created
[root@ocp-master1 ~]# oc get services
NAME                CLUSTER-IP       EXTERNAL-IP   PORT(S)                   AGE
docker-registry     172.30.35.174    <none>        5000/TCP                  8h
glusterfs-cluster   172.30.81.99     <none>        1/TCP                     5s
kubernetes          172.30.0.1       <none>        443/TCP,53/UDP,53/TCP     8h
registry-console    172.30.173.160   <none>        9000/TCP                  8h
router              172.30.139.18    <none>        80/TCP,443/TCP,1936/TCP   8h
[root@ocp-master1 ~]# oc create -f glusterendpoints.yaml
endpoints "glusterfs-cluster" created
[root@ocp-master1 ~]# oc get endpoints
NAME                ENDPOINTS                                                           AGE
docker-registry     10.128.2.3:5000,10.131.0.3:5000                                     8h
glusterfs-cluster   10.19.20.176:1,10.19.20.177:1,10.19.20.178:1                        27s
kubernetes          10.19.20.173:8443,10.19.20.174:8443,10.19.20.175:8443 + 6 more...   8h
registry-console    10.130.0.2:9090                                                     8h
router              10.19.20.170:443,10.19.20.171:443,10.19.20.170:1936 + 3 more...     8h
[root@ocp-master1 ~]# oc create -f glusterpv.yaml
persistentvolume "gluster-default-volume" created
[root@ocp-master1 ~]# oc get pv
NAME                     CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS      CLAIM                    REASON    AGE
gluster-default-volume   20Gi       RWX           Retain          Available                                      16s
registry-volume          10Gi       RWX           Retain          Bound       default/registry-claim             8h
[root@ocp-master1 ~]# oc create -f glusterpvc.yaml
persistentvolumeclaim "gluster-claim" created
[root@ocp-master1 ~]# oc get pvc
NAME             STATUS    VOLUME                   CAPACITY   ACCESSMODES   AGE
gluster-claim    Bound     gluster-default-volume   20Gi       RWX           25s
registry-claim   Bound     registry-volume          10Gi       RWX           8h

Attach the Docker registry to the persistent storage using the oc volume command shown below.

oc volume deploymentconfigs/docker-registry --add --name=v1 -t pvc --claim-name=gluster-claim --overwrite

Review the OpenShift documentation at https://docs.openshift.com/container-platform/3.5/install_config/storage_examples/gluster_backed_registry.html for detailed information on backing the Docker registry with Gluster File Storage.

CNS Storage Class

This section will describe how to create a storage class for provisioning storage from the gluster based Container-native storage. To create an OpenShift storage class you will need the cluster ID and the rest API URL of the CNS cluster. The rest API URL is the same as the HEKETI_CLI_SERVER variable exported earlier. Use the heketi-cli cluster list command to display the cluster ID and the oc get route command to display the heketi server name.

[root@ocp-master1 ~]# heketi-cli cluster list
Clusters:
5f64ac6c1313d2abba1143bb808da586
[root@ocp-master1 ~]# oc get route
NAME      HOST/PORT                                PATH      SERVICES   PORT      TERMINATION   WILDCARD
heketi    heketi-rhgs-cluster.paas.hpecloud.test             heketi     <all>                   None
[root@ocp-master1 ~]# echo $HEKETI_CLI_SERVER
http://heketi-rhgs-cluster.paas.hpecloud.test

The storage class object requires the heketi secret that was created during provisioning of the Container-native storage. In this example, the heketi secret is "n0tP@ssw0rd". This secret will be stored in OpenShift as a secret. The OpenShift secret is created using the oc create command along with a file that defines the secret. First we are going to encrypt the secret with the echo -n <secret> | base64 command, then create a yaml file that will be passed with the oc create command to create the secret in OpenShift as shown below.

[root@ocp-master1 ~]# echo -n n0tP@ssw0rd |base64
bjB0UEBzc3cwcmQ=
[root@ocp-master1 ~]# vi heketi-secret.yaml
[root@ocp-master1 ~]# oc create -f heketi-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
data:
  key: bjB0UEBzc3cwcmQ=
type: kubernetes.io/glusterfs
[root@ocp-master1 ~]# cat heketi-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
data:
  key: bjB0UEBzc3cwcmQ=
type: kubernetes.io/glusterfs
[root@ocp-master1 ~]# oc create -f heketi-secret.yaml
secret "heketi-secret" created
[root@ocp-master1 ~]# oc describe secret heketi-secret
Name:		heketi-secret
Namespace:	default
Labels:		<none>
Annotations:	<none>
Type:	kubernetes.io/glusterfs
Data
====
key:	11 bytes

Next, create a yaml file that will be used to create the storageclass object. This file will contain the cns cluster id, cluster name, and the heketi username and secret.

[root@ocp-master1 ~]# vi cns-storageclass-st1.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: gluster-cns-bronze
provisioner: kubernetes.io/glusterfs
parameters:
  clusterid: 5f64ac6c1313d2abba1143bb808da586
  resturl: http://heketi-rhgs-cluster.paas.hpecloud.test
  restauthenabled: "true"
  restuser: "admin"
  secretName: "heketi-secret"
  secretNamespace: "default"

The file will be passed with the oc create command to create a new storage class object named gluster-cns-bronze.

[root@ocp-master1 ~]# oc create -f cns-storageclass-st1.yaml
storageclass "gluster-cns-bronze" created
[root@ocp-master1 ~]# oc describe storageclass gluster-cns-bronze
Name:		gluster-cns-bronze
IsDefaultClass:	No
Annotations:	<none>
Provisioner:	kubernetes.io/glusterfs
Parameters:	clusterid=5f64ac6c1313d2abba1143bb808da586,restauthenabled=true,resturl=http://heketi-rhgs-cluster.paas.hpecloud.test,restuser=admin,secretName=n0tP@ssw0rd,secretNamespace=default
No events.

Now that the storage class object, gluster-cns-bronze, has been created, create a persistent volume claim against the gluster-cns-bronze storage class. This persistent volume claim will dynamically allocate storage from the gluster-cns-bronze storage class.

Create a yaml file that defines the persistent volume claime against the gluster-cns-bronze storage class object as shown below.

[root@ocp-master1 ~]# vi cns-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: db
 annotations:
   volume.beta.kubernetes.io/storage-class: gluster-cns-bronze
spec:
 accessModes:
  - ReadWriteOnce
 resources:
   requests:
     storage: 10Gi

Create the persistent volume claim using the oc create command and passing the cns-claim.yaml file.

[root@ocp-master1 ~]# oc create -f cns-claim.yaml
persistentvolumeclaim "db" created

Review the provisioning of the persistent storage using the oc describe persistentvolumeclaim command and heketi-cli volume info command

[root@ocp-master1 ~]# oc describe persistentvolumeclaim db
Name:		db
Namespace:	rhgs-cluster
StorageClass:	gluster-cns-bronze
Status:		Bound
Volume:		pvc-7605ae4b-714b-11e7-a319-1402ec825eec
Labels:		<none>
Capacity:	10Gi
Access Modes:	RWO
No events.
[root@ocp-master1 ~]# heketi-cli volume info 93b150f2f7a5ddcb1ed5462cedeb43a9
Name: vol_93b150f2f7a5ddcb1ed5462cedeb43a9
Size: 10
Volume Id: 93b150f2f7a5ddcb1ed5462cedeb43a9
Cluster Id: 5f64ac6c1313d2abba1143bb808da586
Mount: 10.19.20.176:vol_93b150f2f7a5ddcb1ed5462cedeb43a9
Mount Options: backup-volfile-servers=10.19.20.177,10.19.20.178
Durability Type: replicate
Distributed+Replica: 3

In this reference architecture the cakephp application is used as a sample application. This sample application can be found at https://hub.openshift.com/quickstarts/73-cakephp. A list of example applications can be found at https://hub.openshift.com/quickstarts/all or in the OpenShift documentation at https://docs.openshift.com/container-platform/3.5/dev_guide/migrating_applications/quickstart_examples.html.

Access the pubic url endpoint of the OpenShift deployment at https://openshift-master.hpecloud.test:8443/console/command-line for instructions on setting up the command line interface. This link will display instructions on downloading and installing the oc commandline tools as shown in figure 33.

Figure 33: Command Line Tools

Using the oc login command, log into OpenShift using the public URL and the token available from the console command-line interface described above.

oc login https://openshift-master.hpecloud.test:8443 --token=zQN6GDIPOWpklX-Lq4DCCL2CMRRI5jdCYEYIjW5GofA

CakePHP Sample Application

Create a new Project

Use the oc new-project command to create a new OpenShift Container Platform project for the application.

$ oc new-project my-app
Now using project "my-app" on server "https://openshift-master.hpecloud.test:8443".
You can add applications to this project with the 'new-app' command. For example, try:
    oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git
to build a new example application in Ruby.

Create a new Application

Create a new application by executing the oc new-app command and specifying the git repository for the cakephp application.

$ oc new-app https://github.com/openshift/cakephp-ex.git --name=cakephp
--> Found image 79f7d44 (3 weeks old) in image stream "openshift/php" under tag "7.0" for "php"
    Apache 2.4 with PHP 7.0
    -----------------------
    Platform for building and running PHP 7.0 applications
    Tags: builder, php, php70, rh-php70
    * The source repository appears to match: php
    * A source build using source code from https://github.com/openshift/cakephp-ex.git will be created
      * The resulting image will be pushed to image stream "cakephp:latest"
      * Use 'start-build' to trigger a new build
    * This image will be deployed in deployment config "cakephp"
    * Port 8080/tcp will be load balanced by service "cakephp"
      * Other containers can access this service through the hostname "cakephp"
--> Creating resources ...
    imagestream "cakephp" created
    buildconfig "cakephp" created
    deploymentconfig "cakephp" created
    service "cakephp" created
--> Success
    Build scheduled, use 'oc logs -f bc/cakephp' to track its progress.
    Run 'oc status' to view your app.

Expose the Application

Create a route to expose the application using the subdomain that was created during the OpenShift Container Platform deployment and the oc expose command. In our example the subdomain specified was paas.hpecloud.test .

$ oc expose svc cakephp

Access the Application

The cakephp application can now be accessed via a browser, in our example the URL for the cakephp application is http://cakephp-my-app.paas.hpecloud.test/.

Figure 34: Cake Sample Application

This reference architecture provides installation and configuration details for deploying Red Hat OpenShift Container Platform 3.5 on HPE ProLiant DL rack mounted servers. The Red Hat OpenShift Container platform deployment is automated from end to end using Ansible Tower playbooks and workflows, providing a single click deployment from baremetal to a fully configured Red Hat OpenShift Container Platform cluster consisting of three OpenShift Master nodes, three OpenShift infrastructure and application nodes, and three Gluster Container-native storage nodes. The Ansible playbooks used in this reference architecture interact with HPE OneView 3.1 to create server profiles for the baremetal servers, configure the local storage, and update the server firmware to the latest Service Pack for ProLiant (SPP). Ansible playbooks are used to provision the servers with Red Hat Satellite. Red Hat Enterprise Linux 7.3 is installed on each of the nodes and access content repositories for Red Hat OpenShift Container Platform 3.5 and Red Hat Gluster Storage from Red Hat Satellite. Additionally, Red Hat Satellite provide DNS and DHCP services to the deployment. The reference architecture also describes post deployment activities that verify that the Red Hat OpenShift Container Platform is operating correctly. These post deployment validation and configuration processes include assessing the health of the OpenShift and Gluster services, and deploying a sample application (cakephp).