The Master ELB security group allows inbound access on port 443 from the internet to the ELB. The traffic is then allowed to be forwarded to the master instances. See Figure 2.1, “AWS Master ELB Security Group Details - Inbound” diagram and Table 2.5, “AWS Master ELB Security Group Details - Inbound” table below.

Figure 2.1. AWS Master ELB Security Group Details - Inbound

The Internal Master ELB is in the private subnet and utilizes the NAT Gateway. Traffic external from the VPC cannot access the Internal Master ELB.

Figure 2.2. AWS Internal Master ELB Security Group Details - Inbound

The bastion security group allows inbound port SSH traffic from outside the VPC. Any connectivity via SSH to the master, application or infrastructure nodes must go through the bastion host. Ensure the bastion host is secured per your companies security requirements.

The master security group allows traffic to the master instances from the two ELBs and nodes to contact the OpenShift API and DNS.

Figure 2.3. AWS Master Security Group Details - Inbound

The ETCD security group allows for the ETCD service running on the master instances to reach a quorum. The security group allows for the ose-master-sg to communication with the ETCD for the OpenShift master services.

The Router ELB security group allows inbound access on port 80 and 443. If the applications running on the OpenShift cluster are using different ports this can be adjusted as needed.

Figure 2.4. AWS Router ELB Security Group Details - Inbound

The infrastructure nodes security group allows traffic from the router security group.

Figure 2.5. AWS Infrastructure Nodes Security Group Details - Inbound

The node security group only allows traffic from the bastion and traffic relating to OpenShift node services.

Figure 2.6. AWS Nodes Security Group Details - Inbound

The reference architecture environment automatically adds entries into Route53. A Hosted Zone is required for OpenShift. A domain name can either purchased through AWS or another external provider such as Google Domains or GoDaddy. If using a zone from an external provider ensure the NS records point to a Route53 hosted zone. Steps will be detailed in Chapter 3 of this document.

The Red Hat Cloud Access provides gold images for Atomic and Red Hat Enterprise Linux. Since a subscription is required to install OpenShift then it is not necessary to use the Amazon provided image which has a built in charge back for the RHEL subscription.

To register for the Red Hat Cloud Access Gold Image please see Red Hat’s Website and select the tab for Red Hat Gold Image.

The master nodes contain the master components, including the API server, controller manager server and ETCD. The master maintains the clusters configuration, manages nodes in its OpenShift cluster. The master assigns pods to nodes and synchronizes pod information with service configuration. The master is used to define routes, services, and volume claims for pods deployed within the OpenShift environment.

The infrastructure nodes are used for the router and registry pods. These nodes could be used if the optional components Kibana and Hawkular metrics are required. The storage for the Docker registry that is deployed on the infrastructure nodes is S3 which allows for multiple pods to use the same storage. AWS S3 storage is used because it is synchronized between the availability zones, providing data redundancy.

The Application nodes are the instances where non-infrastructure based containers run. Depending on the application, AWS specific storage can be applied such as a Elastic Block Storage which can be assigned using a Persistent Volume Claim for application data that needs to persist between container restarts. A configuration parameter is set on the master which ensures that OpenShift Container Platform user containers will be placed on the application nodes by default.

All OpenShift Container Platform nodes are assigned a label. This allows certain pods to be deployed on specific nodes. For example, nodes labeled infra are Infrastructure nodes. These nodes run the router and registry pods. Nodes with the label app are nodes used for end user Application pods. The configuration parameter 'defaultNodeSelector: "role=app" in /etc/origin/master/master-config.yaml ensures all projects automatically are deployed on Application nodes.

This section describes how the environment should be configured to use Ansible to provision the infrastructure, install OpenShift, and perform post installation tasks.

$ rpm -q python-2.7
$ subscription-manager repos --enable rhel-7-server-optional-rpms
$ subscription-manager repos --enable rhel-7-server-ose-3.5-rpms
$ subscription-manager repos --enable rhel-7-fast-datapath-rpms
$ yum -y install ansible atomic-openshift-utils
$ yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
$ yum -y install python2-boto \
                 python2-boto3 \
                 pyOpenSSL \
                 git \
                 python-netaddr \
                 python-click \
                 python-httplib2

$ cd /home/<user>/git
$ git clone https://github.com/openshift/openshift-ansible-contrib.git

$ yum -y install tree
$ tree /home/<user>/git/

... content abbreviated ...

|-- openshift-ansible-contrib

As mentioned in the previous section, Authentication for the reference architecture deployment is handled by GitHub OAuth. The steps below describe both the process for creating an organization and performing the configuration steps required for GitHub authentication.

3.1.2.2. Configuring OAuth

Browse to https://github.com/settings/applications/new and login to GitHub

The image below will provide an example configuration. Insert values that will be used during the OpenShift deployment.

Figure 3.1. GitHub OAuth Application

• Insert an Application name
• Insert a Homepage URL (This will be the URL used when accessing OpenShift)
• Insert an Application description (Optional)
• Insert an Authorization callback URL (The entry will be the Homepage URL + /oauth2callback/github)
• Click Register application

Figure 3.2. GitHub OAuth Client ID

A Client ID and Client Secret will be presented. These values will be used as variables during the installation of Openshift.

In this reference implementation guide a domain called sysdeseng.com domain was purchased through AWS and managed by Route53. In the example below, the domain sysdeseng.com will be the hosted zone used for the installation of OpenShift. Follow the below instructions to add the main hosted zone.

A subdomain can also be used. The same steps listed above are applicable when using a subdomain. Once the Pubic Zone is created select the radio button for the Domain and copy the Name Servers from the right and add those to the external registrar or top level domain in Route53. Ensure that the Name Servers(NS) records are copied to the root domain or name resolution will not work for the subdomain.

The installation of OpenShift Container Platform (OCP) requires a valid Red Hat subscription. During the installation of OpenShift the Ansible redhat_subscription module will attempt to register the instances. The script will fail if the OpenShift entitlements are exhausted. For the installation of OCP on AWS the following items are required:

Red Hat Subscription Manager User: < Red Hat Username >

Red Hat Subscription Manager Password: < Red Hat Password >

Subscription Name: Red Hat OpenShift Container Platform, Standard, 2-Core

The items above are examples and should reflect subscriptions relevant to the account performing the installation. There are a few different variants of the OpenShift Subscription Name. It is advised to visit https://access.redhat.com/management/subscriptions to find the specific Subscription Name as the values will be used below during the deployment.

The ose-on-aws.py script contains many different configuration options such as the ability to change the AMI, instance size, and the ability to use a currently deployed bastion host. The region can be changed but keep in mind the AMI may need to be changed if the Red Hat Cloud Access gold image AMI ID is different. The Cloudformation stack name is also configurable. Setting --stack-name to a unique value ensures that one Cloudformation stack does not overwrite another Cloudformation stack. The script creates both an auto-generated S3 bucket name and an IAM user account to be used for the registry. These values can be changed before launching. To see all of the potential options the --help trigger is available.

./ose-on-aws.py  --help
Usage: ose-on-aws.py [OPTIONS]

Options:
  --stack-name TEXT               Cloudformation stack name. Must be unique
                                  [default: openshift-infra]
  --console-port INTEGER RANGE    OpenShift web console port  [default: 443]
  --deployment-type [origin|openshift-enterprise]
                                  OpenShift deployment type  [default:
                                  openshift-enterprise]
  --openshift-sdn TEXT            OpenShift SDN (redhat/openshift-ovs-subnet,
                                  redhat/openshift-ovs-multitenant, or other
                                  supported SDN)  [default: redhat/openshift-
                                  ovs-multitenant]
  --region TEXT                   ec2 region  [default: us-east-1]
  --ami TEXT                      ec2 ami  [default: ami-a33668b4]
  --master-instance-type TEXT     ec2 instance type  [default: m4.xlarge]
  --node-instance-type TEXT       ec2 instance type  [default: t2.large]
  --app-instance-type TEXT        ec2 instance type  [default: t2.large]
  --bastion-instance-type TEXT    ec2 instance type  [default: t2.micro]
  --keypair TEXT                  ec2 keypair name
  --create-key TEXT               Create SSH keypair  [default: no]
  --key-path TEXT                 Path to SSH public key. Default is /dev/null
                                  which will skip the step  [default:
                                  /dev/null]
  --create-vpc TEXT               Create VPC  [default: yes]
  --vpc-id TEXT                   Specify an already existing VPC
  --private-subnet-id1 TEXT       Specify a Private subnet within the existing
                                  VPC
  --private-subnet-id2 TEXT       Specify a Private subnet within the existing
                                  VPC
  --private-subnet-id3 TEXT       Specify a Private subnet within the existing
                                  VPC
  --public-subnet-id1 TEXT        Specify a Public subnet within the existing
                                  VPC
  --public-subnet-id2 TEXT        Specify a Public subnet within the existing
                                  VPC
  --public-subnet-id3 TEXT        Specify a Public subnet within the existing
                                  VPC
  --public-hosted-zone TEXT       hosted zone for accessing the environment
  --app-dns-prefix TEXT           application dns prefix  [default: apps]
  --rhsm-user TEXT                Red Hat Subscription Management User
  --rhsm-password TEXT            Red Hat Subscription Management Password
  --rhsm-pool TEXT                Red Hat Subscription Management Pool Name
  --byo-bastion TEXT              skip bastion install when one exists within
                                  the cloud provider  [default: no]
  --bastion-sg TEXT               Specify Bastion Security group used with
                                  byo-bastion  [default: /dev/null]
  --containerized TEXT            Containerized installation of OpenShift
                                  [default: False]
  --s3-bucket-name TEXT           Bucket name for S3 for registry
  --github-client-id TEXT         GitHub OAuth ClientID
  --github-client-secret TEXT     GitHub OAuth Client Secret
  --github-organization TEXT      GitHub Organization
  --s3-username TEXT              S3 user for registry access
  --openshift-metrics-deploy [true|false]
                                  Deploy OpenShift Metrics
  --openshift-logging-deploy [true|false]
                                  Deploy OpenShift Logging
  --openshift-metrics-storage-volume-size TEXT
                                  Size of OptionShift Metrics Persistent
                                  Volume
  --openshift-logging-storage-volume-size TEXT
                                  Size of OptionShift Logging Persistent
                                  Volume
  --no-confirm                    Skip confirmation prompt
  -h, --help                      Show this message and exit.
  -v, --verbose

The -v trigger is available as well. This will allow for Ansible to run more verbose allowing for a more in-depth output of the steps occurring while running ose-on-aws.py.

The OCP installation playbooks allow for OpenShift to be installed in containers. These containers can run on either Atomic Host or RHEL. If the containerized installation of OpenShift is preferred specify --containerized=true while running ose-on-aws.py. If using Atomic Host the configuration trigger --containerized=true must be specified or the installation will fail. Also, when using Atomic Host ensure the AMI being used has Docker 1.10 installed.

OpenShift Cluster Metrics can be deployed during installation. By default, metrics are deployed when running ose-on-aws.py. This includes creating a PVC dynamically which is used for persistent metrics storage. The URL of the Hawkular Metrics endpoint is also set on the masters and defined by an OpenShift route. Metrics components are deployed on nodes with the label of "role=infra"

OpenShift Cluster Logging can be deployed during installation. By default, logging is deployed when running ose-on-aws.py. This includes creating three PVCs dynamically which is used for storing logs in a redundant and durable way. Logging components are deployed on nodes with the label of "role=infra"

Two software-defined networks are provided as choices when deploying OpenShift on AWS openshift-ovs-subnet and openshift-ovs-multitenant. By default the openshift-ovs-subnet is configured as the SDN.

For deploying OpenShift into a new environment, ose-on-aws.py creates instances, load balancers, Route53 entries, and IAM users an ssh key can be entered to be uploaded and used with the new instances. Once the values have been entered into the ose-on-aws.py script all values will be presented and the script will prompt to continue with the values or exit. By default, the Red Hat gold image AMI Section 2.11, “Amazon Machine Images” is used when provisioning instances but can be changed when executing the ose-on-aws.py. The keypair in the example below OSE-key is the keypair name as it appears within the AWS EC2 dashboard. If a keypair has not been created and uploaded to AWS perform the steps below to create, upload, and name the SSH keypair.

Create a Public/Private key

If a user does not currently have a public and private SSH key perform the following.

$ ssh-keygen -t rsa -N '' -f /home/user/.ssh/id_rsa

A message similar to this will be presented indicating they key has been successful created

Your identification has been saved in /home/user/.ssh/id_rsa.
Your public key has been saved in /home/user/.ssh/id_rsa.pub.
The key fingerprint is:
e7:97:c7:e2:0e:f9:0e:fc:c4:d7:cb:e5:31:11:92:14 user@sysdeseng.rdu.redhat.com
The key's randomart image is:
+--[ RSA 2048]----+
|             E.  |
|            . .  |
|             o . |
|              . .|
|        S .    . |
|         + o o ..|
|          * * +oo|
|           O +..=|
|           o*  o.|
+-----------------+

Create a Public/Private key

To deploy the environment using the newly created private/public SSH key which currently does not exist within AWS perform the following.

$ export AWS_ACCESS_KEY_ID=<key_id>
$ export AWS_SECRET_ACCESS_KEY=<access_key>
$ ./ose-on-aws.py --stack-name=dev --public-hosted-zone=sysdeseng.com --rhsm-user=rhsm-user --rhsm-password=rhsm-password --rhsm-pool="Red Hat OpenShift Container Platform, Standard, 2-Core" --github-client-secret=gh-secret --github-client-id=gh-client-id --github-organization=openshift --github-organization=RHSyseng _--keypair=OSE-key --create-key=yes --key-path=/home/user/.ssh/id_rsa.pub

If an SSH key has already been uploaded to AWS specify the name of the keypair as it appears within the AWS EC2 dashboard.

$ export AWS_ACCESS_KEY_ID=<key_id>
$ export AWS_SECRET_ACCESS_KEY=<access_key>
$ ./ose-on-aws.py --stack-name=dev --public-hosted-zone=sysdeseng.com --rhsm-user=rhsm-user --rhsm-password=rhsm-password --rhsm-pool="Red Hat OpenShift Container Platform, Standard, 2-Core" --github-client-secret=gh-secret --github-client-id=gh-client-id --github-organization=openshift --github-organization=RHSyseng _--keypair=OSE-key

Example of Greenfield Deployment values

Configured values:
    stack_name: dev
    ami: ami-a33668b4
    region: us-east-1
    master_instance_type: m4.xlarge
    node_instance_type: t2.large
    app_instance_type: t2.large
    bastion_instance_type: t2.micro
    keypair: OSE-key
    create_key: no
    key_path: /dev/null
    create_vpc: yes
    vpc_id: None
    private_subnet_id1: None
    private_subnet_id2: None
    private_subnet_id3: None
    public_subnet_id1: None
    public_subnet_id2: None
    public_subnet_id3: None
    byo_bastion: no
    bastion_sg: /dev/null
    console port: 443
    deployment_type: openshift-enterprise
    openshift_sdn: redhat/openshift-ovs-subnet
    public_hosted_zone: sysdeseng.com
    app_dns_prefix: apps
    apps_dns: apps.sysdeseng.com
    rhsm_user: rhsm_user
    rhsm_password: *******
    rhsm_pool: Red Hat OpenShift Container Platform, Standard, 2-Core
    containerized: False
    s3_bucket_name: dev-ocp-registry-sysdeseng
    s3_username: dev-s3-openshift-user
    github_client_id: *******
    github_client_secret: *******
    github_organization: openshift,RHSyseng
    deploy_openshift_metrics: true
    deploy_openshift_logging: true


Continue using these values? [y/N]:

The ose-on-aws.py script allows for deployments into an existing environment in which a VPC already exists and at least six subnets, three public and three private, are already created. The script expects three public and three private subnets are created. The private subnets must be able to connect externally which requires a NAT gateway to be deployed. Before running the brownfield deployment ensure that a NAT gateway is deployed and proper Route Table entries are created (private subnets, 0.0.0.0/0 → nat-xxxx; public subnets, 0.0.0.0/0 → igw-xxxx). By default, the Red Hat gold image AMI is used when provisioning instances but can be changed when executing the ose-on-aws.py.

Running the following will prompt for subnets and the VPC to deploy the instances and OpenShift.

$ ./ose-on-aws.py --stack-name=dev --public-hosted-zone=sysdeseng.com --rhsm-user=rhsm-user --rhsm-password=rhsm-password --rhsm-pool="Red Hat OpenShift Container Platform, Standard, 2-Core" --github-client-secret=gh-secret --github-client-id=gh-client-id --github-organization=openshift --github-organization=RHSyseng --keypair=OSE-key --create-vpc=no

Specify the VPC ID: vpc-11d06976
Specify the first Private subnet within the existing VPC: subnet-3e406466
Specify the second Private subnet within the existing VPC: subnet-66ae905b
Specify the third Private subnet within the existing VPC: subnet-4edfd438
Specify the first Public subnet within the existing VPC: subnet-1f416547
Specify the second Public subnet within the existing VPC: subnet-c2ae90ff
Specify the third Public subnet within the existing VPC: subnet-1ddfd46b

In the case that a bastion instance has already been deployed, an option --byo-bastion=yes exists within ose-on-aws.py exists to not deploy the bastion instance.

$ ./ose-on-aws.py --stack-name=dev --public-hosted-zone=sysdeseng.com --rhsm-user=rhsm-user --rhsm-password=rhsm-password --rhsm-pool="Red Hat OpenShift Container Platform, Standard, 2-Core" --github-client-secret=gh-secret --github-client-id=gh-client-id --github-organization=openshift --github-organization=RHSyseng --keypair=OSE-key --create-vpc=no --byo-bastion=yes --bastion-sg=sg-a34ff3af

Specify the VPC ID: vpc-11d06976
Specify the first Private subnet within the existing VPC: subnet-3e406466
Specify the second Private subnet within the existing VPC: subnet-66ae905b
Specify the third Private subnet within the existing VPC: subnet-4edfd438
Specify the first Public subnet within the existing VPC: subnet-1f416547
Specify the second Public subnet within the existing VPC: subnet-c2ae90ff
Specify the third Public subnet within the existing VPC: subnet-1ddfd46b

As stated in the Greenfield deployment the option exists to not use the Red Hat Cloud Access provided gold image AMI. Using the same command from above the --ami= option allows the default value to be changed.

$ ./ose-on-aws.py --stack-name=dev --public-hosted-zone=sysdeseng.com --rhsm-user=rhsm-user --rhsm-password=rhsm-password --rhsm-pool="Red Hat OpenShift Container Platform, Standard, 2-Core" --github-client-secret=gh-secret --github-client-id=gh-client-id --github-organization=openshift --github-organization=RHSyseng --keypair=OSE-key --create-vpc=no --byo-bastion=yes --bastion-sg=sg-a34ff3af --ami=ami-2051294a

Specify the VPC ID: vpc-11d06976
Specify the first Private subnet within the existing VPC: subnet-3e406466
Specify the second Private subnet within the existing VPC: subnet-66ae905b
Specify the third Private subnet within the existing VPC: subnet-4edfd438
Specify the first Public subnet within the existing VPC: subnet-1f416547
Specify the second Public subnet within the existing VPC: subnet-c2ae90ff
Specify the third Public subnet within the existing VPC: subnet-1ddfd46b

Example of Brownfield Deployment values

    stack_name: dev
    ami: ami-a33668b4
    region: us-east-1
    master_instance_type: m4.xlarge
    node_instance_type: t2.large
    app_instance_type: t2.large
    bastion_instance_type: t2.micro
    keypair: OSE-key
    create_key: no
    key_path: /dev/null
    create_vpc: yes
    vpc_id: vpc-11d06976
    private_subnet_id1: subnet-3e406466
    private_subnet_id2: subnet-66ae905b
    private_subnet_id3: subnet-4edfd438
    public_subnet_id1: subnet-1f416547
    public_subnet_id2: subnet-c2ae90ff
    public_subnet_id3: subnet-1ddfd46b
    byo_bastion: no
    bastion_sg: /dev/null
    console port: 443
    deployment_type: openshift-enterprise
    openshift_sdn: redhat/openshift-ovs-subnet
    public_hosted_zone: sysdeseng.com
    app_dns_prefix: apps
    apps_dns: apps.sysdeseng.com
    rhsm_user: rhsm_user
    rhsm_password: *******
    rhsm_pool: Red Hat OpenShift Container Platform, Standard, 2-Core
    containerized: False
    s3_bucket_name: dev-ocp-registry-sysdeseng
    s3_username: dev-s3-openshift-user
    github_client_id: *******
    github_client_secret: *******
    github_organization: openshift,RHSyseng
    deploy_openshift_metrics: true
    deploy_openshift_logging: true

Continue using these values? [y/N]:

Perform the following steps from the local workstation.

Open a browser and access https://openshift-master.sysdeseng.com/console. When logging into the OpenShift web interface the first time the page will redirect and prompt for GitHub credentials. Log into GitHub using an account that is a member of the Organization specified during the install. Next, GitHub will prompt to grant access to authorize the login. If GitHub access is not granted the account will not be able to login to the OpenShift web console.

To deploy an application, click on the New Project button. Provide a Name and click Create. Next, deploy the jenkins-ephemeral instant app by clicking the corresponding box. Accept the defaults and click Create. Instructions along with a URL will be provided for how to access the application on the next screen. Click Continue to Overview and bring up the management page for the application. Click on the link provided and access the application to confirm functionality.

Perform the following steps from your local workstation.

Install the oc client by visiting the public URL of the OpenShift deployment. For example, https://openshift-master.sysdeseng.com/console/command-line and click latest release. When directed to https://access.redhat.com, login with the valid Red Hat customer credentials and download the client relevant to the current workstation. Follow the instructions located on the production documentation site for getting started with the cli.

A token is required to login using GitHub OAuth and OpenShift. The token is presented on the https://openshift-master.sysdeseng.com/console/command-line page. Click the click to show token hyperlink and perform the following on the workstation in which the oc client was installed.

$ oc login https://openshift-master.sysdeseng.com --token=fEAjn7LnZE6v5SOocCSRVmUWGBNIIEKbjD9h-Fv7p09

After the oc client is configured, create a new project and deploy an application.

$ oc new-project test-app

$ oc new-app https://github.com/openshift/cakephp-ex.git --name=php
--> Found image 2997627 (7 days old) in image stream "php" in project "openshift" under tag "5.6" for "php"

    Apache 2.4 with PHP 5.6
    -----------------------
    Platform for building and running PHP 5.6 applications

    Tags: builder, php, php56, rh-php56

    * The source repository appears to match: php
    * A source build using source code from https://github.com/openshift/cakephp-ex.git will be created
      * The resulting image will be pushed to image stream "php:latest"
    * This image will be deployed in deployment config "php"
    * Port 8080/tcp will be load balanced by service "php"
      * Other containers can access this service through the hostname "php"

--> Creating resources with label app=php ...
    imagestream "php" created
    buildconfig "php" created
    deploymentconfig "php" created
    service "php" created
--> Success
    Build scheduled, use 'oc logs -f bc/php' to track its progress.
    Run 'oc status' to view your app.

$ oc expose service php
route "php" exposed

Display the status of the application.

$ oc status
In project test-app on server https://openshift-master.sysdeseng.com

http://test-app.apps.sysdeseng.com to pod port 8080-tcp (svc/php)
  dc/php deploys istag/php:latest <- bc/php builds https://github.com/openshift/cakephp-ex.git with openshift/php:5.6
    deployment #1 deployed about a minute ago - 1 pod

Access the application by accessing the URL provided by oc status. The CakePHP application should be visible now.

If you try to run the following command, it should fail.

# oc get nodes --show-labels
Error from server: User "sysdes-admin" cannot list all nodes in the cluster

The reason it is failing is because the permissions for that user are incorrect. Get the username and configure the permissions.

$ oc whoami

Once the username has been established, log back into a master node and enable the appropriate permissions for your user. Perform the following step from the first master (ose-master01.sysdeseng.com).

# oadm policy add-cluster-role-to-user cluster-admin sysdesadmin

Attempt to list the nodes again and show the labels.

# oc get nodes --show-labels
NAME                          STATUS                     AGE
ip-10-30-1-164.ec2.internal   Ready                      1d
ip-10-30-1-231.ec2.internal   Ready                      1d
ip-10-30-1-251.ec2.internal   Ready,SchedulingDisabled   1d
ip-10-30-2-142.ec2.internal   Ready                      1d
ip-10-30-2-157.ec2.internal   Ready,SchedulingDisabled   1d
ip-10-30-2-97.ec2.internal    Ready                      1d
ip-10-30-3-74.ec2.internal    Ready,SchedulingDisabled   1d

List the router and registry by changing to the default project.

# oc project default
# oc get all
NAME                         REVISION        DESIRED       CURRENT   TRIGGERED BY
dc/docker-registry           1               3             3         config
dc/router                    1               3             3         config
NAME                         DESIRED         CURRENT       AGE
rc/docker-registry-1         3               3             10m
rc/router-1                  3               3             10m
NAME                         CLUSTER-IP      EXTERNAL-IP   PORT(S)                   AGE
svc/docker-registry          172.30.243.63   <none>        5000/TCP                  10m
svc/kubernetes               172.30.0.1      <none>        443/TCP,53/UDP,53/TCP     20m
svc/router                   172.30.224.41   <none>        80/TCP,443/TCP,1936/TCP   10m
NAME                         READY           STATUS        RESTARTS                  AGE
po/docker-registry-1-2a1ho   1/1             Running       0                         8m
po/docker-registry-1-krpix   1/1             Running       0                         8m
po/router-1-1g84e            1/1             Running       0                         8m
po/router-1-t84cy            1/1             Running       0                         8m

Observe the output of oc get all

The OpenShift Ansible playbooks configure three infrastructure nodes that have three registries running. In order to understand the configuration and mapping process of the registry pods, the command 'oc describe' is used. oc describe details how registries are configured and mapped to the Amazon S3 buckets for storage. Using oc describe should help explain how HA works in this environment.

$ oc describe svc/docker-registry
Name:			docker-registry
Namespace:		default
Labels:			docker-registry=default
Selector:		docker-registry=default
Type:			ClusterIP
IP:			172.30.110.31
Port:			5000-tcp	5000/TCP
Endpoints:		172.16.4.2:5000,172.16.4.3:5000
Session Affinity:	ClientIP
No events.

Notice that the registry has two endpoints listed. Each of those endpoints represents a container. The ClusterIP listed is the actual ingress point for the registries.

The oc client allows similar functionality to the docker command. To find out more information about the registry storage perform the following.

# oc get pods
NAME                      READY     STATUS    RESTARTS   AGE
docker-registry-2-8b7c6   1/1       Running   0          2h
docker-registry-2-drhgz   1/1       Running   0          2h
docker-registry-2-2s2ca   1/1       Running   0          2h

# oc exec docker-registry-2-8b7c6 cat /etc/registry/config.yml
version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  cache:
    layerinfo: inmemory
  s3:
    accesskey: "AKIAJZO3LDPPKZFORUQQ"
    secretkey: "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"
    region: us-east-1
    bucket: "1476274760-openshift-docker-registry"
    encrypt: true
    secure: true
    v4auth: true
    rootdirectory: /registry
auth:
  openshift:
    realm: openshift
middleware:
  repository:
    - name: openshift

Observe the S3 stanza. Confirm the bucket name is listed, and access the AWS console. Click on the S3 AWS and locate the bucket. The bucket should contain content. Confirm that the same bucket is mounted to the other registry via the same steps.

This section will explore the Docker storage on an infrastructure node.

The example below can be performed on any node but for this example the infrastructure node(ose-infra-node01.sysdeseng.com) is used.

The output below describing the Storage Driver: docker—​vol-docker—​pool states that docker storage is not using a loop back device.

$ docker info
Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 4
Server Version: 1.10.3
Storage Driver: devicemapper
 Pool Name: docker--vol-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 3.221 GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 1.221 GB
 Data Space Total: 25.5 GB
 Data Space Available: 24.28 GB
 Metadata Space Used: 307.2 kB
 Metadata Space Total: 29.36 MB
 Metadata Space Available: 29.05 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: bridge null host
 Authorization: rhel-push-plugin
Kernel Version: 3.10.0-327.10.1.el7.x86_64
Operating System: Employee SKU
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 2
Total Memory: 7.389 GiB
Name: ip-10-20-3-46.ec2.internal
ID: XDCD:7NAA:N2S5:AMYW:EF33:P2WM:NF5M:XOLN:JHAD:SIHC:IZXP:MOT3
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Registries: registry.access.redhat.com (secure), docker.io (secure)

Verify 3 disks are attached to the instance. The disk /dev/xvda is used for the OS, /dev/xvdb is used for docker storage, and /dev/xvdc is used for emptyDir storage for containers that do not use a persistent volume.

$ fdisk -l
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Disk /dev/xvda: 26.8 GB, 26843545600 bytes, 52428800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt


#         Start          End    Size  Type            Name
 1         2048         4095      1M  BIOS boot parti
 2         4096     52428766     25G  Microsoft basic

Disk /dev/xvdc: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/xvdb: 26.8 GB, 26843545600 bytes, 52428800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdb1            2048    52428799    26213376   8e  Linux LVM

Disk /dev/mapper/docker--vol-docker--pool_tmeta: 29 MB, 29360128 bytes, 57344 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/docker--vol-docker--pool_tdata: 25.5 GB, 25497174016 bytes, 49799168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/docker--vol-docker--pool: 25.5 GB, 25497174016 bytes, 49799168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 131072 bytes / 524288 bytes


Disk /dev/mapper/docker-202:2-75507787-4a813770697f04b1a4e8f5cdaf29ff52073ea66b72a2fbe2546c469b479da9b5: 3221 MB, 3221225472 bytes, 6291456 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 131072 bytes / 524288 bytes


Disk /dev/mapper/docker-202:2-75507787-260bda602f4e740451c428af19bfec870a47270f446ddf7cb427eee52caafdf6: 3221 MB, 3221225472 bytes, 6291456 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 131072 bytes / 524288 bytes

As mentioned earlier in the document several security groups have been created. The purpose of this section is to encourage exploration of the security groups that were created.

On the main AWS console, click on EC2. Next on the left hand navigation panel select the Security Groups. Click through each group and check out both the Inbound and Outbound rules that were created as part of the infrastructure provisioning. For example, notice how the Bastion security group only allows SSH traffic inbound. That can be further restricted to a specific network or host if required. Next take a look at the Master security group and explore all the Inbound and Outbound TCP and UDP rules and the networks from which traffic is allowed.

As mentioned earlier in the document several ELBs have been created. The purpose of this section is to encourage exploration of the ELBs that were created.

On the main AWS console, click on EC2. Next on the left hand navigation panel select the Load Balancers. Select the ose-master load balancer and on the Description page note the Port Configuration and how it is configured for port 443. That is for the OpenShift web console traffic. On the same tab, check the Availability Zones, note how those are Public subnets. Move to the Instances tab. There should be three master instances running with a Status of InService. Next check the Health Check tab and the options that were configured. Further details of the configuration can be viewed by exploring the Ansible playbooks to see exactly what was configured. Finally, change to the ose-internal-master and compare the subnets. The subnets for the ose-internal-master are all private. They are private because that ELB is reserved for traffic coming from the OpenShift infrastructure to the master servers. This results in reduced charges from Amazon because the packets do not have to be processed by the public facing ELB.

As mentioned earlier in the document a Virtual Private Cloud was created. The purpose of this section is to encourage exploration of the VPC that was created.

On the main Amazon Web Services console, click on VPC. Next on the left hand navigation panel select the Your VPCs. Select the VPC recently created and explore the Summary and Tags tabs. Next, on the left hand navigation panel, explore the Subnets, Route Tables, Internet Gateways, DHCP Options Sets, NAT Gateways, Security Groups and Network ACLs. More detail can be looked at with the configuration by exploring the Ansible playbooks to see exactly what was configured.

Log into the AWS console. On the dashboard, click on the EC2 web service and then click Instances. Locate your running ose-master02.sysdeseng.com instance, select it, right click and change the state to stopped.

Ensure the console can still be accessed by opening a browser and accessing openshift-master.sysdeseng.com. At this point, the cluster is in a degraded state because only 2/3 master nodes are running, but complete functionality remains.

SSH into the first master node (ose-master01.sysdeseng.com). Using the output of the command hostname issue the etcdctl command to confirm that the cluster is healthy.

$ ssh ec2-user@ose-master01.sysdeseng.com
$ sudo -i

# hostname
ip-10-20-1-106.ec2.internal
# etcdctl -C https://ip-10-20-1-106.ec2.internal:2379 --ca-file /etc/etcd/ca.crt --cert-file=/etc/origin/master/master.etcd-client.crt --key-file=/etc/origin/master/master.etcd-client.key cluster-health
failed to check the health of member 82c895b7b0de4330 on https://10.20.2.251:2379: Get https://10.20.1.251:2379/health: dial tcp 10.20.1.251:2379: i/o timeout
member 82c895b7b0de4330 is unreachable: [https://10.20.1.251:2379] are all unreachable
member c8e7ac98bb93fe8c is healthy: got healthy result from https://10.20.3.74:2379
member f7bbfc4285f239ba is healthy: got healthy result from https://10.20.1.106:2379
cluster is healthy

Notice how one member of the ETCD cluster is now unreachable. Restart ose-master02.sysdeseng.com by following the same steps in the AWS web console as noted above.

This section shows what to expect when an infrastructure node fails or is brought down intentionally.

$ oc get projects
NAME               DISPLAY NAME   STATUS
openshift                         Active
openshift-infra                   Active
ttester                           Active
test-app1                         Active
default                           Active
management-infra                  Active

$ oc project test-app1
Now using project "test-app1" on server "https://openshift-master.sysdeseng.com".

$ oc status
In project test-app1 on server https://openshift-master.sysdeseng.com

http://php-test-app1.apps.sysdeseng.com to pod port 8080-tcp (svc/php-prod)
  dc/php-prod deploys istag/php-prod:latest <-
    bc/php-prod builds https://github.com/openshift/cakephp-ex.git with openshift/php:5.6
    deployment #1 deployed 27 minutes ago - 1 pod

# oc whoami -t
feAeAgL139uFFF_72bcJlboTv7gi_bo373kf1byaAT8

# docker pull fedora/apache
# docker images

# oc status
In project default on server https://internal-openshift-master.sysdeseng.com:443

https://docker-registry-default.apps.sysdeseng.com (passthrough) (svc/docker-registry)
  dc/docker-registry deploys docker.io/openshift3/ose-docker-registry:v3.5.5.5
    deployment #1 deployed 44 minutes ago - 3 pods

svc/kubernetes - 172.30.0.1 ports 443, 53->8053, 53->8053

https://registry-console-default.apps.sysdeseng.com (passthrough) (svc/registry-console)
  dc/registry-console deploys registry.access.redhat.com/openshift3/registry-console:3.5
    deployment #1 deployed 43 minutes ago - 1 pod

svc/router - 172.30.41.42 ports 80, 443, 1936
  dc/router deploys docker.io/openshift3/ose-haproxy-router:v3.5.5.5
    deployment #1 deployed 45 minutes ago - 3 pods

View details with 'oc describe <resource>/<name>' or list everything with 'oc get all'.

# docker tag docker.io/fedora/apache 172.30.110.31:5000/openshift/prodapache

# docker images

# docker login -u sysdesadmin -e sysdesadmin -p $(oc whoami -t) 172.30.110.31:5000

# oadm policy add-role-to-user admin sysdesadmin -n openshift
# oadm policy add-role-to-user system:registry sysdesadmin
# oadm policy add-role-to-user system:image-builder sysdesadmin

# docker push 172.30.110.222:5000/openshift/prodapache
The push refers to a repository [172.30.110.222:5000/openshift/prodapache]
389eb3601e55: Layer already exists
c56d9d429ea9: Layer already exists
2a6c028a91ff: Layer already exists
11284f349477: Layer already exists
6c992a0e818a: Layer already exists
latest: digest: sha256:ca66f8321243cce9c5dbab48dc79b7c31cf0e1d7e94984de61d37dfdac4e381f size: 6186

$ oc project default
Now using project "default" on server "https://openshift-master.sysdeseng.com".

$ oc get pods -o wide
NAME                      READY     STATUS    RESTARTS   AGE       IP            NODE
docker-registry-2-gmvdr   1/1       Running   1          21h       172.16.4.2    ip-10-30-1-17.ec2.internal
docker-registry-2-jueep   1/1       Running   0          7h        172.16.3.3    ip-10-30-2-208.ec2.internal
router-1-6y5td            1/1       Running   1          21h       172.16.4.4    ip-10-30-1-17.ec2.internal
router-1-rlcwj            1/1       Running   1          21h       172.16.3.5    ip-10-30-2-208.ec2.internal

NAME                      READY     STATUS    RESTARTS   AGE       IP            NODE
docker-registry-2-gmvdr   1/1       Running   1          21h       172.16.3.6    ip-10-30-2-208.ec2.internal
docker-registry-2-jueep   1/1       Running   0          7h        172.16.3.3    ip-10-30-2-208.ec2.internal
router-1-6y5td            1/1       Running   1          21h       172.16.3.7    ip-10-30-2-208.ec2.internal
router-1-rlcwj            1/1       Running   1          21h       172.16.3.5    ip-10-30-2-208.ec2.internal

From the workstation that was used to perform the installation of OpenShift on AWS run the following to ensure that the newest openshift-ansible playbooks and roles are available and to perform the minor upgrade against the deployed environment.

$ yum update atomic-openshift-utils ansible
$ cd ~/git/openshift-ansible-contrib/reference-architecture/aws-ansible
$ ansible-playbook -i inventory/aws/hosts -e 'stack_name=openshift-infra public_hosted_zone=sysdeseng.com console_port=443 region=us-east-1' playbooks/openshift-minor-upgrade.yaml

$ yum update atomic-openshift-utils ansible
$ cd ~/git/openshift-ansible-contrib/reference-architecture/aws-ansible
$ ansible-playbook -i inventory/aws/hosts -e 'stack_name=openshift-infra public_hosted_zone=sysdeseng.com console_port=443 region=us-east-1 containerized=true' playbooks/openshift-minor-upgrade.yaml

The openshift-minor-update.yaml playbook will not restart the instances after updating occurs. Restarting the nodes including the masters can be completed by adding the following line to the minor-update.yaml playbook.

$ cd ~/git/openshift-ansible-contrib/playbooks
$ vi minor-update.yaml
    openshift_rolling_restart_mode: system

The deployed OpenShift environment may not be the latest major version of OpenShift. The minor-update.yaml allows for a variable to be passed to perform an upgrade on previous versions. Below is an example of performing the upgrade on a 3.3 non-containerized environment.

$ yum update atomic-openshift-utils ansible
$ cd ~/git/openshift-ansible-contrib/reference-architecture/aws-ansible
$ ansible-playbook -i inventory/aws/hosts -e 'stack_name=openshift-infra public_hosted_zone=sysdeseng.com console_port=443 region=us-east-1 openshift_vers=v3_4' playbooks/openshift-minor-upgrade.yaml

When requesting cloud provider specific storage the name, zone, and type are configurable items. Ensure that the zone configuration parameter specified when creating the StorageClass object is an AZ that currently hosts application node instances. This will ensure that PVCs will be created in the correct AZ in which containers are running. The cluster-admin or storage-admin can perform the following commands which will allow for dynamically provisioned gp2 storage on demand, in the us-east-1a AZ, and using the name "standard" as the name of the StorageClass object.

$ vi standard-storage-class.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  zone: us-east-1a

The cluster-admin or storage-admin can then create the StorageClass object using the yaml file.

$ oc create -f standard-storage-class.yaml

Multiple StorageClassess objects can be defined depending on the storage needs of the pods within OpenShift.

The example below shows a dynamically provisioned volume being requested from the StorageClass named standard.

$ vi db-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: db
 annotations:
   volume.beta.kubernetes.io/storage-class: standard
spec:
 accessModes:
  - ReadWriteOnce
 resources:
   requests:
     storage: 10Gi

$ oc create -f db-claim.yaml
persistentvolumeclaim "db" created

Deployment of Container-Native Storage (CNS) on OpenShift Container Platform (OCP) requires at least three OpenShift nodes with at least one unused block storage device attached on each of the nodes. Dedicating three OpenShift nodes to CNS will allow for the configuration of one StorageClass object to be used for applications. If two types of CNS storage are required then a minimum of six CNS nodes must be deployed and configured. This is because only a single CNS container per OpenShift node is supported.

If the CNS instances will serve dual roles such as hosting application pods and glusterfs pods ensure the instances have enough resources to support both operations. CNS hardware requirements state that there must be 32GB of RAM per EC2 instance. There is a current limit of 300 volumes or PVs per 3 node CNS cluster. The CNS EC2 instance type may need to be increased to support 300 volumes.

A python script named add-cns-storage.py is provided in the openshift-ansible-contrib git repository which will deploy three nodes, add the nodes to the OpenShift environment with specific OpenShift labels and attach an EBS volume to each node as an available block device to be used for CNS. Do the following from the workstation performing the deployment of the OpenShift Reference Architecture.

If the Reference Architecture deployment version is >= 3.5, use the deployment option --use-cloudformation-facts to auto-populate some values based on the existing Cloudformations stack.

$ cd /home/<user>/git/openshift-ansible-contrib/reference-architecture/aws-ansible/
$ ./add-cns-storage.py --public-hosted-zone=sysdeseng.com \ --region=us-east-1 --gluster-stack=cns \ --rhsm-pool="Red Hat OpenShift Container Platform, Premium, 2-Core" --keypair=OSE-key *./add-cns-storage.py --gluster-stack=cns --public-hosted-zone=sysdeseng.com \ --rhsm-user=username --rhsm-password=password \ --rhsm-pool="Red Hat OpenShift Container Platform, Premium, 2-Core" \ --keypair=OSE-key --existing-stack=openshift-infra --region=us-east-1 \ --use-cloudformation-facts \

If the Reference Architecture deployment version is < 3.5, fill in the values to represent the existing Cloudformations stack.

$ cd /home/<user>/git/openshift-ansible-contrib/reference-architecture/aws-ansible/
$ ./add-cns-storage.py --gluster-stack=cns --public-hosted-zone=sysdeseng.com \ --rhsm-user=username --rhsm-password=password \ --rhsm-pool="Red Hat OpenShift Container Platform, Premium, 2-Core" \ --keypair=OSE-key --existing-stack=openshift-infra --region=us-east-1 \ --private-subnet-id1=subnet-ad2b23f6 --private-subnet-id2=subnet-7cd61a34 \ --private-subnet-id3=subnet-77e89a4b --node-sg=sg-0c7e0f73 \ --iam-role=backup-NodeInstanceProfile-AX9H0AOAINY3

The following ports must be open on the CNS nodes and on the node security group in AWS. Ensure the following ports defined in the table below are open. This configuration is automatically applied on the nodes deployed using the add-cns-storage.py script.

These activities should be done on the master due to the requirement of setting the node selector. The account performing the CNS activities must be a cluster-admin.

The project name used for this example will be storage but the project name can be whatever value an administrator chooses.

If the CNS nodes will only be used for CNS then a node-selector should be supplied.

# oadm new-project storage --node-selector='role=storage'

If the CNS nodes will serve the role of being used for both CNS and application pods then a node-selector does not need to supplied.

# oadm new-project storage

An oadm policy must be set to enable the deployment of the privileged containers as Red Hat Gluster Storage containers can only run in the privileged mode.

# oc project storage
# oadm policy add-scc-to-user privileged -z default

Perform the following steps from CLI on a local or deployment workstation and ensure that the oc client has been installed and configured. An entitlement for Red Hat Gluster Storage is required to install the Gluster services.

$ subscription-manager repos --enable=rh-gluster-3-for-rhel-7-server-rpms
$ subscription-manager repos --enable=rhel-7-server-rpms
$ yum install -y cns-deploy heketi-client

The Container-Native Storage glusterfs and heketi pods, services, and heketi route are created using the cns-deploy tool which was installed during the prerequisite step.

A heketi topology file is used to create the Trusted Storage Pool. The topology describes the OpenShift nodes that will host Red Hat Gluster Storage services and their attached storage devices. A sample topology file topology-sample.json is installed with the heketi-client package in the /usr/share/heketi/ directory. Below a table shows the different instances running in different AZs to distinguish failure domains defined as a zone in heketi. This information will be used to make intelligent decisions about how to structure a volume that is, to create Gluster volume layouts that have no more than one brick or storage device from a single failure domain. This information will also be used when healing degraded volumes in the event of a loss of device or an entire node.

Below is an example of 3 node topology.json file in the us-east-1 region with /dev/xvdd as the EBS volume or device used for CNS. Edit the values of node.hostnames.manage, node.hostnames.storage, and devices in the topology.json file based on the the OpenShift nodes that have been deployed in the previous step.

# vi gluster-topology.json
{
    "clusters": [
        {
            "nodes": [
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ip-10-20-4-163.ec2.internal"
                            ],
                            "storage": [
                                "10.20.4.163"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/xvdd"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ip-10-20-5-247.ec2.internal"
                            ],
                            "storage": [
                                "10.20.5.247"
                            ]
                        },
                        "zone": 2
                    },
                    "devices": [
                        "/dev/xvdd"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ip-10-20-6-191.ec2.internal"
                            ],
                            "storage": [
                                "10.20.6.191"
                            ]
                        },
                        "zone": 3
                    },
                    "devices": [
                        "/dev/xvdd"
                    ]
                }
            ]
        }
    ]
}

Ensure that the storage project is the current project.

# oc project storage
Already on project "storage" on server "https://openshift-master.sysdeseng.com"

To launch the deployment of CNS the script cns-deploy will be used. It is advised to specify an admin-key and user-key for security reasons when launching the topology. Both admin-key and user-key are user defined values, they do not exist before this step. The heketi admin key (password) will later be used to create a heketi-secret in OpenShift. Be sure to note these values as they will be needed in future operations. The cns-deploy script will prompt the user before proceeding.

# cns-deploy -n storage -g gluster-topology.json --admin-key 'myS3cr3tpassw0rd' --user-key 'mys3rs3cr3tpassw0rd'
Welcome to the deployment tool for GlusterFS on Kubernetes and OpenShift.

Before getting started, this script has some requirements of the execution
environment and of the container platform that you should verify.

The client machine that will run this script must have:
 * Administrative access to an existing Kubernetes or OpenShift cluster
 * Access to a python interpreter 'python'
 * Access to the heketi client 'heketi-cli'

Each of the nodes that will host GlusterFS must also have appropriate firewall
rules for the required GlusterFS ports:
 * 2222  - sshd (if running GlusterFS in a pod)
 * 24007 - GlusterFS Daemon
 * 24008 - GlusterFS Management
 * 49152 to 49251 - Each brick for every volume on the host requires its own
   port. For every new brick, one new port will be used starting at 49152. We
   recommend a default range of 49152-49251 on each host, though you can adjust
   this to fit your needs.

In addition, for an OpenShift deployment you must:
 * Have 'cluster_admin' role on the administrative account doing the deployment
 * Add the 'default' and 'router' Service Accounts to the 'privileged' SCC
 * Have a router deployed that is configured to allow apps to access services
   running in the cluster

Do you wish to proceed with deployment?

[Y]es, [N]o? [Default: Y]: y
Using OpenShift CLI.
NAME      STATUS    AGE
storage   Active    9m
Using namespace "storage".
template "deploy-heketi" created
serviceaccount "heketi-service-account" created
template "heketi" created
template "glusterfs" created
node "ip-10-20-4-163.ec2.internal" labeled
node "ip-10-20-5-247.ec2.internal" labeled
node "ip-10-20-6-191.ec2.internal" labeled
daemonset "glusterfs" created
Waiting for GlusterFS pods to start ... OK
service "deploy-heketi" created
route "deploy-heketi" created
deploymentconfig "deploy-heketi" created
Waiting for deploy-heketi pod to start ... OK
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    17  100    17    0     0      3      0  0:00:05  0:00:05 --:--:--     4
Creating cluster ... ID: 372cf750e0b256fbc8565bb7e4afb434
	Creating node ip-10-20-4-163.ec2.internal ... ID: 9683e22a0f98f8c40ed5c3508b2b4a38
		Adding device /dev/xvdd ... OK
	Creating node ip-10-20-5-247.ec2.internal ... ID: b9bb8fc7be62de3152b9164a7cb3a231
		Adding device /dev/xvdd ... OK
	Creating node ip-10-20-6-191.ec2.internal ... ID: 790bff20ac0115584b5cd8225565b868
		Adding device /dev/xvdd ... OK
Saving heketi-storage.json
secret "heketi-storage-secret" created
endpoints "heketi-storage-endpoints" created
service "heketi-storage-endpoints" created
job "heketi-storage-copy-job" created
deploymentconfig "deploy-heketi" deleted
route "deploy-heketi" deleted
service "deploy-heketi" deleted
job "heketi-storage-copy-job" deleted
secret "heketi-storage-secret" deleted
service "heketi" created
route "heketi" created
deploymentconfig "heketi" created
Waiting for heketi pod to start ... OK
Waiting for heketi pod to start ... OK
heketi is now running.
Ready to create and provide GlusterFS volumes.

After successful deploy validate that there are now 3 glusterfs pods and 1 heketi pod in the storage project.

# oc get pods
NAME              READY     STATUS    RESTARTS   AGE
glusterfs-7gr8y   1/1       Running   0          4m
glusterfs-fhy3e   1/1       Running   0          4m
glusterfs-ouay0   1/1       Running   0          4m
heketi-1-twis5    1/1       Running   0          1m

A route will be created for the heketi service that was deployed during the run of the cns-deploy script. The heketi route URL is used by the heketi-client. The same route URL will be used to create StorageClass objects.

The first step is to find the endpoint for the heketi service and then set the environment variables for the route of the heketi server, the heketi cli user, and the heketi cli key.

# oc get routes heketi
NAME      HOST/PORT  PATH      SERVICES   PORT      TERMINATION
heketi    heketi-storage.apps.sysdeseng.com   heketi     <all>
# export HEKETI_CLI_SERVER=http://heketi-storage.apps.sysdeseng.com
# export HEKETI_CLI_USER=admin
# export HEKETI_CLI_KEY=myS3cr3tpassw0rd

To validate that heketi loaded the topology and has the cluster created execute the following commands:

# heketi-cli topology info
... ommitted ...
# heketi-cli cluster list
Clusters:
372cf750e0b256fbc8565bb7e4afb434

Use the output of the cluster list to view the nodes and volumes within the cluster.

# heketi-cli cluster info 372cf750e0b256fbc8565bb7e4afb434

OpenShift allows for the use of secrets so that items do not need to be stored in clear text. The admin password for heketi, specified during installation with cns-deploy, should be stored in base64-encoding. OpenShift can refer to this secret instead of specifying the password in clear text.

To generate the base64-encoded equivalent of the admin password supplied to the cns-deploy command perform the following.

# echo -n myS3cr3tpassw0rd | base64
bXlzZWNyZXRwYXNzdzByZA==

On the master or workstation with the OpenShift client installed and a user with cluster-admin privileges use the base64 password string in the following YAML to define the secret in OpenShift’s default project or namespace.

# vi heketi-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
data:
  key: bXlzZWNyZXRwYXNzdzByZA==
type: kubernetes.io/glusterfs

Create the secret by using the following command.

# oc create -f heketi-secret.yaml
secret "heketi-secret" created

The StorageClass object created using the CNS components is a more robust solution than using cloud provider specific storage due to the fact that the storage is not dependant on AZs. The cluster-admin or storage-admin can perform the following which will allow for dynamically provisioned CNS storage on demand. The key benefit of this storage is that the persistent storage created can be configured with access modes of ReadWriteOnce(RWO), ReadOnlyMany (ROX), or ReadWriteMany (RWX) adding much more flexibility than cloud provider specific storage.

If Multiple types of CNS storage are desired, additional StorageClass objects can be created to realize multiple tiers of storage defining different types of storage behind a single heketi instance. This will involve deploying more glusterfs pods on additional storage nodes (one gluster pod per OpenShift node) with different type and quality of EBS volumes attached to achieve the desired properties of a tier (e.g. io1 for “fast” storage, magnetic for “slow” storage). For the examples below we will assume that only one type of storage is required.

Perform the following steps from CLI on a workstation or master node where the OpenShift client has been configured.

# oc project storage
# oc get routes heketi
NAME      HOST/PORT                           PATH      SERVICES   PORT      TERMINATION
heketi    heketi-storage.apps.sysdeseng.com             heketi     <all>
# export HEKETI_CLI_SERVER=http://heketi-storage.apps.sysdeseng.com
# export HEKETI_CLI_USER=admin
# export HEKETI_CLI_KEY=myS3cr3tpassw0rd

Record the cluster id of the glusterfs pods in heketi.

# heketi-cli cluster list
   Clusters:
 eb08054c3d42c88f0924fc6a57811610

The StorageClass object requires both the cluster id and the heketi route to be defined to successfully created. Use the information from the output of heketi-cli cluster list and oc get routes heketi to fill in the resturl and clusterid. For OpenShift 3.4, the value of clusterid is not supported for the StorageClass object. If a value is provided the StorageClass object will fail to create for OpenShift version 3.4. The failure occurs because OpenShift 3.4 can only have a single TSP or CNS cluster.

OpenShift 3.4

# vi glusterfs-storageclass-st1.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: gluster-cns-slow
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: http://heketi-storage.apps.sysdeseng.com
  restauthenabled: "true"
  restuser: "admin"
  secretNamespace: "default"
  secretName: "heketi-secret"

The StorageClass object can now be created using this yaml file.

# oc create -f glusterfs-storageclass-st1.yaml

OpenShift 3.5

# vi glusterfs-storageclass-st1.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: gluster-cns-slow
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: http://heketi-storage.apps.sysdeseng.com
  clusterid: eb08054c3d42c88f0924fc6a57811610
  restauthenabled: "true"
  restuser: "admin"
  secretNamespace: "default"
  secretName: "heketi-secret"

The StorageClass object can now be created using this yaml file.

# oc create -f glusterfs-storageclass-st1.yaml

To validate the StorageClass object was created perform the following.

# oc get storageclass gluster-cns-slow
NAME             TYPE
gluster-cns-dd   kubernetes.io/glusterfs
# oc describe storageclass gluster-cns-slow
Name:		gluster-cns-slow
IsDefaultClass:	No
Annotations:	<none>
Provisioner:	kubernetes.io/glusterfs
Parameters:	clusterid=e73da525319cbf784ed27df3e8715ea8,restauthenabled=true,resturl=http://heketi-storage.apps.sysdeseng.com,restuser=admin,secretName=heketi-secret,secretNamespace=default
No events.

The StorageClass object created in the previous section allows for storage to be dynamically provisioned using the CNS resources. The example below shows a dynamically provisioned volume being requested from the gluster-cns-slow StorageClass object A sample persistent volume claim is provided below:

$ oc new-project test
$ oc get storageclass
NAME       TYPE
gluster-cns-slow   kubernetes.io/glusterfs

$ vi db-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: db
 annotations:
   volume.beta.kubernetes.io/storage-class: gluster-cns-slow
spec:
 accessModes:
  - ReadWriteOnce
 resources:
   requests:
     storage: 10Gi

$ oc create -f db-claim.yaml
persistentvolumeclaim "db" created

To deploy an additional glusterfs pool OpenShift requires additional nodes to be available that currently are not running glusterfs pods yet. This will require that another three OpenShift nodes are available in the environment using either the add-cns-storage.py script or by manually deploying three instances and installing and configuring those nodes for OpenShift.

Once the new nodes are available, the next step is to get glusterfs pods up and running on the additional nodes. This is achieved by extending the members of the DaemonSet defined in the first CNS deployment. The storagenode=glusterfs label must be applied to the nodes to allow for the scheduling of the glusterfs pods.

First identify the three nodes that will be added to the CNS cluster and then apply the label.

# oc get nodes
NAME                          STATUS                     AGE
...omitted...
ip-10-20-4-189.ec2.internal   Ready                      5m
ip-10-20-5-204.ec2.internal   Ready                      5m
ip-10-20-6-39.ec2.internal    Ready                      5m
...omitted...

# oc label node ip-10-20-4-189.ec2.internal storagenode=glusterfs
# oc label node ip-10-20-5-204.ec2.internal storagenode=glusterfs
# oc label node ip-10-20-6-39.ec2.internal storagenode=glusterfs

Once the label has been applied then the glusterfs pods will scale from 3 pods to 6. The glusterfs pods will be running on both the newly labeled nodes and the existing nodes.

[subs=+quotes]
# *oc get pods*
NAME              READY     STATUS    RESTARTS   AGE
glusterfs-2lcnb   1/1       Running   0          26m
glusterfs-356cf   1/1       Running   0          26m
glusterfs-fh4gm   1/1       Running   0          26m
glusterfs-hg4tk   1/1       Running   0          2m
glusterfs-v759z   1/1       Running   0          1m
glusterfs-x038d   1/1       Running   0          2m
heketi-1-cqjzm    1/1       Running   0          22m

Wait until all of the glusterfs pods are in READY 1/1 state before continuing. The new pods are not yet configured as a CNS cluster. The new glusterfs pods will be a new CNS cluster after the topology.json file is updated to define the new nodes they reside on and the heketi-cli is executed with this new topology.json file as input.

Modify the topology.json file of the first CNS cluster to include a second entry in the “clusters” list containing the additional nodes. The initial nodes have been omitted from the output below but are still required.

# vi gluster-topology.json
{
  "clusters": [
    {
      "nodes": [
        {
		... nodes from initial cns-deploy call ...
        }
      ]
    },
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "ip-10-20-4-189.ec2.internal"
              ],
              "storage": [
                "10.20.4.189"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/xvdd"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "ip-10-20-5-204.ec2.internal"
              ],
              "storage": [
                "10.20.5.204"
              ]
            },
            "zone": 2
          },
          "devices": [
            "/dev/xvdd"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "ip-10-20-6-39.ec2.internal"
              ],
              "storage": [
                "10.20.6.39"
              ]
            },
            "zone": 3
          },
          "devices": [
            "/dev/xvdd"
          ]
        }
      ]
    }
  ]
}

Using heketi-cli load the modified topology.json file via heketi to trigger the creation of a second cluster using the steps below. The first step is to export the values of the heketi server, user, and key. The HEKET_CLI_KEY value should be the same as that created for the first cluster (set using --admin-key for cns-deploy).

# export HEKETI_CLI_SERVER=http://heketi-storage.apps.sysdeseng.com
# export HEKETI_CLI_USER=admin
# export HEKETI_CLI_KEY=myS3cr3tpassw0rd

With these environment variables exported the next step is to load the topology.json.

# heketi-cli topology load --json=gluster-topology.json
	Found node ip-10-20-4-163.ec2.internal on cluster 372cf750e0b256fbc8565bb7e4afb434
		Found device /dev/xvdd
	Found node ip-10-20-5-247.ec2.internal on cluster 372cf750e0b256fbc8565bb7e4afb434
		Found device /dev/xvdd
	Found node ip-10-20-6-191.ec2.internal on cluster 372cf750e0b256fbc8565bb7e4afb434
		Found device /dev/xvdd
Creating cluster ... ID: 269bb26142a15ee10fa8b1cdeb0a37b7
	Creating node ip-10-20-4-189.ec2.internal ... ID: 0bd56937ef5e5689e003f68a7fde7c69
		Adding device /dev/xvdd ... OK
	Creating node ip-10-20-5-204.ec2.internal ... ID: f79524a4de9b799524c87a4feb41545a
		Adding device /dev/xvdd ... OK
	Creating node ip-10-20-6-39.ec2.internal ... ID: 4ee40aee71b60b0627fb57be3dd2c66e
		Adding device /dev/xvdd ... OK

Observe the second cluster being created and verify that there is a new clusterid created in the console output. Verify you now have a second clusterid and that the correct EC2 nodes are in the new cluster.

# heketi-cli cluster list
# heketi-cli topology info

Create a second StorageClass object via a YAML file similar to the first one with the same heketi route and heketi secret but using the new clusterid and a unique StorageClass object name.

# vi glusterfs-storageclass-gp2.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: gluster-cns-fast
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: http://heketi-storage.apps.sysdeseng.com
  clusterid: 269bb26142a15ee10fa8b1cdeb0a37b7
  restauthenabled: "true"
  restuser: "admin"
  secretNamespace: "default"
  secretName: "heketi-secret"

Using the OpenShift client create the StorageClass object.

# oc create -f glusterfs-storageclass-gp2.yaml

The second StorageClass object will now be available to make storage requests using gluster-cns-fast when creating the PVC.

# vi claim2.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: db
 annotations:
   volume.beta.kubernetes.io/storage-class: gluster-cns-fast
spec:
 accessModes:
  - ReadWriteOnce
 resources:
   requests:
     storage: 10Gi

Deployment of Container-Ready Storage (CRS) requires at least 3 AWS instances with at least one unused block storage device or EBS volume on each node. The instances should have at least 4 CPUs, 32GB RAM, and an unused volume 100GB or larger per node. An entitlement for Red Hat Gluster Storage is also required to install the Gluster services.

A python script named add-crs-storage.py is provided in the openshift-ansible-contrib git repository which will deploy three AWS instances, register the instances, and install the prerequisites for CRS for Gluster on each instance. Perform the following from the workstation where the deployment of the OpenShift Reference Architecture was initiated.

If the Reference Architecture deployment is >= OpenShift 3.5

$ cd /home/<user>/git/openshift-ansible-contrib/reference-architecture/aws-ansible/
./add-crs-storage.py --rhsm-user=username --rhsm-password=password --region=us-east-1 --gluster-stack=crs --public-hosted-zone=sysdeseng.com \ --rhsm-pool="Red Hat Gluster Storage , Standard" --keypair=OSE-key --existing-stack=openshift-infra --use-cloudformation-facts

If the Reference Architecture deployment was performed before 3.5.

$ cd /home/<user>/git/openshift-ansible-contrib/reference-architecture/aws-ansible/
./add-crs-storage.py --rhsm-user=username --rhsm-password=PASSWORD --public-hosted-zone=sysdeseng.com \ --rhsm-pool="Red Hat Gluster Storage , Standard" --keypair=OSE-key --existing-stack=openshift-infra \ --private-subnet-id1=subnet-ad2b23f6 --private-subnet-id2=subnet-7cd61a34 --region=us-east-1 \ --private-subnet-id3=subnet-77e89a4b --node-sg=sg-0c7e0f73 -- bastion-sg=sg-1a2b5a23 --gluster-stack=crs

CRS requires the instances to use the Red Hat Gluster Storage entitlement which which allows access to the rh-gluster-3-for-rhel-7-server-rpms repository containing the required RPMs for a successful installation.

If the add-crs-storage.py script was not used perform the following on the 3 CRS instances to enable the required repository. Ensure the pool that is specified matches a pool available to the RHSM credentials provided (example pool ID shown below).

# subscription-manager register
# subscription-manager attach --pool=8a85f98156981319015699f0183a253c
# subscription-manager repos --enable=rhel-7-server-rpms
# subscription-manager repos --enable=rh-gluster-3-for-rhel-7-server-rpms

The following ports must be opened on the CRS nodes and in the AWS gluster-crs-sg security group. Ensure the following ports defined in the table below are opened. Iptables or firewalld can be used depending on the preference of the Administrator. These steps are done as part of the automated provisioning of the instances with the add-crs-storage.py.

The add-crs-gluster.py uses iptables and creates the rules shown in the table above on each of the CRS nodes. The following commands can be ran on the 3 new instances if the instances were built without using the add-crs-gluster.py script.

# yum -y install firewalld
# systemctl enable firewalld
# systemctl disable iptables
# systemctl stop iptables
# systemctl start firewalld
# firewall-cmd --add-port=24007/tcp --add-port=24008/tcp --add-port=2222/tcp \ --add-port=8080/tcp --add-port=49152-49251/tcp --permanent
# firewall-cmd --reload

The redhat-storage-server package and dependencies will install all of the required RPMs for a successful Red Hat Gluster Storage installation. If the add-crs-storage.py script was not used perform the following on the each of the three CRS instances.

# yum install -y redhat-storage-server

After successful installation enable and start the glusterd.service.

# systemctl enable glusterd
# systemctl start glusterd

Heketi is used to manage the Gluster Trusted Storage Pool(TSP). Heketi is used to perform tasks such as adding volumes, removing volumes, and creating the initial TSP. Heketi can be installed on one of the CRS instances or even within OpenShift if desired. Regardless of whether the add-crs-storage.py script was used or not the following must be performed on the CRS instance chosen to run Heketi. For the steps below the first CRS Gluster instance will be used.

# yum install -y heketi heketi-client

Create the heketi private key on the instance designated to run heketi.

# ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''
# chown heketi:heketi /etc/heketi/heketi_key.pub
# chown heketi:heketi /etc/heketi/heketi_key

Copy the contents of the /etc/heketi/heketi_key.pub into a clipboard and login to each CRS node and paste the contents of the clipboard as a new line into the /home/ec2-user/.ssh/authorized_keys file. This must be done on all 3 instances including the CRS node where the heketi services are running. Also, on each of the 3 instances requiretty must be disabled or removed in /etc/sudoers to allow for management of those hosts using sudo. Ensure that the line below either does not exist in sudoers or that it is commented out.

# visudo
... omitted ...
#Defaults    requiretty
... omitted ...

On the node where Heketi was installed, edit the /etc/heketi/heketi.json file to setup the SSH executor and the admin and user keys. The heketi admin key (password) will be used to create a heketi-secret in OpenShift. This secret will then be used during the creation of the StorageClass object.

# vi /etc/heketi/heketi.json
... omitted ...
"_use_auth": "Enable JWT authorization. Please enable for deployment",
"use_auth": true,

"_jwt": "Private keys for access",
"jwt": {
  "_admin": "Admin has access to all APIs",
  "admin": {
    "key": "myS3cr3tpassw0rd"
  },
  "_user": "User only has access to /volumes endpoint",
  "user": {
    "key": "mys3rs3cr3tpassw0rd"
  }
},

 "glusterfs": {
    "_executor_comment": [
      "Execute plugin. Possible choices: mock, ssh",
      "mock: This setting is used for testing and development.",
      "      It will not send commands to any node.",
      "ssh:  This setting will notify Heketi to ssh to the nodes.",
      "      It will need the values in sshexec to be configured.",
      "kubernetes: Communicate with GlusterFS containers over",
      "            Kubernetes exec api."
    ],
    "executor": "ssh",
    "_sshexec_comment": "SSH username and private key file information",
    "sshexec": {
      "keyfile": "/etc/heketi/heketi_key",
      "user": "ec2-user",
      "sudo": true,
      "port": "22",
      "fstab": "/etc/fstab"
    },
... omitted ...

Restart and enable heketi service to use the configured /etc/heketi/heketi.json file.

# systemctl restart heketi
# systemctl enable heketi

The heketi service should now be running. Heketi provides an endpoint to perform a health check. This validation can be done from either an OpenShift master or from any of the CRS instances.

# curl http://ip-10-20-4-40.ec2.internal:8080/hello
Hello from Heketi

The topology.json is used to tell heketi about the environment and which nodes and storage devices it will manage. There is a sample file located in /usr/share/heketi/topology-sample.json and an example shown below for 3 CRS nodes in 3 zones. Both CRS and CNS use the same format for the topology.json file.

# vi topology.json
{
    "clusters": [
        {
            "nodes": [
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ip-10-20-4-40.ec2.internal"
                            ],
                            "storage": [
                                "10.20.4.40"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/xvdb"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ip-10-20-5-104.ec2.internal"
                            ],
                            "storage": [
                                "10.20.5.104"
                            ]
                        },
                        "zone": 2
                    },
                    "devices": [
                        "/dev/xvdb"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "ip-10-20-6-79.ec2.internal"
                            ],
                            "storage": [
                                "10.20.6.79"
                            ]
                        },
                        "zone": 3
                    },
                    "devices": [
                        "/dev/xvdb"
                    ]
                }
            ]
        }
    ]
}

The HEKETI_CLI_SERVER, HEKETI_CLI_USER, and HEKETI_CLI_KEY environment variables are required for heketi-cli commands to be ran. The HEKETI_CLI_SERVER is the AWS instance name where the heketi services are running. The HEKETI_CLI_KEY is the admin key value configured in the /etc/heketi/heketi.json file.

# export HEKETI_CLI_SERVER=http://ip-10-20-4-40.ec2.internal:8080
# export HEKETI_CLI_USER=admin
# export HEKETI_CLI_KEY=myS3cr3tpassw0rd

Using heketi-cli, run the following command to load the topology of your environment.

# heketi-cli topology load --json=topology.json
	Found node ip-10-20-4-40.ec2.internal on cluster c21779dd2a6fb2d665f3a5b025252849
		Adding device /dev/xvdb ... OK
	Creating node ip-10-20-5-104.ec2.internal ... ID: 53f9e1af44cd5471dd40f3349b00b1ed
		Adding device /dev/xvdb ... OK
	Creating node ip-10-20-6-79.ec2.internal ... ID: 328dfe7fab00a989909f6f46303f561c
		Adding device /dev/xvdb ... OK

From the instance where heketi client is installed and the heketi environment variables has been exported create a Gluster volume to verify heketi.

# heketi-cli volume create --size=50
Name: vol_4950679f18b9fad6f118b2b20b0c727e
Size: 50
Volume Id: 4950679f18b9fad6f118b2b20b0c727e
Cluster Id: b708294a5a2b9fed2430af9640e7cae7
Mount: 10.20.4.119:vol_4950679f18b9fad6f118b2b20b0c727e
Mount Options: backup-volfile-servers=10.20.5.14,10.20.6.132
Durability Type: replicate
Distributed+Replica: 3

The command gluster volume info can provide further information on the newly created Gluster volume.

# gluster volume info

Volume Name: vol_f2eec68b2dea1e6c6725d1ca3f9847a4
Type: Replicate
Volume ID: f001d6e9-fee4-4e28-9908-359bbd28b8f5
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.20.5.104:/var/lib/heketi/mounts/vg_9de785372f550942e33d0f3abd8cd9ab/brick_03cfb63f8293238affe791032ec779c2/brick
Brick2: 10.20.4.40:/var/lib/heketi/mounts/vg_ac91b30f6491c571d91022d24185690f/brick_2b60bc032bee1be7341a2f1b5441a37f/brick
Brick3: 10.20.6.79:/var/lib/heketi/mounts/vg_34faf7faaf6fc469298a1c15a0b2fd2f/brick_d7181eb86f3ca37e4116378181e28855/brick
Options Reconfigured:
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on

OpenShift allows for the use of secrets so that items do not need to be stored in clear text. The admin password for heketi, specified during configuration of the heketi.json file, should be stored in base64-encoding. OpenShift can refer to this secret instead of specifying the password in clear text.

To generate the base64-encoded equivalent of the admin password supplied to the cns-deploy command perform the following.

# echo -n myS3cr3tpassw0rd | base64
bXlzZWNyZXRwYXNzdzByZA==

On the master or workstation with the OpenShift client installed with cluster-admin privileges use the base64 password string in the following YAML to define the secret in OpenShift’s default namespace.

# vi heketi-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
data:
  key: bXlzZWNyZXRwYXNzdzByZA==
type: kubernetes.io/glusterfs

Create the secret by using the following command.

# oc create -f heketi-secret.yaml
secret "heketi-secret" created

CRS storage has all of the same benefits that CNS storage has with regard to OpenShift storage. The cluster-admin or storage-admin can perform the following which will allow for dynamically provisioned CRS storage on demand. The key benefit of this storage is that the persistent storage can be created with access modes ReadWriteOnce(RWO), ReadOnlyMany(ROX), or ReadWriteMany(RWX) adding more flexibility than cloud provider specific storage.

A StorageClass object requires certain parameters to be defined to successfully create the resource. Use the values of the exported environment variables from the previous steps to define the resturl, restuser, secretNamespace, and secretName.

# vi storage-crs.json
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: crs-slow-st1
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://ip-10-20-4-40.ec2.internal:8080"
  restauthenabled: "true"
  restuser: "admin"
  secretNamespace: "default"
  secretName: "heketi-secret"

Once the Storage Class json file has been created use the oc create command to create the object in OpenShift.

# oc create -f storage-crs.json

To validate the Storage Class was created perform the following.

# oc get storageclass
NAME             TYPE
crs-slow-st1 kubernetes.io/glusterfs

# oc describe storageclass crs-slow-st1
Name:		crs-slow-st1
IsDefaultClass:	No
Annotations:	storageclass.beta.kubernetes.io/is-default-class=true
Provisioner:	kubernetes.io/glusterfs
Parameters:	restauthenabled=true,resturl=http://ip-10-20-4-40.ec2.internal:8080,
restuser=admin,secretName=heketi-secret,secretNamespace=default
No events.

The Storage Class created in the previous section allows for storage to be dynamically provisioned using the CRS resources. The example below shows a dynamically provisioned volume being requested from the crs-slow-st1 StorageClass object. A sample persistent volume claim is provided below:

$ oc new-project test

$ vi db-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: db
 annotations:
   volume.beta.kubernetes.io/storage-class: crs-slow-st1
spec:
 accessModes:
  - ReadWriteOnce
 resources:
   requests:
     storage: 10Gi

$ oc create -f db-claim.yaml
persistentvolumeclaim "db" created

The .ssh/config will need to reflect both the existing environment and the new environment. Below is an example. The environment of dev will be the existing deployment and prod will be the new deployment.

Host dev
     Hostname                 bastion.dev.sysdeseng.com
     user                       ec2-user
     StrictHostKeyChecking      no
     ProxyCommand               none
     CheckHostIP                no
     ForwardAgent               yes
     IdentityFile               /home/<user>/.ssh/id_rsa

Host *.dev.sysdeseng.com
     ProxyCommand               ssh ec2-user@dev -W %h:%p
     user                       ec2-user
     IdentityFile               /home/<user>/.ssh/id_rsa

Host prod
     Hostname                   bastion.prod.sysdeseng.com
     user                       ec2-user
     StrictHostKeyChecking      no
     ProxyCommand               none
     CheckHostIP                no
     ForwardAgent               yes
     IdentityFile               /home/<user>/.ssh/id_rsa

Host *.prod.sysdeseng.com
     ProxyCommand               ssh ec2-user@prod -W %h:%p
     user                       ec2-user
     IdentityFile               /home/<user>/.ssh/id_rsa