Per Google; "Google Compute Engine delivers virtual machines running in Google’s innovative data centers and worldwide fiber network. Compute Engine’s tooling and workflow support enable scaling from single instances to global, load-balanced cloud computing."

Within this reference environment, the instances are deployed in multiple Regions and Zones in the us-central1-a zone as the default. However, the default region can be changed during the deployment portion of the reference architecture to a zone closer to the user’s geo-location. The master instances for the Container Platform environment are machine type n1-standard-4. The node instances are machine type n1-standard-2 and are provided two extra disks, which are used for Docker storage and OpenShift Container Platform ephemeral volumes. The bastion host is deployed as machine type g1-small. Instance sizing can be changed in the variable file for the installer, which is covered in a later chapter.

For more on sizing and choosing zones and regions, see https://cloud.google.com/compute

Google provides persistent block storage for your virtual machines as both standard (HDD) and solid-state (SSD) disks. This reference implementation will be utilizing Solid-State Persistent disks as they provide better performance for random input/output operations per second.

Google’s persistent storage has the following attributes:

Within this reference environment, the node instances for the OpenShift Container Platform environment are provisioned with an extra 25GB Solid-State Persistent disk for Docker storage and a 50GB disk of the same type used for OpenShift Container Platform ephemeral volumes. Disk sizing can be changed in the variable file for the installer, which is covered in a later chapter.

For more information see https://cloud.google.com/compute/docs/disks

Google provides images of many popular Operating Systems including Red Hat and CentOS, which can be used to deploy infrastructure. They also provide the ability to upload and create custom images, which has the added appeal of being pre-configured with common packages and services common for every OpenShift Container Platform master and node instance. This feature speeds up the couple of initial steps needed for each instance, thus improving the overall deployment time. For this reference architecture a image provided by Red Hat will be uploaded and pre-configured for use as the base image for all OpenShift Container Platform instances.

This reference architecture will also take advantage of GCE’s instance template feature. This allows for all the required information (such as a machine type, disk image, network, and zone) to be turned into templates for each the OpenShift Container Platform instance types. These custom instance templates are then used to successfully launch both OpenShift Container Platform masters and nodes.

For more information see https://cloud.google.com/compute/docs/creating-custom-image and https://cloud.google.com/compute/docs/instance-templates

GCP has a global infrastructure that covers 13 regions and 39 availability zones. Per Google; "Certain Compute Engine resources live in regions or zones. A region is a specific geographical location where you can run your resources. Each region has one or more zones. For example, the us-central1 region denotes a region in the Central United States that has zones us-central1-a, us-central1-b, us-central1-c and us-central1-f."

For this reference architecture, we will default to the "us-central1" region. Minimum requirement for the region is that it has to have at least 3 available zones. To check which region and zone your user is set to use, the following command will provide the answer:

$ gcloud config configurations list
NAME    IS_ACTIVE ACCOUNT             PROJECT     DEFAULT_ZONE  DEFAULT_REGION
default True      chmurphy@redhat.com ose-refarch us-central1-a us-central1

In the next chapter, we will go deeper into regions and zones and discuss how to set them using the gcloud tool as well as in the variables file for the infrastructure setup.

For more information see https://cloud.google.com/compute/docs/zones

A Load Balancing service enables distribution of traffic across multiple instances in the GCE cloud. There are clear advantages to doing this:

GCP provides five kinds of Load Balancing, HTTP(S), SSL Proxy, TCP Proxy, Network based and Internal. This implementation guide will use HTTP(S) and Network Load Balancing.

For more information see https://cloud.google.com/compute/docs/load-balancing-and-autoscaling

GCE LB implements parts of the Section 2.12, “Routing Layer”.

The Public Cloud DNS zone requires a domain name either purchased through Google’s "Domains" service or through one of many 3rd party providers. Once the zone is created in Cloud DNS, the name servers provided by Google will need to be added to the registrar.

For more information see https://domains.google

The master nodes contain the master components, including the API server, controller manager server and ETCD. The master maintains the clusters configuration, manages nodes in its OpenShift Container Platform cluster. The master assigns pods to nodes and synchronizes pod information with service configuration. The master is used to define routes, services, and volume claims for pods deployed within the OpenShift Container Platform environment.

For a highly available OpenShift deployment deploy at least 3 masters behind a load balancer.

The infrastructure nodes are used for the router and registry pods. These nodes could be used if the optional components like Kibana and Hawkular metrics are required. The storage for the Docker registry that is deployed on the infrastructure nodes is backed by a GCS bucket, which allows for multiple pods to use the same storage. GCS storage is used because it is synchronized between the availability zones, providing data redundancy.

To ensure the least impact when performing upgrades of OpenShift deploy at least 3 infrastructure nodes. The router and registry components must be running on all infrastructure nodes to have the least chance of downtime during the upgrade.

The Application nodes are the instances where non-infrastructure based containers run. Depending on the application, GCE specific storage can be applied such as a Block Storage which can be assigned using a Persistent Volume Claim for application data that needs to persist between container restarts. A configuration parameter is set on the master which ensures that OpenShift Container Platform user containers will be placed on the application nodes by default.

All OpenShift Container Platform nodes are assigned a label. This allows certain pods to be deployed on specific nodes. For example, nodes labeled infra are Infrastructure nodes. These nodes run the router and registry pods. Nodes with the label app are nodes used for end user Application pods. The configuration parameter 'defaultNodeSelector: "role=app" in /etc/origin/master/master-config.yaml ensures all projects automatically are deployed on Application nodes.

Node labels can also allow for segmentation of the OpenShift environment. If an organization would like to run development applications on a series of instances and production on another set of instances node labels can be applied to those instances to ensure that applications for development are launched on the development nodes and the production nodes only run those applications that have the label for production. This segmentation also shines with the multitennant SDN plugin which isolates pods and services unless explicitly specified.

The installation of OpenShift Container Platform (OCP) requires a valid Red Hat subscription. During the installation of OpenShift the Ansible redhat_subscription module will attempt to register the instances. The script will fail if the OpenShift entitlements are exhausted. For the installation of OCP on GCP the following items are required:

The items above are examples and should reflect subscriptions relevant to the account performing the installation. There are a few different variants of the OpenShift Container Platform Subscription Name. It is advised to visit https://access.redhat.com/management/subscriptions to find the specific Subscription Name as the value will be used below during the deployment.

Download the latest Red Hat Enterprise Linux 7.x KVM image from the Customer Portal.

Figure 3.1. Red Hat Enterprise Linux 7.x KVM Image Download

It is advised that the downloaded KVM image be placed in the ~/Downloads/ directory. If another directory is used to store the downloaded .qcow2 file, write down the path to that directory for later use in variables file. The KVM image file that gets downloaded will later be used to create the Custom Image for use in GCE.

Documentation for the GCP and the Google Cloud Platform Console can be found on the following link: https://cloud.google.com/compute/docs. To access the Google Cloud Platform Console, a Google account is needed.

This reference architecture assumes that a "Greenfield" deployment is being done in a newly created GCP project. To set that up, log into the GCP console home and select the Project button:

Figure 3.2. Create New Project

Next, name your new project. The name ocp-refarch was used for this reference architecture, and can be used for your project also:

Figure 3.3. Name New Project

The next step will be to set up billing for GCP so you are able to create new resources. Go to the Billing tab in the GCP Console and select Enable Billing. The new project can be linked to an existing project or financial information can be entered:

Figure 3.4. Enable Billing

Figure 3.5. Link to existing account

In this reference implementation guide, the domain sysdeseng.com was purchased through an external provider and the subdomain gce.sysdeseng.com will be managed by Cloud DNS. For the example below, the domain gce.sysdeseng.com will represent the Publicly Hosted Domain Name used for the installation of OpenShift Container Platform, however, your Publicly Hosted Domain Name will need to be substituted for the actual installation. Follow the below instructions to add your Publicly Hosted Domain Name to Cloud DNS.

Once the Pubic Zone is created, select the gce-sysdeseng-com domain and copy the Google Name Servers to add to your external registrar if applicable.

In order to obtain new OAuth credentials in GCP, the OAuth consent screen must be configured. Browse to the API Manager screen in the GCP console and select Credentials. Select the "OAuth consent screen" tab as shown in the image below:

Figure 3.6. Google OAuth consent screen

Clicking save will bring you back to the API Manager → Credentials tab. Next click on the Create Credentials dropdown and select OAuth Client ID

Figure 3.7. Select OAuth Client ID

On the the Create Client ID page:

Figure 3.8. New Client ID Page

Copy down the Client ID and Client Secret for use in the provisioning script.

The set of packages is required on the host running the deployment:

$ sudo yum update
$ sudo yum install curl python which tar qemu-img openssl git ansible python-libcloud python2-jmespath java-1.8.0-openjdk-headless httpd-tools python2-passlib

If you are running the deployment from RHEL 7 machine, enable required repositories and install the OpenShift Ansible Installer package:

$ sudo subscription-manager repos --enable rhel-7-server-rpms --enable rhel-7-server-extras-rpms --enable rhel-7-fast-datapath-rpms --enable rhel-7-server-ose-3.6-rpms
$ sudo yum install atomic-openshift-utils

To manage your Google Cloud Platform resources, download and install the Google Cloud SDK with the gcloud command-line tool.

Google provides repository which can be used to install Google Cloud SDK on RHEL 7 or Fedora based OS:

$ sudo tee -a /etc/yum.repos.d/google-cloud-sdk.repo << EOM
[google-cloud-sdk]
name=Google Cloud SDK
baseurl=https://packages.cloud.google.com/yum/repos/cloud-sdk-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
       https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOM

Now it’s easy to install the SDK and initialize the gcloud utility:

$ sudo yum install google-cloud-sdk
$ gcloud init

More information about the Google Cloud SDK can be found in the documentation.

$ gcloud config set project PROJECT-ID

$ gcloud compute zones list
NAME                    REGION                STATUS  NEXT_MAINTENANCE  TURNDOWN_DATE
asia-east1-b            asia-east1            UP
asia-east1-c            asia-east1            UP
asia-east1-a            asia-east1            UP
asia-northeast1-b       asia-northeast1       UP
asia-northeast1-c       asia-northeast1       UP
asia-northeast1-a       asia-northeast1       UP
asia-south1-c           asia-south1           UP
asia-south1-a           asia-south1           UP
asia-south1-b           asia-south1           UP
asia-southeast1-a       asia-southeast1       UP
asia-southeast1-b       asia-southeast1       UP
australia-southeast1-a  australia-southeast1  UP
australia-southeast1-b  australia-southeast1  UP
australia-southeast1-c  australia-southeast1  UP
europe-west1-c          europe-west1          UP
europe-west1-b          europe-west1          UP
europe-west1-d          europe-west1          UP
europe-west2-a          europe-west2          UP
europe-west2-b          europe-west2          UP
europe-west2-c          europe-west2          UP
europe-west3-c          europe-west3          UP
europe-west3-b          europe-west3          UP
europe-west3-a          europe-west3          UP
southamerica-east1-a    southamerica-east1    UP
southamerica-east1-b    southamerica-east1    UP
southamerica-east1-c    southamerica-east1    UP
us-central1-f           us-central1           UP
us-central1-b           us-central1           UP
us-central1-a           us-central1           UP
us-central1-c           us-central1           UP
us-east1-c              us-east1              UP
us-east1-d              us-east1              UP
us-east1-b              us-east1              UP
us-east4-a              us-east4              UP
us-east4-b              us-east4              UP
us-east4-c              us-east4              UP
us-west1-c              us-west1              UP
us-west1-b              us-west1              UP
us-west1-a              us-west1              UP

The OCP installation playbooks allow for OpenShift to be installed in containers. If the containerized installation of OpenShift is preferred, set containerized: true in the config.yaml file.

OpenShift Cluster Metrics can be deployed during installation. To enable metrics deployment, set openshift_hosted_metrics_deploy: true in the config.yaml. This includes creating a PVC dynamically which is used for persistent metrics storage. The URL of the Hawkular Metrics endpoint is also set on the masters and defined by an OpenShift route. Metrics components are deployed on nodes with the label of "role=infra".

OpenShift Cluster Logging can be deployed during installation. To enable metrics deployment, set openshift_hosted_logging_deploy: true in the config.yaml. This includes creating three PVCs dynamically which are used for storing logs in a redundant and durable way. Logging components are deployed on nodes with the label of "role=infra".

In case you need to re-validate the OpenShift deployment, run:

$ ./ocp-on-gcp.sh --validation

Perform the following steps from your local workstation.

Open a browser and access https://master.gce.sysdeseng.com/console. When logging into the OpenShift Container Platform web interface the first time the page will redirect and prompt for Google OAuth credentials. Log into Google using an account that is a member of the account specified during the install. Next, Google will prompt to grant access to authorize the login. If Google access is not granted the account will not be able to login to the OpenShift Container Platform web console.

To deploy an application, click on the New Project button. Provide a Name and click Create. Next, deploy the jenkins-ephemeral instant app by clicking the corresponding box. Accept the defaults and click Create. Instructions along with a URL will be provided for how to access the application on the next screen. Click Continue to Overview and bring up the management page for the application. Click on the link provided and access the application to confirm functionality.

Perform the following steps from your local workstation.

Install the oc client by visiting the public URL of the OpenShift Container Platform deployment. For example, https://master.gce.sysdeseng.com/console/command-line and click latest release. When directed to https://access.redhat.com, login with the valid Red Hat customer credentials and download the client relevant to the current workstation. Follow the instructions located on the production documentation site for getting started with the cli.

A token is required to login using Google OAuth and OpenShift Container Platform. The token is presented on the https://master.gce.sysdeseng.com/console/command-line URL. Click the link to obtain a token and then perform the following on the workstation in which the oc client was installed.

$ oc login https://master.gce.sysdeseng.com --token=fEAjn7LnZE6v5SOocCSRVmUWGBNIIEKbjD9h-Fv7p09

After the oc client is configured, create a new project and deploy an application.

$ oc new-project test-app

$ oc new-app https://github.com/openshift/cakephp-ex.git --name=php
--> Found image 0d2f8fa (5 weeks old) in image stream "openshift/php" under tag "7.0" for "php"

    Apache 2.4 with PHP 7.0
    -----------------------
    PHP 7.0 available as docker container is a base platform for building and running various PHP 7.0 applications and frameworks. PHP is an HTML-embedded scripting language. PHP attempts to make it easy for developers to write dynamically generated web pages. PHP also offers built-in database integration for several commercial and non-commercial database management systems, so writing a database-enabled webpage with PHP is fairly simple. The most common use of PHP coding is probably as a replacement for CGI scripts.

    Tags: builder, php, php70, rh-php70

    * The source repository appears to match: php
    * A source build using source code from https://github.com/openshift/cakephp-ex.git will be created
      * The resulting image will be pushed to image stream "php:latest"
      * Use 'start-build' to trigger a new build
    * This image will be deployed in deployment config "php"
    * Port 8080/tcp will be load balanced by service "php"
      * Other containers can access this service through the hostname "php"

--> Creating resources ...
    imagestream "php" created
    buildconfig "php" created
    deploymentconfig "php" created
    service "php" created
--> Success
    Build scheduled, use 'oc logs -f bc/php' to track its progress.
    Run 'oc status' to view your app.

$ oc expose service php
route "php" exposed

Display the status of the application.

$ oc status
In project test-app on server https://master.gce.sysdeseng.com:443

http://php-test-app.apps.gce.sysdeseng.com to pod port 8080-tcp (svc/php)
  dc/php deploys istag/php:latest <-
    bc/php source builds https://github.com/openshift/cakephp-ex.git on openshift/php:7.0
    deployment #1 deployed about a minute ago - 1 pod

View details with 'oc describe <resource>/<name>' or list everything with 'oc get all'.

Access the application by accessing the URL provided by oc status. The CakePHP application should be visible now.

If you try to run the following command, it should fail.

$ oc get nodes --show-labels
Error from server: User "user@example.com" cannot list all nodes in the cluster

The reason it is failing is because the permissions for that user are incorrect. Get the username and configure the permissions.

$ oc whoami

Once the username has been established, log back into a master node and enable the appropriate permissions for your user. Perform the following step from the first master (ocp-master-01.gce.sysdeseng.com).

# oadm policy add-cluster-role-to-user cluster-admin user@example.com

Attempt to list the nodes again and show the labels.

$ oc get nodes --show-labels
NAME                  STATUS                     AGE       VERSION             LABELS
ocp-infra-node-4k4l   Ready                      32m       v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=n1-standard-2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-b,kubernetes.io/hostname=ocp-infra-node-4k4l,role=infra
ocp-infra-node-rwjl   Ready                      32m       v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=n1-standard-2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-c,kubernetes.io/hostname=ocp-infra-node-rwjl,role=infra
ocp-infra-node-stl6   Ready                      32m       v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=n1-standard-2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-f,kubernetes.io/hostname=ocp-infra-node-stl6,role=infra
ocp-master-97gr       Ready,SchedulingDisabled   32m       v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=n1-standard-2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-c,kubernetes.io/hostname=ocp-master-97gr,role=master
ocp-master-rmtb       Ready,SchedulingDisabled   32m       v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=n1-standard-2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-b,kubernetes.io/hostname=ocp-master-rmtb,role=master
ocp-master-rscb       Ready,SchedulingDisabled   32m       v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=n1-standard-2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-f,kubernetes.io/hostname=ocp-master-rscb,role=master
ocp-node-9xt0         Ready                      32m       v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=n1-standard-2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-b,kubernetes.io/hostname=ocp-node-9xt0,role=app
ocp-node-mh4w         Ready                      32m       v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=n1-standard-2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-c,kubernetes.io/hostname=ocp-node-mh4w,role=app
ocp-node-z8rv         Ready                      32m       v1.6.1+5115d708d7   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=n1-standard-2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-f,kubernetes.io/hostname=ocp-node-z8rv,role=app

List the router and registry by changing to the default project.

$ oc project default
$ oc get all
NAME                  DOCKER REPO                                                 TAGS      UPDATED
is/registry-console   docker-registry.default.svc:5000/default/registry-console   v3.6      23 minutes ago

NAME                  REVISION   DESIRED   CURRENT   TRIGGERED BY
dc/docker-registry    1          3         3         config
dc/registry-console   1          1         1         config
dc/router             1          3         3         config

NAME                    DESIRED   CURRENT   READY     AGE
rc/docker-registry-1    3         3         3         24m
rc/registry-console-1   1         1         1         23m
rc/router-1             3         3         3         27m

NAME                      HOST/PORT                                         PATH      SERVICES           PORT      TERMINATION   WILDCARD
routes/docker-registry    docker-registry-default.apps.gce.sysdeseng.com              docker-registry    <all>     passthrough   None
routes/registry-console   registry-console-default.apps.gce.sysdeseng.com             registry-console   <all>     passthrough   None

NAME                   CLUSTER-IP       EXTERNAL-IP   PORT(S)                   AGE
svc/docker-registry    172.30.139.160   <none>        5000/TCP                  25m
svc/kubernetes         172.30.0.1       <none>        443/TCP,53/UDP,53/TCP     56m
svc/registry-console   172.30.41.250    <none>        9000/TCP                  23m
svc/router             172.30.91.254    <none>        80/TCP,443/TCP,1936/TCP   27m

NAME                          READY     STATUS    RESTARTS   AGE
po/docker-registry-1-mxqc4    1/1       Running   0          24m
po/docker-registry-1-qp05k    1/1       Running   0          24m
po/docker-registry-1-s3kp9    1/1       Running   0          24m
po/registry-console-1-gn1kg   1/1       Running   0          22m
po/router-1-60q15             1/1       Running   0          27m
po/router-1-m4swp             1/1       Running   0          27m
po/router-1-qq86v             1/1       Running   0          27m

Observe the output of oc get all.

The OpenShift Ansible playbooks configure three infrastructure nodes that have three registries running. In order to understand the configuration and mapping process of the registry pods, the command oc describe is used. The oc describe details how registries are configured and mapped to the GCS for storage. Using oc describe should help explain how HA works in this environment.

$ oc describe svc/docker-registry
Name:              docker-registry
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          docker-registry=default
Type:              ClusterIP
IP:                172.30.139.160
Port:              5000-tcp  5000/TCP
Endpoints:         172.16.14.3:5000,172.16.16.2:5000,172.16.4.3:5000
Session Affinity:  ClientIP
Events:            <none>

Notice that the registry has three endpoints listed. Each of those endpoints represents a container. The ClusterIP listed is the actual ingress point for the registries.

The oc client allows similar functionality to the docker command. To find out more information about the registry storage perform the following.

$ oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-mxqc4    1/1       Running   0          27m
docker-registry-1-qp05k    1/1       Running   0          27m
docker-registry-1-s3kp9    1/1       Running   0          27m
registry-console-1-gn1kg   1/1       Running   0          25m
router-1-60q15             1/1       Running   0          29m
router-1-m4swp             1/1       Running   0          29m
router-1-qq86v             1/1       Running   0          29m

$ oc exec docker-registry-1-mxqc4 cat /etc/registry/config.yml
version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  gcs:
    bucket: ose-refarch-ocp-registry-bucket
    rootdirectory: /registry
auth:
  openshift:
    realm: openshift
middleware:
  registry:
  - name: openshift
  repository:
  - name: openshift
    options:
      pullthrough: True
      acceptschema2: True
      enforcequota: True
  storage:
  - name: openshift

Observe the GCS stanza. Confirm the bucket name is listed, and access the GCP console. Click on the "Storage" tab and locate the bucket. The bucket should contain the /registry content as seen in the image below.

Figure 4.2. Registry Bucket in GCS

Confirm that the same bucket is mounted to the other registries via the same steps.

This section will explore the Docker storage on an infrastructure node.

The example below can be performed on any node but for this example the infrastructure node(ocp-infra-01.gce.sysdeseng.com) is used.

The output below verifies docker storage is not using a loop back device.

# docker info
Containers: 4
 Running: 4
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 1.12.6
Storage Driver: overlay2
 Backing Filesystem: xfs
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: host null bridge overlay
 Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp selinux
Kernel Version: 3.10.0-693.1.1.el7.x86_64
Operating System: Employee SKU
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 7.147 GiB
Name: ocp-infra-node-4k4l
ID: FNH2:VZ2O:LQ4V:CDDY:LLYE:6YVN:VEHU:4735:X4EW:KK6F:KV7H:WLSN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://registry.access.redhat.com/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8
Registries: registry.access.redhat.com (secure), registry.access.redhat.com (secure), docker.io (secure)

Verify 3 disks are attached to the instance. The disk /dev/sda is used for the OS, /dev/sdb is used for docker storage, and /dev/sdc is used for emptyDir storage for containers that do not use a persistent volume.

# fdisk -l
Disk /dev/sda: 26.8 GB, 26843545600 bytes, 52428800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0x000a73bb

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048    52428766    26213359+  83  Linux

Disk /dev/sdc: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdb: 26.8 GB, 26843545600 bytes, 52428800 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

As mentioned earlier in the document several firewall rules have been created. The purpose of this section is to encourage exploration of the firewall rules that were created.

On the main GCP console, click on the left hand navigation panel select the VPC network. Select Firewall rules and check the rules that were created as part of the infrastructure provisioning. For example, notice how the ocp-ssh-external firewall rule only allows SSH traffic inbound. That can be further restricted to a specific network or host if required.

As mentioned earlier in the document several Load Balancers have been created. The purpose of this section is to encourage exploration of the Load Balancers that were created.

On the main GCP console, click on the left hand navigation panel and select the Network services → Load Balancing. Select the ocp-master-https-lb-url-map load balancer and note the Frontend and how it is configured for port 443. That is for the OpenShift Container Platform web console traffic. On the same tab, notice that the ocp-master-https-lb-cert is offloaded at the load balancer level and therefore not terminated on the masters. On the Backend, there should be three healthy master instances running. Next check the ocp-master-network-lb-pool load balancer to see that it contains the three masters that make up the Backend from the ocp-master-https-lb-url-map load balancer. Further details of the configuration can be viewed by exploring the Ansible playbooks to see exactly what was configured.

As mentioned earlier in the document, a network was created especially for OCP. The purpose of this section is to encourage exploration of the network that was created.

On the main GCP console, click on the left hand navigation panel select the VPC network → VPC networks. Select the ocp-network recently created and explore the Subnets, Firewall Rules, and Routes. More detail can be looked at with the configuration by exploring the Ansible playbooks to see exactly what was configured.

On the main GCP console, click on the left hand navigation panel select the VM Instances. Locate your running ocp-master-02.gce.sysdeseng.com instance, select it, and click on STOP.

Ensure the console can still be accessed by opening a browser and accessing https://master.gce.sysdeseng.com/. At this point, the cluster is in a degraded state because only 2/3 master nodes are running, but complete functionality remains.

SSH into the first master node (ocp-master-01.gce.sysdeseng.com). Using the output of the command hostname issue the etcdctl command to confirm that the cluster is healthy.

$ ssh ocp-master-01
$ sudo -i

# hostname -i
10.128.0.8
# ETCD_ENDPOINT=$(hostname) && etcdctl -C https://$ETCD_ENDPOINT:2379 --ca-file /etc/etcd/ca.crt --cert-file=/etc/origin/master/master.etcd-client.crt --key-file=/etc/origin/master/master.etcd-client.key cluster-health
member 8499e5795394a44 is healthy: got healthy result from https://10.128.0.8:2379
failed to check the health of member 36d79a9d80d2baca on https://10.128.0.10:2379: Get https://10.128.0.10:2379/health: dial tcp 10.128.0.10:2379: i/o timeout
member 36d79a9d80d2baca is unreachable: [https://10.128.0.10:2379] are all unreachable
member abdd0db2d8da61c7 is healthy: got healthy result from https://10.128.0.3:2379
cluster is healthy

Notice how one member of the ETCD cluster is now unreachable. Restart ocp-master-02.gce.sysdeseng.com by following the same steps in the GCP web console as noted above.

This section shows what to expect when an infrastructure node fails or is brought down intentionally.

$ oc get projects
NAME               DISPLAY NAME   STATUS
default                           Active
kube-public                       Active
kube-system                       Active
logging                           Active
management-infra                  Active
openshift                         Active
openshift-infra                   Active
test-app                          Active

$ oc project test-app
Now using project "test-app" on server "https://master.gce.sysdeseng.com".

$ oc status
In project test-app on server https://master.gce.sysdeseng.com:443

http://php-test-app.apps.gce.sysdeseng.com to pod port 8080-tcp (svc/php)
  dc/php deploys istag/php:latest <-
    bc/php source builds https://github.com/openshift/cakephp-ex.git on openshift/php:7.0
    deployment #1 deployed 3 minutes ago - 1 pod

View details with 'oc describe <resource>/<name>' or list everything with 'oc get all'.

$ oc whoami -t
AP4y3J6IFGrqtZsliqpIRkncHKaOSV9gRuoIzXzBGtg

$ sudo docker pull fedora/apache
$ sudo docker images

$ oc status
In project default on server https://internal-master.gce.sysdeseng.com:443

https://docker-registry-default.apps.gce.sysdeseng.com (passthrough) (svc/docker-registry)
  dc/docker-registry deploys docker.io/openshift3/ose-docker-registry:v3.6.173.0.21
    deployment #1 deployed 31 minutes ago - 3 pods

svc/kubernetes - 172.30.0.1 ports 443, 53->8053, 53->8053

https://registry-console-default.apps.gce.sysdeseng.com (passthrough) (svc/registry-console)
  dc/registry-console deploys registry.access.redhat.com/openshift3/registry-console:v3.6
    deployment #1 deployed 29 minutes ago - 1 pod

svc/router - 172.30.91.254 ports 80, 443, 1936
  dc/router deploys docker.io/openshift3/ose-haproxy-router:v3.6.173.0.21
    deployment #1 deployed 34 minutes ago - 3 pods

View details with 'oc describe <resource>/<name>' or list everything with 'oc get all'.

$ sudo docker tag docker.io/fedora/apache docker-registry-default.apps.gce.sysdeseng.com/openshift/prodapache

$ sudo docker images

$ sudo docker login -u user@example.com -p AP4y3J6IFGrqtZsliqpIRkncHKaOSV9gRuoIzXzBGtg docker-registry-default.apps.gce.sysdeseng.com
Login Succeeded

$ oadm policy add-role-to-user admin user@example.com -n openshift
$ oadm policy add-role-to-user system:registry user@example.com
$ oadm policy add-role-to-user system:image-builder user@example.com

$ sudo docker push docker-registry-default.apps.gce.sysdeseng.com/openshift/prodapache
The push refers to a repository [docker-registry-default.apps.gce.sysdeseng.com/openshift/prodapache]
3a85ee80fd6c: Pushed
5b0548b012ca: Pushed
a89856341b3d: Pushed
a839f63448f5: Pushed
e4f86288aaf7: Pushed
latest: digest: sha256:ec10a4b8f5acec40f08f7e04c68f19b31ac92af14b0baaadd0def17deb53a229 size: 1362

$ oc project default
Now using project "default" on server "https://master.gce.sysdeseng.com".

$ oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-mxqc4    1/1       Running   0          32m
docker-registry-1-qp05k    1/1       Running   0          32m
docker-registry-1-s3kp9    1/1       Running   0          32m
registry-console-1-gn1kg   1/1       Running   0          30m
router-1-60q15             1/1       Running   0          34m
router-1-m4swp             1/1       Running   0          34m
router-1-qq86v             1/1       Running   0          34m

$ oc describe pod docker-registry-1-mxqc4 | grep Node:
Node:    ocp-infra-node-stl6/10.128.0.6
$ oc describe pod docker-registry-1-qp05k | grep Node:
Node:    ocp-infra-node-4k4l/10.128.0.7
$ oc describe pod docker-registry-1-s3kp9 | grep Node:
Node:    ocp-infra-node-rwjl/10.128.0.4
$ oc describe pod router-1-60q15 | grep Node:
Node:    ocp-infra-node-4k4l/10.128.0.7
$ oc describe pod router-1-m4swp | grep Node:
Node:    ocp-infra-node-stl6/10.128.0.6
$ oc describe pod router-1-qq86v | grep Node:
Node:    ocp-infra-node-rwjl/10.128.0.4

$ oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-3897j    1/1       Running   0          11m
docker-registry-1-qp05k    1/1       Running   0          32m
docker-registry-1-s3kp9    1/1       Running   0          32m
registry-console-1-gn1kg   1/1       Running   0          30m
router-1-60q15             1/1       Running   0          34m
router-1-m4swp             1/1       Running   0          34m
router-2-l90pf             1/1       Running   0          11m

$ oc describe pod docker-registry-1-qp05k | grep Node:
Node:    ocp-infra-node-4k4l/10.128.0.7
$ oc describe pod docker-registry-1-s3kp9 | grep Node:
Node:    ocp-infra-node-rwjl/10.128.0.4
$ oc describe pod docker-registry-1-3897j | grep Node:
Node:    ocp-infra-node-4k4l/10.128.0.7

Run the following to perform the minor upgrade against the deployed environment.

$ ./ocp-on-gcp.sh --minor-upgrade

The minor upgrade playbook will not restart the instances after updating occurs. Restarting the nodes including the masters can be completed by using the following option.

$ ./ocp-on-gcp.sh --minor-upgrade -e 'openshift_rolling_restart_mode=system'

The deployed OpenShift environment may not be the latest major version of OpenShift. The upgrade playbook allows for a variable to be passed to perform an upgrade on previous versions. Below is an example of performing the upgrade on a 3.5 non-containerized environment.

$ ./ocp-on-gcp.sh --minor-upgrade -e 'openshift_vers=v3_5'

To observe available storage classes, run following commands.

$ oc get storageclass
NAME                 TYPE
ssd                  kubernetes.io/gce-pd
standard (default)   kubernetes.io/gce-pd

$ oc describe storageclass standard
Name:            standard
IsDefaultClass:  Yes
Annotations:     storageclass.beta.kubernetes.io/is-default-class=true
Provisioner:     kubernetes.io/gce-pd
Parameters:      type=pd-standard
Events:          <none>

$ oc describe storageclass ssd
Name:            ssd
IsDefaultClass:  No
Annotations:     storageclass.beta.kubernetes.io/is-default-class=false
Provisioner:     kubernetes.io/gce-pd
Parameters:      type=pd-ssd
Events:          <none>

The example below shows a dynamically provisioned volume being requested from the StorageClass named standard.

$ vi db-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: db
 annotations:
   volume.beta.kubernetes.io/storage-class: standard
spec:
 accessModes:
  - ReadWriteOnce
 resources:
   requests:
     storage: 10Gi

$ oc create -f db-claim.yaml
persistentvolumeclaim "db" created

There may become a point in which a PVC is no longer necessary for a project. The following can be done to remove the PVC.

$ oc delete pvc db
persistentvolumeclaim "db" deleted
$ oc get pvc db
Error from server (NotFound): persistentvolumeclaims "db" not found