Chapter 4. Operational Management

Optionally, to be able to connect easily to the VMs, the following SSH configuration file can be applied to the workstation that will perform the SSH commands:

$ cat /home/<user>/.ssh/config

Host bastion
     HostName                 <resourcegroup>b.<region>.cloudapp.azure.com
     User                     <user>
     StrictHostKeyChecking    no
     ProxyCommand             none
     CheckHostIP              no
     ForwardAgent             yes
     IdentityFile             /home/<user>/.ssh/id_rsa

Host master? infranode? node??
     ProxyCommand             ssh <user>@bastion -W %h:%p
     user                     <user>
     IdentityFile             /home/<user>/.ssh/id_rsa

To connect to any VM it is only needed the hostname as:

$ ssh infranode3

With all of the steps that occur during the installation of OpenShift Container Platform, it is possible to lose track of the names of the instances in the recently deployed environment. One option to get these hostnames is to browse to the Azure Resource Group dashboard and select Overview. The filter shows all instances relating to the reference architecture deployment.

To help facilitate the Chapter 4, Operational Management chapter the following hostnames will be used.

To run diagnostics, SSH into the first master node (master1), via the bastion host using the admin user specified in the template:

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@master1
$ sudo -i

Connectivity to the first master node (master1.<region>.cloudapp.azure.com) as the root user should have been established. Run the diagnostics that are included as part of the OpenShift Container Platform installation:

# oadm diagnostics
[Note] Determining if client configuration exists for client/cluster diagnostics
Info:  Successfully read a client config file at '/root/.kube/config'
Info:  Using context for cluster-admin access: 'default/sysdeseng-westus-cloudapp-azure-com:8443/system:admin'
[Note] Performing systemd discovery

[Note] Running diagnostic: ConfigContexts[default/sysdeseng-westus-cloudapp-azure-com:8443/system:admin]
       Description: Validate client config context is complete and has connectivity

Info:  The current client config context is 'default/sysdeseng-westus-cloudapp-azure-com:8443/system:admin':
       The server URL is 'https://sysdeseng.westus.cloudapp.azure.com:8443'
       The user authentication is 'system:admin/sysdeseng-westus-cloudapp-azure-com:8443'
       The current project is 'default'
       Successfully requested project list; has access to project(s):
         [default gsw kube-system logging management-infra openshift openshift-infra]

[Note] Running diagnostic: DiagnosticPod
       Description: Create a pod to run diagnostics from the application standpoint

       [Note] Running diagnostic: PodCheckDns
              Description: Check that DNS within a pod works as expected

       [Note] Summary of diagnostics execution (version v3.6.5.5):
       [Note] Warnings seen: 0
       [Note] Errors seen: 0

[Note] Running diagnostic: NetworkCheck
       Description: Create a pod on all schedulable nodes and run network diagnostics from the application standpoint

       [Note] Running diagnostic: CheckExternalNetwork
              Description: Check that external network is accessible within a pod

       [Note] Running diagnostic: CheckNodeNetwork
              Description: Check that pods in the cluster can access its own node.

       [Note] Running diagnostic: CheckPodNetwork
              Description: Check pod to pod communication in the cluster. In case of ovs-subnet network plugin, all pods should be able to communicate with each other and in case of multitenant network plugin, pods in non-global projects should be isolated and pods in global projects should be able to access any pod in the cluster and vice versa.

       [Note] Running diagnostic: CheckServiceNetwork
              Description: Check pod to service communication in the cluster. In case of ovs-subnet network plugin, all pods should be able to communicate with all services and in case of multitenant network plugin, services in non-global projects should be isolated and pods in global projects should be able to access any service in the cluster.

       [Note] Running diagnostic: CollectNetworkInfo
              Description: Collect network information in the cluster.

       [Note] Summary of diagnostics execution (version v3.6.5.5):
       [Note] Warnings seen: 0


       [Note] Running diagnostic: CheckNodeNetwork
              Description: Check that pods in the cluster can access its own node.

       [Note] Running diagnostic: CheckPodNetwork
              Description: Check pod to pod communication in the cluster. In case of ovs-subnet network plugin, all pods should be able to communicate with each other and in case of multitenant network plugin, pods in non-global projects should be isolated and pods in global projects should be able to access any pod in the cluster and vice versa.

       [Note] Running diagnostic: CheckServiceNetwork
              Description: Check pod to service communication in the cluster. In case of ovs-subnet network plugin, all pods should be able to communicate with all services and in case of multitenant network plugin, services in non-global projects should be isolated and pods in global projects should be able to access any service in the cluster.

       [Note] Running diagnostic: CollectNetworkInfo
              Description: Collect network information in the cluster.

       [Note] Summary of diagnostics execution (version v3.6.5.5):
       [Note] Warnings seen: 0


       [Note] Running diagnostic: CheckNodeNetwork
              Description: Check that pods in the cluster can access its own node.

       [Note] Running diagnostic: CheckPodNetwork
              Description: Check pod to pod communication in the cluster. In case of ovs-subnet network plugin, all pods should be able to communicate with each other and in case of multitenant network plugin, pods in non-global projects should be isolated and pods in global projects should be able to access any pod in the cluster and vice versa.

       [Note] Running diagnostic: CheckServiceNetwork
              Description: Check pod to service communication in the cluster. In case of ovs-subnet network plugin, all pods should be able to communicate with all services and in case of multitenant network plugin, services in non-global projects should be isolated and pods in global projects should be able to access any service in the cluster.

       [Note] Running diagnostic: CollectNetworkInfo
              Description: Collect network information in the cluster.

       [Note] Summary of diagnostics execution (version v3.6.5.5):
       [Note] Warnings seen: 0

[Note] Skipping diagnostic: AggregatedLogging
       Description: Check aggregated logging integration for proper configuration
       Because: No LoggingPublicURL is defined in the master configuration

[Note] Running diagnostic: ClusterRegistry
       Description: Check that there is a working Docker registry

[Note] Running diagnostic: ClusterRoleBindings
       Description: Check that the default ClusterRoleBindings are present and contain the expected subjects

Info:  clusterrolebinding/cluster-readers has more subjects than expected.

       Use the oadm policy reconcile-cluster-role-bindings command to update the role binding to remove extra subjects.

Info:  clusterrolebinding/cluster-readers has extra subject {ServiceAccount management-infra management-admin    }.
Info:  clusterrolebinding/cluster-readers has extra subject {ServiceAccount default router    }.

Info:  clusterrolebinding/self-provisioners has more subjects than expected.

       Use the oadm policy reconcile-cluster-role-bindings command to update the role binding to remove extra subjects.

Info:  clusterrolebinding/self-provisioners has extra subject {ServiceAccount management-infra management-admin    }.

[Note] Running diagnostic: ClusterRoles
       Description: Check that the default ClusterRoles are present and contain the expected permissions

[Note] Running diagnostic: ClusterRouterName
       Description: Check there is a working router

[Note] Running diagnostic: MasterNode
       Description: Check if master is also running node (for Open vSwitch)

WARN:  [DClu3004 from diagnostic MasterNode@openshift/origin/pkg/diagnostics/cluster/master_node.go:164]
       Unable to find a node matching the cluster server IP.
       This may indicate the master is not also running a node, and is unable
       to proxy to pods over the Open vSwitch SDN.

[Note] Skipping diagnostic: MetricsApiProxy
       Description: Check the integrated heapster metrics can be reached via the API proxy
       Because: The heapster service does not exist in the openshift-infra project at this time,
       so it is not available for the Horizontal Pod Autoscaler to use as a source of metrics.

[Note] Running diagnostic: NodeDefinitions
       Description: Check node records on master

WARN:  [DClu0003 from diagnostic NodeDefinition@openshift/origin/pkg/diagnostics/cluster/node_definitions.go:112]
       Node master1 is ready but is marked Unschedulable.
       This is usually set manually for administrative reasons.
       An administrator can mark the node schedulable with:
           oadm manage-node master1 --schedulable=true

       While in this state, pods should not be scheduled to deploy on the node.
       Existing pods will continue to run until completed or evacuated (see
       other options for 'oadm manage-node').

WARN:  [DClu0003 from diagnostic NodeDefinition@openshift/origin/pkg/diagnostics/cluster/node_definitions.go:112]
       Node master2 is ready but is marked Unschedulable.
       This is usually set manually for administrative reasons.
       An administrator can mark the node schedulable with:
           oadm manage-node master2 --schedulable=true

       While in this state, pods should not be scheduled to deploy on the node.
       Existing pods will continue to run until completed or evacuated (see
       other options for 'oadm manage-node').

WARN:  [DClu0003 from diagnostic NodeDefinition@openshift/origin/pkg/diagnostics/cluster/node_definitions.go:112]
       Node master3 is ready but is marked Unschedulable.
       This is usually set manually for administrative reasons.
       An administrator can mark the node schedulable with:
           oadm manage-node master3 --schedulable=true

       While in this state, pods should not be scheduled to deploy on the node.
       Existing pods will continue to run until completed or evacuated (see
       other options for 'oadm manage-node').

[Note] Running diagnostic: ServiceExternalIPs
       Description: Check for existing services with ExternalIPs that are disallowed by master config

[Note] Running diagnostic: AnalyzeLogs
       Description: Check for recent problems in systemd service logs

Info:  Checking journalctl logs for 'atomic-openshift-node' service
Info:  Checking journalctl logs for 'docker' service

[Note] Running diagnostic: MasterConfigCheck
       Description: Check the master config file

WARN:  [DH0005 from diagnostic MasterConfigCheck@openshift/origin/pkg/diagnostics/host/check_master_config.go:52]
       Validation of master config file '/etc/origin/master/master-config.yaml' warned:
       assetConfig.loggingPublicURL: Invalid value: "": required to view aggregated container logs in the console
       assetConfig.metricsPublicURL: Invalid value: "": required to view cluster metrics in the console
       auditConfig.auditFilePath: Required value: audit can now be logged to a separate file

[Note] Running diagnostic: NodeConfigCheck
       Description: Check the node config file

Info:  Found a node config file: /etc/origin/node/node-config.yaml

[Note] Running diagnostic: UnitStatus
       Description: Check status for related systemd units

[Note] Summary of diagnostics execution (version v3.6.5.5):
[Note] Warnings seen: 5
[Note] Errors seen: 0

Based on the results of the diagnostics, actions can be taken to alleviate any issues.

This section focuses on the etcd cluster. It describes the different commands to ensure the cluster is healthy. The internal DNS names of the nodes running etcd must be used.

SSH into the first master node (master1) as before:

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@master1
$ sudo -i

Using the output of the command hostname issue the etcdctl command to confirm that the cluster is healthy.

# etcdctl --endpoints https://master1:2379,https://master2:2379,https://master3:2379 --ca-file /etc/etcd/ca.crt --cert-file=/etc/origin/master/master.etcd-client.crt --key-file=/etc/origin/master/master.etcd-client.key cluster-health
member 82c895b7b0de4330 is healthy: got healthy result from https://10.0.0.4:2379
member c8e7ac98bb93fe8c is healthy: got healthy result from https://10.0.0.5:2379
member f7bbfc4285f239ba is healthy: got healthy result from https://10.0.0.6:2379

As explained in Nodes section, node labels are an important part of the OpenShift Container Platform environment. By default of the reference architecture installation, the default node selector is set to role=apps in /etc/origin/master/master-config.yaml on all of the master nodes. This configuration parameter is set during the installation of OpenShift on all masters.

SSH into the first master node (master1) to verify the defaultNodeSelector is defined.

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@master1
$ sudo -i
# vi /etc/origin/master/master-config.yaml
... [OUTPUT ABBREVIATED] ...
projectConfig:
  defaultNodeSelector: "role=app"
  projectRequestMessage: ""
  projectRequestTemplate: ""
... [OUTPUT ABBREVIATED] ...

Quotas are set on ephemeral volumes within pods to prohibit a pod from becoming too large and impacting the node. There are three places where sizing restrictions should be set. When persistent volume claims are not set a pod has the ability to grow as large as the underlying filesystem will allow. The required modifications are set by automatically.

OpenShift Volume Quota

At launch time a script creates a XFS partition on the block device, adds an entry in /etc/fstab, and mounts the volume with the option of gquota. If gquota is not set the OpenShift Container Platform node will not be able to start with the perFSGroup parameter defined below. This disk and configuration is done on the master, infrastructure, and application nodes.

SSH into the first infrastructure node (infranode1) to verify the entry exists within /etc/fstab

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@infranode1
$ grep "/var/lib/origin/openshift.local.volumes" /etc/fstab
/dev/sdc1 /var/lib/origin/openshift.local.volumes xfs gquota 0 0

OpenShift Emptydir Quota

During installation a value for perFSGroup is set within the node configuration. The perFSGroup setting restricts the ephemeral emptyDir volume from growing larger than 512Mi. This emptyDir quota is done on the master, infrastructure, and application nodes.

SSH into the first infrastructure node (infranode1) to verify /etc/origin/node/node-config.yml matches the information below.

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@infranode1
$ sudo grep -B2 perFSGroup /etc/origin/node/node-config.yaml
volumeConfig:
  localQuota:
     perFSGroup: 512Mi

Docker Storage Setup

The /etc/sysconfig/docker-storage-setup file is created at launch time by the bash script on every node. This file tells the Docker service to use a specific volume group for containers. Docker storage setup is performed on all master, infrastructure, and application nodes.

SSH into the first infrastructure node (infranode1) to verify /etc/sysconfig/docker-storage-setup matches the information below.

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@infranode1
$ cat /etc/sysconfig/docker-storage-setup
DEVS=/dev/sdd
VG=docker_vol
DATA_SIZE=95%VG
STORAGE_DRIVER=overlay2
CONTAINER_ROOT_LV_NAME=dockerlv
CONTAINER_ROOT_LV_MOUNT_PATH=/var/lib/docker
CONTAINER_ROOT_LV_SIZE=100%FREE

In section Required Channels the specific repositories for a successful OpenShift Container Platform installation were defined. All systems except for the bastion host should have the same repositories configured. To verify subscriptions match those defined in Required Channels perform the following. The repositories below are enabled during the rhsm-repos playbook during the installation. The installation will be unsuccessful if the repositories are missing from the system.

SSH into the first infrastructure node (infranode1) and verify the command output matches the information below.

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@infranode1
$ yum repolist
Loaded plugins: langpacks, product-id, search-disabled-repos
repo id                                  repo name                                                     status
rhel-7-fast-datapath-rpms/7Server/x86_64 Red Hat Enterprise Linux Fast Datapath (RHEL 7 Server) (RPMs) 27
rhel-7-server-extras-rpms/x86_64         Red Hat Enterprise Linux 7 Server - Extras (RPMs)             461+4
rhel-7-server-ose-3.6-rpms/x86_64        Red Hat OpenShift Container Platform 3.6 (RPMs)               437+30
rhel-7-server-rpms/7Server/x86_64        Red Hat Enterprise Linux 7 Server (RPMs)                      14.285
repolist: 15.210

This section will cover logging into the OpenShift Container Platform management console via the GUI and the CLI. After logging in via one of these methods applications can then be deployed and managed.

$ oc login https://resourcegroupname.region.cloudapp.azure.com --token=fEAjn7LnZE6v5SOocCSRVmUWGBNIIEKbjD9h-Fv7p09

$ oc new-project test-app
$ oc new-app https://github.com/openshift/cakephp-ex.git --name=php
--> Found image 2997627 (7 days old) in image stream "php" in project "openshift" under tag "5.6" for "php"

    Apache 2.4 with PHP 5.6
    -----------------------
    Platform for building and running PHP 5.6 applications

    Tags: builder, php, php56, rh-php56

    * The source repository appears to match: php
    * A source build using source code from https://github.com/openshift/cakephp-ex.git will be created
      * The resulting image will be pushed to image stream "php:latest"
    * This image will be deployed in deployment config "php"
    * Port 8080/tcp will be load balanced by service "php"
      * Other containers can access this service through the hostname "php"

--> Creating resources with label app=php ...
    imagestream "php" created
    buildconfig "php" created
    deploymentconfig "php" created
    service "php" created
--> Success
    Build scheduled, use 'oc logs -f bc/php' to track its progress.
    Run 'oc status' to view your app.

$ oc expose service php
route "php" exposed

$ oc status
In project test-app on server https://resourcegroupname.region.cloudapp.azure.com

http://test-app.apps.13.93.162.100.nip.io to pod port 8080-tcp (svc/php)
  dc/php deploys istag/php:latest <- bc/php builds https://github.com/openshift/cakephp-ex.git with openshift/php:5.6
    deployment #1 deployed about a minute ago - 1 pod

In this section, reactions to failure are explored. After a successful install and some of the smoke tests noted above have been completed, failure testing is executed.

$ ssh <user>@<resourcegroup>b.<region>.cloudapp.azure.com
$ ssh <user>@master1
$ sudo -i
# etcdctl --endpoints https://master1:2379,https://master2:2379,https://master3:2379 --ca-file /etc/etcd/ca.crt --cert-file=/etc/origin/master/master.etcd-client.crt --key-file=/etc/origin/master/master.etcd-client.key cluster-health
failed to check the health of member 82c895b7b0de4330 on https://10.20.2.251:2379: Get https://10.20.1.251:2379/health: dial tcp 10.20.1.251:2379: i/o timeout
member 82c895b7b0de4330 is unreachable: [https://10.20.1.251:2379] are all unreachable
member c8e7ac98bb93fe8c is healthy: got healthy result from https://10.20.3.74:2379
member f7bbfc4285f239ba is healthy: got healthy result from https://10.20.1.106:2379
cluster is healthy

$ oc get projects
NAME               DISPLAY NAME   STATUS
openshift                         Active
openshift-infra                   Active
ttester                           Active
test-app1                         Active
default                           Active
management-infra                  Active

$ oc project test-app1
Now using project "test-app1" on server "https://resourcegroupname.region.cloudapp.azure.com".

$ oc status
In project test-app1 on server https://resourcegroupname.region.cloudapp.azure.com

http://test-app1.apps.13.93.162.100.nip.io to pod port 8080-tcp (svc/php-prod)
  dc/php-prod deploys istag/php-prod:latest <-
    bc/php-prod builds https://github.com/openshift/cakephp-ex.git with openshift/php:5.6
    deployment #1 deployed 27 minutes ago - 1 pod

$ oc get pods -n default | grep docker-registry
docker-registry-4-9r033    1/1       Running   0          2h

$ oc exec docker-registry-4-9r033 cat /etc/secrets/registry.crt >> /tmp/my-docker-registry-certificate.crt

$ oc get route docker-registry -n default
NAME              HOST/PORT                                      PATH      SERVICES          PORT      TERMINATION   WILDCARD
docker-registry   docker-registry-default.13.64.245.134.nip.io             docker-registry   <all>     passthrough   None

$ sudo mkdir -p /etc/docker/certs.d/docker-registry-default.13.64.245.134.nip.io

$ sudo mv /tmp/my-docker-registry-certificate.crt /etc/docker/certs.d/docker-registry-default.13.64.245.134.nip.io/ca.crt
$ sudo systemctl restart docker

$ oc whoami -t
feAeAgL139uFFF_72bcJlboTv7gi_bo373kf1byaAT8

$ docker pull fedora/apache
$ docker images | grep fedora/apache
docker.io/fedora/apache  latest  c786010769a8  3 months ago  396.4 MB

$ docker tag docker.io/fedora/apache docker-registry-default.13.64.245.134.nip.io/openshift/prodapache

$ docker images | grep openshift/prodapache
docker-registry-default.13.64.245.134.nip.io/openshift/prodapache   latest              c786010769a8        3 months ago        396.4 MB

$ docker login -u $(oc whoami) -e <email> -p $(oc whoami -t) docker-registry-default.13.64.245.134.nip.io
Login Succeeded

$ docker push docker-registry-default.13.64.245.134.nip.io/openshift/prodapache
The push refers to a repository [docker-registry-default.13.64.245.134.nip.io/openshift/prodapache]
3a85ee80fd6c: Pushed
5b0548b012ca: Pushed
a89856341b3d: Pushed
a839f63448f5: Pushed
e4f86288aaf7: Pushed
latest: digest: sha256:e2a15a809ce2fe1a692b2728bd07f58fbf06429a79143b96b5f3e3ba0d1ce6b5 size: 7536

$ oc get pods -o wide -n default
NAME                       READY     STATUS    RESTARTS   AGE       IP           NODE
docker-registry-4-9r033    1/1       Running   0          2h        10.128.6.5   infranode3
registry-console-1-zwzsl   1/1       Running   0          5d        10.131.4.2   infranode2
router-1-09x4g             1/1       Running   0          5d        10.0.2.5     infranode2
router-1-6135c             1/1       Running   0          5d        10.0.2.4     infranode1
router-1-l2562             1/1       Running   0          5d        10.0.2.6     infranode3

$ oc get pods -o wide -n default | grep docker-registry
docker-registry-4-kd40f    1/1       Running   0          1m        10.130.4.3   infranode1

Red Hat OpenShift Container Platform metrics components enable additional features in the Red Hat OpenShift Container Platform web interface. If the environment has been deployed choosing to deploy metrics, there will be a new tab in the pod section named "Metrics" where it shows usage data of CPU, memory and network resources for a period of time:

Using the CLI, the cluster-admin can observe the usage of the pods and nodes using the following commands as well:

$ oc adm top pod --heapster-namespace="openshift-infra" --heapster-scheme="https" --all-namespaces
NAMESPACE         NAME                                   CPU(cores)   MEMORY(bytes)
openshift-infra   hawkular-cassandra-1-h9mrq             161m         1423Mi
logging           logging-fluentd-g5jqw                  8m           92Mi
logging           logging-es-ops-b44n3gav-1-zkl3r        19m          861Mi
... [OUTPUT ABBREVIATED] ...
$ oc adm top node --heapster-namespace="openshift-infra" --heapster-scheme="https"
NAME         CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%
infranode3   372m         9%        4657Mi          33%
master3      68m          1%        1923Mi          13%
node02       43m          1%        1437Mi          5%
... [OUTPUT ABBREVIATED] ...

$ oc new-project autoscaletest
Now using project "autoscaletest" on server "https://myocp.eastus2.cloudapp.azure.com:8443".
... [OUTPUT ABBREVIATED] ...

$ oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git
--> Found Docker image d9c9735 (10 days old) from Docker Hub for "centos/ruby-22-centos7"
... [OUTPUT ABBREVIATED] ...

$ oc patch dc/ruby-ex -p \'{"spec":{"template":{"spec":{"containers":[{"name":"ruby-ex","resources":{"limits":{"cpu":"80m"}}}]}}}}'
"ruby-ex" patched

$ oc get pods
NAME              READY     STATUS      RESTARTS   AGE
ruby-ex-1-210l9   1/1       Running     0          2m
ruby-ex-1-build   0/1       Completed   0          4m

$ oc describe pod ruby-ex-1-210l9
Name:			ruby-ex-1-210l9
... [OUTPUT ABBREVIATED] ...
    Limits:
      cpu:	80m
    Requests:
      cpu:		80m

$ oc autoscale dc/ruby-ex --min 1 --max 10 --cpu-percent=50
deploymentconfig "ruby-ex" autoscaled
$ oc get horizontalpodautoscaler
NAME      REFERENCE                  TARGET    CURRENT   MINPODS   MAXPODS   AGE
ruby-ex   DeploymentConfig/ruby-ex   50%       0%        1         10        53s

$ oc rsh ruby-ex-1-210l9

sh-4.2$ while true; do echo "cpu hog" >> mytempfile; rm -f mytempfile; done

$ oc get events -w
LASTSEEN                        FIRSTSEEN                       COUNT     NAME      KIND                      SUBOBJECT   TYPE      REASON                    SOURCE                         MESSAGE
2017-07-13 13:28:35 +0000 UTC   2017-07-13 13:26:30 +0000 UTC   7         ruby-ex   HorizontalPodAutoscaler               Normal    DesiredReplicasComputed   {horizontal-pod-autoscaler }   Computed the desired num of replicas: 0 (avgCPUutil: 0, current replicas: 1)
2017-07-13 13:29:05 +0000 UTC   2017-07-13 13:29:05 +0000 UTC   1         ruby-ex   HorizontalPodAutoscaler             Normal    DesiredReplicasComputed   {horizontal-pod-autoscaler }   Computed the desired num of replicas: 2 (avgCPUutil: 67, current replicas: 1)
2017-07-13 13:29:05 +0000 UTC   2017-07-13 13:29:05 +0000 UTC   1         ruby-ex   DeploymentConfig             Normal    ReplicationControllerScaled   {deploymentconfig-controller }   Scaled replication controller "ruby-ex-1" from 1 to 2
2017-07-13 13:29:05 +0000 UTC   2017-07-13 13:29:05 +0000 UTC   1         ruby-ex   HorizontalPodAutoscaler             Normal    SuccessfulRescale   {horizontal-pod-autoscaler }   New size: 2; reason: CPU utilization above target
2017-07-13 13:29:05 +0000 UTC   2017-07-13 13:29:05 +0000 UTC   1         ruby-ex-1-zwmxd   Pod                 Normal    Scheduled   {default-scheduler }   Successfully assigned ruby-ex-1-zwmxd to node02

$ oc get pods
NAME              READY     STATUS      RESTARTS   AGE
ruby-ex-1-210l9   1/1       Running     0          8m
ruby-ex-1-build   0/1       Completed   0          9m
ruby-ex-1-zwmxd   1/1       Running     0          58s

$ oc get events -w
LASTSEEN                        FIRSTSEEN                       COUNT     NAME      KIND                      SUBOBJECT   TYPE      REASON                    SOURCE                         MESSAGE
2017-07-13 13:34:05 +0000 UTC   2017-07-13 13:34:05 +0000 UTC   1         ruby-ex   HorizontalPodAutoscaler             Normal    SuccessfulRescale   {horizontal-pod-autoscaler }   New size: 1; reason: All metrics below target
2017-07-13 13:34:05 +0000 UTC   2017-07-13 13:34:05 +0000 UTC   1         ruby-ex   DeploymentConfig             Normal    ReplicationControllerScaled   {deploymentconfig-controller }   Scaled replication controller "ruby-ex-1" from 2 to 1
2017-07-13 13:34:05 +0000 UTC   2017-07-13 13:34:05 +0000 UTC   1         ruby-ex-1   ReplicationController             Normal    SuccessfulDelete   {replication-controller }   Deleted pod: ruby-ex-1-zwmxd

Red Hat OpenShift Container Platform aggregated logging components enable additional features in the Red Hat OpenShift Container Platform web interface. If the environment has been deployed choosing to deploy logging, there will be a new link in the pod logs section named "View Archive" that will redirect to the Kibana web interface for the user to see the pods logs, create queries, filters, etc.

In case the "opslogging" cluster has been deployed, there will be a route "kibana-ops" in the "logging" project where cluster-admin users can browse infrastructure logs.