This section applies to Core and MBaaS components.

The status levels for CPU and Memory are

The following describes how to check that CPU/Memory usage is within the desired threshold for all containers in the current project.

In order to calculate the current containers CPU/Memory usage, the check needs to run twice in order to gather adequate data and make the calculation. Every check records the current usage and uptime of the process and then compares it to the previous check.

This check must be executed twice before it will provide CPU/Memory usage for containers. A status of UNKNOWN is displayed for the first check.

An occasional spike in CPU/Memory usage for a particular component might not be a cause for concern, especially if the value slightly exceeds the WARNING threshold. For example, it is common for a higher CPU/Memory usage to be reported on component startup or after an upgrade, in which case no action is required.

If the check is reporting issues, consider one of the following actions

$ oc logs -f <pod id> -c <container id>

  CPU example - 	OK: mysql-1-8puwv:mysql: - usage: 0.17%
  Memory example - 	OK: mysql-1-8puwv:mysql: - usage: 57.1%

$ oc delete pod <component pod id>
$ watch 'oc get pods | grep <component>'

<component>-2-9xeva    1/1       Running   0          10s

The CPU check must be executed at least twice before it can gather adequate data to calculate the usage of a container. If a container has not previously been checked, usage cannot be calculated and Nagios displays a status of UNKNOWN. Forcing service checks on all containers ensures that all service checks are executed at least twice.

To force host service checks

$ oc exec -ti "$(oc get pods | grep nagios | awk '{print $1}')" -- /opt/rhmap/host-svc-check

An UNKNOWN status should never occur and usually indicates an internal error in the check. Contact Support and include the status information displayed for the check.

Millicore loads some of the binary objects into memory and stores them in mysql when completing app builds and also when using the app store. This can spike the reported memory usage. This issue is not related to JVM memory usage, but to the reported container memory usage and will be addressed in a future release.

If the reported memory usage stays high or gets to critical you may want to do a redeploy of millicore.

$ oc deploy dc/millicore --latest
$ oc deploy dc/mysql --latest

This section applies to Core and MBaaS components.

The status levels for Disk Usage are

The following describes how to check disk usage for all volumes that are mounted in all containers and are also within the desired threshold for the current project.

For each volume, retrieve the disk usage from the container as a percentage for both the blocks and inodes. Compare the highest percentage against the threshold to determine the status.

$ oc logs -f <pod id> -c <container id>

The following applies to MBaaS only.

The status levels for MongoDB are

The following describes how to check that the MongoDB replicaset is correctly configured and healthy. The course of action required is determined by the current status and the information returned by the check. For each individual check, view this information using the Nagios dashboard.

$ oc delete pod <mongodb-pod-name>

An UNKNOWN status should never occur and usually indicates an internal error in the check. Contact Support and include the status information displayed for the check.

The following applies to Core only.

The status levels for MySQL are

The following describes how to check a connection to MySQL and also gather various metric values that are useful for diagnosing issues. Although the check does gather statistics, it does not validate the stats against a threshold. If you can connect, authenticate and run the metric collection commands against MySQL - it will return OK.

Sample output

  Status Information:	Uptime: 1461116 Threads: 8 Questions: 758388 Slow queries: 0 Opens: 588 Flush tables: 1 Open tables: 100 Queries per second avg: 0.519
  Performance Data:	Connections=175748c;;; Open_files=20;;;
  Open_tables=100;;; Qcache_free_memory=0;;; Qcache_hits=0c;;;
  Qcache_inserts=0c;;; Qcache_lowmem_prunes=0c;;; Qcache_not_cached=0c;;;
  Qcache_queries_in_cache=0;;; Queries=758389c;;; Questions=753634c;;;
  Table_locks_waited=0c;;; Threads_connected=8;;; Threads_running=1;;;
  Uptime=1461116c;;;

$ oc scale --replicas=0 dc mysql

$ oc scale --replicas=1 dc mysql

The following applies to Core and MBaaS.

The status levels for Health Check are

The component health endpoint is responding, that is an output can be viewed, but an issue with a dependent component is reported in the output.

A critical test item encountered an error. Please investigate this. See the "details" object for specifics.

Check: fh-supercore::health

A critical test item encountered an error. Please investigate this. See the "details" object for specifics.
Test: Check fh-metrics - Status: crit - Details: {u'status': u'crit', u'id': u'fh-fhmetrics', u'error': {u'syscall': u'connect', u'errno': u'EHOSTUNREACH', u'code': u'EHOSTUNREACH', u'port': 8080, u'address': u'172.30.112.65'}}

The component health endpoint has failed to respond.

RequestError: The health endpoint at "<component-health-endpoint>" is not contactable. Error: <urlopen error [Errno 113] No route to host>

The following applies to Core and MBaaS.

The status levels for Ping Check are

The following describes how to check that a HTTP connection can be established to the RHMAP component service.

Check the current status and the status information returned by the check using the Nagios dashboard.

No Route To host HTTP CRITICAL - Unable to open TCP socket

A WARNING status will most commonly happen if the service is running and can be contacted but the client appears to have failed (4xx HTTP status).

An UNKNOWN status should never occur and usually indicates an internal error in the check. Contact Support and include the status information displayed for the check.