How to use user-workload-monitoring with MySQL instance.

Solution Verified - Updated -

Environment

  • Azure Red Hat OpenShift 4 (ARO)
  • Red Hat OpenShift Container Platform (RHOCP)

Issue

  • Can not find user application metrics from openshift console --> Observe --> Metrics tab
  • Provide a sample to monitor user application like mysql in user-workload-monitoring on openshift

Resolution

oc new-project mariadb
oc new-app mariadb-ephemeral
...
     * With parameters:
        * Memory Limit=512Mi
        * Namespace=openshift
        * Database Service Name=mariadb
        * MariaDB Connection Username=user # generated
        * MariaDB Connection Password=pass # generated
        * MariaDB root Password=rootpass # generated
        * MariaDB Database Name=sampledb
        * Version of MariaDB Image=10.3-el8
  • Create an exporter using the service name and username password generated above
oc new-app docker.io/prom/mysqld-exporter -e DATA_SOURCE_NAME="datasource(mariadb.mysqldb.svc.cluster.local:3306)/"
  • Create a Router towards DB and check related URL
oc create route edge --service=mysqld-exporter
oc get route
  • Check metrics gathered by mysql exporter
curl https://mysqld-exporter-mariadb.apps.xxx.eastus.aroapp.io/metrics

# HELP go_gc_cycles_automatic_gc_cycles_total Count of completed GC cycles generated by the Go runtime.
# TYPE go_gc_cycles_automatic_gc_cycles_total counter
go_gc_cycles_automatic_gc_cycles_total 0
# HELP go_gc_cycles_forced_gc_cycles_total Count of completed GC cycles forced by the application.
# TYPE go_gc_cycles_forced_gc_cycles_total counter
go_gc_cycles_forced_gc_cycles_total 0
# HELP go_gc_cycles_total_gc_cycles_total Count of all completed GC cycles.
# TYPE go_gc_cycles_total_gc_cycles_total counter
go_gc_cycles_total_gc_cycles_total 0
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
..........
  • Create the ServiceMonitor CR to connect the newly created exporter to OpenShift’s monitoring stack.
oc apply -f servicemonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: mysqld-exporter
  name: mysqld-exporter
  namespace: mariadb
spec:
  endpoints:
  - interval: 30s
    port: 9104-tcp
    scheme: http
  selector:
    matchLabels:
      app: mysqld-exporter
apiVersion: monitoring.coreos.com/v1
  • Connect to the Administration Portal in the OpenShift web console.
  • MariaDB's metrics will be available in the Observe -> Metrics tab. Inspect by typing mysql_up, and click Run Queries.

Root Cause

In order to OpenShift be able to collect metrics, it is a pre-requisite that the given workload does have exposed metrics. OpenShift then can display those metrics once enabled monitoring for user-defined projects by setting the enableUserWorkload: true. Full details at documentation: Enabling monitoring for user-defined projects. For customized dashboards with workload metrics, a separate configuration is required. Solution Custom Grafana Dashboard for custom metrics, provides overall guidance.

Diagnostic Steps

  • Check if openshift-user-workload-monitoring is enabled in the cluster.
$ oc get prometheus -A
NAMESPACE                            NAME            VERSION   REPLICAS   AGE
openshift-monitoring                 k8s             2.36.2    2          2d11h
openshift-user-workload-monitoring   user-workload   2.36.2    2          2m17s

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments