Data Grid exposes metrics that can be used by Prometheus and Grafana for monitoring and visualizing the cluster state.
This documentation explains how to set up monitoring on OpenShift Container Platform. If you’re working with community Prometheus deployments, you might find these instructions useful as a general guide. However you should refer to the Prometheus documentation for installation and usage instructions.
See the Prometheus Operator documentation.
Creating a Prometheus service monitor
Data Grid Operator automatically creates a Prometheus ServiceMonitor
that scrapes metrics from your Data Grid cluster.
Enable monitoring for user-defined projects on OpenShift Container Platform.
When the Operator detects an Infinispan
CR with the monitoring annotation set to true
, which is the default, Data Grid Operator does the following:
-
Creates a
ServiceMonitor
named<cluster_name>-monitor
. -
Adds the
infinispan.org/monitoring: 'true'
annotation to yourInfinispan
CR metadata, if the value is not already explicitly set:apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: infinispan annotations: infinispan.org/monitoring: 'true'
To authenticate with Data Grid, Prometheus uses the operator
credentials.
You can check that Prometheus is scraping Data Grid metrics as follows:
-
In the OpenShift Web Console, select the </> Developer perspective and then select Monitoring.
-
Open the Dashboard tab for the namespace where your Data Grid cluster runs.
-
Open the Metrics tab and confirm that you can query Data Grid metrics such as:
vendor_cache_manager_default_cluster_size
Disabling the Prometheus service monitor
You can disable the ServiceMonitor
if you do not want Prometheus to scrape metrics for your Data Grid cluster.
-
Set
'false'
as the value for theinfinispan.org/monitoring
annotation in yourInfinispan
CR.apiVersion: infinispan.org/v1 kind: Infinispan metadata: name: infinispan annotations: infinispan.org/monitoring: 'false'
-
Apply the changes.
Installing the Grafana Operator
To support various needs, Data Grid Operator integrates with the community version of the Grafana Operator to create dashboards for Data Grid services.
Until Grafana is integrated with OpenShift user workload monitoring, the only option is to rely on the community version. You can install the Grafana Operator on OpenShift from the OperatorHub and should create a subscription for the alpha channel.
However, as is the policy for all Community Operators, Red Hat does not certify the Grafana Operator and does not provide support for it in combination with Data Grid. When you install the Grafana Operator you are prompted to acknowledge a warning about the community version before you can continue.
Creating Grafana data sources
Create a GrafanaDatasource
CR so you can visualize Data Grid metrics in Grafana dashboards.
-
Have an
oc
client. -
Have
cluster-admin
access to OpenShift Container Platform. -
Enable monitoring for user-defined projects on OpenShift Container Platform.
-
Install the Grafana Operator from the alpha channel and create a
Grafana
CR.
-
Create a
ServiceAccount
that lets Grafana read Data Grid metrics from Prometheus.apiVersion: v1 kind: ServiceAccount metadata: name: infinispan-monitoring
-
Apply the
ServiceAccount
.$ oc apply -f service-account.yaml
-
Grant
cluster-monitoring-view
permissions to theServiceAccount
.$ oc adm policy add-cluster-role-to-user cluster-monitoring-view -z infinispan-monitoring
-
-
Create a Grafana data source.
-
Retrieve the token for the
ServiceAccount
.$ oc serviceaccounts get-token infinispan-monitoring eyJhbGciOiJSUzI1NiIsImtpZCI6Imc4O...
-
Define a
GrafanaDataSource
that includes the token in thespec.datasources.secureJsonData.httpHeaderValue1
field, as in the following example:apiVersion: integreatly.org/v1alpha1 kind: GrafanaDataSource metadata: name: grafanadatasource spec: name: datasource.yaml datasources: - access: proxy editable: true isDefault: true jsonData: httpHeaderName1: Authorization timeInterval: 5s tlsSkipVerify: true name: Prometheus secureJsonData: httpHeaderValue1: >- Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Imc4O... type: prometheus url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'
-
-
Apply the
GrafanaDataSource
.$ oc apply -f grafana-datasource.yaml
Enable Grafana dashboards with the Data Grid Operator configuration properties.
Configuring Data Grid dashboards
Data Grid Operator provides global configuration properties that let you configure Grafana dashboards for Data Grid clusters.
You can modify global configuration properties while Data Grid Operator is running.
-
Data Grid Operator must watch the namespace where the Grafana Operator is running.
-
Create a
ConfigMap
namedinfinispan-operator-config
in the Data Grid Operator namespace.apiVersion: v1 kind: ConfigMap metadata: name: infinispan-operator-config data: grafana.dashboard.namespace: infinispan grafana.dashboard.name: infinispan grafana.dashboard.monitoring.key: middleware
-
Specify the namespace of your Data Grid cluster with the
data.grafana.dashboard.namespace
property.Deleting the value for this property removes the dashboard. Changing the value moves the dashboard to that namespace.
-
Specify a name for the dashboard with the
data.grafana.dashboard.name
property. -
If necessary, specify a monitoring key with the
data.grafana.dashboard.monitoring.key
property. -
Create
infinispan-operator-config
or update the configuration.$ oc apply -f infinispan-operator-config.yaml
-
Open the Grafana UI, which is available at:
$ oc get routes grafana-route -o jsonpath=https://"{.spec.host}"