Chapter 2. Monitoring 3scale

Prometheus is container-native software built for storing historical data and for monitoring large, scalable systems. It gathers data over an extended time, rather than just for the currently running session. Alerting rules in Prometheus are managed by Alertmanager.

You use Prometheus and Alertmanager to monitor and store 3scale data so that you can use a graphical tool, such as Grafana, to visualize and run queries on the data.

Important

Prometheus is an open-source system monitoring toolkit and Grafana is an open-source dashboard toolkit. Red Hat support for Prometheus and Grafana is limited to the configuration recommendations provided in Red Hat product documentation.

The 3scale operator allows you to use an existing Prometheus and Grafana operator installation to monitor 3scale usage and resources.

Note

The 3scale operator creates monitoring resources, but does not prevent modification of those resources.

Prerequisites

  • The 3scale operator is installed.
  • The Prometheus operator is installed in the cluster. The Prometheus operator is an operator for creating and managing Prometheus instances. It provides the Prometheus custom resource definition required by 3scale monitoring.

    The following Prometheus operator and image versions are tested with 3scale:

    • Prometheus operator v0.37.0
    • Prometheus image: quay.io/prometheus/prometheus:v2.16.0
  • The Grafana operator is installed in the cluster. The Grafana operator is an operator for creating and managing Grafana instances. It provides the GrafanaDashboard custom resource definition required by 3scale monitoring.

    The following Grafana operator and image versions are tested with 3scale:

    • Grafana operator v3.6.0
    • Grafana image: registry.hub.docker.com/grafana/grafana:7.1.1
Important

If your cluster is exposed on the Internet, make sure to protect the Prometheus and Grafana services.

This section describes how to enable monitoring of a 3scale instance, so that you can view the Grafana dashboards.

2.1. Enabling monitoring for 3scale

To monitor 3scale, you must enable monitoring by setting an APIManager custom resource.

Procedure

  1. Configure 3scale to enable monitoring by setting the spec.monitoring.enabled parameter of the 3scale deployment YAML to true. For example:

    1. . Create an APIManager custom resource named 3scale-monitoring.yml to enable monitoring:

      apiVersion: apps.3scale.net/v1alpha1
      kind: APIManager
      metadata:
        name: apimanager1
      spec:
        wildcardDomain: example.com
        monitoring:
          enabled: true
    2. Log in to your OpenShift cluster. You must log in as a user with an edit cluster role in the OpenShift project of the 3scale, for example, cluster-admin.

      oc login
    3. Switch to your 3scale project.

      oc project <project_name>
    4. Deploy the custom resource:

      $ oc apply -f 3scale-monitoring.yml

Additional resources

2.2. Configuring Prometheus to monitor 3scale

You must deploy and configure Prometheus using the Prometheus custom resource to enable monitoring of 3scale.

Note

Make sure permissions are set correctly as described in Prometheus documentation.

Procedure

  1. Deploy the Prometheus custom resource as follows depending on whether you want to monitor all resources in the cluster or only 3scale resources:

    • To monitor all resources in the cluster, set the spec.podMonitorSelector attribute to {} and set the spec.ruleSelector attribute to {}. For example, apply the following custom resource:

      apiVersion: monitoring.coreos.com/v1
      kind: Prometheus
      metadata:
        name: example
      spec:
        podMonitorSelector: {}
        ruleSelector: {}
    • If you deployed 3scale and the Prometheus operator in the same OpenShift project, and assuming the value of APP_LABEL is set to the default 3scale-api-management, monitor 3scale resources using the following steps:

      1. Set the spec.podMonitorSelector attribute to:

         podMonitorSelector:
          matchExpressions:
          - key: app
              operator: In
              values:
              - 3scale-api-management
      2. Set the spec.ruleSelector attribute to:

           matchExpressions:
           - key: app
             operator: In
             values:
             - 3scale-api-management

        For example, apply the following custom resource:

        apiVersion: monitoring.coreos.com/v1
        kind: Prometheus
        metadata:
          name: example
        spec:
         podMonitorSelector:
          matchExpressions:
          - key: app
              operator: In
              values:
              - 3scale-api-management
         ruleSelector:
           matchExpressions:
           - key: app
             operator: In
             values:
             - 3scale-api-management
    • If you deployed 3scale and the Prometheus operator in different OpenShift projects, monitor 3scale resources using the following steps:

      1. Label the OpenShift project where 3scale is deployed with MYLABELKEY=MYLABELVALUE
      2. Use a podMonitorNamespaceSelector filter to select the 3scale pods. For example, apply the following custom resource:

        apiVersion: monitoring.coreos.com/v1
        kind: Prometheus
        metadata:
          name: example
        spec:
         podMonitorSelector: {}
         ruleSelector: {}
         podMonitorNamespaceSelector:
           matchExpressions:
           - key: MYLABELKEY
             operator: In
             values:
             - MYLABELVALUE
  2. To ensure that dashboards and alerts work as expected, you must incorporate Kubernetes metrics, that is, kube-state-metrics, by performing one of the following:

    • Federate the Prometheus instance with the cluster default Prometheus instance.
    • Configure your own scraping jobs to get metrics from kubelet, etcd and others.

Additional resources

2.3. Configuring Grafana to monitor 3scale

You must configure Grafana in order to enable monitoring of 3scale.

Procedure

  1. Make sure Grafana services are configured to monitor the GrafanaDashboards resources by overwriting the app=3scale-api-management label. For example, apply the following custom resource:

    apiVersion: integreatly.org/v1alpha1
    kind: Grafana
    metadata:
      name: grafana
    spec:
      dashboardLabelSelector:
      - matchExpressions:
        - key: app
          operator: In
          values:
          - 3scale-api-management

    Grafana Dashboards created by the 3scale operator are labeled as follows:

    app: 3scale-api-management
    monitoring-key: middleware
  2. If the Grafana operator is installed in a different namespace than 3scale, configure it to monitor resources outside the namespace using the --namespaces or --scan-all operator flags. See the Grafana documentation for more information about the operator flags.
  3. Create a GrafanaDataSource custom resource of type prometheus to make the Prometheus data available in Grafana. For example:

    apiVersion: integreatly.org/v1alpha1
    kind: GrafanaDataSource
    metadata:
      name: prometheus
    spec:
      name: middleware
      datasources:
        - name: Prometheus
          type: prometheus
          access: proxy
          url: http://prometheus-operated:9090
          isDefault: true
          version: 1
          editable: true
          jsonData:
            timeInterval: "5s"

    where http://prometheus-operated:9090 is the Prometheus route.

  4. Make sure permissions are set correctly as described in the Grafana documentation.

Additional resources

2.4. Viewing metrics for 3scale

After configuring 3scale, Prometheus, and Grafana you can view the metrics described in this section.

Procedure

  1. Log into the Grafana console.
  2. Check that you can view metrics for the following:

    • Kubernetes resources at pod and namespace level where 3scale is installed
    • APIcast Staging
    • APIcast Production
    • Backend worker
    • Backend listener
    • System
    • Zync

Additional resources