Chapter 3. Accessing Prometheus to monitor Fuse Online

You can use Prometheus to monitor Fuse Online infrastructure and Fuse Online applications as described in these sections:

You can configure an external Prometheus instance to monitor Fuse Online applications as described in this section:

3.1. Monitoring Fuse Online infrastructure components

You can use Prometheus to monitor the metrics exposed by the following Fuse Online infrastructure components:

Syndesis Server
The syndesis-server component has been instrumented with Micrometer and exposes all of the JVM metrics Micrometer automatically by default. Additionally, syndesis-server exposes metrics about the REST API endpoints, such as request rate, error rate, and latency.
Syndesis Meta
The syndesis-meta component has been instrumented with Micrometer and exposes all of the JVM metrics Micrometer automatically by default. It also exposes metrics about its REST API endpoints.
Syndesis DB
Metrics for the Fuse Online Postgres database are exported using a third-party Prometheus exporter.
Integrations
The integration metrics are exported using the official JMX exporter, which exposes several JVM metrics by default. Additionally, integration metrics expose metrics that are specific to Apache Camel, such as message rate and error rate.

You can also use a Grafana dashboard to visualize the metrics gathered by Prometheus.

Prerequisites

  • You have cluster admin access to the OpenShift cluster.
  • Deploy Prometheus and Grafana with the Application Monitoring operator (specifically, tag v1.1.6) by following these installation instructions.
Note

Grafana is a community-supported feature. Deploying Grafana to monitor Red Hat Integration products is not supported with Red Hat production service level agreements (SLAs).

Procedure

  1. In the Fuse Online namespace, set the monitoring-key=middleware label by using the following command: 

    oc label namespace <fuse-online-namespace> monitoring-key=middleware

  2. Verify that your Fuse Online installation added the application monitoring configuration resources to the OpenShift cluster:

    1. In the OpenShift web console, go to the applicaton-monitoring project and then open the prometheus-route URL.

    2. In the Prometheus console, go to the Status → Targets page.

      If a Syndesis target is listed, then Fuse Online is configured for monitoring and you can skip to Step 4.

      If a Syndesis target is not listed, continue to Step 3.

  3. If the infrastructure resources are not on the OpenShift cluster, to enable Prometheus monitoring:

    1. Go to the Fuse Online namespace:

      OpenShift 4.xOpenShift 3.11
      1. In the OpenShift web console, go to the Fuse Online (syndesis) project.
      2. Select Catalog > Installed Operators and then click Fuse Online Operator.
      3. Click Syndesis CRD and then click app.
      4. Click Yaml to open the yaml file in the editor.
      1. Select Resources > Other Resources.
      2. From the dropdown menu, select the Syndesis resources type.
      3. For the app resource, click Actions and then select Edit YAML to open the yaml file in the editor.

    2. Edit the yaml file to set the Syndesis.Spec.Addons.Ops.Enable value to true by adding the follow lines: 

      spec:
  addons:
    ops:
      enabled: "true"
    3. Save the file.

    4. Wait for the pod to restart.

      Note

      The infrastructure resources are not available immediately after you run the install commands. You might need to wait before you can see the Fuse Online (Syndesis) targets in the Prometheus Targets page.

  4. To access Prometheus:

    1. In the OpenShift console for the project where the application monitoring operator is installed, open the list of routes.
    2. Next to the prometheus-route entry, click the hostname URL to open the Prometheus console.
    3. To view a list of the alert rules configured for Fuse Online infrastructure components, click the Alerts menu item.

  5. To access Grafana dashboards:

    1. In the OpenShift console for the project where the application monitoring operator is installed, open the list of routes.
    2. Next to the grafana-route entry, click the hostname URL to open the Grafana console.

    3. At the top of the Grafana console, click the dashboard selector and then select one of the following infrastructure dashboards:

      Infrastructure - DB
      Displays metrics related to the Fuse Online Postgres instance.
      Infrastructure - JVM
      Displays metrics about the running JVM for the syndesis-meta or syndesis-server applications. Chose the application that you want to monitor from the Application drop down list at the top of the dashboard.
      Infrastructure - REST APIs
      Displays metrics relating to the Fuse Online infrastructure API endpoints, such as request throughput and latency. Chose the application that you want to monitor from the Application drop down list at the top of the dashboard.

  6. To access Prometheus Alertmanager:

    1. In the OpenShift console for the project where the application monitoring operator is installed, open the list of routes.

    2. Next to the alertmanager-route entry, click the hostname URL to open the Alertmanager console.

      If the Fuse Online infrastructure is healthy, the default view is empty.

      If any of the infrastructure components are unhealthy, any active alerts that have been fired are listed, along with the option to silence them.

3.2. Monitoring Fuse Online integration applications

You can use the Application Monitoring operator to deploy Prometheus so that you can monitor Fuse Online integration applications. You can also use Grafana dashboards to visualize the metrics gathered by Prometheus.

Prerequisites

  • You have cluster admin access to the OpenShift cluster.
  • Deploy Prometheus and Grafana with the Application Monitoring operator (specifically, tag v1.1.6) by following these installation instructions.
Note

Grafana is a community-supported feature. Deploying Grafana to monitor Red Hat Integration products is not supported with Red Hat production service level agreements (SLAs).

Procedure

  1. In the Fuse Online namespace, set the monitoring-key=middleware label by using the following command: 

    oc label namespace <fuse-online-namespace> monitoring-key=middleware

  2. Verify that your Fuse Online installation added the application monitoring configuration resources to the OpenShift cluster.

    1. In the OpenShift web console, go to the applicaton-monitoring project and then open the prometheus-route URL.

    2. In the Prometheus console, go to the Status > Targets page.

      If a Syndesis target is listed, then Fuse Online has been configured for monitoring and you can skip to Step 4.

      If a Syndesis target is not listed, continue to Step 3.

  3. If the infrastructure resources are not on the OpenShift cluster, to enable Prometheus monitoring:

    1. Go to the Fuse Online namespace:

      OpenShift 4.xOpenShift 3.11
      1. In the OpenShift web console, go to the Fuse Online (syndesis) project.
      2. Select Catalog > Installed Operators and then click Fuse Online Operator.
      3. Click Syndesis CRD and then click app.
      4. Click Yaml to open the yaml file in the editor.
      1. Select Resources > Other Resources.
      2. From the dropdown menu, select the Syndesis resources type.
      3. For the app resource, click Actions and then select Edit YAML to open the yaml file in the editor.

    2. Edit the yaml file to set the Syndesis.Spec.Addons.Ops.Enable value to true by adding the follow lines: 

      spec:
  addons:
    ops:
      enabled: "true"
    3. Save the file.

    4. Wait for the pod to restart.

      Note

      The infrastructure resources are not available immediately after you run the install commands. You might need to wait before you can see the Fuse Online (Syndesis) targets in the Prometheus Targets page.

  4. To access Prometheus:

    1. In the OpenShift console for the project where the application monitoring operator is installed, open the list of routes.
    2. Next to the prometheus-route entry, click the hostname URL to open the Prometheus console.
    3. To view a list of the alert rules configured for Fuse Online infrastructure components, clicking the Alerts menu item.

  5. To access a Grafana dashboard:

    1. In the OpenShift console for the project where the application monitoring operator is installed, open the list of routes.
    2. Next to the grafana-route entry, click the hostname URL to open the Grafana console.

    3. At the top of the Grafana console, click the dashboard selector and then select Integration - Camel.

      This dashboard displays the standard metrics exposed by Apache Camel integration applications.

3.3. Configuring an external Prometheus instance to monitor Fuse Online applications

When you install Fuse Online on OpenShift Container Platform 3.11, a syndesis-prometheus instance is included by default. For detailed instructions on how to install Fuse Online on premise OpenShift Container Platform, see Integrating Applications with Fuse Online.

However, if you already have an existing external Prometheus instance, you can configure that external instance to also monitor Fuse Online Integration applications that are deployed on OpenShift Container Platform.

The steps depend on whether you used the Prometheus operator to install your external Prometheus instance.

3.3.1. Configuring a Prometheus instance (with the Prometheus operator)

If you installed a Prometheus instance by using the Prometheus operator, updating your external Prometheus configuration to monitor Fuse Online integration involves adding and then editing a service monitor.

Prerequisite

You installed Prometheus as described in Section 2.1, “Setting up Prometheus”.

Procedure

  1. In a terminal window, add a service monitor to the namespace (project) in which you installed Prometheus: 

    oc process -f  {$templates-base-url}/fuse-servicemonitor.yml -p NAMESPACE=<YOUR-NAMESPACE> -p FUSE_SERVICE_NAME=fuseonline | oc create -f -
  2. In the OpenShift console, open the project and then select Applications → Services.
  3. Click the fuseonline service and then select Actions → Edit YAML.
  4. In the Editor, change the selector section of the YAML file by replacing app: fuseonline with syndesis.io/type: integration.
  5. Click Save.

The Prometheus operator updates its configuration to monitor all Fuse Online integrations.

You can now view Fuse Online integrations in the Prometheus instance. The following example shows a Prometheus instance that monitors two integrations:

prometheus targets

3.3.2. Configuring a Prometheus instance (without the Prometheus operator)

If you installed an external Prometheus instance without using the Prometheus operator, updating the configuration to monitor Fuse Online integration involves editing your Prometheus configuration file and updating the Prometheus pod configuration.

Prerequisite

You must have write access to the Prometheus configuration file (prometheus-config.yml). In the OpenShift web console, the configuration file is located in Resources → Config Maps.

Procedure

  1. Edit your Prometheus configuration file (prometheus-config.yml) as follows:

    1. Set the scrape interval to 5 seconds: 

      global:
        scrape_interval:     5s
        evaluation_interval: 5s

    2. Add a scrape config job named integration-pod with the Kubernetes service discovery configuration that configures it to scrape pods in the ${OPENSHIFT_PROJECT}, which is typically the syndesis namespace: 

       - job_name: integration-pods

          kubernetes_sd_configs:
          - role: pod
            namespaces:
              names:
                - ${OPENSHIFT_PROJECT}

    3. Add a relabel_configs section that:

      • Only scrapes integration pods with the prometheus.io/scrape label set to true.
      • Sets the metrics_path and address labels, that are used to scrape the JMX exporter in the integration pod, by using values from the prometheus.io/path and prometheus.io/port labels.
      • Adds pod labels and annotations as Prometheus labels.

      • Creates kubernetes_namespace and kubernetes_pod_name labels.

        Here is an example relabel_configs section: 

        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - action: labelmap
            regex: __meta_kubernetes_pod_annotation_(syndesis.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: kubernetes_pod_name

    4. Fuse Online integrations expose a large number of metrics from the JVM, Camel and CXF. To reduce the amount of storage needed for metrics, add the following metric_relabel_configs section to filter out metrics that are not displayed in the Fuse Online console. 

       metric_relabel_configs:
          - source_labels: [__name__]
            regex: jmx_(.+)
            action: drop
          - source_labels: [__name__]
            regex: jvm_(.+)
            action: drop
          - source_labels: [__name__]
            regex: process_(.+)
            action: drop
          - source_labels: [type, __name__]
            separator: ':'
            regex: context:(org_apache_camel_ExchangesTotal|org_apache_camel_ExchangesFailed|io_syndesis_camel_StartTimestamp|io_syndesis_camel_LastExchangeCompletedTimestamp|io_syndesis_camel_LastExchangeFailureTimestamp)
            action: keep
      Note

      The last configuration line explicitly lists metrics to be added in Prometheus metrics store that are critical to the statistics shown in the Fuse Online web console. Your Prometheus instance must explicitly allow these metrics to be collected if other metrics are being filtered.

  2. Update the Prometheus pod configuration to store 30 days worth of metric data as follows: 

    args:
            - '--config.file=/etc/prometheus/prometheus.yml'
            - '--storage.tsdb.retention=30d'