Chapter 9. Monitoring your brokers

To monitor runtime data for brokers in your deployment, use one of these approaches:

In general, using Prometheus is the recommended approach. However, you might choose to use the Jolokia REST interface to JMX if a metric that you need to monitor is not exported by the Prometheus plugin. For more information about the broker runtime metrics that the Prometheus plugin exports, see Section 9.1.1, “Overview of broker metrics”.

9.1. Monitoring broker runtime metrics using Prometheus

The sections that follow describe how to configure the Prometheus metrics plugin for AMQ Broker on OpenShift Container Platform. You can use the plugin to monitor and store broker runtime metrics. You might also use a graphical tool such as Grafana to configure more advanced visualizations and dashboards of the data that the Prometheus plugin collects.

Note

The Prometheus metrics plugin enables you to collect and export broker metrics in Prometheus format. However, Red Hat does not provide support for installation or configuration of Prometheus itself, nor of visualization tools such as Grafana. If you require support with installing, configuring, or running Prometheus or Grafana, visit the product websites for resources such as community support and documentation.

9.1.1. Overview of broker metrics

To monitor the health and performance of your broker instances, you can use the Prometheus plugin for AMQ Broker to monitor and store broker runtime metrics. The AMQ Broker Prometheus plugin exports the broker runtime metrics to Prometheus format, enabling you to use Prometheus itself to visualize and run queries on the data.

You can also use a graphical tool, such as Grafana, to configure more advanced visualizations and dashboards for the metrics that the Prometheus plugin collects.

The metrics that the plugin exports to Prometheus format are listed below. A description of each metric is exported along with the metric itself.

Broker Metrics

  • address.memory.usage
  • connection.count
  • total.connection.count

Address Metrics

  • routed.message.count
  • unrouted.message.count

Queue Metrics

  • consumer.count
  • delivering.durable.message.count
  • delivering.durable.persistent.size
  • delivering.message.count
  • delivering.persistent.size
  • durable.message.count
  • durable.persistent.size
  • messages.acknowledged
  • messages.added
  • message.count
  • messages.killed
  • messages.expired
  • persistent.size
  • scheduled.durable.message.count
  • scheduled.durable.persistent.size
  • scheduled.message.count
  • scheduled.persistent.size

For higher-level broker metrics that are not listed above, you can calculate these by aggregating lower-level metrics. For example, to calculate total message count, you can aggregate the message.count metrics from all queues in your broker deployment.

Java Virtual Machine (JVM) metrics are also exported to Prometheus format.

9.1.2. Enabling the Prometheus plugin for a running broker deployment

This procedure shows how to enable the Prometheus plugin for a broker Pod in a given deployment.

Prerequisites

  • You can enable the Prometheus plugin for a broker Pod created with application templates or with the AMQ Broker Operator. However, your deployed broker must use the broker container image for AMQ Broker 7.5 or later. For more information about ensuring that your broker deployment uses the latest broker container image, see Chapter 8, Upgrading a template-based broker deployment.

Procedure

  1. Log in to the OpenShift Container Platform web console with administrator privileges for the project that contains your broker deployment.
  2. In the web console, click Home → Projects (OpenShift Container Platform 4.1 or later) or the drop-down list in the top-left corner (OpenShift Container Platform 3.11). Choose the project that contains your broker deployment.

  3. To see the Stateful Sets or Deployment Configs in your project, click:

    1. Workloads → Stateful Sets or Workloads → Deployment Configs (OpenShift Container Platform 4.1 or later).
    2. Applications → Stateful Sets or Applications → Deployments (OpenShift Container Platform 3.11).
  4. Click the Stateful Set or Deployment Config that corresponds to your broker deployment.
  5. To access the environment variables for your broker deployment, click the Environment tab.

  6. Add a new environment variable, AMQ_ENABLE_METRICS_PLUGIN. Set the value of the variable to true.

    When you set the AMQ_ENABLE_METRICS_PLUGIN environment variable, OpenShift restarts each broker Pod in the Stateful Set or Deployment Config. When there are multiple Pods in the deployment, OpenShift restarts each Pod in turn. When each broker Pod restarts, the Prometheus plugin for that broker starts to gather broker runtime metrics.

Note

The AMQ_ENABLE_METRICS_PLUGIN environment variable is included by default in the application templates for AMQ Broker 7.5 or later. To enable the plugin for each broker in a new template-based deployment, ensure that the value of AMQ_ENABLE_METRICS_PLUGIN is set to true when deploying the application template.

Additional resources

9.1.3. Accessing Prometheus metrics for a running broker Pod

This procedure shows how to access Prometheus metrics for a running broker Pod.

Prerequisites

Procedure

  1. For the broker Pod whose metrics you want to access, you need to identify the Route you previously created to connect the Pod to the AMQ Broker management console. The Route name forms part of the URL needed to access the metrics.

    1. Click Networking → Routes (OpenShift Container Platform 4.1 or later) or Applications → Routes (OpenShift Container Platform 3.11).

    2. For your chosen broker Pod, identify the Route created to connect the Pod to the AMQ Broker management console. Under Hostname, note the complete URL that is shown. For example: 

      http://rte-console-access-pod1.openshiftdomain

  2. To access Prometheus metrics, in a web browser, enter the previously noted Route name appended with “/metrics”. For example: 

    http://rte-console-access-pod1.openshiftdomain/metrics
Note

If your console configuration does not use SSL, specify http in the URL. In this case, DNS resolution of the host name directs traffic to port 80 of the OpenShift router. If your console configuration uses SSL, specify https in the URL. In this case, your browser defaults to port 443 of the OpenShift router. This enables a successful connection to the console if the OpenShift router also uses port 443 for SSL traffic, which the router does by default.

9.2. Monitoring broker runtime data using JMX

This example shows how to monitor a broker using the Jolokia REST interface to JMX.

Prerequisites

Procedure

  1. Get the list of running pods: 

    $ oc get pods

NAME                 READY     STATUS    RESTARTS   AGE
broker-amq-1-ftqmk   1/1       Running   0          14d

  2. Run the oc logs command: 

    $ oc logs -f broker-amq-1-ftqmk

Running /amq-broker-71-openshift image, version 1.3-5
INFO: Loading '/opt/amq/bin/env'
INFO: Using java '/usr/lib/jvm/java-1.8.0/bin/java'
INFO: Starting in foreground, this is just for debugging purposes (stop process by pressing CTRL+C)
...
INFO | Listening for connections at: tcp://broker-amq-1-ftqmk:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
INFO | Connector openwire started
INFO | Starting OpenShift discovery agent for service broker-amq-tcp transport type tcp
INFO | Network Connector DiscoveryNetworkConnector:NC:BrokerService[broker-amq-1-ftqmk] started
INFO | Apache ActiveMQ 5.11.0.redhat-621084 (broker-amq-1-ftqmk, ID:broker-amq-1-ftqmk-41433-1491445582960-0:1) started
INFO | For help or more information please see: http://activemq.apache.org
WARN | Store limit is 102400 mb (current store usage is 0 mb). The data directory: /opt/amq/data/kahadb only has 9684 mb of usable space - resetting to maximum available disk space: 9684 mb
WARN | Temporary Store limit is 51200 mb, whilst the temporary data directory: /opt/amq/data/broker-amq-1-ftqmk/tmp_storage only has 9684 mb of usable space - resetting to maximum available 9684 mb.

  3. Run your query to monitor your broker for MaxConsumers: 

    $ curl -k -u admin:admin http://console-broker.amq-demo.apps.example.com/console/jolokia/read/org.apache.activemq.artemis:broker=%22broker%22,component=addresses,address=%22TESTQUEUE%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22TESTQUEUE%22/MaxConsumers

{"request":{"mbean":"org.apache.activemq.artemis:address=\"TESTQUEUE\",broker=\"broker\",component=addresses,queue=\"TESTQUEUE\",routing-type=\"anycast\",subcomponent=queues","attribute":"MaxConsumers","type":"read"},"value":-1,"timestamp":1528297825,"status":200}