Chapter 4. Monitoring broker runtime metrics

When you install AMQ Broker, a Prometheus metrics plugin is included in your installation. Prometheus is software built for monitoring large, scalable systems and storing historical runtime data over an extended time period. You must modify the broker configuration to enable the plugin. When enabled, the plugin collects runtime metrics for the broker and exports these to Prometheus format. You can then use Prometheus to review the metrics. You might also use a graphical tool such as Grafana to configure more advanced visualizations of the data.

Note

The Prometheus metrics plugin enables you to collect and export broker metrics in Prometheus format. However, Red Hat does not provide support for installation or configuration of Prometheus itself, nor of visualization tools such as Grafana. If you require support with installing, configuring, or running Prometheus or Grafana, visit the product websites for resources such as community support and documentation.

In addition to the broker metrics collected by the Prometheus plugin, you can modify the broker configuration to capture standard sets of metrics relating to the host Java Virtual Machine (JVM) for the broker. Specifically, you can capture JVM metrics for Garbage Collection (GC), memory, and threads.

The sections that follow describe:

4.1. Metrics overview

To monitor the health and performance of your broker instances, you can use the Prometheus plugin for AMQ Broker to monitor and store broker runtime metrics. The AMQ Broker Prometheus plugin exports the broker runtime metrics to Prometheus format, enabling you to use Prometheus itself to visualize and run queries on the data.

You can also use a graphical tool, such as Grafana, to configure more advanced visualizations and dashboards for the metrics that the Prometheus plugin collects.

The metrics that the plugin exports to Prometheus format are described below.

Broker metrics

artemis_address_memory_usage
Number of bytes used by all addresses on this broker for in-memory messages.
artemis_address_memory_usage_percentage
Memory used by all the addresses on this broker as a percentage of the global-max-size parameter.
artemis_connection_count
Number of clients connected to this broker.
artemis_total_connection_count
Number of clients that have connected to this broker since it was started.

Address metrics

artemis_routed_message_count
Number of messages routed to one or more queue bindings.
artemis_unrouted_message_count
Number of messages not routed to any queue bindings.

Queue metrics

artemis_consumer_count
Number of clients consuming messages from a given queue.
artemis_delivering_durable_message_count
Number of durable messages that a given queue is currently delivering to consumers.
artemis_delivering_durable_persistent_size
Persistent size of durable messages that a given queue is currently delivering to consumers.
artemis_delivering_message_count
Number of messages that a given queue is currently delivering to consumers.
artemis_delivering_persistent_size
Persistent size of messages that a given queue is currently delivering to consumers.
artemis_durable_message_count
Number of durable messages currently in a given queue. This includes scheduled, paged, and in-delivery messages.
artemis_durable_persistent_size
Persistent size of durable messages currently in a given queue. This includes scheduled, paged, and in-delivery messages.
artemis_messages_acknowledged
Number of messages acknowledged from a given queue since the queue was created.
artemis_messages_added
Number of messages added to a given queue since the queue was created.
artemis_message_count
Number of messages currently in a given queue. This includes scheduled, paged, and in-delivery messages.
artemis_messages_killed
Number of messages removed from a given queue since the queue was created. The broker kills a message when the message exceeds the configured maximum number of delivery attempts.
artemis_messages_expired
Number of messages expired from a given queue since the queue was created.
artemis_persistent_size
Persistent size of all messages (both durable and non-durable) currently in a given queue. This includes scheduled, paged, and in-delivery messages.
artemis_scheduled_durable_message_count
Number of durable, scheduled messages in a given queue.
artemis_scheduled_durable_persistent_size
Persistent size of durable, scheduled messages in a given queue.
artemis_scheduled_message_count
Number of scheduled messages in a given queue.
artemis_scheduled_persistent_size
Persistent size of scheduled messages in a given queue.

For higher-level broker metrics that are not listed above, you can calculate these by aggregating lower-level metrics. For example, to calculate total message count, you can aggregate the artemis_message_count metrics from all queues in your broker deployment.

For an on-premise deployment of AMQ Broker, metrics for the Java Virtual Machine (JVM) hosting the broker are also exported to Prometheus format. This does not apply to a deployment of AMQ Broker on OpenShift Container Platform.

4.2. Enabling the Prometheus metrics plugin for AMQ Broker

When you install AMQ Broker, a Prometheus metrics plugin is included in your installation. Although the plugin is already configured for use, you need to enable the plugin in your broker configuration. When enabled, the plugin collects runtime metrics for the broker and exports these to Prometheus format.

The following procedure shows how to enable the Prometheus plugin for AMQ Broker.

Procedure

  1. Copy the Prometheus metrics plugin .jar file from your AMQ Broker 7.8 extracted archive to the lib directory of your broker instance.

    $ cp amq-broker-7.8.0/lib/artemis-prometheus-metrics-plugin-1.0.0.CR1-redhat-00010.jar <broker-instance-dir>/lib
  2. Open the <broker-instance-dir>/etc/broker.xml configuration file.
  3. Enable the Prometheus plugin in the broker configuration. Add a <metrics> element with a <plugin> sub-element, configured as shown below.

    <metrics>
        <plugin class-name="org.apache.activemq.artemis.core.server.metrics.plugins.ArtemisPrometheusMetricsPlugin"/>
    </metrics>
  4. Save the broker.xml configuration file. The metrics plugin starts to gather broker runtime metrics in Prometheus format.

4.3. Configuring the broker to collect JVM metrics

The following procedure shows how to configure the broker to collect Java Virtual Machine (JVM) metrics for Garbage Collection (GC), memory, and threads.

Prerequisites

Procedure

  1. Open the <broker-instance-dir>/etc/broker.xml configuration file.
  2. In the <metrics> element that you added to the configuration when enabling the Prometheus metrics plugin, specify whether the broker collects JVM metrics for Garbage Collection (GC), memory, and threads. For example:

    <metrics>
        <jvm-gc>true</jvm-gc>
        <jvm-memory>true</jvm-memory
        <jvm-threads>true</jvm-threads>
        <plugin class-name="org.apache.activemq.artemis.core.server.metrics.plugins.ArtemisPrometheusMetricsPlugin"/>
    </metrics>
    Note

    If you do not explicitly add the jvm-memory parameter to your configuration and specify a value, the broker uses a default value of true. This means that the broker exports JVM memory metrics by default. The default values of the jvm-gc and jvm-threads parameters are false.

  3. Save the broker.xml configuration file. The broker starts to gather the JVM metrics that you have enabled. These metrics are also exported to Prometheus format.

4.4. Disabling metrics collection for specific addresses

When you configure a metrics plugin for AMQ Broker (for example, the Prometheus metrics plugin), metrics collection is enabled by default. However, within the address-setting configuration element of a specific address or set of addresses, you can explicitly disable metrics collection.

The following procedure shows how disable metrics collection for a specific address or set of addresses.

Procedure

  1. Open the <broker-instance-dir>/etc/broker.xml configuration file.
  2. In the address-setting element of a matching address or set of addresses, add the enable-metrics parameter and set the value of the parameter to false. For example, the following configuration disables metrics collection for an address called orders.

    <configuration>
      <core>
        ...
        <address-settings>
           <address-setting match="orders">
              ...
              <enable-metrics>false</enable-metrics>
              ...
           </address-setting>
        </address-settings>
        ...
      </core>
    </configuration>

4.5. Accessing broker runtime data using Prometheus

Prerequisites

  • To query and visualize the broker runtime data collected by the Prometheus plugin, you need to install Prometheus. For more information, see Installing Prometheus in the Prometheus documentation.

Procedure

  1. From your Prometheus installation directory, open the prometheus.yml configuration file.
  2. In the static_configs section of the configuration file, change the targets element to localhost:8161. This location is where the broker runs its web server. By default, /metrics is appended to this host name, forming the full path to the metrics stored on the broker web server.
  3. To view the broker runtime metrics collected by the Prometheus plugin, open localhost:8161/metrics in a web browser.

    On the resulting web page, you see the current values of the metrics collected by the plugin, based on the queues and addresses that you have configured on the broker. If you have more than one running broker instance in your JVM, you see metrics for each broker.

  4. From your Prometheus installation directory, run Prometheus.

    $ ./prometheus

    When Prometheus starts, the shell output includes the following line:

    component=web, msg=”Start listening for connections” address=0.0.0.0:9090

    The preceding line indicates that Prometheus is listening for HTTP traffic on port 9090.

  5. To access the Prometheus web console, open 127.0.0.1:9090 in a web browser.
  6. In the Prometheus web console, you can use the Expression field to create a query on your broker data. The queries you create are based on the Prometheus query language, PromQL. Broker metrics that are available to insert in your query are in the Insert metric drop-down list.

    As a simple example, suppose you want to query the message count on the DLQ queue, over time. In this case, select artemis_message_count from the metrics drop-down list. Complete your query by specifying the DLQ queue name and address. This example query is shown below.

    artemis_message_count{address=“DLQ”, queue=“DLQ”}

    For more advanced visualizations, you can use regular expressions to create complex queries that overlay several metrics, for example. Or, you can perform mathematical operations on a number of metrics, such as aggregating them. For more information about creating Prometheus queries, see Querying Prometheus in the Prometheus documentation.