Chapter 10. Monitoring your brokers

10.1. Viewing brokers in Fuse Console

You can configure an Operator-based broker deployment to use Fuse Console for OpenShift instead of the AMQ Management Console. When you have configured your broker deployment appropriately, Fuse Console discovers the brokers and displays them on a dedicated Artemis tab. You can view the same broker runtime data that you do in the AMQ Management Console. You can also perform the same basic management operations, such as creating addresses and queues.

The following procedure describes how to configure the Custom Resource (CR) instance for a broker deployment to enable Fuse Console for OpenShift to discover and display brokers in the deployment.

Important

Viewing brokers from Fuse Console is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Prerequisites

  • Fuse Console for OpenShift must be deployed to an OCP cluster, or to a specific namespace on that cluster. If you have deployed the console to a specific namespace, your broker deployment must be in the same namespace, to enable the console to discover the brokers. Otherwise, it is sufficient for Fuse Console and the brokers to be deployed on the same OCP cluster. For more information on installing Fuse Online on OCP, see Installing and Operating Fuse Online on OpenShift Container Platform.
  • You must have already created a broker deployment. For example, to learn how to use a Custom Resource (CR) instance to create a basic Operator-based deployment, see Section 3.4.1, “Deploying a basic broker instance”.

Procedure

  1. Open the CR instance that you used for your broker deployment. For example, the CR for a basic deployment might resemble the following:

    apiVersion: broker.amq.io/v2alpha4
    kind: ActiveMQArtemis
    metadata:
      name: ex-aao
      application: ex-aao-app
    spec:
        version: 7.8.0
        deploymentPlan:
            size: 4
            image: registry.redhat.io/amq7/amq-broker:7.8
            ...
  2. In the deploymentPlan section, add the jolokiaAgentEnabled and managementRBACEnabled properties and specify values, as shown below.

    apiVersion: broker.amq.io/v2alpha4
    kind: ActiveMQArtemis
    metadata:
      name: ex-aao
      application: ex-aao-app
    spec:
        version: 7.8.0
        deploymentPlan:
            size: 4
            image: registry.redhat.io/amq7/amq-broker:7.8
            ...
            jolokiaAgentEnabled: true
            managementRBACEnabled: false
    jolokiaAgentEnabled
    Specifies whether Fuse Console can discover and display runtime data for the brokers in the deployment. To use Fuse Console, set the value to true.
    managementRBACEnabled

    Specifies whether role-based access control (RBAC) is enabled for the brokers in the deployment. You must set the value to false to use Fuse Console because Fuse Console uses its own role-based access control.

    Important

    If you set the value of managementRBACEnabled to false to enable use of Fuse Console, management MBeans for the brokers no longer require authorization. You should not use the AMQ management console while managementRBACEnabled is set to false because this potentially exposes all management operations on the brokers to unauthorized use.

  3. Save the CR instance.
  4. Switch to the project in which you previously created your broker deployment.

    $ oc project <project-name>
  5. At the command line, apply the change.

    $ oc apply -f <path/to/custom-resource-instance>.yaml
  6. In Fuse Console, to view Fuse applications, click the Online tab. To view running brokers, in the left navigation menu, click Artemis.

Additional resources

10.2. Monitoring broker runtime metrics using Prometheus

The sections that follow describe how to configure the Prometheus metrics plugin for AMQ Broker on OpenShift Container Platform. You can use the plugin to monitor and store broker runtime metrics. You might also use a graphical tool such as Grafana to configure more advanced visualizations and dashboards of the data that the Prometheus plugin collects.

Note

The Prometheus metrics plugin enables you to collect and export broker metrics in Prometheus format. However, Red Hat does not provide support for installation or configuration of Prometheus itself, nor of visualization tools such as Grafana. If you require support with installing, configuring, or running Prometheus or Grafana, visit the product websites for resources such as community support and documentation.

10.2.1. Metrics overview

To monitor the health and performance of your broker instances, you can use the Prometheus plugin for AMQ Broker to monitor and store broker runtime metrics. The AMQ Broker Prometheus plugin exports the broker runtime metrics to Prometheus format, enabling you to use Prometheus itself to visualize and run queries on the data.

You can also use a graphical tool, such as Grafana, to configure more advanced visualizations and dashboards for the metrics that the Prometheus plugin collects.

The metrics that the plugin exports to Prometheus format are described below.

Broker metrics

artemis_address_memory_usage
Number of bytes used by all addresses on this broker for in-memory messages.
artemis_address_memory_usage_percentage
Memory used by all the addresses on this broker as a percentage of the global-max-size parameter.
artemis_connection_count
Number of clients connected to this broker.
artemis_total_connection_count
Number of clients that have connected to this broker since it was started.

Address metrics

artemis_routed_message_count
Number of messages routed to one or more queue bindings.
artemis_unrouted_message_count
Number of messages not routed to any queue bindings.

Queue metrics

artemis_consumer_count
Number of clients consuming messages from a given queue.
artemis_delivering_durable_message_count
Number of durable messages that a given queue is currently delivering to consumers.
artemis_delivering_durable_persistent_size
Persistent size of durable messages that a given queue is currently delivering to consumers.
artemis_delivering_message_count
Number of messages that a given queue is currently delivering to consumers.
artemis_delivering_persistent_size
Persistent size of messages that a given queue is currently delivering to consumers.
artemis_durable_message_count
Number of durable messages currently in a given queue. This includes scheduled, paged, and in-delivery messages.
artemis_durable_persistent_size
Persistent size of durable messages currently in a given queue. This includes scheduled, paged, and in-delivery messages.
artemis_messages_acknowledged
Number of messages acknowledged from a given queue since the queue was created.
artemis_messages_added
Number of messages added to a given queue since the queue was created.
artemis_message_count
Number of messages currently in a given queue. This includes scheduled, paged, and in-delivery messages.
artemis_messages_killed
Number of messages removed from a given queue since the queue was created. The broker kills a message when the message exceeds the configured maximum number of delivery attempts.
artemis_messages_expired
Number of messages expired from a given queue since the queue was created.
artemis_persistent_size
Persistent size of all messages (both durable and non-durable) currently in a given queue. This includes scheduled, paged, and in-delivery messages.
artemis_scheduled_durable_message_count
Number of durable, scheduled messages in a given queue.
artemis_scheduled_durable_persistent_size
Persistent size of durable, scheduled messages in a given queue.
artemis_scheduled_message_count
Number of scheduled messages in a given queue.
artemis_scheduled_persistent_size
Persistent size of scheduled messages in a given queue.

For higher-level broker metrics that are not listed above, you can calculate these by aggregating lower-level metrics. For example, to calculate total message count, you can aggregate the artemis_message_count metrics from all queues in your broker deployment.

For an on-premise deployment of AMQ Broker, metrics for the Java Virtual Machine (JVM) hosting the broker are also exported to Prometheus format. This does not apply to a deployment of AMQ Broker on OpenShift Container Platform.

10.2.2. Enabling the Prometheus plugin for a running broker deployment

This procedure shows how to enable the Prometheus plugin for a broker Pod in a given deployment.

Prerequisites

  • You can enable the Prometheus plugin for a broker Pod created with application templates or with the AMQ Broker Operator. However, your deployed broker must use the broker container image for AMQ Broker 7.5 or later. For more information about ensuring that your broker deployment uses the latest broker container image, see Chapter 9, Upgrading a template-based broker deployment.

Procedure

  1. Log in to the OpenShift Container Platform web console with administrator privileges for the project that contains your broker deployment.
  2. In the web console, click HomeProjects (OpenShift Container Platform 4.5 or later) or the drop-down list in the top-left corner (OpenShift Container Platform 3.11). Choose the project that contains your broker deployment.
  3. To see the StatefulSets or DeploymentConfigs in your project, click:

    1. WorkloadsStatefulSets or WorkloadsDeploymentConfigs (OpenShift Container Platform 4.5 or later).
    2. ApplicationsStatefulSets or ApplicationsDeployments (OpenShift Container Platform 3.11).
  4. Click the StatefulSet or DeploymentConfig that corresponds to your broker deployment.
  5. To access the environment variables for your broker deployment, click the Environment tab.
  6. Add a new environment variable, AMQ_ENABLE_METRICS_PLUGIN. Set the value of the variable to true.

    When you set the AMQ_ENABLE_METRICS_PLUGIN environment variable, OpenShift restarts each broker Pod in the StatefulSet or DeploymentConfig. When there are multiple Pods in the deployment, OpenShift restarts each Pod in turn. When each broker Pod restarts, the Prometheus plugin for that broker starts to gather broker runtime metrics.

Note

The AMQ_ENABLE_METRICS_PLUGIN environment variable is included by default in the application templates for AMQ Broker 7.5 or later. To enable the plugin for each broker in a new template-based deployment, ensure that the value of AMQ_ENABLE_METRICS_PLUGIN is set to true when deploying the application template.

Additional resources

10.2.3. Accessing Prometheus metrics for a running broker Pod

This procedure shows how to access Prometheus metrics for a running broker Pod.

Prerequisites

Procedure

  1. For the broker Pod whose metrics you want to access, you need to identify the Route you previously created to connect the Pod to the AMQ Broker management console. The Route name forms part of the URL needed to access the metrics.

    1. Click NetworkingRoutes (OpenShift Container Platform 4.5 or later) or ApplicationsRoutes (OpenShift Container Platform 3.11).
    2. For your chosen broker Pod, identify the Route created to connect the Pod to the AMQ Broker management console. Under Hostname, note the complete URL that is shown. For example:

      http://rte-console-access-pod1.openshiftdomain
  2. To access Prometheus metrics, in a web browser, enter the previously noted Route name appended with “/metrics”. For example:

    http://rte-console-access-pod1.openshiftdomain/metrics
Note

If your console configuration does not use SSL, specify http in the URL. In this case, DNS resolution of the host name directs traffic to port 80 of the OpenShift router. If your console configuration uses SSL, specify https in the URL. In this case, your browser defaults to port 443 of the OpenShift router. This enables a successful connection to the console if the OpenShift router also uses port 443 for SSL traffic, which the router does by default.

10.3. Monitoring broker runtime data using JMX

This example shows how to monitor a broker using the Jolokia REST interface to JMX.

Prerequisites

Procedure

  1. Get the list of running pods:

    $ oc get pods
    
    NAME                 READY     STATUS    RESTARTS   AGE
    broker-amq-1-ftqmk   1/1       Running   0          14d
  2. Run the oc logs command:

    $ oc logs -f broker-amq-1-ftqmk
    
    Running /amq-broker-71-openshift image, version 1.3-5
    INFO: Loading '/opt/amq/bin/env'
    INFO: Using java '/usr/lib/jvm/java-1.8.0/bin/java'
    INFO: Starting in foreground, this is just for debugging purposes (stop process by pressing CTRL+C)
    ...
    INFO | Listening for connections at: tcp://broker-amq-1-ftqmk:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
    INFO | Connector openwire started
    INFO | Starting OpenShift discovery agent for service broker-amq-tcp transport type tcp
    INFO | Network Connector DiscoveryNetworkConnector:NC:BrokerService[broker-amq-1-ftqmk] started
    INFO | Apache ActiveMQ 5.11.0.redhat-621084 (broker-amq-1-ftqmk, ID:broker-amq-1-ftqmk-41433-1491445582960-0:1) started
    INFO | For help or more information please see: http://activemq.apache.org
    WARN | Store limit is 102400 mb (current store usage is 0 mb). The data directory: /opt/amq/data/kahadb only has 9684 mb of usable space - resetting to maximum available disk space: 9684 mb
    WARN | Temporary Store limit is 51200 mb, whilst the temporary data directory: /opt/amq/data/broker-amq-1-ftqmk/tmp_storage only has 9684 mb of usable space - resetting to maximum available 9684 mb.
  3. Run your query to monitor your broker for MaxConsumers:

    $ curl -k -u admin:admin http://console-broker.amq-demo.apps.example.com/console/jolokia/read/org.apache.activemq.artemis:broker=%22broker%22,component=addresses,address=%22TESTQUEUE%22,subcomponent=queues,routing-type=%22anycast%22,queue=%22TESTQUEUE%22/MaxConsumers
    
    {"request":{"mbean":"org.apache.activemq.artemis:address=\"TESTQUEUE\",broker=\"broker\",component=addresses,queue=\"TESTQUEUE\",routing-type=\"anycast\",subcomponent=queues","attribute":"MaxConsumers","type":"read"},"value":-1,"timestamp":1528297825,"status":200}