Red Hat AMQ Online is an OpenShift-based mechanism for delivering messaging as a managed service. With Red Hat AMQ Online, administrators can configure a cloud-native, multi-tenant messaging service either in the cloud or on premise. Developers can provision messaging using the Red Hat AMQ Console. Multiple development teams can provision the brokers and queues from the Console, without requiring each team to install, configure, deploy, maintain, or patch any software.

AMQ Online can provision different types of messaging depending on your use case. A user can request messaging resources by creating an address space. AMQ Online currently supports two address space types, standard and brokered, each with different semantics. The following diagrams illustrate the high-level architecture of each address space type:

Figure 1.1. Standard address space

Figure 1.2. Brokered address space

The following table shows the supported features for AMQ Online 1.5:

AMQ Online users can be defined broadly in terms of two user roles: service administrator and messaging tenant. Depending on the size of your organization, these roles might be performed by the same person or different people.

The service administrator performs the initial installation and any subsequent upgrades. The service administrator might also deploy and manage the messaging infrastructure, such as monitoring the routers, brokers, and administration components; and creating the address space plans and address plans. Installing and Managing AMQ Online on OpenShift provides information about how to set up and manage AMQ Online as well as configure the infrastructure and plans as a service administrator.

The messaging tenant can request messaging resources, using both cloud-native APIs and tools. The messaging tenant can also manage the users and permissions of a particular address space within the messaging system as well as create address spaces and addresses. For more information about how to manage address spaces, addresses, and users, see Using AMQ Online on OpenShift Container Platform.

For more information about AMQ Online supported configurations see Red Hat AMQ 7 Supported Configurations.

The simplest way to install AMQ Online is to use the predefined YAML bundles.

Installing AMQ Online using Ansible requires creating an inventory file with the variables for configuring the system. Example inventory files can be found in the ansible/inventory folder.

The following example inventory file enables a minimal installation of AMQ Online:

[enmasse]
localhost ansible_connection=local

[enmasse:vars]
namespace=amq-online-infra
enable_rbac=False
api_server=True
service_catalog=False
register_api_server=True
keycloak_admin_password=admin
authentication_services=["standard"]
standard_authentication_service_postgresql=False
monitoring_namespace=enmasse-monitoring
monitoring_operator=False
monitoring=False

The following Ansible configuration settings are supported:

You can use the Operator Lifecycle Manager to install and configure an instance of AMQ Online.

In OpenShift 4.x, the Operator Lifecycle Manager (OLM) helps users install, update, and manage the life cycle of all Operators and their associated services running across their clusters. It is part of the Operator Framework, an open source toolkit designed to manage Kubernetes native applications (Operators) in an effective, automated, and scalable way.

The OLM runs by default in OpenShift 4.x, which aids cluster administrators in installing, upgrading, and granting access to Operators running on their cluster. The OpenShift console provides management screens for cluster administrators to install Operators and grant specific projects access to use the catalog of Operators available on the cluster.

OperatorHub is the graphical interface that OpenShift cluster administrators use to discover, install, and upgrade Operators. With one click, these Operators can be pulled from OperatorHub, installed on the cluster, and managed by the OLM, ready for engineering teams to self-service manage the software in development, test, and production environments.

This method uninstalls AMQ Online that was installed using the YAML bundle.

Uninstalling AMQ Online using Ansible requires using the same inventory file that was used for installing AMQ Online.

This method uninstalls AMQ Online that was installed using the Operator Lifecycle Manager (OLM).

This method uninstalls AMQ Online that was installed using the Operator Lifecycle Manager (OLM) in the OpenShift Container Platform console.

Configuring AMQ Online for production takes some time and consideration. The following procedure will get you started with a minimal service configuration. For a more complete example, navigate to the install/components/example-plans folder of the AMQ Online distribution. All of the commands must be run in the namespace where AMQ Online is installed.

Address space plans are used to configure quotas and control the resources consumed by address spaces. Address space plans are configured by the AMQ Online service operator and are selected by the messaging tenant when creating an address space.

AMQ Online includes a default set of plans that are sufficient for most use cases.

Plans are configured as custom resources. The following example shows a plan for the standard address space:

apiVersion: admin.enmasse.io/v1beta2
kind: AddressSpacePlan
metadata:
  name: restrictive-plan
  labels:
    app: enmasse
spec:
  displayName: Restrictive Plan
  displayOrder: 0
  infraConfigRef: default 1
  shortDescription: A plan with restrictive quotas
  longDescription: A plan with restrictive quotas for the standard address space
  addressSpaceType: standard 2
  addressPlans: 3
  - small-queue
  - small-anycast
  resourceLimits: 4
    router: 2.0
    broker: 2.0
    aggregate: 2.0

The other fields are used by the Red Hat AMQ Console UI. Note the field spec.infraConfigRef, which points to an infrastructure configuration that must exist when an address space using this plan is created. For more information about infrastructure configurations, see Infrastructure configuration.

Address plans specify the expected resource usage of a given address. The sum of the resource usage for all resource types determines the amount of infrastructure provisioned for an address space. A single router and broker pod has a maximum usage of one. If a new address requires additional resources and the resource consumption is within the address space plan limits, a new pod will be created automatically to handle the increased load.

Address plans are configured by the AMQ Online service operator and are selected when creating an address.

AMQ Online includes a default set of address plans that are sufficient for most use cases.

In the Address space plans section, the address space plan references two address plans: small-queue and small-anycast. These address plans are stored as custom resources and are defined as follows:

apiVersion: admin.enmasse.io/v1beta2
kind: AddressPlan
metadata:
  name: small-queue
  labels:
    app: enmasse
spec:
  displayName: Small queue plan
  displayOrder: 0
  shortDescription: A plan for small queues
  longDescription: A plan for small queues that consume little resources
  addressType: queue 1
  resources: 2
    router: 0.2
    broker: 0.3
  partitions: 1 3
  messageTtl: 4
    minimim: 30000
    maximum: 300000

The other fields are used by the Red Hat AMQ Console UI.

A single router can support five instances of addresses and broker can support three instances of addresses with this plan. If the number of addresses with this plan increases to four, another broker is created. If it increases further to six, another router is created as well.

In the standard address space, address plans for the queue address type may contain a field partitions, which allows a queue to be sharded across multiple brokers for HA and improved performance. Specifying an amount of broker resource above 1 will automatically cause a queue to be partitioned.

The messageTtl field is used to restrict the effective absolute-expiry-time of any message put to a queue or topic. The maximum and minimum values are defined in milliseconds. The system adjusts the TTL value of an incoming message to a particular address based on these values:

Messages that arrive without a TTL defined are considered to have a TTL value of infinity.

Expired messages will be automatically removed from the queue, subscription or temporary topic subscription periodically. These messages are lost. This occurs every 30 seconds.

Although the example address space plan in Address space plans allows two routers and two brokers to be deployed, it only allows two pods to be deployed in total. This means that the address space is restricted to three addresses with the small-queue plan.

The small-anycast plan does not consume any broker resources, and can provision two routers at the expense of not being able to create any brokers:

apiVersion: admin.enmasse.io/v1beta2
kind: AddressPlan
metadata:
  name: small-anycast
  labels:
    app: enmasse
spec:
  addressType: anycast
  resources:
    router: 0.2

With this plan, up to 10 addresses can be created.

AMQ Online creates infrastructure components such as routers, brokers, and consoles. These components can be configured while the system is running, and AMQ Online automatically updates the components with the new settings. The AMQ Online service operator can edit the AMQ Online default infrastructure configuration or create new configurations.

Infrastructure configurations can be referred to from one or more address space plans. For more information about address space plans, see Address space plans.

Infrastructure configuration can be managed for both brokered and standard infrastructure using BrokeredInfraConfig and StandardInfraConfig resources.

apiVersion: admin.enmasse.io/v1beta1
kind: BrokeredInfraConfig
metadata:
  name: brokered-infra-config-example
spec:
  version: "0.32" 1
  admin: 2
    resources:
      memory: 256Mi
    podTemplate:
      metadata:
        labels:
          key: value
  broker: 3
    resources:
      memory: 2Gi
      storage: 100Gi
    addressFullPolicy: PAGE
    globalMaxSize: 256Mb
    podTemplate: 4
      spec:
        priorityClassName: messaging

apiVersion: admin.enmasse.io/v1beta1
kind: BrokeredInfraConfig
metadata:
  name: brokered-infra-config
spec:
  broker:
    ...
    podTemplate:
      spec:
        containers:
          - name: broker 1
            readinessProbe:
              failureThreshold: 6 2
              initialDelaySeconds: 20 3

apiVersion: admin.enmasse.io/v1beta1
kind: StandardInfraConfig
metadata:
  name: myconfig
spec:
  version: "0.32" 1
  admin: 2
    resources:
      memory: 256Mi
  broker: 3
    resources:
      cpu: 0.5
      memory: 2Gi
      storage: 100Gi
    addressFullPolicy: PAGE
  router: 4
    resources:
      cpu: 1
      memory: 256Mi
    linkCapacity: 1000
    minReplicas: 1
    policy:
      maxConnections: 1000
      maxConnectionsPerHost: 1
      maxConnectionsPerUser: 10
      maxSessionsPerConnection: 10
      maxSendersPerConnection: 5
      maxReceiversPerConnection: 5
    podTemplate: 5
      spec:
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              preference:
              matchExpressions:
              - key: e2e-az-EastWest
                operator: In
                values:
                - e2e-az-East
                - e2e-az-West

apiVersion: admin.enmasse.io/v1beta1
kind: StandardInfraConfig
metadata:
  name: standard-infra-config
spec:
  broker:
    ...
    podTemplate:
      spec:
        containers:
          - name: broker 1
            readinessProbe:
              failureThreshold: 6 2
              initialDelaySeconds: 20 3

  router:
    ...
    podTemplate:
      spec:
        containers:
          - name: router 4
            livenessProbe:
              failureThreshold: 6 5
              initialDelaySeconds: 20 6

You can create a new infrastructure configuration or edit an existing one. For more information, see Infrastructure configuration.

Authentication services are used to configure the authentication and authorization endpoints available to messaging clients. The authentication services are configured by the AMQ Online service operator, and are specified when creating an address space.

Authentication services are configured as Custom Resources. An authentication service has a type, which can be standard, external, or none.

apiVersion: admin.enmasse.io/v1beta1
kind: AuthenticationService
metadata:
  name: standard
spec:
  type: standard 1
  standard:
    credentialsSecret: 2
      name: my-admin-credentials
    certificateSecret 3
      name: my-authservice-certificate
    resources: 4
      requests:
        memory: 2Gi
      limits:
        memory: 2Gi
    storage: 5
      type: persistent-claim
      size: 5Gi
    datasource: 6
      type: postgresql
      host: example.com
      port: 5432
      database: authdb

apiVersion: admin.enmasse.io/v1beta1
kind: AuthenticationService
metadata:
  name: my-external-1
spec:
  type: external
  realm: myrealm 1
  external:
    host: example.com 2
    port: 5671 3
    caCertSecret: 4
      name: my-ca-cert

apiVersion: admin.enmasse.io/v1beta1
kind: AuthenticationService
metadata:
  name: my-external-2
spec:
  type: external
  realm: myrealm 1
  external:
    host: example.org 2
    port: 5671 3
    caCertSecret: 4
      name: my-ca-cert
    allowOverride: true 5

{
    "authenticated-identity": {
        "sub": "myid",
        "preferred_username": "myuser"
    }
}

{
    "address-authz": {
        "myqueue": [
          "send",
          "recv"
        ],
        "mytopic": [
          "send"
        ]
    }
}

AMQ Online provides the following example roles that you can use directly or use as models to create your own roles.

For more information about service administrator resources, see the AMQ Online service administrator resources table.

For more information about messaging tenant resources, see the AMQ Online messaging tenant resources table.

In order to monitor AMQ Online on OpenShift 4 using the existing monitoring stack, user-workload monitoring must be enabled.

To monitor AMQ Online, an operator that acts on the monitoring Custom Resource Definitions must be deployed. You may skip this step if you have such an operator installed on your OpenShift cluster.

You can monitor AMQ Online pods using the kube-state-metrics agent.

If you are not using a default installation configuration, the simplest way to deploy monitoring is to enable the monitoring environment variable on the enmasse-operator deployment.

To configure alert notifications, such as emails, you must change the default configuration of Alertmanager.

Metrics from brokers and routers can be exposed to tenants without exposing system-admin metrics. To expose tenant metrics create a service monitor in any non-amq-online-infra namespace, ideally the namespace of the concerned address space(s).

You can use qdstat to monitor the AMQ Online service.

Restarting AMQ Online components is required to get image updates for CVEs. The scripts are provided in the AMQ Online installation files within the script folder. To restart all components, run all scripts.

For the standard address space type, you can view the router logs to troubleshoot issues with clients not connecting or issues with sending and receiving messages.

For the brokered or standard address space type, you can view the broker logs to troubleshoot issues with clients not connecting or issues with sending and receiving messages.

For diagnostic purposes, you can enable an AMQP protocol trace for a router. This can be helpful when troubleshooting issues related to client connectivity or with sending and receiving messages. There are two methods for enabling a protocol trace for the router.

For diagnostic purposes, you can enable an AMQP protocol trace for a broker. This can be helpful when troubleshooting issues with sending or receiving messages.

To enable the protocol trace, you apply configuration to the standardinfraconfig (for standard address spaces) or brokeredinfraconfig (for brokered address spaces) that enables the protocol trace for all brokers of all address spaces using that configuration. Applying this configuration will cause the brokers to restart.

If a problem is suspected with a Broker associated with an address space, you can examine the state of the broker directly using its built-in management interfaces. AMQ Online exposes the AMQ Broker’s CLI and JMX (via Jolokia). It does not expose the AMQ Broker Console.

Brokers are configured using the BrokeredInfraConfig and StandardInfraConfig resources, depending on the type of address space. When sizing a broker, consider:

The broker persists all messages to disk. When the BLOCK, FAIL, or DROP address full policy is specified, the number of messages that can be persisted is limited to the amount of memory in the broker. By using the PAGE address full policy, more messages can be stored than can be held in memory, at the expense of a potential performance degradation from reading data from disk. Therefore, paging is useful in the case of large messages or a large backlog of messages in your system.

10 queues * 1000 messages * (128 + (128 kB * 1024)) = 1.25 GB

apiVersion: admin.enmasse.io/v1beta1
kind: BrokeredInfraConfig
metadata:
  name: cfg1
spec:
  broker:
    addressFullPolicy: FAIL
    globalMaxSize: 1.35Gb
    resources:
      memory: 8Gi
      storage: 2Gi
  ...

(1000 messages * 1000 * 128 kB) + (10 queues * 128 kB * 1024) = 123.5 MB

apiVersion: admin.enmasse.io/v1beta1
kind: BrokeredInfraConfig
metadata:
  name: cfg1
spec:
  broker:
    addressFullPolicy: PAGE
    globalMaxSize: 124Mb
    resources:
      memory: 1Gi
      storage: 2Gi
  ...

apiVersion: admin.enmasse.io/v1beta2
kind: AddressSpacePlan
metadata:
  name: cfg1
spec:
  resourceLimits:
    broker: 4.0
  ...

apiVersion: admin.enmasse.io/v1beta2
kind: AddressPlan
metadata:
  name: plan1
spec:
  ...
  resources:
    broker: 0.01

Routers are configured in the StandardInfraConfig resource. In determining router sizing, consider:

The router does not persist any state and therefore does not require persistent storage.

Address configuration itself does not require a significant amount of router memory. However, queues and subscriptions require an additional two links between the router and broker per address.

The total number of links is then two times the number of queues/subscriptions plus the number of client links. Each link requires metadata and buffers in the router to handle routing messages for that link.

The router link capacity affects how many messages the router can handle per link. Setting the link capacity to a higher value might improve performance, but at the cost of potentially more memory being used to hold in-flight messages if senders are filling the links. If you have many connections and links, consider specifying a lower value to balance the memory usage.

In addition, the router has to parse the message headers, manage dispositions and settlements of messages, and other per-link activities. The per-link cost can be derived using a constant factor of the link capacity and message size. This factor varies depending on the message size. The following table provides an approximation of this factor for different message size ranges:

500 anycast addresses * 7 kB overhead per address = 3.5 MB

(1000 queued addresses * 32,768) + (2000 * 18,000 link multiplication factor * 100 links) = 374 MB

10,000 clients * 10 link capacity * 18,000 link multiplication factor = 1717 MB

apiVersion: admin.enmasse.io/v1beta1
kind: StandardInfraConfig
metadata:
  name: cfg1
spec:
  router:
    resources:
      memory: 1100Mi
    linkCapacity: 10
    policy:
      maxConnections: 5000
      maxSessionsPerConnection: 1
      maxSendersPerConnection: 1
      maxReceiversPerConnection: 1
  ...

apiVersion: admin.enmasse.io/v1beta1
kind: StandardInfraConfig
metadata:
  name: cfg1
spec:
  router:
    minReplicas: 2
  ...
---
apiVersion: admin.enmasse.io/v1beta2
kind: AddressSpacePlan
metadata:
  name: plan1
spec:
  infraConfigRef: cfg1
  resourceLimits:
    router: 4
  ...

apiVersion: admin.enmasse.io/v1beta2
kind: AddressPlan
metadata:
  name: plan1
spec:
  ...
  resources:
    router: 0.01

The operator component is tasked with reading all address configuration and applying these configurations to the routers and brokers. It is important to size the operator component proportionally to the number of addresses.

In the standard address space, the admin Pod contains two processes, agent and standard-controller. These processes cannot be sized individually, but the memory usage of both is proportional to the number of addresses. In the brokered address space, only a single agent process exists.

apiVersion: admin.enmasse.io/v1beta1
kind: StandardInfraConfig
metadata:
  name: cfg1
spec:
  admin:
    resources:
      memory: 300Mi
  ...

Plans enable dynamic scaling in the standard address space, as shown in the broker and router sizing sections. At the cluster level, the combination of plans and infrastructure configuration settings determines the maximum number of Pods that can be deployed on the cluster. Since AMQ Online does not support limiting the number of address spaces that can be created, it is a best practice to apply a policy to limit who is allowed to create address spaces. Such policy configuration can be handled through the standard OpenShift policies.

From a capacity-planning perspective, it is useful to calculate the maximum number of Pods and the maximum amount of memory that can be consumed for a given address space. To make this calculation using a script, see Running the check-memory calculation script.

Per address broker memory limits are calculated from the address plan configuration. AMQ Online determines the maximum size allowed for each queue by multiplying the broker configuration globalMaxSize (specified in the standardinfraconfig or brokeredinfraconfig) by the address plan’s broker resource limit. The behavior when the queue reaches its memory limit is governed by the address full policy. For more information on the address full policy, see Broker component sizing.

For example, if the broker’s configuration specifies globalMaxSize = 124 MB and the address plan configuration specifies addressplan.spec.resources.broker = 0.2, the maximum size allowed for each queue is 25 MB (124 * 0.2 = 25 MB).

Before you begin configuring resources for AMQ Online, you must first understand the concepts of an address space and an address in AMQ Online.

The service administrator configures AMQ Online by creating Custom Resources that comprise the "service configuration." This service configuration contains instances of the following Custom Resource types:

When created, these Custom Resources define the configuration that is available to the messaging tenants.

The following diagram illustrates the relationship between the different service configuration resources and how they are referenced by the messaging tenant resources.

To help illustrate how the service configuration resources can be defined to satisfy a particular use case, the requirements of Company X for using AMQ Online are outlined. This use case is referenced throughout the following documentation describing the service configuration resource types in further detail.

Company X has the following requirements:

Each of these requirements and how they can be met by configuring the appropriate resources is discussed in the following sections.

apiVersion: admin.enmasse.io/v1beta1
kind: BrokeredInfraConfig
metadata:
  name: engineering
spec:
  broker:
    resources:
      memory: 512Mi
      storage: 20Mi

apiVersion: admin.enmasse.io/v1beta1
kind: BrokeredInfraConfig
metadata:
  name: qa
spec:
  broker:
    resources:
      memory: 4Gi
      storage: 50Gi

apiVersion: admin.enmasse.io/v1beta1
kind: BrokeredInfraConfig
metadata:
  name: engineering
spec:
  networkPolicy:
    ingress:
      - from:
        - namespaceSelector:
            matchLabels:
              org: engineering
  broker:
    resources:
      memory: 512Mi
      storage: 20Mi

apiVersion: admin.enmasse.io/v1beta2
kind: AddressSpacePlan
metadata:
  name: engineering
spec:
  infraConfigRef: engineering
  addressSpaceType: brokered
  addressPlans:
  - brokered-queue
  - brokered-topic

apiVersion: admin.enmasse.io/v1beta1
kind: AuthenticationService
metadata:
  name: engineering
spec:
  type: none

apiVersion: admin.enmasse.io/v1beta1
kind: AuthenticationService
metadata:
  name: qa
spec:
  type: standard
  standard:
    storage:
      type: persistent-claim
      size: 5Gi
    datasource:
      type: postgresql
      host: db.example.com
      port: 5432
      database: authdb