You can use AMQ Interconnect to flexibly route messages between any AMQP-enabled endpoints, including clients, servers, and message brokers. AMQ Interconnect provides the following benefits:

AMQ Interconnect supports the following industry-recognized standards and network protocols:

AMQ Interconnect is supported on Red Hat Enterprise Linux 6, 7, and 8. See Red Hat AMQ Supported Configurations for more information.

Before using AMQ Interconnect, you should be familiar with AMQP and understand some key concepts about AMQ Interconnect.

AMQ Interconnect 1.5 is distributed as a set of RPM packages, which are available through your Red Hat subscription.

The router’s configuration file (qdrouterd.conf) controls the way in which the router functions. The default configuration file contains the minimum number of settings required for the router to run. As you become more familiar with the router, you can add to or change these settings, or create your own configuration files.

By default, the router configuration file defines the following settings for the router:

After installing AMQ Interconnect, you start the router by using the qdrouterd command.

After starting the router, send some test messages to see how the router can connect two endpoints by distributing messages between them.

This procedure demonstrates a simple configuration consisting of a single router with two clients connected to it: a sender and a receiver. The receiver wants to receive messages on a specific address, and the sender sends messages to that address.

A broker is not used in this procedure, so there is no "store and forward" mechanism in the middle. Instead, the messages flow from the sender, through the router, to the receiver only if the receiver is online, and the sender can confirm that the messages have arrived at their destination.

After you successfully install a standalone router and use it to distribute messages between two clients, you can configure a router network topology and route messages.

To plan your router network and design the network topology, you must first understand the different router modes and how you can use them to create different types of networks.

After planning the router network topology, you implement it by adding each router to the router network. You add routers one at a time.

This procedure describes the workflow required to add a router to the router network.

By default, AMQ Interconnect operates in standalone mode with a randomly-generated ID. If you want to use this router in a router network, you must change these properties.

AMQ Interconnect connects clients, servers, AMQP services, and other routers through network connections. To connect the router to other messaging endpoints, you configure listeners to accept connections, and connectors to make outbound connections. However, connections are bidirectional - once the connection is established, message traffic flows in both directions.

You can configure the following types of connections:

You can configure AMQ Interconnect to communicate with clients, routers, and brokers in a secure way by authenticating and encrypting the router’s connections. AMQ Interconnect supports the following security protocols:

To secure the router network, you configure SSL/TLS, SASL (or a combination of both) to secure each of the following types of connections:

You can configure policies to secure messaging resources in your messaging environment. Policies ensure that only authorized users can access messaging endpoints through the router network, and that the resources on those endpoints are used in an authorized way.

AMQ Interconnect provides the following types of policies:

The resource limits defined in global and vhost policies are applied to user connections only. The limits do not affect inter-router connections or router connections that are outbound to waypoints.

sources: *

targets:

sources: tmp_${user}, temp*, ${user}-home-*

sourcePattern: tmp.${user}, temp/#, ${user}.home/*

[
    ["vhost", {
        "hostname": "example.com",  1
        "maxConnectionsPerUser": 10,  2
        "allowUnknownUser": true,  3
        "groups": {
            "admin": {
                "users": ["admin1", "admin2"],  4
                "remoteHosts": ["127.0.0.1", "::1"],  5
                "sources": "*",  6
                "targets": "*"  7
            },
            "$default": {
                "remoteHosts": "*",  8
                "sources": ["news*", "sports*" "chat*"],  9
                "targets": "chat*"  10
            }
        }
    }]
]

[
    ["vhost", {
        "hostname": "traders.com",  1
        "groups": {
            "traders": {
                "users": ["trader1", "trader2"],  2
                "maxFrameSize": 10000,
                "maxSessionWindow": 5000000,  3
                "maxSessions": 1  4
            },
            "feeds": {
                "users": ["nyse-feed", "nasdaq-feed"],  5
                "maxFrameSize": 60000,
                "maxSessionWindow": 1200000000,  6
                "maxSessions": 3  7
            }
        }
    }]
]

While you can use either message routing or link routing to deliver messages to a destination, they differ in several important ways. Understanding these differences will enable you to choose the proper routing approach for any particular use case.

With message routing, routing is performed on messages as producers send them to a router. When a message arrives on a router, the router routes the message and its settlement based on the message’s address and routing pattern.

With message routing, you can do the following:

5.2.7. Routing Messages Through a Broker Queue

You can route messages to and from a broker queue to provide clients with access to the queue through a router. In this scenario, clients connect to a router to send and receive messages, and the router routes the messages to or from the broker queue.

You can route messages to a queue hosted on a single broker, or route messages to a sharded queue distributed across multiple brokers.

Figure 5.6. Brokered Messaging

In this diagram, the sender connects to the router and sends messages to my_queue. The router attaches an outgoing link to the broker, and then sends the messages to my_queue. Later, the receiver connects to the router and requests messages from my_queue. The router attaches an incoming link to the broker to receive the messages from my_queue, and then delivers them to the receiver.

You can also route messages to a sharded queue, which is a single, logical queue comprised of multiple, underlying physical queues. Using queue sharding, it is possible to distribute a single queue over multiple brokers. Clients can connect to any of the brokers that hold a shard to send and receive messages.

Figure 5.7. Brokered Messaging with Sharded Queue

In this diagram, a sharded queue (my_queue) is distributed across two brokers. The router is connected to the clients and to both brokers. The sender connects to the router and sends messages to my_queue. The router attaches an outgoing link to each broker, and then sends messages to each shard (by default, the routing distribution is balanced). Later, the receiver connects to the router and requests all of the messages from my_queue. The router attaches an incoming link to one of the brokers to receive the messages from my_queue, and then delivers them to the receiver.

Procedure

1. Add a waypoint address.

   This address identifies the queue to which you want to route messages.

2. Add autolinks to connect the router to the broker.

   Autolinks connect the router to the broker queue identified by the waypoint address.

3. If the queue is sharded, add autolinks for each additional broker that hosts a shard.

5.2.7.1. Configuring Waypoint Addresses

A waypoint address identifies a queue on a broker to which you want to route messages. You need to configure the waypoint address on each router that needs to use the address. For example, if a client is connected to Router A to send messages to the broker queue, and another client is connected to Router B to receive those messages, then you would need to configure the waypoint address on both Router A and Router B.

Prerequisites

An incoming connection (listener) to which the clients can connect should be configured. This connection defines how the producers and consumers connect to the router to send and receive messages. For more information, see Section 4.2.2, “Listening for client connections”.

Procedure

• Create waypoint addresses on each router that needs to use the address:

  address {
      prefix: ADDRESS_PREFIX
      waypoint: yes
  }

  prefix | pattern

  The address prefix or pattern that matches the broker queue to which you want to send messages. You can specify a prefix to match an exact address or beginning segment of an address. Alternatively, you can specify a pattern to match an address using wildcards.

  A prefix matches either an exact address or the beginning segment within an address that is delimited by either a . or / character. For example, the prefix my_address would match the address my_address as well as my_address.1 and my_address/1. However, it would not match my_address1.

  A pattern matches an address that corresponds to a pattern. A pattern is a sequence of words delimited by either a . or / character. You can use wildcard characters to represent a word. The * character matches exactly one word, and the # character matches any sequence of zero or more words.

  The * and # characters are reserved as wildcards. Therefore, you should not use them in the message address.

  For more information about creating address patterns, see Section 5.3.6, “Pattern Matching for Addresses”.

  Note

  You can convert a prefix value to a pattern by appending /# to it. For example, the prefix a/b/c is equivalent to the pattern a/b/c/#.

  waypoint

  Set this attribute to yes so that the router handles messages sent to this address as a waypoint.

5.2.7.2. Connecting a Router to the Broker

After you add waypoint addresses to identify the broker queue, you must connect a router to the broker using autolinks.

With autolinks, client traffic is handled on the router, not the broker. Clients attach their links to the router, and then the router uses internal autolinks to connect to the queue on the broker. Therefore, the queue will always have a single producer and a single consumer regardless of how many clients are attached to the router.

Note

If the connection to the broker fails, AMQ Interconnect automatically attempts to reestablish the connection and reroute message deliveries to any available alternate destinations. However, some deliveries could be returned to the sender with a RELEASED or MODIFIED disposition. Therefore, you should ensure that your clients can handle these deliveries appropriately (generally by resending them).

1. If this router is different than the router that is connected to the clients, then add the waypoint address.

2. Add an outgoing connection to the broker:

   connector {
       name: NAME
       host: HOST_NAME/ADDRESS
       port: PORT_NUMBER/NAME
       role: route-container
       ...
   }

   name

   The name of the connector. Specify a name that describes the broker.

   host

   Either an IP address (IPv4 or IPv6) or hostname on which the router should connect to the broker.

   port

   The port number or symbolic service name on which the router should connect to the broker.

   role

   Specify route-container to indicate that this connection is for an external container (broker).

   For information about additional attributes, see connector in the qdrouterd.conf man page.

3. If you want to send messages to the broker queue, create an outgoing autolink to the broker queue:

   autoLink {
       address: ADDRESS
       connection: CONNECTOR_NAME
       direction: out
       ...
   }

   address

   The address of the broker queue. When the autolink is created, it will be attached to this address.

   externalAddress

   An optional alternate address for the broker queue. You use an external address if the broker queue should have a different address than that which the sender uses. In this scenario, senders send messages to the addr address, and then the router routes them to the broker queue represented by the externalAddress address.

   connection | containerID

   How the router should connect to the broker. You can specify either an outgoing connection (connection) or the container ID of the broker (containerID).

   direction

   Set this attribute to out to specify that this autolink can send messages from the router to the broker.

   For information about additional attributes, see autoLink in the qdrouterd.conf man page.

4. If you want to receive messages from the broker queue, create an incoming autolink from the broker queue:

   autoLink {
       address: ADDRESS
       connection: CONNECTOR_NAME
       direction: in
       ...
   }

   address

   The address of the broker queue. When the autolink is created, it will be attached to this address.

   externalAddress

   An optional alternate address for the broker queue. You use an external address if the broker queue should have a different address than that which the receiver uses. In this scenario, receivers receive messages from the addr address, and the router retrieves them from the broker queue represented by the externalAddress address.

   connection | containerID

   How the router should connect to the broker. You can specify either an outgoing connection (connection) or the container ID of the broker (containerID).

   direction

   Set this attribute to in to specify that this autolink can receive messages from the broker to the router.

   For information about additional attributes, see autoLink in the qdrouterd.conf man page.

connector {  1
    name: broker
    role: route-container
    host: 198.51.100.1
    port: 61617
    saslMechanisms: ANONYMOUS
}

address {  2
    prefix: queue
    waypoint: yes
}

autoLink {  3
    address: queue.first
    direction: in
    connection: broker
}

autoLink {  4
    address: queue.first
    direction: out
    connection: broker
}

autoLink {  5
    address: queue.second
    direction: in
    connection: broker
}

autoLink {  6
    address: queue.second
    direction: out
    connection: broker
}

$ qdstat --autolinks
AutoLinks
  addr          dir  phs  extAddr  link  status    lastErr
  ========================================================
  queue.first   in   1                   inactive
  queue.first   out  0                   inactive
  queue.second  in   1                   inactive
  queue.second  out  0                   inactive

$ qdstat --autolinks
AutoLinks
  addr          dir  phs  extAddr  link  status  lastErr
  ===========================================================================
  queue.first   in   1             6     active
  queue.first   out  0             7     active
  queue.second  in   1                   failed  Node not found: queue.second
  queue.second  out  0                   failed  Node not found: queue.second

$ python simple_send.py -a 127.0.0.1/queue.first -m3
all messages confirmed

$ qdstat -a
Router Addresses
  class   addr           phs  distrib   in-proc  local  remote  cntnr  in  out  thru  to-proc  from-proc
  ========================================================================================================
  mobile  queue.first    1    balanced  0        0      0       0      0   0    0     0        0
  mobile  queue.first    0    balanced  0        1      0       0      3   3    0     0        0

$ python simple_recv.py -a 127.0.0.1:5672/queue.first -m3
{u'sequence': int32(1)}
{u'sequence': int32(2)}
{u'sequence': int32(3)}

$ qdstat -a
Router Addresses
  class   addr           phs  distrib   in-proc  local  remote  cntnr  in  out  thru  to-proc  from-proc
  ========================================================================================================
  mobile  queue.first    1    balanced  0        0      0       0      3   3    0     0        0
  mobile  queue.first    0    balanced  0        1      0       0      3   3    0     0        0

Link routing provides an alternative strategy for brokered messaging. A link route represents a private messaging path between a sender and a receiver in which the router passes the messages between end points. You can think of a link route as a "virtual connection" or "tunnel" that travels from a sender, through the router network, to a receiver.

With link routing, routing is performed on link-attach frames, which are chained together to form a virtual messaging path that directly connects a sender and receiver. Once a link route is established, the transfer of message deliveries, flow frames, and dispositions is performed across the link route.

                        Public Network
                       +-----------------+
                       |      +-----+    |
                       | B1   | Rp  |    |
                       |      +/--\-+    |
                       |      /    \     |
                       |     /      \    |
                       +----/--------\---+
                           /          \
                          /            \
                         /              \
         Private Net A  /                \ Private Net B
        +--------------/--+           +---\-------------+
        |         +---/-+ |           | +--\--+         |
        |  B2     | Ra  | |           | | Rb  |   C1    |
        |         +-----+ |           | +-----+         |
        |                 |           |                 |
        |                 |           |                 |
        +-----------------+           +-----------------+

connector {  1
    name: broker
    role: route-container
    host: 198.51.100.1
    port: 61617
    saslMechanisms: ANONYMOUS
}

linkRoute {  2
    prefix: b2
    direction: in
    connection: broker
}

linkRoute {  3
    prefix: b2
    direction: out
    connection: broker
}

Logging enables you to diagnose error and performance issues with AMQ Interconnect.

AMQ Interconnect consists of internal modules that provide important information about the router. For each module, you can specify logging levels, the format of the log file, and the location to which the logs should be written.

2019-04-05 14:54:38.037248 -0400 ROUTER (info) [C1] Connection Opened: dir=in host=127.0.0.1:55440 vhost= encrypted=no auth=no user=anonymous container_id=95e55424-6c0a-4a5c-8848-65a3ea5cc25a props= 1
2019-04-05 14:54:38.038137 -0400 ROUTER (info) [C1][L6] Link attached: dir=in source={<none> expire:sess} target={$management expire:sess} 2
2019-04-05 14:54:38.041103 -0400 ROUTER (info) [C1][L6] Link lost: del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0 3
2019-04-05 14:54:38.041154 -0400 ROUTER (info) [C1] Connection Closed 4

Tue Jun  7 13:50:21 2016 ROUTER_HELLO (trace) RCVD: HELLO(id=Router.B area=0 inst=1465307413 seen=['Router.A', 'Router.C']) 1
Tue Jun  7 13:50:21 2016 ROUTER_HELLO (trace) SENT: HELLO(id=Router.A area=0 inst=1465307416 seen=['Router.B']) 2
Tue Jun  7 13:50:22 2016 ROUTER_HELLO (trace) RCVD: HELLO(id=Router.B area=0 inst=1465307413 seen=['Router.A', 'Router.C'])
Tue Jun  7 13:50:22 2016 ROUTER_HELLO (trace) SENT: HELLO(id=Router.A area=0 inst=1465307416 seen=['Router.B'])

Tue Jun  7 13:50:18 2016 ROUTER_HELLO (trace) SENT: HELLO(id=Router.B area=0 inst=1465307413 seen=['Router.A', 'Router.C']) 1
Tue Jun  7 13:50:18 2016 ROUTER_HELLO (trace) RCVD: HELLO(id=Router.A area=0 inst=1465307416 seen=['Router.B']) 2
Tue Jun  7 13:50:19 2016 ROUTER_HELLO (trace) RCVD: HELLO(id=Router.C area=0 inst=1465307411 seen=['Router.B']) 3

Tue Jun  7 14:10:02 2016 ROUTER_LS (trace) SENT: LSR(id=Router.A area=0) to: Router.C
Tue Jun  7 14:10:02 2016 ROUTER_LS (trace) SENT: LSR(id=Router.A area=0) to: Router.B
Tue Jun  7 14:10:02 2016 ROUTER_LS (trace) SENT: RA(id=Router.A area=0 inst=1465308600 ls_seq=1 mobile_seq=1) 1
Tue Jun  7 14:10:02 2016 ROUTER_LS (trace) RCVD: LSU(id=Router.B area=0 inst=1465308595 ls_seq=2 ls=LS(id=Router.B area=0 ls_seq=2 peers={'Router.A': 1L, 'Router.C': 1L})) 2
Tue Jun  7 14:10:02 2016 ROUTER_LS (trace) RCVD: LSR(id=Router.B area=0)
Tue Jun  7 14:10:02 2016 ROUTER_LS (trace) SENT: LSU(id=Router.A area=0 inst=1465308600 ls_seq=1 ls=LS(id=Router.A area=0 ls_seq=1 peers={'Router.B': 1}))
Tue Jun  7 14:10:02 2016 ROUTER_LS (trace) RCVD: RA(id=Router.C area=0 inst=1465308592 ls_seq=1 mobile_seq=0)
Tue Jun  7 14:10:02 2016 ROUTER_LS (trace) SENT: LSR(id=Router.A area=0) to: Router.C
Tue Jun  7 14:10:02 2016 ROUTER_LS (trace) RCVD: LSR(id=Router.C area=0) 3
Tue Jun  7 14:10:02 2016 ROUTER_LS (trace) SENT: LSU(id=Router.A area=0 inst=1465308600 ls_seq=1 ls=LS(id=Router.A area=0 ls_seq=1 peers={'Router.B': 1}))
Tue Jun  7 14:10:02 2016 ROUTER_LS (trace) RCVD: LSU(id=Router.C area=0 inst=1465308592 ls_seq=1 ls=LS(id=Router.C area=0 ls_seq=1 peers={'Router.B': 1L})) 4
Tue Jun  7 14:10:03 2016 ROUTER_LS (trace) Computed next hops: {'Router.C': 'Router.B', 'Router.B': 'Router.B'} 5
Tue Jun  7 14:10:03 2016 ROUTER_LS (trace) Computed costs: {'Router.C': 2L, 'Router.B': 1}
Tue Jun  7 14:10:03 2016 ROUTER_LS (trace) Computed valid origins: {'Router.C': [], 'Router.B': []}

Tue Jun  7 14:27:20 2016 ROUTER_MA (trace) SENT: MAU(id=Router.A area=0 mobile_seq=1 add=['Cmy_queue', 'Dmy_queue', 'M0my_queue_wp'] del=[]) 1
Tue Jun  7 14:27:21 2016 ROUTER_MA (trace) RCVD: MAR(id=Router.C area=0 have_seq=0) 2
Tue Jun  7 14:27:21 2016 ROUTER_MA (trace) SENT: MAU(id=Router.A area=0 mobile_seq=1 add=['Cmy_queue', 'Dmy_queue', 'M0my_queue_wp'] del=[])
Tue Jun  7 14:27:22 2016 ROUTER_MA (trace) RCVD: MAR(id=Router.B area=0 have_seq=0) 3
Tue Jun  7 14:27:22 2016 ROUTER_MA (trace) SENT: MAU(id=Router.A area=0 mobile_seq=1 add=['Cmy_queue', 'Dmy_queue', 'M0my_queue_wp'] del=[])
Tue Jun  7 14:27:39 2016 ROUTER_MA (trace) RCVD: MAU(id=Router.C area=0 mobile_seq=1 add=['M0my_test'] del=[]) 4
Tue Jun  7 14:27:51 2016 ROUTER_MA (trace) RCVD: MAU(id=Router.C area=0 mobile_seq=2 add=[] del=['M0my_test']) 5

Tue Jun  7 14:36:54 2016 MESSAGE (trace) Sending Message{to='amqp:/_topo/0/Router.B/qdrouter' body='\d1\00\00\00\1b\00\00\00\04\a1\02id\a1\08R'} on link qdlink.p9XmBm19uDqx50R
Tue Jun  7 14:36:54 2016 MESSAGE (trace) Received Message{to='amqp:/_topo/0/Router.A/qdrouter' body='\d1\00\00\00\8e\00\00\00
\a1\06ls_se'} on link qdlink.phMsJOq7YaFsGAG
Tue Jun  7 14:36:54 2016 MESSAGE (trace) Received Message{ body='\d1\00\00\00\10\00\00\00\02\a1\08seque'} on link qdlink.FYHqBX+TtwXZHfV
Tue Jun  7 14:36:54 2016 MESSAGE (trace) Sending Message{ body='\d1\00\00\00\10\00\00\00\02\a1\08seque'} on link qdlink.yU1tnPs5KbMlieM
Tue Jun  7 14:36:54 2016 MESSAGE (trace) Sending Message{to='amqp:/_local/qdhello' body='\d1\00\00\00G\00\00\00\08\a1\04seen\d0'} on link qdlink.p9XmBm19uDqx50R
Tue Jun  7 14:36:54 2016 MESSAGE (trace) Sending Message{to='amqp:/_topo/0/Router.C/qdrouter' body='\d1\00\00\00\1b\00\00\00\04\a1\02id\a1\08R'} on link qdlink.p9XmBm19uDqx50R

Tue Jun  7 14:39:52 2016 SERVER (trace) [2]:  <- AMQP
Tue Jun  7 14:39:52 2016 SERVER (trace) [1]:  <- AMQP
Tue Jun  7 14:39:52 2016 SERVER (trace) [1]:0 <- @open(16) [container-id="Router.B", max-frame-size=16384, channel-max=32767, idle-time-out=8000, offered-capabilities=:"ANONYMOUS-RELAY", properties={:product="qpid-dispatch-router", :version="0.6.0"}]
Tue Jun  7 14:39:52 2016 SERVER (trace) [1]:0 -> @begin(17) [next-outgoing-id=0, incoming-window=15, outgoing-window=2147483647]
Tue Jun  7 14:39:52 2016 SERVER (trace) [1]:RAW: "\x00\x00\x00\x1e\x02\x00\x00\x00\x00S\x11\xd0\x00\x00\x00\x0e\x00\x00\x00\x04@R\x00R\x0fp\x7f\xff\xff\xff"
Tue Jun  7 14:39:52 2016 SERVER (trace) [1]:1 -> @begin(17) [next-outgoing-id=0, incoming-window=15, outgoing-window=2147483647]
Tue Jun  7 14:39:52 2016 SERVER (trace) [1]:RAW: "\x00\x00\x00\x1e\x02\x00\x00\x01\x00S\x11\xd0\x00\x00\x00\x0e\x00\x00\x00\x04@R\x00R\x0fp\x7f\xff\xff\xff"
Tue Jun  7 14:39:52 2016 SERVER (trace) [1]:0 -> @attach(18) [name="qdlink.uSSeXPSfTHhxo8d", handle=0, role=true, snd-settle-mode=2, rcv-settle-mode=0, source=@source(40) [durable=0, expiry-policy=:"link-detach", timeout=0, dynamic=false, capabilities=:"qd.router"], target=@target(41) [durable=0, expiry-policy=:"link-detach", timeout=0, dynamic=false, capabilities=:"qd.router"], initial-delivery-count=0]
Tue Jun  7 14:39:52 2016 SERVER (trace) [1]:RAW: "\x00\x00\x00\x91\x02\x00\x00\x00\x00S\x12\xd0\x00\x00\x00\x81\x00\x00\x00\x0a\xa1\x16qdlink.uSSeXPSfTHhxo8dR\x00AP\x02P\x00\x00S(\xd0\x00\x00\x00'\x00\x00\x00\x0b@R\x00\xa3\x0blink-detachR\x00B@@@@@\xa3\x09qd.router\x00S)\xd0\x00\x00\x00#\x00\x00\x00\x07@R\x00\xa3\x0blink-detachR\x00B@\xa3\x09qd.router@@R\x00"

Tue Jun  7 15:07:32 2016 AGENT (debug) Add entity: ConnectorEntity(addr=127.0.0.1, allowRedirect=True, cost=1, host=127.0.0.1, identity=connector/127.0.0.1:5672:BROKER, idleTimeoutSeconds=16, maxFrameSize=65536, name=BROKER, port=5672, role=route-container, stripAnnotations=both, type=org.apache.qpid.dispatch.connector, verifyHostname=True)
Tue Jun  7 15:07:32 2016 AGENT (debug) Add entity: RouterConfigAddressEntity(distribution=closest, identity=router.config.address/0, name=router.config.address/0, prefix=my_address, type=org.apache.qpid.dispatch.router.config.address, waypoint=False)
Tue Jun  7 15:07:32 2016 AGENT (debug) Add entity: RouterConfigAddressEntity(distribution=balanced, identity=router.config.address/1, name=router.config.address/1, prefix=my_queue_wp, type=org.apache.qpid.dispatch.router.config.address, waypoint=True)
Tue Jun  7 15:07:32 2016 AGENT (debug) Add entity: RouterConfigLinkrouteEntity(connection=BROKER, direction=in, distribution=linkBalanced, identity=router.config.linkRoute/0, name=router.config.linkRoute/0, prefix=my_queue, type=org.apache.qpid.dispatch.router.config.linkRoute)
Tue Jun  7 15:07:32 2016 AGENT (debug) Add entity: RouterConfigLinkrouteEntity(connection=BROKER, direction=out, distribution=linkBalanced, identity=router.config.linkRoute/1, name=router.config.linkRoute/1, prefix=my_queue, type=org.apache.qpid.dispatch.router.config.linkRoute)
Tue Jun  7 15:07:32 2016 AGENT (debug) Add entity: RouterConfigAutolinkEntity(address=my_queue_wp, connection=BROKER, direction=in, identity=router.config.autoLink/0, name=router.config.autoLink/0, type=org.apache.qpid.dispatch.router.config.autoLink)
Tue Jun  7 15:07:32 2016 AGENT (debug) Add entity: RouterConfigAutolinkEntity(address=my_queue_wp, connection=BROKER, direction=out, identity=router.config.autoLink/1, name=router.config.autoLink/1, type=org.apache.qpid.dispatch.router.config.autoLink)

Tue Jun  7 14:46:18 2016 CONTAINER (trace) Container Initialized
Tue Jun  7 14:46:18 2016 CONTAINER (trace) Node Type Registered - router
Tue Jun  7 14:46:18 2016 CONTAINER (trace) Node of type 'router' installed as default node

$ sudo qdrouterd --conf xxx
Wed Jun 15 09:53:28 2016 ERROR (error) Python: Exception: Cannot load configuration file xxx: [Errno 2] No such file or directory: 'xxx'
Wed Jun 15 09:53:28 2016 ERROR (error) Traceback (most recent call last):
  File "/usr/lib/qpid-dispatch/python/qpid_dispatch_internal/management/config.py", line 155, in configure_dispatch
    config = Config(filename)
  File "/usr/lib/qpid-dispatch/python/qpid_dispatch_internal/management/config.py", line 41, in __init__
    self.load(filename, raw_json)
  File "/usr/lib/qpid-dispatch/python/qpid_dispatch_internal/management/config.py", line 123, in load
    with open(source) as f:
Exception: Cannot load configuration file xxx: [Errno 2] No such file or directory: 'xxx'

Wed Jun 15 09:53:28 2016 MAIN (critical) Router start-up failed: Python: Exception: Cannot load configuration file xxx: [Errno 2] No such file or directory: 'xxx'
qdrouterd: Python: Exception: Cannot load configuration file xxx: [Errno 2] No such file or directory: 'xxx'

Tue Jun  7 15:07:32 2016 POLICY (info) Policy configured maximumConnections: 0, policyFolder: '', access rules enabled: 'false'
Tue Jun  7 15:07:32 2016 POLICY (info) Policy fallback defaultApplication is disabled

AMQ Console is a web console for monitoring the status and performance of AMQ Interconnect router networks.

You can use qdstat to view the status of routers on your router network. For example, you can view information about the attached links and configured addresses, available connections, and nodes in the router network.

$ qdstat OPTION [CONNECTION_OPTIONS] [SECURE_CONNECTION_OPTIONS]

You can use qdmanage to view and modify the configuration of a running router at runtime. Specifically, qdmanage enables you to create, read, update, and delete the sections and attributes in the router’s configuration file without having to restart the router.

$ qdmanage [CONNECTION_OPTIONS] OPERATION [OPTIONS]

$ qdmanage query --type listener
[
  {
    "stripAnnotations": "both",
    "addr": "127.0.0.1",
    "multiTenant": false,
    "requireSsl": false,
    "idleTimeoutSeconds": 16,
    "saslMechanisms": "ANONYMOUS",
    "maxFrameSize": 16384,
    "requireEncryption": false,
    "host": "0.0.0.0",
    "cost": 1,
    "role": "normal",
    "http": false,
    "maxSessions": 32768,
    "authenticatePeer": false,
    "type": "org.apache.qpid.dispatch.listener",
    "port": "amqp",
    "identity": "listener/0.0.0.0:amqp",
    "name": "listener/0.0.0.0:amqp"
  }
]

qdmanage query --type=listener

qdmanage query role port --type=listener

qdmanage read --name=LISTENER_NAME

qdmanage create --type=listener --ATTRIBUTE=VALUE ...

qdmanage update --type=listener --ATTRIBUTE=VALUE ...

qdmanage update --type=listener --ATTRIBUTE

qdmanage delete --name=LISTENER_NAME

qdmanage query --type=connector

qdmanage query role port --type=connector

qdmanage query failoverUrls --type=connector --name=CONNECTOR_NAME

qdmanage read --name=CONNECTOR_NAME

qdmanage create --type=connector --ATTRIBUTE=VALUE ...

qdmanage update --type=connector --ATTRIBUTE=VALUE ...

qdmanage update --type=connector --ATTRIBUTE

qdmanage delete --name=CONNECTOR_NAME

qdmanage query --type=sslProfile

qdmanage create --type=sslProfile --name=NAME --ATTRIBUTE=VALUE ...

qdmanage update --name=LISTENER_NAME --sslProfile=NAME --requireSsl=yes

qdmanage update --name=LISTENER_NAME --ATTRIBUTE=VALUE ...

qdmanage update --name=LISTENER_NAME --authenticatePeer=yes

qdmanage update --name=LISTENER_NAME --authenticatePeer=no

qdmanage update --name=CONNECTOR_NAME --sslProfile=NAME

qdmanage update --name=CONNECTOR_NAME --sslProfile

qdmanage delete --name=SSL_PROFILE_NAME

qdmanage update --type=router --saslConfigDir=PATH --saslConfigName=NAME

qdmanage update --name=LISTENER_NAME --authenticatePeer=yes --saslMechanisms=MECHANISMS

qdmanage update --name=LISTENER_NAME --saslMechanisms=MECHANISMS

qdmanage update --name=CONNECTOR_NAME --saslMechanisms=MECHANISMS --saslUsername=USERNAME --saslPassword=PASSWORD

qdmanage update --name=CONNECTOR_NAME --saslMechanisms=MECHANISMS

qdmanage update --name=LISTENER_NAME --requireEncryption=yes --saslMechanisms=MECHANISMS

qdmanage update --name=LISTENER_NAME --saslMechanisms=MECHANISMS

qdmanage update --name=LISTENER_NAME --requireEncryption=no --saslMechanisms

qdmanage update --type=router --saslConfigDir --saslConfigName

qdmanage query --type=address

qdmanage read --name=ADDRESS_NAME

qdmanage query prefix distribution --type=address

qdmanage query prefix --type=address --waypoint=yes

qdmanage query --type=autolink

qdmanage create --type=address --prefix=ADDRESS_PREFIX --distribution=DISTRIBUTION_PATTERN ...

qdmanage update --name=ADDRESS_NAME --ATTRIBUTE=VALUE ...

qdmanage update --name=AUTOLINK_NAME --ATTRIBUTE=VALUE ...

qdmanage delete --name=ADDRESS_NAME

qdmanage delete --name=AUTOLINK_NAME

qdmanage query --type=linkRoute

qdmanage read --name=LINK_ROUTE_NAME

qdmanage update --name=LINK_ROUTE_NAME --ATTRIBUTE=VALUE ...

qdmanage delete --name=INCOMING_LINK_ROUTE_NAME
qdmanage delete --name=OUTGOING_LINK_ROUTE_NAME

qdmanage query --type=log

qdmanage read --type=log --name=log/LOGGING_MODULE_NAME

qdmanage update --type=log --name=log/DEFAULT enable=LOGGING_LEVEL includeTimestamp=yes ATTRIBUTE=VALUE

qdmanage update --type=log --name=log/LOGGING_MODULE_NAME enable=LOGGING_LEVEL ATTRIBUTE=VALUE ...

qdmanage update --type=log --name=log/LOGGING_MODULE_NAME ATTRIBUTE=VALUE ...

You should upgrade AMQ Interconnect to the latest version to ensure that you have the latest enhancements and fixes. The upgrade process involves installing the new AMQ Interconnect packages and restarting your routers.

You can use these instructions to upgrade AMQ Interconnect to a new minor release or maintenance release.

Offering path redundancy means designing the network topology in a way that even when one or more routers go down or even connections between them, each destination is always reachable following alternate paths through the routers that are still part of the network.

Consider the following simple scenario :

Figure 7.1. Path Redundancy Enabled Topology

The "Router.A" configuration is something like following.

router {
    mode: interior
    id: Router.A
}

listener {
    host: 0.0.0.0
    port: 6000
    authenticatePeer: no
}

connector {
    name: INTER_ROUTER_B
    addr: 127.0.0.1
    port: 5001
    role: inter-router
}

connector {
    name: INTER_ROUTER_C
    addr: 127.0.0.1
    port: 5002
    role: inter-router
}

There is only one listener in order to accept client connections and two connector entities for connecting to the other two routers.

The "Router.B" configuration is the following.

router {
    mode: interior
    id: Router.B
}

listener {
    addr: 0.0.0.0
    port: 5001
    authenticatePeer: no
    role: inter-router
}

listener {
    host: 0.0.0.0
    port: 6001
    authenticatePeer: no
}

It has two listener entities in order to listen for connections from clients and from other routers in the network (in this case from the "Router.A" and "Router.C").

Finally, quite similar is the "Router.C" configuration.

router {
    mode: interior
    id: Router.C
}

listener {
    addr: 0.0.0.0
    port: 5002
    authenticatePeer: no
    role: inter-router
}

listener {
    host: 0.0.0.0
    port: 6002
    authenticatePeer: no
}

connector {
    name: INTER_ROUTER_B
    addr: 127.0.0.1
    port: 5001
    role: inter-router
}

It has two listener entities in order to listen for connections from clients and from other routers in the network (in this case from the "Router.A") and finally it has a connector (for connecting to the "Router.B")

Consider a sender client connected to "Router.A" and attached to my_address address which start to send messages (that is, 10 messages) and a receiver client connected to the "Router.B" and attached to the same address.

Starting the receiver, it waits for messages with no output on the console.

$ sudo python simple_recv.py -a localhost:6001/my_queue -m 10

Starting the sender, all the messages flow through "Router.A" and "Router.B" reaching the receiver; at this point the messages are all confirmed at sender side.

$ sudo python simple_send.py -a localhost:6001/my_queue -m 10
all messages confirmed

At same time, the receivers shows the messages received through the "Router.B".

{u'sequence': 1L}
{u'sequence': 2L}
{u'sequence': 3L}
{u'sequence': 4L}
{u'sequence': 5L}
{u'sequence': 6L}
{u'sequence': 7L}
{u'sequence': 8L}
{u'sequence': 9L}
{u'sequence': 10L}

The path redundancy is provided by the other available path through the "Router.A", "Router.C" and then "Router.B". It means that if the connection between "Router.A" and "Router.B" goes down, the alternative path is used to reach the receiver.

Now, consider a fault on the "Router.B"; the receiver is not reachable anymore on that path but it can connect to the "Router.C" in order to continue to receive messages from the sender which does not know what’s happened and it can continue to send messages to the "Router.A" in order to reach the receiver.

Figure 7.2. Path Redundancy after Router Failure

The receiver is still reachable in order to get messages from the sender as displayed in the console output.

$ sudo python simple_recv.py -a localhost:6002/my_queue -m 10
{u'sequence': 1L}
{u'sequence': 2L}
{u'sequence': 3L}
{u'sequence': 4L}
{u'sequence': 5L}
{u'sequence': 6L}
{u'sequence': 7L}
{u'sequence': 8L}
{u'sequence': 9L}
{u'sequence': 10L}

In order to have temporal decoupling in a solution based on AMQ Interconnect, adding one or more brokers is a must for its "store and forward" feature. Choosing the right topology, it is possible to have a solution which offers reliability with both path redundancy and permanent storing for messages.

Consider the following simple scenario :

Figure 7.3. Path Redundancy and Temporal Decoupling Enabled Topology

The receiver client can be offline when the sender starts to send messages because they’ll be stored into the queue permanently; coming back online, the receiver can get messages from the queue itself without message loss.

The "Router.A" configuration is something like following.

router {
    mode: interior
    id: Router.A
}

listener {
    host: 0.0.0.0
    port: 6000
    authenticatePeer: no
}

connector {
    name: INTER_ROUTER_B
    addr: 127.0.0.1
    port: 5001
    role: inter-router
}

connector {
    name: INTER_ROUTER_C
    addr: 127.0.0.1
    port: 5002
    role: inter-router
}

address {
    prefix: my_queue
    waypoint: yes
}

It has a listener for accepting incoming connections from clients and two connector entities in order to connect to the other routers. The queue named my_queue on the broker is exposed by a waypoint.

The "Router.B" configuration is the following.

router {
    mode: interior
    id: Router.B
}

listener {
    addr: 0.0.0.0
    port: 5001
    authenticatePeer: no
    role: inter-router
}

listener {
    host: 0.0.0.0
    port: 6001
    authenticatePeer: no
}

connector {
    name: BROKER
    addr: 127.0.0.1
    port: 5672
    role: route-container
}

address {
    prefix: my_queue
    waypoint: yes
}

autoLink {
    address: my_queue
    connection: BROKER
    direction: in
}

autoLink {
    address: my_queue
    connection: BROKER
    direction: out
}

It can accept incoming connections from clients and from other routers (in this case the "Router.A") and connects to the broker. The queue named my_queue on the broker is exposed by a waypoint with the related auto-links in both directions in order to send and receive messages to/from the queue itself.

Finally, the simple "Router.C" configuration.

router {
    mode: interior
    id: Router.C
}

listener {
    addr: 0.0.0.0
    port: 5002
    authenticatePeer: no
    role: inter-router
}

listener {
    host: 0.0.0.0
    port: 6002
    authenticatePeer: no
}

It can accept incoming connections from clients and from other routers (in this case the "Router.A"). Initially there is no connection between this router and the broker.

First of all, thanks to the broker and its "store and forward" feature, the sender can connect to the "Router.A" and start to send messages even if the receiver is not online in that moment. Using the Python sample from the Qpid Proton library, the console output is like following.

$ sudo python simple_send.py -a localhost:6000/my_queue -m 10
all messages confirmed

All messages are confirmed because they reached the queue inside the broker through "Router.A" and "Router.B"; it is confirmed using the qdstat tool.

$ sudo qdstat -b localhost:6001 -a
Router Addresses
  class   addr                   phs  distrib    in-proc  local  remote  cntnr  in  out  thru  to-proc  from-proc
  =================================================================================================================
  local   $_management_internal       closest    1        0      0       0      0   0    0     0        0
  local   $displayname                closest    1        0      0       0      0   0    0     0        0
  mobile  $management            0    closest    1        0      0       0      1   0    0     1        0
  local   $management                 closest    1        0      0       0      0   0    0     0        0
  router  Router.A                    closest    0        0      1       0      0   0    6     0        6
  router  Router.C                    closest    0        0      1       0      0   0    4     0        4
  mobile  my_queue               1    balanced   0        0      0       0      0   0    0     0        0
  mobile  my_queue               0    balanced   0        1      0       0      0   10   0     0        0
  local   qdhello                     flood      1        1      0       0      0   0    0     97       117
  local   qdrouter                    flood      1        0      0       0      0   0    0     7        0
  topo    qdrouter                    flood      1        0      2       0      0   0    8     13       9
  local   qdrouter.ma                 multicast  1        0      0       0      0   0    0     2        0
  topo    qdrouter.ma                 multicast  1        0      2       0      0   0    0     0        1
  local   temp.7f2u0zv9_U6QC5e        closest    0        1      0       0      0   0    0     0        0

For the "Router.B", there are 10 messages as output (from the router to the broker) on the my_queue address.

Starting the receiver connected to the "Router.A", it gets all the available messages from the queue.

$ sudo python simple_recv.py -a localhost:6000/my_queue -m 10
{u'sequence': 1L}
{u'sequence': 2L}
{u'sequence': 3L}
{u'sequence': 4L}
{u'sequence': 5L}
{u'sequence': 6L}
{u'sequence': 7L}
{u'sequence': 8L}
{u'sequence': 9L}
{u'sequence': 10L}

Using the qdstat tool on the "Router.B" another time, the output is like following.

$ sudo qdstat -b localhost:6001 -a
Router Addresses
  class   addr                   phs  distrib    in-proc  local  remote  cntnr  in  out  thru  to-proc  from-proc
  =================================================================================================================
  local   $_management_internal       closest    1        0      0       0      0   0    0     0        0
  local   $displayname                closest    1        0      0       0      0   0    0     0        0
  mobile  $management            0    closest    1        0      0       0      2   0    0     2        0
  local   $management                 closest    1        0      0       0      0   0    0     0        0
  router  Router.A                    closest    0        0      1       0      0   0    6     0        6
  router  Router.C                    closest    0        0      1       0      0   0    4     0        4
  mobile  my_queue               1    balanced   0        0      0       0      10  0    10    0        0
  mobile  my_queue               0    balanced   0        1      0       0      0   10   0     0        0
  local   qdhello                     flood      1        1      0       0      0   0    0     156      182
  local   qdrouter                    flood      1        0      0       0      0   0    0     7        0
  topo    qdrouter                    flood      1        0      2       0      0   0    10    18       11
  local   qdrouter.ma                 multicast  1        0      0       0      0   0    0     2        0
  topo    qdrouter.ma                 multicast  1        0      2       0      0   0    0     2        1
  local   temp.Xov_ZUcyti3jjXY        closest    0        1      0       0      0   0    0     0        0

For the "Router.B", there are 10 messages as input (from the broker to the router) on the my_queue address.

Now, consider a fault on the "Router.B"; in this case the broker is not reachable but it is possible to set up path redundancy through the "Router.C".

Figure 7.4. Path Redundancy and Temporal Decoupling after Router Failure

Using the qdmanage tool, it is possible to configure the waypoint on my_queue address, the related auto-links in both directions and finally the connector instance in order to enable the connection to the broker.

$ sudo qdmanage -b localhost:6002 create --stdin
[
{ "type":"connector", "name":"BROKER", "port":5672, "role":"route-container" },
{ "type":"address", "prefix":"my_queue", "waypoint":"yes" },
{ "type":"autoLink", "address":"my_queue", "connection":"BROKER", "direction":"in" },
{ "type":"autoLink", "address":"my_queue", "connection":"BROKER", "direction":"out" }
]
[
  {
    "verifyHostname": true,
    "stripAnnotations": "both",
    "name": "BROKER",
    "allowRedirect": true,
    "idleTimeoutSeconds": 16,
    "maxFrameSize": 65536,
    "host": "127.0.0.1",
    "cost": 1,
    "role": "route-container",
    "maxSessions": 32768,
    "type": "org.apache.qpid.dispatch.connector",
    "port": "5672",
    "identity": "connector/127.0.0.1:5672:BROKER",
    "addr": "127.0.0.1"
  },
  {
    "name": null,
    "prefix": "my_queue",
    "ingressPhase": 0,
    "waypoint": false,
    "distribution": "balanced",
    "type": "org.apache.qpid.dispatch.router.config.address",
    "identity": "7",
    "egressPhase": 0
  },
  {
    "address": "my_queue",
    "name": null,
    "linkRef": null,
    "type": "org.apache.qpid.dispatch.router.config.autoLink",
    "operStatus": "inactive",
    "connection": "BROKER",
    "direction": "in",
    "phase": 1,
    "lastError": null,
    "externalAddress": null,
    "identity": "8",
    "containerId": null
  },
  {
    "address": "my_queue",
    "name": null,
    "linkRef": null,
    "type": "org.apache.qpid.dispatch.router.config.autoLink",
    "operStatus": "inactive",
    "connection": "BROKER",
    "direction": "out",
    "phase": 0,
    "lastError": null,
    "externalAddress": null,
    "identity": "9",
    "containerId": null
  }
]

The "Router.C" configuration changes in the same way as "Router.B". It can accept incoming connections from clients and from other routers (in this case the "Router.A") and connects to the broker. The queue named my_queue on the broker is exposed by a waypoint with the related auto-links in both directions in order to send and receive messages to/from the queue itself.

At this point, the sender can connect to the "Router.A" for sending messages to the queue in the broker thanks to the "Router.C".

$ sudo python simple_send.py -a localhost:6000/my_queue -m 10
all messages confirmed

All messages are confirmed because they reached the queue inside the broker through "Router.A" and "Router.C"; it is confirmed using the qdstat tool.

$ sudo qdstat -b localhost:6002 -a
Router Addresses
  class   addr                   phs  distrib    in-proc  local  remote  cntnr  in  out  thru  to-proc  from-proc
  =================================================================================================================
  local   $_management_internal       closest    1        0      0       0      0   0    0     1        1
  local   $displayname                closest    1        0      0       0      0   0    0     0        0
  mobile  $management            0    closest    1        0      0       0      5   0    0     5        0
  local   $management                 closest    1        0      0       0      0   0    0     0        0
  router  Router.A                    closest    0        0      1       0      0   0    5     0        5
  mobile  my_queue               0    balanced   0        1      0       0      0   10   0     0        0
  mobile  my_queue               1    balanced   0        0      0       0      0   0    0     0        0
  local   qdhello                     flood      1        1      0       0      0   0    0     665      647
  local   qdrouter                    flood      1        0      0       0      0   0    0     8        0
  topo    qdrouter                    flood      1        0      1       0      0   0    31    52       32
  local   qdrouter.ma                 multicast  1        0      0       0      0   0    0     1        0
  topo    qdrouter.ma                 multicast  1        0      1       0      0   0    1     2        1
  local   temp.k6UMaS4P0JmtSlL        closest    0        1      0       0      0   0    0     0        0

For the "Router.C", there are 10 messages as output (from the router to the broker) on the my_queue address.

Starting the receiver connected to the "Router.A", it gets all the available messages from the queue.

$ sudo python simple_recv.py -a localhost:6000/my_queue -m 10
{u'sequence': 1L}
{u'sequence': 2L}
{u'sequence': 3L}
{u'sequence': 4L}
{u'sequence': 5L}
{u'sequence': 6L}
{u'sequence': 7L}
{u'sequence': 8L}
{u'sequence': 9L}
{u'sequence': 10L}

Using the qdstat tool on the "Router.C" another time, the output is like following.

$ sudo qdstat -b localhost:6002 -a
Router Addresses
  class   addr                   phs  distrib    in-proc  local  remote  cntnr  in  out  thru  to-proc  from-proc
  =================================================================================================================
  local   $_management_internal       closest    1        0      0       0      0   0    0     1        1
  local   $displayname                closest    1        0      0       0      0   0    0     0        0
  mobile  $management            0    closest    1        0      0       0      6   0    0     6        0
  local   $management                 closest    1        0      0       0      0   0    0     0        0
  router  Router.A                    closest    0        0      1       0      0   0    5     0        5
  mobile  my_queue               0    balanced   0        1      0       0      0   10   0     0        0
  mobile  my_queue               1    balanced   0        0      0       0      10  0    10    0        0
  local   qdhello                     flood      1        1      0       0      0   0    0     746      726
  local   qdrouter                    flood      1        0      0       0      0   0    0     8        0
  topo    qdrouter                    flood      1        0      1       0      0   0    34    55       35
  local   qdrouter.ma                 multicast  1        0      0       0      0   0    0     1        0
  topo    qdrouter.ma                 multicast  1        0      1       0      0   0    1     4        1
  local   temp.Hso3moy3l+Sn+Fy        closest    0        1      0       0      0   0    0     0        0

For the "Router.C", there are 10 messages as input (from the broker to the router) on the my_queue address.

Every broker has limits in terms of queue size but in order to overcome this problem, one possible solution is "sharding" queues : in that way a single queue is divided in more "shards" (chunks) each on a different broker. It means that such solution needs more than one broker instance in order to host a shard on each of them. Of course, a sender connected to one of these brokers can send messages to the shard hosted only on that broker. At same time, a receiver connected to a broker can get messages from the shard that is hosted on that broker and can not see available messages in the shards hosted on the other brokers, even if they are all parts of the same queue.

The big problem in this scenario, designed only with brokers, is that a receiver can be stucked on an empty shard without reading any messages while the shards on the other brokers have messages to deliver. it is a real problem because the receiver is interested in receiving messages from the whole queue and it does not take care if it is shared or not. Because of this problem, the receiver sees the queue as empty even if it is not so true due to the sharding and the messages available on the other shards.

The above problem can be solved adding a AMQ Interconnect instance in the network in front of the brokers and leverage on its waypoint feature with related auto-links.

Consider the following simple scenario :

Figure 7.5. Sharded Queue Enabled Topology

With such solution and connecting to the "Router.A", sender and receiver do not know anything about sharding; they want send and receive messages to/from the whole queue that is the only thing they are aware of. They are both connected to the router and see only one address (related to the queue).

The "Router.A" configuration is something like following.

router {
    mode: standalone
    id: Router.A
}

listener {
    host: 0.0.0.0
    port: 6000
    authenticatePeer: no
}

connector {
    name: BROKER1
    addr: 127.0.0.1
    port: 5672
    role: route-container
}

connector {
    name: BROKER2
    addr: 127.0.0.1
    port: 5673
    role: route-container
}

address {
    prefix: my_queue
    waypoint: yes
}

autoLink {
    address: my_queue
    connection: BROKER1
    direction: in
}

autoLink {
    address: my_queue
    connection: BROKER1
    direction: out
}

autoLink {
    address: my_queue
    connection: BROKER2
    direction: in
}

autoLink {
    address: my_queue
    connection: BROKER2
    direction: out
}

The router has a listener for incoming connection from clients and two connector instances in order to connect to both brokers. The whole queue is named my_queue hosted in terms of shards on both brokers and the router is configured with a waypoint for that address. Finally, there are two auto-links in both directions for that queue on both brokers.

Using the Python sample from the Qpid Proton library, the sender can connect to the "Router.A" and start to send messages to the queue; the console output is like following.

$ sudo python simple_send.py -a localhost:6000/my_queue -m 10
all messages confirmed

All messages are confirmed because they reached the queue and, thanks to the default balanced distribution on the address, the messages are delivered to both shards on the brokers (5 messages per shard). Using the qdstat tool on the router, the distribution is clear.

$ sudo qdstat -b localhost:6000 -l
Router Links
  type      dir  conn id  id  peer  class   addr                  phs  cap  undel  unsettled  deliveries  admin    oper
  =======================================================================================================================
  endpoint  in   1        6         mobile  my_queue              1    250  0      0          0           enabled  up
  endpoint  out  1        7         mobile  my_queue              0    250  0      0          5           enabled  up
  endpoint  in   2        8         mobile  my_queue              1    250  0      0          0           enabled  up
  endpoint  out  2        9         mobile  my_queue              0    250  0      0          5           enabled  up
  endpoint  in   8        19        mobile  $management           0    250  0      0          1           enabled  up
  endpoint  out  8        20        local   temp.qCGHruCa4UIvYrS       250  0      0          0           enabled  up

There are the out links (from router to brokers) for the my_queue address (id values 7 and 9) which have each 5 deliveries. It shows messages distributed across brokers and related shards for the queue; it is confirmed by the different connections they are tied (conn id values 1 and 2).

Starting the receiver connected to the "Router.A", it gets all the available messages from the queue.

$ sudo python simple_recv.py -a localhost:6000/my_queue -m 10
{u'sequence': 1L}
{u'sequence': 2L}
{u'sequence': 3L}
{u'sequence': 4L}
{u'sequence': 5L}
{u'sequence': 6L}
{u'sequence': 7L}
{u'sequence': 8L}
{u'sequence': 9L}
{u'sequence': 10L}

As for the sender, they are received through both the brokers and related shards. it is confirmed using the qdstat tool.

$ sudo qdstat -b localhost:6000 -l
Router Links
  type      dir  conn id  id  peer  class   addr                  phs  cap  undel  unsettled  deliveries  admin    oper
  =======================================================================================================================
  endpoint  in   1        6         mobile  my_queue              1    250  0      0          5           enabled  up
  endpoint  out  1        7         mobile  my_queue              0    250  0      0          5           enabled  up
  endpoint  in   2        8         mobile  my_queue              1    250  0      0          5           enabled  up
  endpoint  out  2        9         mobile  my_queue              0    250  0      0          5           enabled  up
  endpoint  in   10       22        mobile  $management           0    250  0      0          1           enabled  up
  endpoint  out  10       23        local   temp.HT+f3ZilGP5o3wo       250  0      0          0           enabled  up

There are the in links (from brokers to router) for the my_queue address (id values 6 and 8) which have each 5 deliveries. It shows messages distributed across brokers and related shards for the queue; it is confirmed by the different connections they are tied (conn id values 1 and 2).

One disadvantage of sharded queues is that the receiver might receive messages "out of order" even with very good performance.