qpid broker forgot a federation route

Solution Verified - Updated -

Environment

  • Red Hat Enterprise MRG Messaging 2.3

Issue

  • configured plenty of federation routes among three qpid brokers
  • one federation route with routing key "#" seems to be properly provisioned but the brokers do not route the traffic via it

Resolution

As the root cause is unknown, it is recommended to delete and re-create the failing federation route. Following the setup in Diagnostic Steps, it means to execute commands:

qpid-route -v route del broker1:5672 broker2:5672 my.topic.exchange \#
qpid-route -v route del broker2:5672 broker1:5672 my.topic.exchange \#

sleep 10      # just to be sure the deletion has been completed

qpid-route --timeout 30 -d -v route add broker1:5672 broker2:5672 my.topic.exchange \#
qpid-route --timeout 30 -d -v route add broker2:5672 broker1:5672 my.topic.exchange \#

If the federation route will not work either, try to restart both brokers to clean some cached obsolete data.

Root Cause

Unknown at the moment. Possible causes:

  • somebody deleted by a mistake the auxiliary queue for the federation route named like ..
  • the federation topology was provisioned with some exchange route between non-existing exchanges - repetitive attempts to re-create this route might influenced the properly configured routes

Diagnostic Steps

  • federation route map shows (among others) exchange route between broker1 and broker2:
$ qpid-route route map broker1

Finding Linked Brokers:
    broker1:5672... Ok
    broker2:5672... Ok
    broker3:5672... Ok

Dynamic Routes:
  none found

Static Routes:
  broker1:5672(ex=my.topic.exchange) <= broker2:5672(ex=my.topic.exchange) key=#
  broker1:5672(ex=my.topic.exchange) <= broker2:5672(ex=my.topic.exchange) key=boo.#
  broker1:5672(ex=my.topic.exchange) <= broker2:5672(ex=my.topic.exchange) key=foo2.#

  broker1:5672(ex=my.topic.exchange) <= broker3:5672(ex=my.topic.exchange) key=#
  broker1:5672(ex=my.topic.exchange) <= broker3:5672(ex=my.topic.exchange) key=boo.#
  broker1:5672(ex=my.topic.exchange) <= broker3:5672(ex=my.topic.exchange) key=foo2.#

  broker2:5672(ex=my.topic.exchange) <= broker1:5672(ex=my.topic.exchange) key=#
  broker2:5672(ex=my.topic.exchange) <= broker1:5672(ex=my.topic.exchange) key=boo.#
  broker2:5672(ex=my.topic.exchange) <= broker1:5672(ex=my.topic.exchange) key=foo2.#

  broker2:5672(ex=my.topic.exchange) <= broker3:5672(ex=my.topic.exchange) key=#
  broker2:5672(ex=my.topic.exchange) <= broker3:5672(ex=my.topic.exchange) key=boo.#
  broker2:5672(ex=my.topic.exchange) <= broker3:5672(ex=my.topic.exchange) key=foo2.#

  broker3:5672(ex=my.topic.exchange) <= broker1:5672(ex=my.topic.exchange) key=#
  broker3:5672(ex=my.topic.exchange) <= broker1:5672(ex=my.topic.exchange) key=boo.#
  broker3:5672(ex=my.topic.exchange) <= broker1:5672(ex=my.topic.exchange) key=foo2.#

  broker3:5672(ex=my.topic.exchange) <= broker2:5672(ex=my.topic.exchange) key=#
  broker3:5672(ex=my.topic.exchange) <= broker2:5672(ex=my.topic.exchange) key=boo.#
  broker3:5672(ex=my.topic.exchange) <= broker2:5672(ex=my.topic.exchange) key=foo2.#


$
  • but there is no auxiliary queue for the bridge on either broker1 or broker2 for key "#" there, just for the keys "boo.#" and "foo2.#":
$ qpid-config queues -b broker1 | grep "qpid.bridge"
qpid.bridge_queue_qpid.tcp:broker1:5672!my.topic.exchange!my.topic.exchange!boo.#_1d288398-f5f0-4c57-a815-72ee3186fda2                     auto-del excl --argument qpid.trace.exclude=1d288398-f5f0-4c57-a815-72ee3186fda2 --argument qpid.trace.id=c0773ecb-f82c-4c78-9c01-f0c1dbc1c814
qpid.bridge_queue_qpid.tcp:broker1:5672!my.topic.exchange!my.topic.exchange!foo2.#_1d288398-f5f0-4c57-a815-72ee3186fda2                 auto-del excl --argument qpid.trace.exclude=1d288398-f5f0-4c57-a815-72ee3186fda2 --argument qpid.trace.id=c0773ecb-f82c-4c78-9c01-f0c1dbc1c814
$ qpid-config queues -b broker2 | grep "qpid.bridge"
qpid.bridge_queue_qpid.tcp:broker2:5672!my.topic.exchange!my.topic.exchange!boo.#_c0773ecb-f82c-4c78-9c01-f0c1dbc1c814                     auto-del excl --argument qpid.trace.exclude=c0773ecb-f82c-4c78-9c01-f0c1dbc1c814 --argument qpid.trace.id=1d288398-f5f0-4c57-a815-72ee3186fda2
qpid.bridge_queue_qpid.tcp:broker2:5672!my.topic.exchange!my.topic.exchange!foo2.#_c0773ecb-f82c-4c78-9c01-f0c1dbc1c814                 auto-del excl --argument qpid.trace.exclude=c0773ecb-f82c-4c78-9c01-f0c1dbc1c814 --argument qpid.trace.id=1d288398-f5f0-4c57-a815-72ee3186fda2
$

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments