qpid commands don't work on long running queries

Latest response

Hello all,
I'm new to qpid and messaging. I hope you could give me some clue how to troubleshoot this issue.

We have a federated environment with other enterprises and we run Red Hat Enterprise MRG Messaging 2.5 for Red Hat Enterprise Linux 6.

With -m tag, qpid-stat works for me. All other tags like -c|-e|-q which usually takes longer to process, all hang and doesn't return any result.
This works:
qpid-stat -b xxx.xxx.xxx.xxx:10000 -m

This does not:
qpid-stat -b xxx.xxx.xxx.xxx:10000 -q

Tcpdump shows that on the network level, it's all working. Qpid was just waiting for the response from broker.

There's no ACL configured currently on the other broker and allow anonymous authentication.

Thank you.
HH

Responses

Hello, I guess you have a huge number of management objects (like queues, connections, sessions or exchanges).

qpid-stat raises the same type of QMF method, with the only difference in querying different QMF class (memory / queue / connection / ..). So the basic call flow within qpid broker is the same for all qpid-stat [-c|-q|-m].

The broker however prints memory and broker (-g)stats immediately, while it needs to traverse through all management objects to select those of proper type first. Then it "just" sends the filtered objects (what can take some time as well, if the list is huge). See https://github.com/apache/qpid-cpp/blob/master/src/qpid/management/ManagementAgent.cpp#L1864-#L1886 for relevant code.

I guess qpid-stat -g would finish soon and it will provide you information about number of connections, sessions, exchanges and queues. That can give you hint if/where is some excessive number of objects.

An alternative but not probable explanation could be some networking issue/config preventing to send large packets with huge list of objects (i.e. MTU set too low somewhere on the path from the broker to the qpid-stat client).

Thanks Pavel.

I'm really leaning towards the MTU issue you raised. Unfortunately my network guy is on vacation. Is there away I can test with qpid to determine whether this is MTU?

I have other already federated brokers which has their own network tunnels, i can use those networks as a baseline/compare tests.

I tested the MTU limit on the problematic broker using ping -s and figured that its limit is 1476.

Let enable qpidd debugs either by https://access.redhat.com/solutions/67020 (without a need to restart the broker) or by adding to its qpidd.conf:

log-enable=debug+:Management
log-enable=notice+
log-to-file=/path/to/qpidd.log

and restart the service.

Once you trigger qpid-stat request, the broker will log something like:

2017-04-18 15:00:01 [Management] debug RECV GetQuery (v2): map={_schema_id:{_class_name:queue}, _what:OBJECT} seq=1

and once the response is sent, it logs:

2017-04-18 15:00:01 [Management] debug SENT QueryResponse (query by schema_id) to=qmf.default.topic/direct.5b736690-922a-4362-9732-5d3641ce4ed4 len=1622

So let correlate invoking qpid-stat and (not) getting response with presence of these logs. If the 2nd log will be present, the broker sent the response but the client hasnt received it - some networking problem is expected (and the len=.. suggests the length of the response (that should be expected to split among more IP frames if necessary, though)).

If the 2nd log will be missing (but be aware, the pitty is there is no correlation among the 2 logs), the broker is stuck in generating the response - then it is worth taking coredump (gcore <pid-of-qpidd-broker>) and attach it to a new support case opened.

hello Pavel, Thank you for your help. The problem is solved. After mentioning the MTU concern to my network guy and showed him the trace you suggested to put on above thread, he updated the setting "adjust-mss" on the router and it works now. Thanks again. Not sure how redhat works, but this ticket can be closed. Cheers, HH

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.