MRG: qpid max connection count limit exceeded

Solution Verified - Updated -

Environment

  • Red Hat Enterprise MRG Messaging 2.0, 2.1
    • qpid-cpp RPM version until 0.14-22

Issue

  • Running qpid broker in a cluster
  • Trying to connect to qpid broker that bounces the attempt with "max connection count limit exceeded: 500 conections" log, but netstat shows much less connections established

Resolution

  • The bug is tracked in Red Hat Bugzilla and fixed in MRG 2.2 release (qpid-cpp 0.14-22)
  • There is no effective workaround for the problem, except for preventing failures in authentication

Root Cause

There is a bug in qpid broker that increases internal counter of active connections when a new connection is rejected due to an authentication failure. Attempting to connect 500 times to qpidd (i.e., using default configuration) will cause the counter to reach its default limit. Any further attempt of a connection - even with valid credentials - is then bounced

Diagnostic Steps

  • Setup qpid broker to run in a cluster (to see the problem faster allow further at most 10 connections) - add the following options to /etc/qpidd.conf:
cluster-name=test
max-connections=10
  • Start openais/corosync and qpidd services.
  • Simulate 10 authentication failures - try to log using invalid credentials and let drain program to retry:
# cd /usr/share/qpidc/examples/messaging/
# make drain
..
# ./drain test --connection-options '{sasl_mechanisms:PLAIN, username:blah, password:blah }'
2012-01-10 15:19:13 warning Broker closed connection: 320, connection-forced: Authentication failed
2012-01-10 15:19:23 warning Broker closed connection: 320, connection-forced: Authentication failed
2012-01-10 15:19:33 warning Broker closed connection: 320, connection-forced: Authentication failed
2012-01-10 15:19:43 warning Broker closed connection: 320, connection-forced: Authentication failed
2012-01-10 15:19:53 warning Broker closed connection: 320, connection-forced: Authentication failed
2012-01-10 15:20:03 warning Broker closed connection: 320, connection-forced: Authentication failed
2012-01-10 15:20:13 warning Broker closed connection: 320, connection-forced: Authentication failed
2012-01-10 15:20:23 warning Broker closed connection: 320, connection-forced: Authentication failed
2012-01-10 15:20:33 warning Broker closed connection: 320, connection-forced: Authentication failed
2012-01-10 15:20:43 warning Broker closed connection: 320, connection-forced: Authentication failed
2012-01-10 15:20:53 warning Connection [49103 127.0.0.1:5672] closed
2012-01-10 15:21:03 warning Connection [49104 127.0.0.1:5672] closed
..
#
  • See the change in log after 10 successive failures. Now qpid rejects the connections due to maximum connections reached. Double check it in qpid logs:
# tail /var/log/messages
Jan 10 15:20:53 hostname1 qpidd[23499]: 2012-01-10 15:20:53 error Client max connection count limit exceeded: 10 connection refused
Jan 10 15:21:03 hostname1 qpidd[23499]: 2012-01-10 15:21:03 error Client max connection count limit exceeded: 10 connection refused
Jan 10 15:21:13 hostname1 qpidd[23499]: 2012-01-10 15:21:13 error Client max connection count limit exceeded: 10 connection refused
  • Check that no TCP connection is really established:
# netstat -an | grep 5672
tcp        0      0 0.0.0.0:5672                0.0.0.0:*                   LISTEN     
#

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments