1.2.3. Messaging Broker Memory Requirements

The amount of memory required by a Broker is a function of the number and size of messages it will process simultaneously.
The size of a message is the combination of the size of the message header and the size of the message body.

Calculate message size

Note: Transactions increase the size of messages.

Procedure 1.4. Estimate message size

This method allows you to calculate message size theoretically.
  1. Default message header content (such as Java timestamp and message-id): 55 bytes
  2. Routing Key (for example: a routing key of "testQ" = 5 bytes)
  3. Java clients add:
    • content-type (for "text/plain" it is 10 bytes)
    • user-id (user name passed to SASL for authentication, number of bytes equal to string size)
  4. Application headers:
    • Application header overhead: 8 bytes
    • For any textual header properties: property_name_size + property_value_size + 4 bytes
For example, sending a message using spout such as the following: 
./run_example.sh org.apache.qpid.example.Spout -c=1 -b="guest:guest@localhost:5672" -P=property1=value1 -P=property2=value2 "testQ; {create:always}" "123456789"
sends an AMQP message with message body size 9, while the message header will be of size:
  • 55 bytes for the default size
  • 5 bytes for the routing key "testQ"
  • 10 bytes for content-type "text/plain"
  • 5 bytes for user-id "guest"
  • 8 bytes for using application headers
  • 9+6+4 bytes for the first property
  • 9+6+4 bytes for the second property
  • Total header size: 121 bytes
  • Total message size: 130 bytes

Procedure 1.5. Determine message size from logs

This method allows you to measure message sizes from a running broker.
  1. Enable trace logging by adding the following to /etc/qpid/qpidd.conf: 
    log-enable=trace+:qpid::SessionState::receiverRecord
log-enable=info+
log-to-file=/path/to/file.log
    Note that this logging will consume significant disk space, and should be turned off by removing these lines after the test is performed.
  2. (Re)start the Broker.
  3. Send a sample message pattern from a qpid client. This sample message pattern should correspond to your normal utilization, so that the message header and body average sizes match your projected real-world use case.
  4. After the message pattern is sent, grep in the logfile for log records: 
    2012-10-16 08:56:20 trace guest@QPID.2fa0df51-6131-463e-90cc-45895bea072c: recv cmd 2: header (121 bytes); properties={{MessageProperties: content-length=9; message-id=d096f253-56b9-33df-9673-61c55dcba4ae; content-type=text/plain; user-id=guest; application-headers={property1:V2:6:str16(value1),property2:V2:6:str16(value2)}; }{DeliveryProperties: priority=4; delivery-mode=2; timestamp=1350370580363; exchange=; routing-key=testQ; }}
    This example log entry contains both header size (121 bytes in this case) and message body size (9 bytes in this case, as content-length=9).

Message memory utilization on Broker

On the broker, memory is utilized to hold the message. In addition:
  • A second instance of the message header is kept - one is stored as raw bytes, the other as a map.
  • The Message object uses 600 bytes.
  • Each message is guarded by three mutexes and monitor. These require 208 bytes.
So the total calculation of memory usage for a message is:
message_body + (message_header * 2) + 808 bytes
Using an average value for message body and header size, and multiplying this figure by the sum of queue depths will give you a saturated memory load figure for the Broker.
Note: an in-memory optimization for exchanges uses one copy of a message for all subscribed queues when the message is delivered to an exchange. This means that a broker with exchanges delivering to multiple queues will use significantly lower amounts of memory in normal operation. However, if the broker is restarted and the queued messages are read from disk, they are read per-queue and the full memory impact is experienced.
Report a bug