qpid broker fails to start on store recovery: with error "Cannot allocate memory"

Solution Verified - Updated -

Issue

  • after stopping qpid broker, unable to start it again
  • several particular error messages seen, most often with Cannot allocate memory:
    • Error on recovery (MessageStoreImpl.cpp:701): Dbc::get: Cannot allocate memory
    • JERR__AIO: AIO error. (io_queue_init() failed: errno=12 (Cannot allocate memory)
    • JERR_JDIR_STAT: File stat failed. (file="/var/lib/qpidd/.qpidd/qls/jrnl/pulp.agent.ea302561-219c-42a7-bd18-2e3038349769" errno=2 (No such file or directory)
    • JERR__MALLOC: Buffer memory allocation failed. (posix_memalign(): alignment=4096 size=1048576 errno=12 (Cannot allocate memory))
  • the machine has much memory that is not used
  • ulimit or kernel tunables tuned appropriatelly
  • after wiping out provisioning (under /var/lib/qpidd/ directory), broker can start (but without the stored provisioning and journals - unacceptable solution)

Environment

  • Red Hat Enterprise Linux (RHEL) 6 or 7
  • Red Hat Enterprise MRG Messaging
    • store module used - both legacy store (qpid-cpp-server-store) and linear store (qpid-cpp-server-linearstore) could be affected

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In