Warnings in rabbitmq log "rabbit_sysmon_handler busy_dist_port"

Solution In Progress - Updated -

Issue

  • In our Openstack PROD environment we are getting constant warnings "rabbit_sysmon_handler busy_dist_port" (some capture of log file below) on all 3 controller nodes.
    We tried to change rabbitmq configuration by adding parameter RABBITMQ_DISTRIBUTION_BUFFER_SIZE=192000 into rabbitmq-env.conf on 3 rabbitmq instances, however looks like this doesn't help.

  • Example:

2022-06-17 12:18:59.852 [warning] <0.1152.0> rabbit_sysmon_handler busy_dist_port <0.4622.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.37>,unknown}
2022-06-17 12:24:51.229 [warning] <0.1152.0> rabbit_sysmon_handler busy_dist_port <0.15073.5> [{initial_call,{rabbit_channel,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.37>,unknown}
2022-06-17 12:29:33.351 [warning] <0.1152.0> rabbit_sysmon_handler busy_dist_port <0.2408.5> [{initial_call,{rabbit_channel,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.37>,unknown}
2022-06-17 12:32:00.492 [warning] <0.1152.0> rabbit_sysmon_handler busy_dist_port <0.2408.5> [{initial_call,{rabbit_channel,init,1}},{gen_server2,drain,1},{message_queue_len,0}] {#Port<0.37>,unknown}
2022-06-17 12:36:21.968 [warning] <0.1152.0> rabbit_sysmon_handler busy_dist_port <0.11892.5> [{initial_call,{rabbit_channel,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.37>,unknown}
2022-06-17 12:37:16.989 [warning] <0.1152.0> rabbit_sysmon_handler busy_dist_port <0.2228.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.9726>,unknown}
2022-06-17 12:37:17.964 [warning] <0.1152.0> rabbit_sysmon_handler busy_dist_port <0.2228.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.9726>,unknown}
2022-06-17 12:40:42.425 [warning] <0.1152.0> rabbit_sysmon_handler busy_dist_port <0.6409.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.37>,unknown}
2022-06-17 12:40:43.388 [warning] <0.1152.0> rabbit_sysmon_handler busy_dist_port <0.2408.5> [{initial_call,{rabbit_channel,init,1}},{erlang,bif_return_trap,2},{message_queue_len,0}] {#Port<0.37>,unknown}
2022-06-17 12:40:43.389 [warning] <0.1152.0> rabbit_sysmon_handler busy_dist_port <0.2408.5> [{initial_call,{rabbit_channel,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.9726>,unknown}
2022-06-17 12:45:48.715 [warning] <0.1152.0> rabbit_sysmon_handler busy_dist_port <0.4778.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.9726>,unknown}
2022-06-17 12:45:49.725 [warning] <0.1152.0> rabbit_sysmon_handler busy_dist_port <0.4778.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.9726>,unknown}
2022-06-17 12:46:11.863 [warning] <0.1152.0> rabbit_sysmon_handler busy_dist_port <0.7760.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.37>,unknown}
2022-06-17 12:46:12.736 [warning] <0.1152.0> rabbit_sysmon_handler busy_dist_port <0.7760.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.37>,unknown}
  • At some point we were not able to launch new VMs, however after restarting rabbitmq this issue is gone. However we are still not sure how these warning can impact our environment further.

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content