Improving UDP Performance by Configuring OS UDP Buffer Limits

By default, the JGroups channels in JBoss Enterprise Application Platform use the UDP transport protocol to take advantage of IP multicast. However, one disadvantage of UDP is it does not come with the reliable delivery guarantees provided by TCP. The protocols discussed in Section 28.1.5, “Reliable Delivery Protocols” allow JGroups to guarantee delivery of UDP messages, but those protocols are implemented in Java, not at the operating system network layer. For peak performance from a UDP-based JGroups channel it is important to limit the need for JGroups to re-transmit messages by limiting UDP datagram loss.
One of the most common causes of lost UDP datagrams is an undersized receive buffer on the socket. The UDP protocol's mcast_recv_buf_size and ucast_recv_buf_size configuration attributes are used to specify the amount of receive buffer JGroups requests from the operating system, but the actual size of the buffer the operating system provides is limited by operating system-level maximums. These maximums are often very low:

Table 28.1. Default Max UDP Buffer Sizes

Operating System Default Max UDP Buffer (in bytes)
Linux 131071
Windows No known limit
Solaris 262144
FreeBSD, Darwin 262144
AIX 1048576
The command used to increase the above limits is operating system-specific. The table below shows the command required to increase the maximum buffer to 25 megabytes. In all cases, root privileges are required:

Table 28.2. Commands to Change Max UDP Buffer Sizes

Operating System Command
Linux sysctl -w net.core.rmem_max=26214400
Solaris ndd -set /dev/udp udp_max_buf 26214400
FreeBSD, Darwin sysctl -w kern.ipc.maxsockbuf=26214400
AIX no -o sb_max=8388608 (AIX will only allow 1 megabyte, 4 megabytes or 8 megabytes).