Chapter 29. Improving network latency using TCP_NODELAY

By default, TCP uses Nagle’s algorithm to collect small outgoing packets to send all at once. This can cause higher rates of latency.

Prerequisites

  • You have administrator privileges.

29.1. The effects of using TCP_NODELAY

Applications that require low latency on every packet sent must be run on sockets with the TCP_NODELAY option enabled. This sends buffer writes to the kernel as soon as an event occurs.

Note
For TCP_NODELAY to be effective, applications must avoid doing small, logically related buffer writes. Otherwise, these small writes cause TCP to send these multiple buffers as individual packets, resulting in poor overall performance.

If applications have several buffers that are logically related and must be sent as one packet, apply one of the following workarounds to avoid poor performance:

  • Build a contiguous packet in memory and then send the logical packet to TCP on a socket configured with TCP_NODELAY.
  • Create an I/O vector and pass it to the kernel using the writev command on a socket configured with TCP_NODELAY.
  • Use the TCP_CORK option. TCP_CORK tells TCP to wait for the application to remove the cork before sending any packets. This command causes the buffers it receives to be appended to the existing buffers. This allows applications to build a packet in kernel space, which can be required when using different libraries that provide abstractions for layers.

When a logical packet has been built in the kernel by the various components in the application, the socket should be uncorked, allowing TCP to send the accumulated logical packet immediately.

29.2. Enabling TCP_NODELAY

The TCP_NODELAY option sends buffer writes to the kernel when events occur, with no delays. Enable TCP_NODELAY using the setsockopt() function.

Procedure

  1. Add the following lines to the TCP application’s .c file. 

    int one = 1;
setsockopt(descriptor, SOL_TCP, TCP_NODELAY, &one, sizeof(one));
  2. Save the file and exit the editor.

  3. Apply one of the following workarounds to prevent poor performance.

    • Build a contiguous packet in memory and then send the logical packet to TCP on a socket configured with TCP_NODELAY.
    • Create an I/O vector and pass it to the kernel using writev on a socket configured with TCP_NODELAY.

29.3. Enabling TCP_CORK

The TCP_CORK option prevents TCP from sending any packets until the socket is "uncorked".

Procedure

  1. Add the following lines to the TCP application’s .c file. 

    int one = 1;
setsockopt(descriptor, SOL_TCP, TCP_CORK, &one, sizeof(one));
  2. Save the file and exit the editor.

  3. After the logical packet has been built in the kernel by the various components in the application, disable TCP_CORK. 

    int zero = 0;
setsockopt(descriptor, SOL_TCP, TCP_CORK, &zero, sizeof(zero));

    TCP sends the accumulated logical packet immediately, without waiting for any further packets from the application.

29.4. Additional resources

  • tcp(7) man page
  • setsockopt(3p) man page
  • setsockopt(2) man page