4.4. TCP_NODELAY and Small Buffer Writes

As discussed briefly in Transmission Control Protocol (TCP), by default TCP uses Nagle's algorithm to collect small outgoing packets to send all at once. This can have a detrimental effect on latency.

Using TCP_NODELAY and TCP_CORK to improve network latency

  1. Applications that require lower latency on every packet sent should be run on sockets with TCP_NODELAY enabled. It can be enabled through the setsockopt command with the sockets API:
    # int one = 1;
    
    # setsockopt(descriptor, SOL_TCP, TCP_NODELAY, &one, sizeof(one));
    
  2. For this to be used effectively, applications must avoid doing small, logically related buffer writes. Because TCP_NODELAY is enabled, these small writes will make TCP send these multiple buffers as individual packets, which can result in poor overall performance.
    If applications have several buffers that are logically related and that should be sent as one packet it could be possible to build a contiguous packet in memory and then send the logical packet to TCP, on a socket configured with TCP_NODELAY.
    Alternatively, create an I/O vector and pass it to the kernel using writev on a socket configured with TCP_NODELAY.
  3. Another option is to use TCP_CORK, which tells TCP to wait for the application to remove the cork before sending any packets. This command will cause the buffers it receives to be appended to the existing buffers. This allows applications to build a packet in kernel space, which may be required when using different libraries that provides abstractions for layers. To enable TCP_CORK, set it to a value of 1 using the setsockopt sockets API (this is known as "corking the socket"):
    # int one = 1;
    
    # setsockopt(descriptor, SOL_TCP, TCP_CORK, &one, sizeof(one));
    
  4. When the logical packet has been built in the kernel by the various components in the application, tell TCP to remove the cork. TCP will send the accumulated logical packet right away, without waiting for any further packets from the application.
    # int zero = 0;
    
    # setsockopt(descriptor, SOL_TCP, TCP_CORK, &zero, sizeof(zero));
    
Related Manual Pages
For more information, or for further reading, the following man pages are related to the information given in this section.
  • tcp(7)
  • setsockopt(3p)
  • setsockopt(2)