2.9. Network Determinism Tips

Transmission Control Protocol (TCP)

TCP can have a large effect on latency. TCP adds latency in order to obtain efficiency, control congestion, and to ensure reliable delivery. When tuning, consider the following points:

  • Do you need ordered delivery?
  • Do you need to guard against packet loss?
    Transmitting packets more than once can cause delays.
  • If you must use TCP, consider disabling the Nagle buffering algorithm by using TCP_NODELAY on your socket. The Nagle algorithm collects small outgoing packets to send all at once, and can have a detrimental effect on latency.

Network Tuning

There are numerous tools for tuning the network. Here are some of the more useful:
Interrupt Coalescing
To reduce network traffic, packets can be collected and a single interrupt generated.
In systems that transfer large amounts of data where bandwidth use is a priority, using the default value or increasing coalesce can increase bandwidth use and lower system use. For systems requiring a rapid network response, reducing or disabling coalesce is advised.
Use the -C (--coalesce) option with the ethtool command to enable.
Congestion
Often, I/O switches can be subject to back-pressure, where network data builds up as a result of full buffers.
Use the -A (--pause) option with the ethtool command to change pause parameters and avoid network congestion.
Infiniband (IB)
Infiniband is a type of communications architecture often used to increase bandwidth and provide quality of service and failover. It can also be used to improve latency through Remote Direct Memory Access (RDMA) capabilities.
Network Protocol Statistics
Use the -s (--statistics) option with the netstat command to monitor network traffic.
See also Section 2.12, “Reduce TCP Performance Spikes” and Section 3.7, “Reducing the TCP Delayed ACK Timeout”.
Related Manual Pages

For more information, or for further reading, the following man pages are related to the information given in this section.

  • ethtool(8)
  • netstat(8)