TCP tuning for high bandwidth delay product network

Latest response

We are trying to copy some files over a very high latency connection of about 325ms round-trip. Using UDP, I'm able to get throughput speeds of about 80Mb/s, but using various TCP-based methods such as SCP or http, I am only getting about 25Mb/s. In addition to our "production" environment, we have also used a network emulator in a lab environment to replicate the bandwidth and latency, and we are getting the same results. After doing some web searching, I tried doing the following things:

  • Sending system:
    Net.core.wmem_default set to 8000000 (2x BDP)
    Net.core.wmem_max set to 16000000
    Net.ipv4.tcp_wmem set to 4096 8000000 16000000
    Rebooted, verified parameters
    Ethernet interface ring buffer sizes increased from 512 default to
    4096 maximum
    increased txqueuelen to 10000

• Receiving system:
Net.core.rmem_default set to 8000000
Net.core.rmem_max set to 16000000
Net.ipv4.tcp_rmem set to 4096 8000000 16000000
Rebooted, verified parameters
Ethernet interface ring buffer set to 4096 from 512

Not only did this not fix the problem, we aren't seeing any appreciable difference in throughput. One thing we have tried is multi-threading, which gave us some moderate improvement, but I'm wondering if anyone has had success with tuning for a high latency connection, and if so, what are we missing? Thanks for your help!

TR

Responses