tcp delays (seconds), high load, high memory machine
Issue
- A system has delays writing to a TCP socket
- Responding even syn/ack, lost/resent syn's to the problematic machine)
- Intermittent network traffic delays, up to several seconds
Environment
-
System is productive, possibly oracle or some other heavy duty memory consumer; however, this can even be caused by massive cache on high memory machines
-
It occurs after sustained productive load, it may not be present in testing stage, if there is no sustained load-testing (sometimes it takes days for the issue to occur)
-
the issue usually degrades further under sustained load, once it appears (becomes more frequent, longer delays, etc)
-
Delays are isolated to the TCP stack: application protocol, interface driver, offloading features make no difference (it can be reproduced over lo, with any TCP server/client)
-
note: UDP will likely suffer from the same issues, but it will be harder to observe.
-
memory usage is high, but may seem low in relative terms (50% in use only, but in absolute terms 50G)
-
Observed on Red Hat Enterprise Linux 5.
-
Red Hat Enterprise Linux 6 has transparent hugepages support, which might help in some cases (still not impossible for this to happen on rhel6)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.
