tcp delays (seconds), high load, high memory machine

Solution In Progress - Updated -

Issue

  • A system has delays writing to a TCP socket
  • Responding even syn/ack, lost/resent syn's to the problematic machine)
  • Intermittent network traffic delays, up to several seconds

Environment

  • System is productive, possibly oracle or some other heavy duty memory consumer; however, this can even be caused by massive cache on high memory machines

  • It occurs after sustained productive load, it may not be present in testing stage, if there is no sustained load-testing (sometimes it takes days for the issue to occur)

  • the issue usually degrades further under sustained load, once it appears (becomes more frequent, longer delays, etc)

  • Delays are isolated to the TCP stack: application protocol, interface driver, offloading features make no difference (it can be reproduced over lo, with any TCP server/client)

  • note: UDP will likely suffer from the same issues, but it will be harder to observe.

  • memory usage is high, but may seem low in relative terms (50% in use only, but in absolute terms 50G)

  • Observed on Red Hat Enterprise Linux 5.

  • Red Hat Enterprise Linux 6 has transparent hugepages support, which might help in some cases (still not impossible for this to happen on rhel6)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.