HAproxy routers performance seems to deteriorate over time and haproxy mapping files doesn't match

Solution In Progress - Updated -

Issue

  • We have two haproxy routers deployed with the same routes (each one of them is in a different node), when a route is published it's published almost instantly in etcd, then routes.json in each node and then haproxy.config and mapping files are reloaded as expected.
  • Over time this performance seems to deteriorate:
    • Routes aren't published that fast, it takes at least 2-5 minutes to publish routes in one haproxy, and 5-10 minutes in the other. For some reason it seems one haproxy is always refreshing its configuration taking twice the time.
    • HAproxy mapping files doesn't match. After a router restart the mapping files shows the same routes (as expected), but when the performance decreases there are routes published in one haproxy that aren't published in the other, causing we can't reach some pods when accessing through the haproxy with missing routes.
  • Workaround is restarting haproxys routers, then they work as expected.

Environment

  • OpenShift 3.0
    -OpenStack
  • Red Hat Enterprise Linux (RHEL) 7.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content