OCP 4 - egress behaviour when node reboots (e.g. during cluster-update)

Latest response

Hey folks,

our new ocp-cluster is in production since a couple of weeks and we have now performed our first update (4.7.9 -> 4.7.21). Unfortunately, we had a short service-outage while the workernodes were rebooting.
The failed service (two replicas on different nodes) needs a constant jdbc-connection to an external database, so we have already implemented an according liveness-check which already restarts the pod succesfully.
The problem is, that this connection interrupts and there is still a timeperiod between the first failed connection and the liveness initiated pod restarts.
We use projectbased egressIPs, annotated to our four worker-nodes, for outgoing traffic ( oc patch netnamespace mynamespace--type=merge -p '{"egressIPs": ["A.B.C.D"]}' )

My assumptions:
1. An egressIP can only exist on one node
2. After or while the egressIP floats from one node to another, the clusteroperator "network" is processing.
3. There is a time period while the egressIP can`t be used by the assigned namespaces

My questions:
1. Are my assumptions right?
2. What can we do to prevent this situation? We want to update our cluster without any service downtime.

Maybe the problem is on a total different place, if so, please let me know.
Thank you!

Kind regards,
Sascha

Responses

Sascha, I had/have the same questions. While I am not authoritative on this subject, I might have a little light/hope to shine.

  1. Your assumptions are solid. (those are my assumptions and match what I see in running clusters)
  2. Assumption number 3 troubled us as well. Many of our services are highly-available and need near 100% uptime including planned outages and patching.

Now the "light/hope" ... There is a rumor that load-balancing traffic from 1 namespace via 2 egress IPs (assigned to disparate nodes) could show up "soon". Imho, watch for announced features and the PRs in the GH/OCP repos.

Currently, we are pre-emptivelty moving the egress IP addresses from an yet-to-be-upgraded-node to an already-upgrade-node during an update. This reduces the number of egress outages to once most of the time, and only twice in the worst case scenario. The odds for that depend on the number of worker nodes. In my personal experience, the egress outage is 2 - 15 seconds. YMMV.

We have a Red Hat TAM for OpenShift. I highly recommend having one; they are indispensable.

If you have a TAM, contact them about this. The more requests there are for a feature, the sooner that feature becomes available. Have a great day!

Hello Brandon,

thx for your response and the TAM-hint, i am glad that i`m not alone with my problem :-). So you are moving the egressIP during the updateprocess?

e.g. (egressIP is on Worker 3) 1. Worker 1 Updates/Reboots 2. Worker 2 Updates/Reboots -> Meanwhile move egressIP from Worker 3 to 1 (manual process) 3. Worker 3 Updates/Reboots 4. Worker 4 Updates/Reboots

How do you know the order or at least which worker-nodes is rebooting first? On our last update, worker 4 rebooted before worker 3.

Thank you!