OCP4 on premise distributed across two DCs

Hey Folks,

we use OCP 4.8, hosted on vmware across two datacenters. We now want to improve the high availibility with the best possible configuration.

Our current node placement looks as followed:

Our goal is to minimize (or better eliminate) the downtime of our services in case of an datacenter outage, so we want to ensure that all required components run on both datacenters. To achieve this, we have already done the following things:

• Two f5-loadbalancers in active-active mode (one per DC) who handle all requests and forward them to the ingress-router.
• Percona MySQL-Cluster with two proxies (one per DC). All services use this dbms if required.
• Trident filebased storage for PVs on mirrored systems (one per DC). All services use this storage if required.
• EgressIPs on each worker-node.
• Additional labels, indicating on which datacenter the node is running (dc=1 on Worker 1 & 2, dc=2 on Worker 3 & 4)
• At least two independant replicas for each service. The distribution on both datacenters is guaranteed via deploymenconfiguration, based on dc-labels.
• vSphere HA, so when one datacenter is going down, the lost nodes will be rebooted on the availabe datacenter.

One homework-task we still have to do is to reconfigure the ingress-routers. The configuration is default, so we have two replicas.

Questions:

1. Is there an way to change the ingress-deployment so that it will be also guaranteed that there is on replica per dc?

2. Is it a good idea to increase ingress to four replicas (one per node)?

3. Is there anything missing to get the best possible HA for our services?

4. When DC 1 (including master-quorum) goes down, then of course, the master components (etcd and co) will be not available but all other components (ingress, egress, services...) will stay where they are. So i think the service availability, with lower resources, is given. Is my assumption correct?

Thank you!

Kind regards,
Sascha