Symmetric Routing Configuration for MetalLB LoadBalancer IPs Using Secondary Interface on OpenShift

Issue

• MetalLB advertises LB type IP from the secondary interface, the requirement is to configure the OCP nodes to receive application traffic on the MetalLB provided LB IP via secondary interface and send response via the secondary interface.

• When RP_filter is enabled on the system in strict mode(net.ipv4.conf.default.rp_filter = 1 and net.ipv4.conf.all.rp_filter = 1 ) , it verifies whether the source of a received packet is reachable via the same interface it arrived on. If not, the packet is dropped. If it is disabled or enabled as lose mode (net.ipv4.conf.all.rp_filter = 2 and net.ipv4.conf.default.rp_filter = 2), system tries to respond to the source via the default gateway using the default route or through another gateway/route via a different interface as per the routing configuration. On OCP nodes rp_filter is disabled by default.

Example Lab/Requirement

• MetalLB serves IP address from the ip-addresspool-90 for the test-vlan-mb namespace.

  $ oc get ipaddresspools.metallb.io ip-addresspool-90 -o yaml
  
  apiVersion: metallb.io/v1beta1
  kind: IPAddressPool
  metadata:
    name: ip-addresspool-90
    namespace: metallb-system
    resourceVersion: "265512386"
    uid: a146b8be-70ae-41de-a7d0-f876ad4dc67c
  spec:
    addresses:
    - 192.168.90.60-192.168.90.70
    autoAssign: true
    avoidBuggyIPs: true
    serviceAllocation:
      namespaces:
      - test-vlan-mb
  

• MetalLB announces the LB IP from the bond0.3208 vlan interface from worker-1 node.

  $ oc get l2advertisements.metallb.io l2-adv-90 -o yaml
  
  apiVersion: metallb.io/v1beta1
  kind: L2Advertisement
  metadata:
    name: l2-adv-90
    namespace: metallb-system
  spec:
    interfaces:
    - bond0.3208
    ipAddressPools:
    - ip-addresspool-90
    nodeSelectors:
    - matchLabels:
        kubernetes.io/hostname: worker-1
  

• Requirement is to configure the OCP node worker-1 to receive application traffic via bond0.3208 vlan interface and send response via the same interface.

Additional LAB details

MetalLB  IP: 192.168.90.60
SVC IP of MetalLB IP : 172.30.114.10 (externalTrafficPolicy: Cluster)
Primary Interface: enp1s0(br-ex replicates from enp1s0 during boot)
Secondary VLAN Interface: bond0.3208
SVC subnet: 172.30.0.0/16 
SVC Masquerade GW: 169.254.169.4

$ oc get svc -n test-vlan-mb 
NAME   TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
web1   LoadBalancer   172.30.114.10   192.168.90.60   8080:30102/TCP   17d

DNAT Table for the MetalLB IP on worker-1. This is automatically added by MetalLB for each LB IP.

[root@worker-1 ~]# iptables -L -n -t nat | grep 192.168.90.60
DNAT       6    --  0.0.0.0/0            192.168.90.60        tcp dpt:8080 to:172.30.114.10:8080

Prerequisites

• Enable Local gateway mode(route via host) and IP forwarding in the cluster network operator.