How Force ARO Outbound Traffic To Use Always The Same Public IP?

Solution Verified - Updated -

Environment

  • Microsoft Azure Red Hat Openshift
    • 4

Issue

  • Need to force all outbound traffic in an ARO cluster to always use the same public IP
  • Application outbound traffic is using different IPs, need to make it consistent

Resolution

Setting disableOutboundSNAT to true will force all egress to always use the IP in the load balancer outbound rule.

The steps below will force the outbound traffic to use the outbound rule 'outbound-rule-v4' by disabling outbound SNAT on all the load balancer rules, forcing the use of the default LB IP for all cluster outbound traffic.

  1. Whitelist the IP addresses referenced by the outboundRule, each IP in this list will need to be whitelisted though there should only be one named "public-lb-ip-v4". If using the preview feature for scaling outbound IPs, make sure to whitelist IPs in the following format uuid-outbound-pip-v4 in addition to "public-lb-ip-v4"
az network lb show -n <loadbalancer-name> -g <cluster-resource-group> | jq .outboundRules[].frontendIPConfigurations
  1. Check the cloud-provider-config and verify disableOutboundSNAT is set to true. If not, set it to true and don't change anything in the data section:
oc edit cm cloud-provider-config -n openshift-config

data:
  config: "{\n\t\"cloud\": \"AzurePublicCloud\",\n\t\"tenantId\": \"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\n\t\"aadClientId\":
    \"\",\n\t\"aadClientSecret\": \"\",\n\t\"aadClientCertPath\": \"\",\n\t\"aadClientCertPassword\":
    \"\",\n\t\"useManagedIdentityExtension\": false,\n\t\"userAssignedIdentityID\":
    \"\",\n\t\"subscriptionId\": \"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\n\t\"resourceGroup\":
    \"xxxxxxxxx\",\n\t\"location\": \"eastus\",\n\t\"vnetName\": \"xxxxxxx\",\n\t\"vnetResourceGroup\":
    \"shared-cluster\",\n\t\"subnetName\": \"xxxxx\",\n\t\"securityGroupName\":
    \"xxxxxxx\",\n\t\"routeTableName\": \"xxxxxx\",\n\t\"primaryAvailabilitySetName\":
    \"\",\n\t\"vmType\": \"\",\n\t\"primaryScaleSetName\": \"\",\n\t\"cloudProviderBackoff\":
    true,\n\t\"cloudProviderBackoffRetries\": 0,\n\t\"cloudProviderBackoffExponent\":
    0,\n\t\"cloudProviderBackoffDuration\": 6,\n\t\"cloudProviderBackoffJitter\":
    0,\n\t\"cloudProviderRateLimit\": false,\n\t\"cloudProviderRateLimitQPS\": 0,\n\t\"cloudProviderRateLimitBucket\":
    0,\n\t\"cloudProviderRateLimitQPSWrite\": 0,\n\t\"cloudProviderRateLimitBucketWrite\":
    0,\n\t\"useInstanceMetadata\": true,\n\t\"loadBalancerSku\": \"standard\",\n\t\"excludeMasterFromStandardLB\":
    false,\n\t\"disableOutboundSNAT\": true,\n\t\"maximumLoadBalancerRuleCount\":
    0\n}\n"
  1. After applying the change, monitor the load balancer rules for 5-10 minutes to ensure disableOutboundSnat is updated to true, you can use the below command
az network lb show  -n <loadbalancer-name> -g <cluster-resource-group> | jq .loadBalancingRules[].disableOutboundSnat

Root Cause

Before ARO-3112 was applied, the disableOutboundSNAT option was set to false by default, so the Azure lb would pick up an IP from either the outbound rule or the load balancing rules for outbound traffic. After the rollout of this fix, disableOutboundSNAT should be set to true by default. If that is not the case for you cluster, open a case with Red Hat support.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments