Error syncing load balancer: failed to ensure load balancer: could not find any suitable subnets for creating the ELB

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform [RHOCP]
    • 4.x
  • Red Hat OpenShift Service on AWS [ROSA]

Issue

  • While creating a LoadBalancer on an AWS EKS cluster without any public subnet it will get stuck on the pending state and won't get any external IP/DNS name for it.

Resolution

  • Add below annotation to load balancer.
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
  • Another way of doing this
 $ oc annotate svc <svc_name> -n <namespace>"service.beta.kubernetes.io/aws-load-balancer-internal"="true"
  • Or by editing yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
  name: <svc_name>
  namespace: <namespace>
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: Loadbal
  type: LoadBalancer

Root Cause

  • By default AWS EKS only attaches load balancers to public subnets.

Diagnostic Steps

  • Check the svc
$ oc get svc

NAME                                   TYPE           CLUSTER-IP       EXTERNAL-IP                                                                        PORT(S)                      AGE
router-lb-coustom-internet            LoadBalancer   172.30.x.110    <pending>                                                                          80:31292/TCP,443:32393/TCP   30s
  • Check events
$ oc get events

LAST SEEN   TYPE      REASON                   OBJECT                                              MESSAGE
47s         Normal    Scheduled                pod/router-x-coustom-internet-85c755cc79-5ftvn    Successfully assigned openshift-ingress/router-x-coustom-internet-85c755cc79-5ftvn to ip-10-201-34-222.ec2.internal
45s         Normal    AddedInterface           pod/router-x-coustom-internet-85c755cc79-5ftvn    Add eth0 [10.130.2.98/23]
45s         Normal    Pulled                   pod/router-x-coustom-internet-85c755cc79-5ftvn    Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b6b843a7744e371a4d1c6c9dd59288bfd277f63318fd2011b25fc1a2efe82439" already present on machine
45s         Normal    Created                  pod/router-x-coustom-internet-85c755cc79-5ftvn    Created container router
45s         Normal    Started                  pod/router-x-coustom-internet-85c755cc79-5ftvn    Started container router
47s         Normal    Scheduled                pod/router-x-coustom-internet-85c755cc79-xbdcg    Successfully assigned openshift-ingress/router-x-coustom-internet-85c755cc79-xbdcg to ip-10-201-35-35.ec2.internal
47s         Warning   FailedMount              pod/router-x-coustom-internet-85c755cc79-xbdcg    MountVolume.SetUp failed for volume "default-certificate" : secret "router-certs-x-coustom-internet" not found
45s         Normal    AddedInterface           pod/router-x-coustom-internet-85c755cc79-xbdcg    Add eth0 [10.x.2.209/23]
44s         Normal    Pulled                   pod/router-x-coustom-internet-85c755cc79-xbdcg    Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b6b843a7744e37c9dd59288bfd277f63318fd2011b25fc1a2efe82439" already present on machine
44s         Normal    Created                  pod/router-x-coustom-internet-85c755cc79-xbdcg    Created container router
44s         Normal    Started                  pod/router-x-coustom-internet-85c755cc79-xbdcg    Started container router
47s         Normal    Scheduled                pod/router-x-coustom-internet-85c755cc79-z52dv    Successfully assigned openshift-ingress/router-x-coustom-internet-85c755cc79-z52dv to ip-10-201-33-84.ec2.internal
47s         Warning   FailedMount              pod/router-x-coustom-internet-85c755cc79-z52dv    MountVolume.SetUp failed for volume "default-certificate" : secret "router-certs-x-coustom-internet" not found
45s         Normal    AddedInterface           pod/router-x-coustom-internet-85c755cc79-z52dv    Add eth0 [10.128.4.116/23]
45s         Normal    Pulled                   pod/router-x-coustom-internet-85c755cc79-z52dv    Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b6b843ac1c6c9dd59288bfd277f63318fd2011b25fc1a2efe82439" already present on machine
45s         Normal    Created                  pod/router-x-coustom-internet-85c755cc79-z52dv    Created container router
45s         Normal    Started                  pod/router-x-coustom-internet-85c755cc79-z52dv    Started container router
47s         Normal    SuccessfulCreate         replicaset/router-x-coustom-internet-85c755cc79   Created pod: router-x-coustom-internet-85c7x5cc79-z52dv
47s         Normal    SuccessfulCreate         replicaset/router-x-coustom-internet-85c755cc79   Created pod: router-x-coustom-internet-85c75xcc79-5ftvn
47s         Normal    SuccessfulCreate         replicaset/router-x-coustom-internet-85c755cc79   Created pod: router-x-coustom-internet-8xc755cc79-xbdcg
48s         Normal    ScalingReplicaSet        deployment/router-x-coustom-internet              Scaled up replica set router-x-coustom-internet-85c755cc79 to 3
22s         Normal    EnsuringLoadBalancer     service/router-x-coustom-internet                 Ensuring load balancer
22s         Warning   SyncLoadBalancerFailed   service/router-x-coustom-internet                 Error syncing load balancer: failed to ensure load balancer: could not find any suitable subnets for creating the ELB

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments