OpenShift service mesh operator istio basic not starting due to authentication errors

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA)
  • Red Hat OpenShift Dedicated
  • Istio Service Mesh 2.1
  • Istio Service Mesh 2.3

Issue

  • After following the installation guides for OpenShift Service Mesh operator, when going to a new project to setup an Istio Service Mesh Control Plane it keeps the istio-ingressgateway and istio-egressgateway pods pending
  • The istiod pod is having the following repeating error:
2021-11-11T16:36:47.343764Z error   ads Failed to authenticate client from 10.x.x.x:34572: Authenticator ClientCertAuthenticator: no verified chain is found; Authenticator KubeJWTAuthenticator: failed to validate the JWT from cluster "Kubernetes": the service account authentication returns an error: [invalid bearer token, token audiences ["https://example.0.myregion.example.com"] is invalid for the target audiences ["istio-ca"]]
2021-11-11T16:36:47.528181Z warn    serverca    Authentication failed for 10.x.x.x:33410: Authenticator ClientCertAuthenticator at index 0 got error: no verified chain is found. Authenticator KubeJWTAuthenticator at index 1 got error: failed to validate the JWT from cluster "Kubernetes": the service account authentication returns an error: [invalid bearer token, token audiences ["https://example.0.myregion.example.com"] is invalid for the target audiences ["istio-ca"]].
  • istio-ingressgateway, istio-egressgateway have following error logs.
2023-03-23T20:38:17.138427Z info    xdsproxy    connected to upstream XDS server: istiod-basic.istio-system.svc:15012
2023-03-23T20:38:17.142877Z warn    xdsproxy    upstream [1006] terminated with unexpected error rpc error: code = Unauthenticated desc = authentication failure
2023-03-23T20:38:17.143215Z warning envoy config    StreamAggregatedResources gRPC config stream closed: 16, authentication failure
2023-03-23T20:38:19.842195Z error   failed scraping envoy metrics: error scraping http://localhost:15090/stats/prometheus: Get "http://localhost:15090/stats/prometheus": dial tcp [::1]:15090: connect: connection refused
  • When checking the resources of that deployment ("istiod-basic" replicaset) it expects "cacerts" and "istio-kubeconfig" to exist, but they are not autogenerated when the SMCP resource is created.

Resolution

There may be an issue with the use of first party tokens, or third party tokens may be required (for example, in ROSA clusters).

Configure istiod to use ThirdParty in the ServiceMeshControlPlane as:

spec:
  security:
    identity:
      type: ThirdParty
      thirdParty:
        audience: istio-ca

Note, if audience is unspecified, istio-ca will be used by default.

If the above does not work, another a workaround is to use Istio Service Mesh 2.0. To track a solution for Istio 2.1 please open a support case with Red Hat Support.

For more information, refer to Installing Service Mesh on ROSA the Service Mesh documentation.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments