istio-proxy taking almost 1 minute for scraping envoy metrics in OpenShift 4 cluster running over GCP

Issue

• The istio-proxy for default SMCP pods as well as application pods taking 60 to 75 seconds to become ready specifically for OpenShift 4 clusters running over GCP.

$ oc logs istio-ingressgateway-845695d4c6-sswzx
2023-05-04T12:44:11.468122095Z 2023-05-04T12:44:11.463542Z  info    citadelclient   Citadel client using custom root cert: var/run/secrets/istio/root-cert.pem
2023-05-04T12:44:11.505282032Z 2023-05-04T12:44:11.505197Z  info    ads All caches have been synced up in 93.044624ms, marking server ready
2023-05-04T12:44:11.512864937Z 2023-05-04T12:44:11.510446Z  info    sds SDS server for workload certificates started, listening on "./var/run/secrets/workload-spiffe-uds/socket"
2023-05-04T12:44:11.512864937Z 2023-05-04T12:44:11.510501Z  info    xdsproxy    Initializing with upstream address "istiod-basic.istio-system.svc:15012" and cluster "Kubernetes"
2023-05-04T12:44:11.513154887Z 2023-05-04T12:44:11.513054Z  info    sds Starting SDS grpc server
2023-05-04T12:44:11.787320901Z 2023-05-04T12:44:11.785732Z  info    cache   generated new workload certificate  latency=274.017118ms ttl=23h59m59.214285398s
2023-05-04T12:44:11.787320901Z 2023-05-04T12:44:11.785783Z  info    cache   Root cert has changed, start rotating root cert
2023-05-04T12:44:11.787320901Z 2023-05-04T12:44:11.785819Z  info    ads XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2023-05-04T12:44:11.787320901Z 2023-05-04T12:44:11.785897Z  info    cache   returned workload trust anchor from cache   ttl=23h59m59.214107264s
2023-05-04T12:44:17.288755414Z 2023-05-04T12:44:17.288632Z  error   failed scraping envoy metrics: error scraping http://localhost:15090/stats/prometheus: Get "http://localhost:15090/stats/prometheus": dial tcp [::1]:15090: connect: connection refused
2023-05-04T12:44:32.287879259Z 2023-05-04T12:44:32.287812Z  error   failed scraping envoy metrics: error scraping http://localhost:15090/stats/prometheus: Get "http://localhost:15090/stats/prometheus": dial tcp [::1]:15090: connect: connection refused
2023-05-04T12:44:47.287925888Z 2023-05-04T12:44:47.287847Z  error   failed scraping envoy metrics: error scraping http://localhost:15090/stats/prometheus: Get "http://localhost:15090/stats/prometheus": dial tcp [::1]:15090: connect: connection refused
2023-05-04T12:45:02.288080294Z 2023-05-04T12:45:02.287575Z  error   failed scraping envoy metrics: error scraping http://localhost:15090/stats/prometheus: Get "http://localhost:15090/stats/prometheus": dial tcp [::1]:15090: connect: connection refused
2023-05-04T12:45:17.287165311Z 2023-05-04T12:45:17.287105Z  error   failed scraping envoy metrics: error scraping http://localhost:15090/stats/prometheus: Get "http://localhost:15090/stats/prometheus": dial tcp [::1]:15090: connect: connection refused
2023-05-04T12:45:32.288012629Z 2023-05-04T12:45:32.287252Z  error   failed scraping envoy metrics: error scraping http://localhost:15090/stats/prometheus: Get "http://localhost:15090/stats/prometheus": dial tcp [::1]:15090: connect: connection refused
2023-05-04T12:45:33.747686323Z 2023-05-04T12:45:33.747616Z  warn    Error fetching GCP zone: Get "http://169.254.169.254/computeMetadata/v1/instance/zone": dial tcp 169.254.169.254:80: connect: connection refused
2023-05-04T12:45:41.491612103Z 2023-05-04T12:45:41.491517Z  warn    Error fetching GCP zone: Get "http://169.254.169.254/computeMetadata/v1/instance/zone": dial tcp 169.254.169.254:80: connect: connection refused
2023-05-04T12:45:41.589819816Z 2023-05-04T12:45:41.589744Z  info    xdsproxy    connected to upstream XDS server: istiod-basic.istio-system.svc:15012
2023-05-04T12:45:41.627454436Z 2023-05-04T12:45:41.627051Z  info    ads ADS: new connection for node:istio-ingressgateway-845695d4c6-sswzx.istio-system-1
2023-05-04T12:45:41.627454436Z 2023-05-04T12:45:41.627149Z  info    cache   returned workload trust anchor from cache   ttl=23h58m29.372857934s
2023-05-04T12:45:41.627621704Z 2023-05-04T12:45:41.627528Z  info    ads SDS: PUSH request for node:istio-ingressgateway-845695d4c6-sswzx.istio-system resources:1 size:1.1kB resource:ROOTCA
2023-05-04T12:45:41.667451381Z 2023-05-04T12:45:41.667360Z  info    ads ADS: new connection for node:istio-ingressgateway-845695d4c6-sswzx.istio-system-2
2023-05-04T12:45:41.667570020Z 2023-05-04T12:45:41.667541Z  info    cache   returned workload certificate from cache    ttl=23h58m29.332468792s
2023-05-04T12:45:41.667770184Z 2023-05-04T12:45:41.667734Z  info    ads SDS: PUSH request for node:istio-ingressgateway-845695d4c6-sswzx.istio-system resources:1 size:4.0kB resource:default