keda-metrics-apiserver failing with connection refused and invalid memory address

Issue

• We have two errors with the keda operator, we have to constantly restart the pods to temporarily fix the errors, but after a short time the error reappears.

The HPA was unable to compute the replica count: unable to get external metric 
example/p0-prometheus/$LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: 
prometheus-scaleobject-batch},MatchExpressions:[]LabelSelectorRequirement{}}: unable to fetch metrics from external metrics 
API: the server is currently unable to handle the request (get 
s0-prometheus.external.metrics.k8s.io)

• Keda apiserver pod logs show two errors:

E0902 12:15:21.526236       1 timeout.go:142] post-timeout activity - time-elapsed: 75.677933ms, GET "/apis/external.metrics.k8s.io/v1beta1/namespaces/example/s0-prometheus" result: runtime error: invalid memory address or nil pointer dereference
...
W0902 12:42:59.666838       1 logging.go:59] [core] [Channel #1 SubChannel #49] grpc: addrConn.createTransport failed to connect to {Addr: "keda-operator.example.svc.cluster.local:9666", ServerName: "keda-operator.example.svc.cluster.local:9666", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.0.0.1:9666: connect: connection refused"

• At other times, the connection succeeds:

I0830 20:51:36.122572       1 provider.go:66] keda_metrics_adapter/provider "msg"="Connection to KEDA Metrics Service gRPC server has been successfully established" "server"="keda-operator.example.svc.cluster.local:9666"