noProxy URL not available in Prometheus k8s CR after configuring remote-write

Solution Verified - Updated -

Issue

  • Unable to send metrics to remote-write endpoint after upgrading RHOCP cluster to 4.16+ version.

  • Promtheus-k8s pods stream below logs indefinitely:

    $ oc logs -c prometheus prometheus-k8s-1 -n openshift-monitoring 
...
2024-08-06T14:56:36.607159251Z ts=2024-08-06T14:56:36.607Z caller=dedupe.go:112 component=remote level=error remote_name=8dee62 url=http://example-prometheus.test.svc.cluster.local:9090/api/v1/receive msg="non-recoverable error" count=9 exemplarCount=0 err="server returned HTTP status 403 Forbidden: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd\">"
2024-08-06T14:56:36.607239708Z ts=2024-08-06T14:56:36.607Z caller=dedupe.go:112 component=remote level=error remote_name=8dee62 url=http://example-prometheus.test.svc.cluster.local:9090/api/v1/receive msg="non-recoverab

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.14+

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content