Prometheus scrape fails with "metric name [..] does not support exemplars" in RHOCP 4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.11
    • 4.12
    • 4.13
    • 4.14
  • Prometheus

Issue

  • In the Console, a metrics targets in Openshift show that their status is Down
  • Metrics are missing

  • The error seen in the logs is the following:

    Scrape failed
metric name http_server_requests_seconds_count does not support exemplars`

Resolution

There are changes on how exemplars are handled included in Prometheus version 2.43 or later. Red Hat investigated this issue in bug report:

RHOL Release Bug Fixed version Errata
4.14 OCPBUGS-26421 4.14.9 RHSA-2024:0204
4.13 OCPBUGS-26423 4.13.31 RHSA-2024:0484

If this issue still occurs in the environment after updating, open a new support case case in the Red Hat Customer Portal referring to this solution.

Workaround

As a workaround, change the metric type to that of a Counter or Histogram bucket or return Prometheus metrics in the text/plain format, which is not affected by this issue.

Root Cause

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Prior to Prometheus version 2.43 exemplars are only supported on Counters and Histogram buckets.

Diagnostic Steps

  • check the Targetdown alert, The following error will be present:

    Scrape failed: Metrics name $METRIC_NAME_count does not support exemplars

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments