Prometheus - error label name 'xxx' is not unique: Invalid Sample

Solution In Progress - Updated -

Environment

  • OCP v4x
  • ARO v4x
  • OSD v4x
  • ROSA v4x

Issue

  • Grafana is not showing any data
  • Prometheus reporting the following error
    • error label name 'xxx' is not unique: Invalid Sample

Resolution

  • Prometheus views label names with "underscore (_)" and "dash (-)" as the same and requires the labels to be unique.
    • One malformed pod can cause kube-state-metrics to stop exposing metrics for all other pods.
  • Any two labels that share the same name but are differentiated only by the use of a dash "-" and an underscore "_" will be treated as the same label by Prometheus.
  • You will need to review the labels and alter them so that they are not affected by this restrictions.

Root Cause

  • More information can be found here:
    • https://github.com/kubernetes/kube-state-metrics/issues/1115

Diagnostic Steps

  • Identifying the offending Prometheus labels can be accomplished through the following steps:
    • Access the Prometheus Dashboard, then locate Targets and then Unhealthy
    • Check for 'label name=' in Error column.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments