Getting an PrometheusDuplicateTimestamps alert due to duplicate IP in endpoint in RHOCP4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform
    • 4

Issue

  • Getting an PrometheusDuplicateTimestamps alert due to duplicate IP in machine-config-daemon endpoint.

  • Prometheus pods logs show below error:

    025-06-30T20:38:01.259191268Z ts=2025-06-30T20:38:01.259Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/openshift-monitoring/kube-state-metrics/0 target=https://10.131.6.4:8443/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=1

  • Prometheus pod debug logs shows below error:

    2025-06-30T23:15:58.300340590Z ts=2025-06-30T23:15:58.300Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/openshift-monitoring/kube-state-metrics/0 target=https://10.131.6.4:8443/metrics msg="Duplicate sample for timestamp" series="kube_endpoint_address{namespace=\"openshift-machine-config-operator\",endpoint=\"machine-config-daemon\",ip=\"10.42.x.x\",ready=\"true\"}"

Resolution

  • To resolve this issue, Kindly restart the machine-config-daemon daemonSet pods.

    • Use the below command to force a resync of endpoints and pod IP assignments.
    $ oc -n openshift-machine-config-operator rollout restart daemonset machine-config-daemon

Diagnostic Steps

  • Look for below error message in the prometheus-k8s-0 pod logs.

    $ oc logs prometheus-k8s-0 -c prometheus | grep -i error
2025-06-30T20:38:01.259191268Z ts=2025-06-30T20:38:01.259Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/openshift-monitoring/kube-state-metrics/0 target=https://10.131.6.4:8443/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=1

  • In the prometheus-k8s-0 pod debug logs, check for the below duplicate sample message.

    $ oc logs prometheus-k8s-0 -c prometheus | grep -i error
2025-06-30T23:15:58.300340590Z ts=2025-06-30T23:15:58.300Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/openshift-monitoring/kube-state-metrics/0 target=https://10.131.6.4:8443/metrics msg="Duplicate sample for timestamp" series="kube_endpoint_address{namespace=\"openshift-machine-config-operator\",endpoint=\"machine-config-daemon\",ip=\"10.42.x.x\",ready=\"true\"}"
2025-06-30T23:16:58.317388052Z ts=2025-06-30T23:16:58.317Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/openshift-monitoring/kube-state-metrics/0 target=https://10.131.6.4:8443/metrics msg="Duplicate sample for timestamp" series="kube_endpoint_address{namespace=\"openshift-machine-config-operator\",endpoint=\"machine-config-daemon\",ip=\"10.42.x.x\",ready=\"true\"}"

  • Check the machine-config-daemon endpoint, Verify that whether IP is used for two nodes is similar.

    $ oc get ep machine-config-daemon -oyaml -n openshift-machine-config-operator
- ip: 10.42.x.x
nodeName: infra-node-1
targetRef:
  kind: Pod
  name: machine-config-daemon-1
  namespace: openshift-machine-config-operator
  uid: ae2c407d-ec01-47d2-bba6-8ec443b52f1b
- ip: 10.42.x.x
nodeName: infra-node-2
targetRef:
  kind: Pod
  name: machine-config-daemon-2
  namespace: openshift-machine-config-operator
  uid: dfb37065-7f2b-4cba-bce6-0ae06d5d2cd3

  • Verify whether the machine-config-daemon pods are reporting the same Internal IPs.

    $ oc get pods -owide | grep -i machine-config-daemon
machine-config-daemon-1                                  2/2     Running   2          13d   10.42.x.x   infra-node-2   <none>           <none>
machine-config-daemon-2                                  2/2     Running   0          7d    10.42.x.x   infra-node-1       <none>           <none>

  • Verify the InternalIP of each associated node. Ensure that the nodes have different InternalIPs, but observe that in the endpoint both nodes are showing the same IP.

    $ oc get node infra-node-1 -oyaml
addresses:
- address: 10.42.x.x
type: InternalIP
- address: infra-node-1
$ oc get node infra-node-2 -oyaml
addresses:
- address: 10.42.x.y
type: InternalIP
- address: infra-node-2

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments