- OpenShift Container Platform
- Prometheus is throwing a "server returned HTTP status 500 Internal Server Error".
- This is happening on one or more nodes when attempting to access monitoring targets or endpoints that should normally be accessible.
- Looking at journalctl on one of the failing nodes, we can see errors related to metrics collection:
10 error(s) occurred: * collected metric kubelet_container_log_filesystem_used_bytes label:<name:"container" value:"dns" > label:<name:"namespace" value:"openshift-dns" > label:<name:"pod" value:"dns-default-dpvdr" > gauge:<value:12288 > was collected before with the same name and label values * collected metric kubelet_container_log_filesystem_used_bytes label:<name:"container" value:"machine-config-daemon" > label:<name:"namespace" value:"openshift-machine-config-operator" > label:<name:"pod" value:"machine-config-daemon-7clnj" > gauge:<value:12288 > was collected before with the same name and label values
- There is a bug report for 4.1.0 specifically concerning Prometheus endpoints and 500 errors. There are backports as well as upstream reports available in the bug report.
- Deploy OCP 4.1.0 on vmware/bare metal environment with monitoring enabled.
- Let the cluster run for a few days.
- Load the Prometheus page, and observe the error.
- Red Hat OpenShift Container Platform
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.