Prometheus failing to report data due to "no space left on device"

Issue

• Web Console showing status of resources, but not metrics data, e.g. CPU/Memory usage, Pods, etc.
• Monitoring Dashboards shows "no datapoints found".

• Prometheus pods are streaming below logs:

  2022-07-07T14:55:40.065158526Z level=error ts=2022-07-07T14:55:40.065Z caller=scrape.go:1190 component="scrape manager" scrape_pool=serviceMonitor/openshift-multus/monitor-network/0 target=https://x.x.x.x:8443/metrics msg="Scrape commit failed" err="write to WAL: log samples: write /prometheus/wal/00017534: no space left on device"
  2022-07-07T14:55:40.160856275Z level=error ts=2022-07-07T14:55:40.160Z caller=scrape.go:1190 component="scrape manager" scrape_pool=serviceMonitor/openshift-monitoring/kubelet/3 target=http://x.x.x.x:9537/metrics msg="Scrape commit failed" err="write to WAL: log samples: write /prometheus/wal/00017534: no space left on device"
  2022-07-07T14:55:40.294090053Z level=error ts=2022-07-07T14:55:40.294Z caller=scrape.go:1190 component="scrape manager" scrape_pool=serviceMonitor/openshift-insights/insights-operator/0 target=https://x.x.x.x:8443/metrics msg="Scrape commit failed" err="write to WAL: log samples: write /prometheus/wal/00017534: no space left on device"
  

• oc adm top nodes returns below error:

  error: metrics not available yet
  

• Prometheus is failing to commit metrics data.