Loki doesn't trigger an alert when not able to write the chunks/tables to the local filesystem
Issue
-
The Loki ingester is not able to write the tables and chunks to the local filesystem as an input/output error exists writing to the filesystem, but not an alert is triggered for Loki
2024-07-15T07:01:18.141369178Z level=error ts=2024-07-15T07:01:18.050447873Z caller=table_manager.go:143 index-store=boltdb-shipper-2024-04-24 msg="failed to upload table" table=index_19907 err="open /tmp/loki/index/index_19907/1720033200.temp: input/output error" 2024-07-15T07:01:30.641416067Z level=error ts=2024-07-15T07:01:30.604133647Z caller=wlog.go:243 msg="Failed to calculate size of \"wal\" dir" err="lstat /tmp/wal: input/output error" 2024-07-15T07:01:44.541289292Z level=error ts=2024-07-15T07:01:44.512366666Z caller=wlog.go:243 msg="Failed to calculate size of \"wal\" dir" err="lstat /tmp/wal: input/output error" 2024-07-15T07:01:45.241215638Z level=error ts=2024-07-15T07:01:45.14634612Z caller=flush.go:143 org_id=infrastructure msg="failed to flush" err="failed to flush chunks: store put chunk: open /tmp/loki/index/index_19907/1721026800: input/output error, num_chunks: 1, labels: {kubernetes_container_name=\"ovn-controller\", kubernetes_host=\"xxxxxxxxxxxxxxxxxxxxx-infra-e820000004001a-az2-s8qk7\", kubernetes_namespace_name=\"openshift-ovn-kubernetes\", kubernetes_pod_name=\"ovnkube-node-wskq9\", log_type=\"infrastructure\"}"
Environment
- Red Hat OpenShift Container Platform (RHOCP)
- 4
- Red Hat OpenShift Logging (RHOL)
- 5.8
- 5.9
- 6
- Loki
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.