OTelExporterSendFailingSRE alert happens on the ARO cluster

Solution Verified - Updated -

Environment

  • Azure Red Hat OpenShift (ARO)
    • 4

Issue

  • OTelExporterSendFailingSRE alert happens on the ARO cluster

Resolution

  • This alert is ignorable. Logs can still be retrieved even if the batch process to ship to centralized logging fails, and no alerts rely on logs for SRE actions. The processes drop logs more aggressively to ensure gateway health because we have other capabilities to retrieve them, and it does not result in service degradation. We'll review the current gateway health and root cause the refusals, but the customer shouldn't be concerned about our ability to meet SLA

  • There's no service outage or degradation associated with the logs not shipping.  These are failing by design, as the intention is that the backoff happens aggressively to prevent unreasonable load on the centralized regional infrastructure. SRE uses these logs only for service observability, and we have other methods of getting these logs if and when we need them. There's no impact to customers using the service.

  • Red Hat is aware of this issue, and an internal bug has been filed for further investigation.

  • Since the alert is harmless for customer applications and does not affect SRE monitoring, you can try the steps below to silence the alert:

Step 1: Check the exact labels of the firing alerts

# Get into the alertmanager pod
oc exec -n openshift-monitoring alertmanager-main-0 -- \
  amtool alert query --alertmanager.url=http://localhost:9093 \
  alertname=OTelExporterQueueSaturatedSRE

Do the same for the other two

oc exec -n openshift-monitoring alertmanager-main-0 -- \
  amtool alert query --alertmanager.url=http://localhost:9093 \
  alertname=OTelExporterNoLogsShippedSRE

oc exec -n openshift-monitoring alertmanager-main-0 -- \
  amtool alert query --alertmanager.url=http://localhost:9093 \
  alertname=OTelExporterSendFailingSRE

This shows the full label set — you need all of these to construct a working silence.

Step 2: Create a correct silence using regex to cover all three alerts

oc exec -n openshift-monitoring alertmanager-main-0 -- \
  amtool silence add \
  --alertmanager.url=http://localhost:9093 \
  --author="your-name" \
  --comment="Silencing OTel exporter alerts until underlying issue resolved" \
  --duration=168h \
  'alertname=~"OTelExporter.*SRE"'

The =~ regex matcher covers all three alert names in one silence. Adjust --duration as needed (168h = 7 days).

Step 3: Verify the silence is matching

oc exec -n openshift-monitoring alertmanager-main-0 -- \
  amtool silence query --alertmanager.url=http://localhost:9093

Confirm alerts are now silenced (Status should show "Suppressed")

oc exec -n openshift-monitoring alertmanager-main-0 -- \
  amtool alert query --alertmanager.url=http://localhost:9093 \
  alertname=OTelExporterQueueSaturatedSRE

Root Cause

  • This issue is due to a bug which is being investigated internally.

Diagnostic Steps

  • Checked information below indicates OTelExporterSendFailingSRE alert keeps firing
$ oc prometheus alertrule | grep -iE "otel|telem"

 RULE               SEVERITY  STATE   AGE  ALERTS  ACTIVE SINCE
.......
 OTelExporterSendFailingSRE   warning  pending  xx  3    xx xx xx xx:xx UTC <-- ACTIVE

They also received below 

 $ oc logs -n openshift-azure-logging otel-exporter-worker-xxxx 2>&1 \

   | grep -iE "error|fail|timeout|unavail|refused|reset" \

   | grep -v "recombine\|source_identifier"



 [ERROR 1 — Primary] Gateway refusing data: HIGH MEMORY USAGE on Azure backend

 -------------------------------------------------------------------------------

 2026-xxx-xx Exporting failed. Will retry after interval.

  error: "rpc error: code = Unavailable

      desc = data refused due to high memory usage; data refused due to high memory usage"


 [ERROR 2] Gateway forcibly resetting established gRPC connections

 -------------------------------------------------------------------------------

 2026-xx-xx Exporting failed. Will retry after interval.

  error: "rpc error: code = Unavailable

      desc = error reading from server:

      read tcp xxxxxx->xxxxxx: read: connection reset by peer"

 [ERROR 3] Gateway completely unreachable (full outage windows)

 -------------------------------------------------------------------------------

 2026-xx-xxx [core] grpc: addrConn.createTransport failed to connect

  Addr: "xxxxxx"

  ServerName: "xxxxx.xxxxx.aro.azure.com:xxxx"

  Err: connection error: desc = "transport: Error while dialing:

     dial tcp xxxxx:xxxx: i/o timeout"

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments