Azure Redhat OpenShift - AlertmanagerClusterFailedToSendAlerts

Solution Verified - Updated -

Environment

  • Azure Red Hat OpenShift 4 (ARO )

Issue

  • Do we need to allow traffic from the cluster to these addresses [40.100.XXX.XXX, 52.98.X.XXX] on port 587?
  • Alert receiver was configured but failed to send to Microsoft Exchange server.

Sample Errors:

From the alertmanager pod logs

<truncated>

level=error ts=2023-04-22T04:28:50.411Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=10 err="XXXXXX-paging/email[0]: notify retry canceled after 2 attempts: establish connection to server: dial tcp 52.98.X.XXX:587: i/o timeout"
level=error ts=2023-04-22T04:44:46.830Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="XXXXXX-paging/email[0]: notify retry canceled after 2 attempts: establish connection to server: dial tcp 40.100.XXX.X:587: i/o timeout"

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

  • The user has the ability to set up where alert notifications are sent using SMTP, which is explained in this documentation. After completing this step, it is the cluster-admin/admin responsibility to inquire with their network or security teams if any outbound/egress traffic needs to be allowed from their proxy/firewall to ensure proper functioning. Redhat is not responsible for this configuration since it falls outside the scope of their support, as indicated in the Support Matrix referenced here.

  • If the user plans on using Microsoft Exchange Online, they can refer to Microsoft's documentation for their worldwide endpoints here. Although the IP addresses may vary based on the region, the public IP address should match the published Microsoft CIDR range provided in the link.

Root Cause

  • Network Firewall is blocking the egress/outbound traffic.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments