Test required connectivity from OSD/ROSA cluster

Solution Unverified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA)
    • 4
  • Red Hat OpenShift Dedicated on AWS (OSD)
    • 4

Issue

  • The OSD/ROSA cluster is in Limited Support status with a Network misconfiguration Service Log:

    Cluster is in Limited Support status and Red Hat requires you to take action. SRE has observed that there have been changes made to network configuration which impact normal working of the cluster: Route table rtb-xxxxx and rtb-yyyyy have been modified. Please revert changes.

    Your cluster requires you to take action. SRE has observed that there have been changes made to the network configuration which impacts normal working of the cluster, including lack of network egress to these internet-based resources which are required for the cluster operation and support: registry.access.redhat.com:443, route53domains.us-east-1.amazonaws.com:443, ec2.us-west-2.amazonaws.com:443, registry.redhat.io:443, mirror.openshift.com:443, api.access.redhat.com:443, among others.

Resolution

Check the connectivity to all the required URLs as explained in the Diagnostic Steps section, and follow the documentation regarding the required prerequisites:

Root Cause

A network misconfiguration has been detected by the SRE automation and a Service Log was sent.

Diagnostic Steps

For these steps, we are going to use a ROSA clusters as an example. Please, check the product documentation for more information.

For ROSA clusters, it is possible to run the network check via two modes:

  1. Using rosa cli command. Reference: Network verification for ROSA clusters.

  2. Or, using the manual mode by specifying the required URLs.

  • These commands can be run from any node. Please change [node_name] with the name of the node you want to test connectivity with the external world:

    $ oc get nodes
[...]

    $ oc debug node/[node_name]

sh-4.4# chroot /host bash
[root@node_name /]# echo; for url in \
quay.io:443 \
ec2.us-west-2.amazonaws.com:443 \
ec2.eu-west-1.amazonaws.com:443 \
elasticloadbalancing.us-west-2.amazonaws.com:443 \
events.pagerduty.com:443 \
storage.googleapis.com:443 \
mirror.openshift.com:443 \
registry.access.redhat.com:443 \
console.redhat.com:443 \
api.access.redhat.com:443 \
cert-api.access.redhat.com:443 \
http-inputs-osdsecuritylogs.splunkcloud.com:443 \
sts.amazonaws.com:443 \
sts.us-west-2.amazonaws.com:443 \
route53.amazonaws.com:443 \
route53domains.us-east-1.amazonaws.com:443 \
ec2.amazonaws.com:443 \
tagging.us-east-1.amazonaws.com:443 \
iam.amazonaws.com:443 \
nosnch.in:443 \
api.openshift.com:443 \
observatorium.api.openshift.com:443 \
api.deadmanssnitch.com:443 \
pull.q1w2.quay.rhcloud.com:443 \
infogw.api.openshift.com:443 \
sso.redhat.com:80 \
okd.io:443; do nc -z -w 3 `echo $url | tr -s ':' ' '`; echo -n $?; echo " - $url"; done; echo; exit

  • Also, from a debug pod. Please, ensure to use a namespace where a debug pod can be created:

    $ oc debug -n [namespace_name] --image=quay.io/openshift/origin-network-tools

sh-4.4# echo; for url in \
quay.io:443 \
ec2.us-west-2.amazonaws.com:443 \
elasticloadbalancing.us-west-2.amazonaws.com:443 \
events.pagerduty.com:443 \
storage.googleapis.com:443 \
events.us-west-2.amazonaws.com:443 \
mirror.openshift.com:443 \
registry.access.redhat.com:443 \
console.redhat.com:443 \
api.access.redhat.com:443 \
cert-api.access.redhat.com:443 \
http-inputs-osdsecuritylogs.splunkcloud.com:443 \
registry.redhat.io:443 \
sts.amazonaws.com:443 \
sts.us-west-2.amazonaws.com:443 \
route53.amazonaws.com:443 \
route53domains.us-east-1.amazonaws.com:443 \
ec2.amazonaws.com:443 \
tagging.us-east-1.amazonaws.com:443 \
iam.amazonaws.com:443 \
nosnch.in:443 \
sso.redhat.com:443 \
api.openshift.com:443 \
observatorium.api.openshift.com:443 \
api.deadmanssnitch.com:443 \
pull.q1w2.quay.rhcloud.com:443 \
infogw.api.openshift.com:443 \
sso.redhat.com:80 \
okd.io:443 \
inputs1.osdsecuritylogs.splunkcloud.com:9997; do nc -z -w 3 `echo $url | tr -s ':' ' '`; echo -n $?; echo " - $url"; done; echo; exit

  • The output should include a 0 at the beginning of each line. For example:

    0 - quay.io:443

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments