OpenShift Dynatrace Alerts - TCP connectivity rate for process

Solution Verified - Updated -

Environment

  • Red Hat OpenShift on AWS (ROSA)
    • 4.x
  • Red Hat OpenShift Dedicated (OSD)
    • 4.x
  • Azure Red Hat OpenShift (ARO)
    • 4.x

Issue

  • We are seeing quite a lot of problems generated with the message"TCP connectivity rate for process podname on host xxxxxxxx has decreased to 33 %" (sometimes also 0%), but the service is up and running.

Resolution

  • You may log a case with Dynatrace support on how to tweak and configure this type of alerts.

Root Cause

  • If your host has no connectivity problems and network traffic is fine, then there is nothing to worry about it.

  • This alert comes from the Dynatrace monitoring dashboard and its not generated by OpenShift.

Diagnostic Steps

  • Check your cluster operators and make sure everything is fine.

Sample Azure Redhat OpenShift Cluster:

#oc get clusteroperators
NAME                                       VERSION        AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
aro                                        v20230802.01   True        False         False      3d14h   
authentication                             4.10.40        True        False         False      2d23h   
baremetal                                  4.10.40        True        False         False      3d14h   
cloud-controller-manager                   4.10.40        True        False         False      3d14h   
cloud-credential                           4.10.40        True        False         False      3d14h   
cluster-autoscaler                         4.10.40        True        False         False      3d14h   
config-operator                            4.10.40        True        False         False      3d14h   
console                                    4.10.40        True        False         False      3d14h   
csi-snapshot-controller                    4.10.40        True        False         False      3d14h   
dns                                        4.10.40        True        False         False      3d14h   
etcd                                       4.10.40        True        False         False      3d14h   
image-registry                             4.10.40        True        False         False      3d14h   
ingress                                    4.10.40        True        False         False      3d14h   
insights                                   4.10.40        True        False         False      29s     
kube-apiserver                             4.10.40        True        False         False      3d14h   
kube-controller-manager                    4.10.40        True        False         False      3d14h   
kube-scheduler                             4.10.40        True        False         False      3d14h   
kube-storage-version-migrator              4.10.40        True        False         False      3d14h   
machine-api                                4.10.40        True        False         False      3d14h   
machine-approver                           4.10.40        True        False         False      3d14h   
machine-config                             4.10.40        True        False         False      3d14h   
marketplace                                4.10.40        True        False         False      3d14h   
monitoring                                 4.10.40        True        False         False      3d14h   
network                                    4.10.40        True        False         False      3d14h   
node-tuning                                4.10.40        True        False         False      3d14h   
openshift-apiserver                        4.10.40        True        False         False      3d14h   
openshift-controller-manager               4.10.40        True        False         False      3d14h   
openshift-samples                          4.10.40        True        False         False      3d14h   
operator-lifecycle-manager                 4.10.40        True        False         False      3d14h   
operator-lifecycle-manager-catalog         4.10.40        True        False         False      3d14h   
operator-lifecycle-manager-packageserver   4.10.40        True        False         False      3d14h   
service-ca                                 4.10.40        True        False         False      3d14h   
storage                                    4.10.40        True        False         False      3d14h
  • Check if the pods related to the cluster operators are running properly and the nodes are not over committed or it is having enough resources.
  • Check for any recent events related to your cluster operators.
#oc describe node/nodename 
#oc adm top nodes
#oc get pods -n namespace
#oc get events --all-namespaces --sort-by='.lastTimestamp'

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments