Increased Network Traffic in ROSA Clusters After Upgrading to 4.12 Due to Third-Party Operators SCCs

Solution In Progress - Updated -

Environment

  • Red Hat OpenShift on AWS (ROSA)
    • 4.12
  • Red Hat OpenShift Dedicated (OSD)
    • 4.12

Issue

  • Customers have reported increased cross-AZ network usage costs after upgrading their clusters from 4.11 to 4.12.
  • The surge in traffic is attributed to containers in the openshift-marketplace namespace
  • The Openshift Cluster has third-party operators, such as Datadog and Twistlock, which create SSCs installed in the cluster.

Resolution

  1. Immediate Workaround: Consider uninstalling the Datadog operator or any other problematic third-party operator including the SCC they created to check if the network traffic returns to normal levels.

  2. Long-term Solution: An investigation into this issue is ongoing. Customers are advised to monitor the OCPBUGS-20347 ticket for updates and potential fixes.

Root Cause

The root cause appears to be linked to certain SCCs installed by third-party operators, primarily Datadog. This leads to an unusual increase in traffic between the packageserver pods in the openshift-operator-lifecycle-manager namespace and *-operators pods in the openshift-marketplace namespace, resulting in higher network usage costs.

This issue has been observed in multiple ROSA clusters after upgrading to 4.12.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments