Wrong subnet for Pods on a new OpenShift Node

Solution Unverified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift Service on AWS (ROSA)
  • OVNKubernetes

Issue

  • On a new OpenShift Container Platform Worker Node, Pods are not started with the correct subnet set:

    k8s.ovn.org/host-addresses: '["10.178.67.42"]'
k8s.ovn.org/node-subnets: '{"default":["10.179.206.0/23"]}'

    openshift-ingress-canary                           ingress-canary-26q4g   [..]   10.179.204.9     ip-10-178-67-42.node.example.com [..]

  • Pods on the new OpenShift Container Platform Worker Node do not have network connectivity and time out when trying to reach other resources on the network.

Resolution

Red Hat is aware of this issue and tracking the resolution in the following Bugs:

Target Minor Release Bug Fixed Version Errata
4.16 OCPBUGS-25733 4.16.0 RHSA-2024:0041
4.15 OCPBUGS-33294 4.15.15 RHSA-2024:3327
4.14 OCPBUGS-34076 4.14.29 RHBA-2024:3697

Workaround for previous releases

The workaround for this issue is to delete the Node and to recreate it.

Root Cause

The subnet is wrongly assigned to the nodes.

Diagnostic Steps

  • Review the OVN labels and annotations on the affected Node:

    $ oc get node ip-10-178-67-42.node.example.com -o yaml | grep k8s.ovn.org
  k8s.ovn.org/host-addresses: '["10.178.67.42"]'
  [...]
  k8s.ovn.org/node-subnets: '{"default":["10.179.206.0/23"]}'

  • Check the IPs of the pods running on the node:

    $ oc get pods -o wide --all-namespaces | grep "ip-10-178-67-42"
[...]
openshift-ingress-canary                           ingress-canary-26q4g   [..]   10.179.204.9     ip-10-178-67-42.node.example.com [..]
[...]

  • Review the IPs of the Pods in the OVN database:

    # ovn-nbctl show ip-10-178-67-42.node.example.com
[...]
  • Component
  • ovn

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments