GPU Pods Failing to Schedule After OpenShift Cluster Upgrade Due to 'node.cloudprovider.kubernetes.io/uninitialized' Taint

Issue

• After upgrading the OpenShift cluster, the GPU pods were unable to schedule onto newly provisioned GPU nodes. Although the GPU pods were created successfully, they remained in the Pending state due to unavailability of the GPU node.

• GPU pods failed to get scheduled on node with below error:

    message: '0/10 nodes are available: 1 node(s) had untolerated taint {node.cloudprovider.kubernetes.io/uninitialized:
      true}, 2 node(s) had untolerated taint {node-role.kubernetes.io/infra: }, 3
      node(s) had untolerated taint {node-role.kubernetes.io/master: }, 4 node(s)
      didn''t match Pod''s node affinity/selector. preemption: 0/10 nodes are available:
      10 Preemption is not helpful for scheduling.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending