Creating a custom mcp without worker role causes the worker node to enter the “NotReady” state

Solution Verified - Updated -

Environment

  • OpenShift Container Platform
    • 4.14

Issue

  • Creating a custom mcp without worker role causes the worker node to enter the “NotReady” state.

Resolution

  • Delete the unexpected MCP && MC.
$ oc delete mc <MC-name>
$ oc delete mcp <MCP-name>
  • Rebuild the unexpected worker node, and join to the OCP.

Diagnostic Steps

  • Verify worker node status
$ oc get no|grep NotReady
worker03.ocp4.danliu.com   NotReady,SchedulingDisabled   worker,worker-t   70d    v1.27.13+e709aa5
  • Check the MCP status
 $ oc get mcp
NAME       CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master     rendered-master-66416683c6e6a6097cbe1f9c91536c59   True      False      False      3              3                   3                     0                      454d
worker     rendered-worker-6730a9197bcb1dde15daf066a4e09d3a   True      False      False      2              2                   2                     0                      454d
worker-t                                                      False     True       False      1              0                   0                     0                      108m
  • Check the events in the openshift-machine-config-operator project
$ oc get ev -n  openshift-machine-config-operator
6m50s       Normal    RenderedConfigGenerated                 machineconfigpool/worker-t                          rendered-worker-t-44c26ffa742bdb25eea6874cb0bb25ae successfully generated (release version: 4.14.26, controller version: 83d66a9228fef00885826e00ad3d392c5525bfab)
6m45s       Normal    SetDesiredConfig                        machineconfigpool/worker-t                          Targeted node worker03.ocp4.danliu.com to MachineConfig 6m39s       Normal    ConfigDriftMonitorStopped               node/worker03.ocp4.danliu.com                       Config Drift Monitor stopped
6m39s       Normal    Cordon                                  node/worker03.ocp4.danliu.com                       Cordoned node to apply update
6m39s       Normal    Drain                                   node/worker03.ocp4.danliu.com                       Draining node to update config.
6m19s       Normal    OSUpdateStarted                         node/worker03.ocp4.danliu.com                       Changing kernel arguments
6m18s       Normal    OSUpgradeSkipped                        node/worker03.ocp4.danliu.com                       OS upgrade skipped; new MachineConfig (rendered-worker-t-44c26ffa742bdb25eea6874cb0bb25ae) has same OS image (quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b661952b113f439dffc23ba4a8815f003fd01808fda78c56b26e4438f5f4ea4) as old MachineConfig (rendered-worker-6730a9197bcb1dde15daf066a4e09d3a)
6m12s       Normal    Reboot                                  node/worker03.ocp4.danliu.com                       Node will reboot into config rendered-worker-t-44c26ffa742bdb25eea6874cb0bb25ae

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments