AWS cloudprovider rejects a node after instance resize with message "No longer present in cloud provider"

Solution Verified - Updated -

Issue

  • After following AWS instance resize procedure and restarting the node, it becomes no longer present in the cluster and there are some controller errors like the following:
W0128 15:36:30.484938       1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="ip-10-0-131-87.example.com" does not exist
I0128 15:36:33.167795       1 node_lifecycle_controller.go:558] Controller observed a new Node: "ip-10-0-131-87.example.com"
I0128 15:36:33.167827       1 controller_utils.go:212] Recording Registered Node ip-10-0-131-87.example.com in Controller event message for node ip-10-0-131-87.example.com
W0128 15:36:33.167955       1 node_lifecycle_controller.go:782] Missing timestamp for Node ip-10-0-131-87.example.com. Assuming now as a timestamp.
I0128 15:36:33.281926       1 node_lifecycle_controller.go:712] Deleting node (no longer present in cloud provider): ip-10-0-131-87.example.com
I0128 15:36:33.285410       1 controller_utils.go:212] Recording Deleting Node ip-10-0-131-87.example.com because it's not present according to cloud provider event message for node ip-10-0-131-87.example.com
I0128 15:36:38.285705       1 node_lifecycle_controller.go:570] Controller observed a Node deletion: ip-10-0-131-87.example.com
I0128 15:36:38.285737       1 controller_utils.go:212] Recording Removing Node ip-10-0-131-87.example.com from Controller event message for node ip-10-0-131-87.example.com
I0128 15:36:51.580276       1 gc_controller.go:162] Found orphaned Pod sync-qs8h7 assigned to the Node ip-10-0-131-87.example.com. Deleting.
I0128 15:36:51.592087       1 gc_controller.go:162] Found orphaned Pod sdn-42wzx assigned to the Node ip-10-0-131-87.example.com. Deleting.
I0128 15:36:51.608574       1 gc_controller.go:162] Found orphaned Pod ovs-fvff2 assigned to the Node ip-10-0-131-87.example.com. Deleting.
I0128 15:36:51.622391       1 gc_controller.go:162] Found orphaned Pod node-exporter-j7m5h assigned to the Node ip-10-0-131-87.example.com. Deleting.

NOTE: The node could also appear momentarily within the cluster and even present some errors trying to attach pending PVCs, but at the end it dissapears again.

Environment

  • OpenShift Container Platform
    • 3.11
  • AWS cloudprovider

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content