AWS cloudprovider rejects a node after instance resize with message "No longer present in cloud provider"
Issue
- After following AWS instance resize procedure and restarting the node, it becomes no longer present in the cluster and there are some controller errors like the following:
W0128 15:36:30.484938 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="ip-10-0-131-87.example.com" does not exist
I0128 15:36:33.167795 1 node_lifecycle_controller.go:558] Controller observed a new Node: "ip-10-0-131-87.example.com"
I0128 15:36:33.167827 1 controller_utils.go:212] Recording Registered Node ip-10-0-131-87.example.com in Controller event message for node ip-10-0-131-87.example.com
W0128 15:36:33.167955 1 node_lifecycle_controller.go:782] Missing timestamp for Node ip-10-0-131-87.example.com. Assuming now as a timestamp.
I0128 15:36:33.281926 1 node_lifecycle_controller.go:712] Deleting node (no longer present in cloud provider): ip-10-0-131-87.example.com
I0128 15:36:33.285410 1 controller_utils.go:212] Recording Deleting Node ip-10-0-131-87.example.com because it's not present according to cloud provider event message for node ip-10-0-131-87.example.com
I0128 15:36:38.285705 1 node_lifecycle_controller.go:570] Controller observed a Node deletion: ip-10-0-131-87.example.com
I0128 15:36:38.285737 1 controller_utils.go:212] Recording Removing Node ip-10-0-131-87.example.com from Controller event message for node ip-10-0-131-87.example.com
I0128 15:36:51.580276 1 gc_controller.go:162] Found orphaned Pod sync-qs8h7 assigned to the Node ip-10-0-131-87.example.com. Deleting.
I0128 15:36:51.592087 1 gc_controller.go:162] Found orphaned Pod sdn-42wzx assigned to the Node ip-10-0-131-87.example.com. Deleting.
I0128 15:36:51.608574 1 gc_controller.go:162] Found orphaned Pod ovs-fvff2 assigned to the Node ip-10-0-131-87.example.com. Deleting.
I0128 15:36:51.622391 1 gc_controller.go:162] Found orphaned Pod node-exporter-j7m5h assigned to the Node ip-10-0-131-87.example.com. Deleting.
NOTE: The node could also appear momentarily within the cluster and even present some errors trying to attach pending PVCs, but at the end it dissapears again.
Environment
- OpenShift Container Platform
- 3.11
- AWS cloudprovider
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.