Pod Stuck in terminating state and unable to remove finalizers
Issue
- After hardware failure pod has remained in a
Terminatingstate even though the node no longer exists - Force delete pod with
--grace-period=0is not working - Unable to remove the finalizers
- A
namespacelabelSPK-DNSis enabled and preventing the finalizers to be removed spk-webhook-deploymentpod underkube-systemnamespace is trying to update thednsPolicyand thednsConfigfrom the troubled pod that is inTerminatingstate.
2023-05-05T05:09:26.997311959Z I0505 05:09:26.997280 1 webhook.go:144] AdmissionReview for Kind=/v1, Kind=Pod, Namespace=<namespace> Name=<pod_name> (pod_name) UID=<UID> patchOperation=UPDATE UserInfo={system:serviceaccount:openshift-operators:<user> 2b25a3fc-59d1-4e7c-9757-306d11590cc5 [system:serviceaccounts system:serviceaccounts:openshift-operators system:authenticated] map[]}
2023-05-05T05:09:26.997354404Z I0505 05:09:26.997324 1 webhook.go:156] AdmissionResponse: patch=[{“op”:“replace”,“path”:“/spec/dnsPolicy”,“value”:“None”},{“op”:“add”,“path”:“/spec/dnsConfig”,“value”:{“nameservers”:[“<IP_Address_of_the_NameServer>”],“searches”:[“<name>.svc.cluster.local”,“svc.cluster.local”,“cluster.local”],“options”:[{“name”:“ndots”,“value”:“5"}]}}]
Error
$ oc patch pod <Pod_Name> -p '{"metadata":{"finalizers":null}}'
The Pod "<Pod_name>" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)
core.PodSpec{
... // 5 identical fields
TerminationGracePeriodSeconds: &30,
ActiveDeadlineSeconds: nil,
- DNSPolicy: "None",
+ DNSPolicy: "ClusterFirst",
NodeSelector: nil,
ServiceAccountName: "default",
... // 12 identical fields
Priority: &0,
PreemptionPolicy: &"PreemptLowerPriority",
- DNSConfig: &core.PodDNSConfig{
- Nameservers: []string{"IP_Address_of_nameserver"},
- Searches: []string{
- "<name>.svc.cluster.local", "svc.cluster.local",
- "cluster.local",
- },
- Options: []core.PodDNSConfigOption{{Name: "ndots", Value: &"5"}},
- },
+ DNSConfig: nil,
ReadinessGates: nil,
RuntimeClassName: nil,
... // 3 identical fields
}
Environment
- Red Hat OpenShift Container Platform 4.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.