Pod Stuck in terminating state and unable to remove finalizers

Solution Verified - Updated -

Issue

  • After hardware failure pod has remained in a Terminating state even though the node no longer exists
  • Force delete pod with --grace-period=0 is not working
  • Unable to remove the finalizers
  • A namespace label SPK-DNS is enabled and preventing the finalizers to be removed
  • spk-webhook-deployment pod under kube-system namespace is trying to update the dnsPolicy and the dnsConfig from the troubled pod that is in Terminating state.
2023-05-05T05:09:26.997311959Z I0505 05:09:26.997280       1 webhook.go:144] AdmissionReview for Kind=/v1, Kind=Pod, Namespace=<namespace> Name=<pod_name> (pod_name) UID=<UID> patchOperation=UPDATE UserInfo={system:serviceaccount:openshift-operators:<user> 2b25a3fc-59d1-4e7c-9757-306d11590cc5 [system:serviceaccounts system:serviceaccounts:openshift-operators system:authenticated] map[]}
2023-05-05T05:09:26.997354404Z I0505 05:09:26.997324       1 webhook.go:156] AdmissionResponse: patch=[{“op”:“replace”,“path”:“/spec/dnsPolicy”,“value”:“None”},{“op”:“add”,“path”:“/spec/dnsConfig”,“value”:{“nameservers”:[“<IP_Address_of_the_NameServer>”],“searches”:[“<name>.svc.cluster.local”,“svc.cluster.local”,“cluster.local”],“options”:[{“name”:“ndots”,“value”:“5"}]}}]

Error

$ oc patch pod <Pod_Name> -p '{"metadata":{"finalizers":null}}'
The Pod "<Pod_name>" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)
  core.PodSpec{
        ... // 5 identical fields
        TerminationGracePeriodSeconds: &30,
        ActiveDeadlineSeconds:         nil,
-       DNSPolicy:                     "None",
+       DNSPolicy:                     "ClusterFirst",
        NodeSelector:                  nil,
        ServiceAccountName:            "default",
        ... // 12 identical fields
        Priority:         &0,
        PreemptionPolicy: &"PreemptLowerPriority",
-       DNSConfig: &core.PodDNSConfig{
-               Nameservers: []string{"IP_Address_of_nameserver"},
-               Searches: []string{
-                       "<name>.svc.cluster.local", "svc.cluster.local",
-                       "cluster.local",
-               },
-               Options: []core.PodDNSConfigOption{{Name: "ndots", Value: &"5"}},
-       },
+       DNSConfig:        nil,
        ReadinessGates:   nil,
        RuntimeClassName: nil,
        ... // 3 identical fields
  }

Environment

  • Red Hat OpenShift Container Platform 4.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content