How to modify kube-controller-manager argument terminated-pod-gc-threshold on OpenShift

Issue

In some cases the pods may be evicted and the terminated pods remained in ContainerStatusUnknown state.

[root@hub-helper ~]# oc get pods
NAME                              READY   STATUS                   RESTARTS   AGE
bad-deployment-5d574d6c9f-2ldqj   0/1     ContainerStatusUnknown   1          12m
bad-deployment-5d574d6c9f-4vzct   0/1     ContainerStatusUnknown   1          13m
bad-deployment-5d574d6c9f-5hp4t   0/1     ContainerStatusUnknown   1          75m
bad-deployment-5d574d6c9f-5njxs   0/1     ContainerStatusUnknown   1          72m
bad-deployment-5d574d6c9f-65xfl   0/1     ContainerStatusUnknown   1          92m
bad-deployment-5d574d6c9f-6cfc4   0/1     ContainerStatusUnknown   1          4m25s
bad-deployment-5d574d6c9f-6n6fx   0/1     ContainerStatusUnknown   1          15m
bad-deployment-5d574d6c9f-6p78f   0/1     Error                    0          96m
bad-deployment-5d574d6c9f-7q2qf   0/1     ContainerStatusUnknown   1          21m
bad-deployment-5d574d6c9f-7wsbz   0/1     Error                    0          41m
bad-deployment-5d574d6c9f-82k7t   0/1     ContainerStatusUnknown   1          5m24s
bad-deployment-5d574d6c9f-87ccn   0/1     ContainerStatusUnknown   1          82m
bad-deployment-5d574d6c9f-8fp2g   0/1     ContainerStatusUnknown   1          74m
bad-deployment-5d574d6c9f-8gbwh   0/1     Error                    0          10m
bad-deployment-5d574d6c9f-b26jd   0/1     ContainerStatusUnknown   1          61m
bad-deployment-5d574d6c9f-bj8qz   0/1     ContainerStatusUnknown   1          95m
bad-deployment-5d574d6c9f-bjj5j   0/1     ContainerStatusUnknown   1          55m
bad-deployment-5d574d6c9f-c4pt7   0/1     Error                    0          69m
bad-deployment-5d574d6c9f-c8v5p   0/1     ContainerStatusUnknown   1          14m
bad-deployment-5d574d6c9f-cbpwd   0/1     ContainerStatusUnknown   1          89m
bad-deployment-5d574d6c9f-ctxkc   0/1     Error                    0          27m
bad-deployment-5d574d6c9f-cz7wn   0/1     Error                    0          16m
bad-deployment-5d574d6c9f-dbww7   0/1     ContainerStatusUnknown   1          7m42s
bad-deployment-5d574d6c9f-dkp5g   0/1     Error                    0          85m
bad-deployment-5d574d6c9f-dplkp   0/1     Error                    0          3m23s
bad-deployment-5d574d6c9f-dqj7w   1/1     Running                  0          66s
bad-deployment-5d574d6c9f-dv4db   0/1     ContainerStatusUnknown   1          59m
bad-deployment-5d574d6c9f-f5wl9   0/1     ContainerStatusUnknown   1          56m
bad-deployment-5d574d6c9f-fdkww   0/1     Error                    0          20m
bad-deployment-5d574d6c9f-fk6js   0/1     ContainerStatusUnknown   1          63m

Until pod garbage collector (PodGC) gets involved or a human(oc delete pods) or controller process explicitly removes them, the terminated pod objects will be remained in the cluster's API.

The pod garbage collector (PodGC) cleans up terminated pods (with a phase of Succeeded or Failed), when the number of Pods exceeds the configured threshold (determined by terminated-pod-gc-threshold in the kube-controller-manager).

The default value of terminated-pod-gc-threshold is 12500, which is relatively big especially for the SNO(Single Node OpenShift) which may give the feeling that the terminated pods remained in the cluster and won't be cleanup for ever.

This KCS provides a procedure to modify kube-controller-manager argument terminated-pod-gc-threshold so that the PodGC can be triggered more frenquently.