[OpenShift] atomic-openshift-node.service getting restarted on all nodes every three minutes

Solution Verified - Updated -

Issue

  • Modification times of files in /etc/origin/node like volume-config.yaml become changed every 3 minutes, although their contents keep constant.
-rw-------. 1 root root 2.2K Jul 16 10:17 /etc/origin/node/node-config.yaml
-rw-------. 1 root root   94 Jul 16 10:17 /etc/origin/node/volume-config.yaml
....
-rw-------. 1 root root 2.2K Jul 16 10:20 /etc/origin/node/node-config.yaml
-rw-------. 1 root root   94 Jul 16 10:20 /etc/origin/node/volume-config.yaml
....
-rw-------. 1 root root 2.2K Jul 16 10:23 /etc/origin/node/node-config.yaml
-rw-------. 1 root root   94 Jul 16 10:23 /etc/origin/node/volume-config.yaml
  • Sync pods restart atomic-openshift-node.service on all the nodes every 3 minutes.
2019-07-15T05:23:17.518342186Z info: Applying node labels node-role.kubernetes.io/compute=true logging-infra-fluentd=allowed registry=allowed node-role.kubernetes.io/infra=true 
2019-07-15T05:23:17.74207203Z node/ip-10-200-2-28.eu-central-1.compute.internal not labeled
2019-07-15T05:23:18.017371704Z node/ip-10-200-2-28.eu-central-1.compute.internal annotated
2019-07-15T05:26:20.477917096Z info: Configuration changed, restarting kubelet
2019-07-15T05:26:20.523725187Z info: Applying node labels node-role.kubernetes.io/compute=true logging-infra-fluentd=allowed registry=allowed node-role.kubernetes.io/infra=true 
2019-07-15T05:26:20.760464599Z node/ip-10-200-2-28.eu-central-1.compute.internal not labeled
2019-07-15T05:26:21.029823231Z node/ip-10-200-2-28.eu-central-1.compute.internal annotated
2019-07-15T05:29:23.554951263Z info: Configuration changed, restarting kubelet
2019-07-15T05:29:23.600854253Z info: Applying node labels node-role.kubernetes.io/compute=true logging-infra-fluentd=allowed registry=allowed node-role.kubernetes.io/infra=true 
2019-07-15T05:29:23.853819999Z node/ip-10-200-2-28.eu-central-1.compute.internal not labeled
2019-07-15T05:29:24.182170716Z node/ip-10-200-2-28.eu-central-1.compute.internal annotated
2019-07-15T05:32:26.753933573Z info: Configuration changed, restarting kubelet
  • Problem seems to occur when openshift_node_local_quota_per_fsgroup is configured on inventory. That implies that a file /etc/origin/node/volume-config.yaml is created on every node.

Environment

  • OpenShift Container Platform
    • Versions: from 3.11.82 until 3.11.117.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content