[OpenShift] atomic-openshift-node.service getting restarted on all nodes every three minutes

Solution Verified - Updated -

Issue

  • Modification times of files in /etc/origin/node like volume-config.yaml become changed every 3 minutes, although their contents keep constant.
-rw-------. 1 root root 2.2K Jul 16 10:17 /etc/origin/node/node-config.yaml
-rw-------. 1 root root   94 Jul 16 10:17 /etc/origin/node/volume-config.yaml
....
-rw-------. 1 root root 2.2K Jul 16 10:20 /etc/origin/node/node-config.yaml
-rw-------. 1 root root   94 Jul 16 10:20 /etc/origin/node/volume-config.yaml
....
-rw-------. 1 root root 2.2K Jul 16 10:23 /etc/origin/node/node-config.yaml
-rw-------. 1 root root   94 Jul 16 10:23 /etc/origin/node/volume-config.yaml
  • Sync pods restart atomic-openshift-node.service on all the nodes every 3 minutes.
2019-07-15T05:23:17.518342186Z info: Applying node labels node-role.kubernetes.io/compute=true logging-infra-fluentd=allowed registry=allowed node-role.kubernetes.io/infra=true 
2019-07-15T05:23:17.74207203Z node/ip-10-200-2-28.eu-central-1.compute.internal not labeled
2019-07-15T05:23:18.017371704Z node/ip-10-200-2-28.eu-central-1.compute.internal annotated
2019-07-15T05:26:20.477917096Z info: Configuration changed, restarting kubelet
2019-07-15T05:26:20.523725187Z info: Applying node labels node-role.kubernetes.io/compute=true logging-infra-fluentd=allowed registry=allowed node-role.kubernetes.io/infra=true 
2019-07-15T05:26:20.760464599Z node/ip-10-200-2-28.eu-central-1.compute.internal not labeled
2019-07-15T05:26:21.029823231Z node/ip-10-200-2-28.eu-central-1.compute.internal annotated
2019-07-15T05:29:23.554951263Z info: Configuration changed, restarting kubelet
2019-07-15T05:29:23.600854253Z info: Applying node labels node-role.kubernetes.io/compute=true logging-infra-fluentd=allowed registry=allowed node-role.kubernetes.io/infra=true 
2019-07-15T05:29:23.853819999Z node/ip-10-200-2-28.eu-central-1.compute.internal not labeled
2019-07-15T05:29:24.182170716Z node/ip-10-200-2-28.eu-central-1.compute.internal annotated
2019-07-15T05:32:26.753933573Z info: Configuration changed, restarting kubelet
  • Problem seems to occur when openshift_node_local_quota_per_fsgroup is configured on inventory. That implies that a file /etc/origin/node/volume-config.yaml is created on every node.

Environment

  • OpenShift Container Platform
    • Versions: from 3.11.82 until 3.11.117.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In