Gluster pod getting evicted due to DiskPressure on openshift node
Issue
-
gluster pods are getting in pending state continuously due to the disk pressure on the openshift node trowing below error
The gluster and other pods on node NodeXX and NodeYY are getting evicted due to "node has conditions: [DiskPressure]" which seems to be due to /var/lib/docker/ (i.e / root disk) reaching 90% usage. The NodeXX and NodeYY reaching node configuration "nodefs.available<10%".
Apr 24 04:36:43 NodeXX atomic-openshift-node[109786]: I0424 04:36:43.207933 109786 kubelet.go:1836] SyncLoop (DELETE, "api"): "glusterfs-storage-gdqfs_glusterfs(6e0e49b9-85d4-11ea-ad50-XXX)"
Apr 24 04:36:43 NodeXX atomic-openshift-node[109786]: I0424 04:36:43.212741 109786 kubelet.go:1830] SyncLoop (REMOVE, "api"): "glusterfs-storage-gdqfs_glusterfs(6e0e49b9-85d4-11ea-ad50-XXX)"
Apr 24 04:36:43 NodeXX atomic-openshift-node[109786]: I0424 04:36:43.222242 109786 kubelet.go:2032] Failed to delete pod "glusterfs-storage-gdqfs_glusterfs(6e0e49b9-85d4-11ea-ad50-0XXX)", err: pod not found
Apr 24 04:36:43 NodeXX atomic-openshift-node[109786]: I0424 04:36:43.234439 109786 kubelet.go:1820] SyncLoop (ADD, "api"): "glusterfs-storage-qdw7l_glusterfs(6f20f678-85d4-11ea-ad50-XXX)"
Apr 24 04:36:43 NodeXX atomic-openshift-node[109786]: W0424 04:36:43.234823 109786 eviction_manager.go:156] Failed to admit pod glusterfs-storage-qdw7l_glusterfs(6f20f678-85d4-11ea-ad50-XXX) - node has conditions: [DiskPressure]
$ oc get pods -o wide -n glusterfs
NAME READY STATUS RESTARTS AGE IP
glusterblock-storage-provisioner-dc-2-8m6ck 1/1 Running 0 3d 10.130.14.X
glusterfs-storage-48r9l 0/1 Pending 0 2s <none>
glusterfs-storage-64bzr 1/1 Running 0 6h 192.198.226.X
glusterfs-storage-z6chv 0/1 Pending 0 1s <none>
heketi-storage-6-gwnft 1/1 Running 58 9h 10.131.6.X
Environment
RHOCS 3.11
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.