Gluster pod getting evicted due to DiskPressure on openshift node

Solution Verified - Updated -

Issue

  • gluster pods are getting in pending state continuously due to the disk pressure on the openshift node trowing below error

    The gluster and other pods on node NodeXX and NodeYY are getting evicted due to "node has conditions: [DiskPressure]" which seems to be due to /var/lib/docker/ (i.e / root disk) reaching 90% usage. The NodeXX and NodeYY reaching node configuration "nodefs.available<10%".

Apr 24 04:36:43 NodeXX atomic-openshift-node[109786]: I0424 04:36:43.207933  109786 kubelet.go:1836] SyncLoop (DELETE, "api"): "glusterfs-storage-gdqfs_glusterfs(6e0e49b9-85d4-11ea-ad50-XXX)"
Apr 24 04:36:43 NodeXX atomic-openshift-node[109786]: I0424 04:36:43.212741  109786 kubelet.go:1830] SyncLoop (REMOVE, "api"): "glusterfs-storage-gdqfs_glusterfs(6e0e49b9-85d4-11ea-ad50-XXX)"
Apr 24 04:36:43 NodeXX atomic-openshift-node[109786]: I0424 04:36:43.222242  109786 kubelet.go:2032] Failed to delete pod "glusterfs-storage-gdqfs_glusterfs(6e0e49b9-85d4-11ea-ad50-0XXX)", err: pod not found
Apr 24 04:36:43 NodeXX atomic-openshift-node[109786]: I0424 04:36:43.234439  109786 kubelet.go:1820] SyncLoop (ADD, "api"): "glusterfs-storage-qdw7l_glusterfs(6f20f678-85d4-11ea-ad50-XXX)"
Apr 24 04:36:43 NodeXX atomic-openshift-node[109786]: W0424 04:36:43.234823  109786 eviction_manager.go:156] Failed to admit pod glusterfs-storage-qdw7l_glusterfs(6f20f678-85d4-11ea-ad50-XXX) - node has conditions: [DiskPressure]
$ oc get pods -o wide -n glusterfs
NAME                                          READY     STATUS    RESTARTS   AGE       IP              
glusterblock-storage-provisioner-dc-2-8m6ck   1/1       Running   0          3d        10.130.14.X    
glusterfs-storage-48r9l                       0/1       Pending   0          2s        <none>           
glusterfs-storage-64bzr                       1/1       Running   0          6h        192.198.226.X  
glusterfs-storage-z6chv                       0/1       Pending   0          1s        <none>           
heketi-storage-6-gwnft                        1/1       Running   58         9h        10.131.6.X     

Environment

RHOCS 3.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content