PersistentVolumeClaim for glusterfs storage in pending state due to "Server busy" error
Issue
-
A newly created PersistentVolumeClaim (PVC) stays in
Status: Pending
. -
Examining the
Status: Pending
PVC, there are events like:Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 2m (x64 over 17m) persistentvolume-controller Failed to provision volume with StorageClass "glusterfs-storage": failed to create volume: failed to create volume: Server busy. Retry operation later.
-
Looking into
heketi
logs, there are messages withToo Many Requests
andoperations in-flight (8) exceeds limit (8)
:[heketi] WARNING 2019/11/19 11:49:52 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 800.958µs [negroni] Started POST /volumes [heketi] WARNING 2019/11/19 11:49:52 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 182.937µs [negroni] Started POST /volumes [heketi] WARNING 2019/11/19 11:49:52 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 282.027µs [negroni] Started POST /volumes [heketi] WARNING 2019/11/19 11:49:52 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 286.491µs
Environment
- Red Hat OpenShift Container Storage (OCS) 3.11
- Red Hat OpenShift Container Platform (OCP) 3.9, 3.10, 3.11
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.