PersistentVolumeClaim for glusterfs storage in pending state due to "Server busy" error
Issue
-
A newly created PersistentVolumeClaim (PVC) stays in
Status: Pending. -
Examining the
Status: PendingPVC, there are events like:Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 2m (x64 over 17m) persistentvolume-controller Failed to provision volume with StorageClass "glusterfs-storage": failed to create volume: failed to create volume: Server busy. Retry operation later. -
Looking into
heketilogs, there are messages withToo Many Requestsandoperations in-flight (8) exceeds limit (8):[heketi] WARNING 2019/11/19 11:49:52 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 800.958µs [negroni] Started POST /volumes [heketi] WARNING 2019/11/19 11:49:52 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 182.937µs [negroni] Started POST /volumes [heketi] WARNING 2019/11/19 11:49:52 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 282.027µs [negroni] Started POST /volumes [heketi] WARNING 2019/11/19 11:49:52 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 286.491µs
Environment
- Red Hat OpenShift Container Storage (OCS) 3.11
- Red Hat OpenShift Container Platform (OCP) 3.9, 3.10, 3.11
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.