PersistentVolumeClaim for glusterfs storage in pending state due to "Server busy" error

Solution Verified - Updated -

Issue

  • A newly created PersistentVolumeClaim (PVC) stays in Status: Pending.

  • Examining the Status: Pending PVC, there are events like:

    Events:
      Type     Reason              Age                From                         Message
       ----     ------              ----               ----                         -------
      Warning  ProvisioningFailed  2m (x64 over 17m)  persistentvolume-controller  Failed to provision volume with StorageClass "glusterfs-storage": failed to create volume: failed to create volume: Server busy. Retry operation later.
    
  • Looking into heketilogs, there are messages with Too Many Requests and operations in-flight (8) exceeds limit (8):

    [heketi] WARNING 2019/11/19 11:49:52 operations in-flight (8) exceeds limit (8)
    [negroni] Completed 429 Too Many Requests in 800.958µs
    [negroni] Started POST /volumes
    [heketi] WARNING 2019/11/19 11:49:52 operations in-flight (8) exceeds limit (8)
    [negroni] Completed 429 Too Many Requests in 182.937µs
    [negroni] Started POST /volumes
    [heketi] WARNING 2019/11/19 11:49:52 operations in-flight (8) exceeds limit (8)
    [negroni] Completed 429 Too Many Requests in 282.027µs
    [negroni] Started POST /volumes
    [heketi] WARNING 2019/11/19 11:49:52 operations in-flight (8) exceeds limit (8)
    [negroni] Completed 429 Too Many Requests in 286.491µs
    

Environment

  • Red Hat OpenShift Container Storage (OCS) 3.11
  • Red Hat OpenShift Container Platform (OCP) 3.9, 3.10, 3.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content