Unable to delete or create volumes in HCI environment

Solution In Progress - Updated -


  • We are unable to create or or delete volumes. This is an outage for the environment. We need to get the environment back up. We tried rebooting the entire environment, but had no success. The environment looks healthy. pcs cluster status, docker containers on controller look to be running, ceph is healthy.

  • Ceph status is HEALTH_WARN and many PGs statuses are unknown:

ceph status
    id:     8dbc7e62-2bff-11e9-84a1-509a4ca3c6c2
    health: HEALTH_WARN
            Reduced data availability: 503 pgs inactive

    mon: 3 daemons, quorum overcloud-controller-0,overcloud-controller-1,overcloud-controller-2
    mgr: overcloud-controller-0(active), standbys: overcloud-controller-1, overcloud-controller-2
    mds: cephfs-1/1/1 up  {0=dev1-controller-1=up:active}, 2 up:standby
    osd: 44 osds: 44 up, 44 in
    rgw: 3 daemons active

    pools:   12 pools, 1508 pgs
    objects: 506.04k objects, 1.91TiB
    usage:   5.90TiB used, 53.0TiB / 58.9TiB avail
    pgs:     33.355% pgs unknown
             1005 active+clean
             503  unknown

    client:   1.17KiB/s rd, 175KiB/s wr, 5op/s rd, 6op/s wr


  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content