Unable to delete or create volumes in HCI environment

Issue

• We are unable to create or or delete volumes. This is an outage for the environment. We need to get the environment back up. We tried rebooting the entire environment, but had no success. The environment looks healthy. pcs cluster status, docker containers on controller look to be running, ceph is healthy.

• Ceph status is HEALTH_WARN and many PGs statuses are unknown:

ceph status
  cluster:
    id:     8dbc7e62-2bff-11e9-84a1-509a4ca3c6c2
    health: HEALTH_WARN
            Reduced data availability: 503 pgs inactive

  services:
    mon: 3 daemons, quorum overcloud-controller-0,overcloud-controller-1,overcloud-controller-2
    mgr: overcloud-controller-0(active), standbys: overcloud-controller-1, overcloud-controller-2
    mds: cephfs-1/1/1 up  {0=dev1-controller-1=up:active}, 2 up:standby
    osd: 44 osds: 44 up, 44 in
    rgw: 3 daemons active

  data:
    pools:   12 pools, 1508 pgs
    objects: 506.04k objects, 1.91TiB
    usage:   5.90TiB used, 53.0TiB / 58.9TiB avail
    pgs:     33.355% pgs unknown
             1005 active+clean
             503  unknown

  io:
    client:   1.17KiB/s rd, 175KiB/s wr, 5op/s rd, 6op/s wr