Ceph - Cannot create or delete Cinder volumes or Glance images

Solution Verified - Updated -

Environment

  • Red Hat Ceph Storage 1.3.z
  • Red Hat Ceph Storage 2.x
  • Red Hat Ceph Storage 3.x

Issue

  • I am unable to create or delete images and volumes when using Ceph as a backend storage for Openstack's Cinder or glance.

Resolution

  • In one situation observed, this was the result of the ceph osd pause flag being set within the Ceph cluster environment:
pauserd,pausewr,nodown flag(s) set

  • Removing this flag results in resending pending requests, and allow future IO requests to be processed.

  • This flag can be removed using the following commnad:

# ceph osd unset pause

Root Cause

Diagnostic Steps

  • Check the health state of the Ceph Cluster to see that the OSD pause flag has been set using the ceph -s command.
     health HEALTH_WARN
            1420 pgs degraded
            1420 pgs stuck degraded
            1472 pgs stuck unclean
            1420 pgs stuck undersized
            1420 pgs undersized
            recovery 2222/6666 objects degraded (33.333%)
            too many PGs per OSD (736 > max 300)
            pauserd,pausewr,nodown flag(s) set    <--------------------------------------
     monmap e1: 3 mons at {mon01=192.0.2.1:6789/0,mon02=192.0.2.2:6789/0,mon03=192.0.2.3:6789/0}
            election epoch 18, quorum 0,1,2 mon03,mon01,mon02
     osdmap e62: 8 osds: 4 up, 4 in; 52 remapped pgs
            flags pauserd,pausewr,nodown,sortbitwise,require_jewel_osds
      pgmap v20761: 1472 pgs, 12 pools, 6734 MB data, 2222 objects
            13699 MB used, 4455 GB / 4468 GB avail
            2222/6666 objects degraded (33.333%)
                1420 active+undersized+degraded
                  52 active+remapped

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.