A Ceph cluster shows a status of 'HEALTH_WARN' warning with the message "too many PGs per OSD", why?

Issue

• An RHCS/Ceph cluster shows a status of 'HEALTH_WARN' warning with the message "too many PGs per OSD", why?

• This can normally happen in two cases :

  • A perfectly normal RHCS cluster (usually 1.3.x) may see this after adding new pools, or adding more placement groups to existing pools.

  • A healthy RHCS1.2.3 cluster upgraded to RHCS1.3.x, provided the PG density (PG per OSD) was already high before upgrade.

• A ceph -s should print the cluster health.

# ceph -s
    cluster cccccccc-eeee-eeee-ffff-7aceb97642a0
     health HEALTH_WARN
            too many PGs per OSD (496 > max 300)