Ceph - During running "ansible-playbook site.yml" on deployed cluster fails on TASK "restart ceph osds daemon(s) - non container"

Solution In Progress - Updated -

Issue

We were introducing some changes on our RHCS environment running "ansible-playbook site.yml" and it keeps failing on TASK "restart ceph osds daemon(s) - non container" without obvious reason:

failed: [osd1 -> osd1] (item=osd1) => {"changed": true, "cmd": ["/usr/bin/env", "bash", "/tmp/restart_osd_daemon.sh"], "delta": "0:20:53.733572", "end": "2018-09-07 13:24:09.880074", "item": "osd1", "msg": "non-zero return code", "rc": 1, "start": "2018-09-07 13:03:16.146502", "stderr": "", "stderr_lines": [], "stdout": "Error while running 'ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --cluster ceph -s', PGs were not reported as active+clean\nIt is possible that the cluster has less OSDs than the replica configuration\nWill refuse to continue\n  cluster:\n    id:     XXX\n    health: HEALTH_WARN\n            1 MDSs have many clients failing to respond to cache pressure\n \n  services:\n    mon: 3 daemons, quorum mon1,mon2,mon3\n    mgr: mon1(active), standbys: mon2, mon3\n    mds: cephfs-1/1/1 up  {0=mon1=up:active}, 1 up:standby\n    osd: 188 osds: 188 up, 188 in\n \n  data:\n    pools:   3 pools, 12288 pgs\n    objects: 154M objects, 218 TB\n    usage:   656 TB used, 366 TB / 1023 TB avail\n    pgs:     12277 active+clean\n             6     active+clean+scrubbing+deep\n             5     active+clean+scrubbing\n \n  io:\n    client:   148 MB/s rd, 38526 kB/s wr, 6597 op/s rd, 175 op/s wr\n ", "stdout_lines": ["Error while running 'ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --cluster ceph -s', PGs were not reported as active+clean", "It is possible that the cluster has less OSDs than the replica configuration", "Will refuse to continue", "  cluster:", "    id:     XXX", "    health: HEALTH_WARN", "            1 MDSs have many clients failing to respond to cache pressure", " ", "  services:", "    mon: 3 daemons, quorum mon1,mon2,mon3", "    mgr: mon1(active), standbys: mon2, mon3", "    mds: cephfs-1/1/1 up  {0=mon1=up:active}, 1 up:standby", "    osd: 188 osds: 188 up, 188 in", " ", "  data:", "    pools:   3 pools, 12288 pgs", "    objects: 154M objects, 218 TB", "    usage:   656 TB used, 366 TB / 1023 TB avail", "    pgs:     12277 active+clean", "             6     active+clean+scrubbing+deep", "             5     active+clean+scrubbing", " ", "  io:", "    client:   148 MB/s rd, 38526 kB/s wr, 6597 op/s rd, 175 op/s wr", " "]}

Environment

Red Hat Ceph Storage 2
Red Hat Ceph Storage 3
ceph-ansible versions prior to version "ceph-ansible-3.1.8"

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content