Ceph - During running "ansible-playbook site.yml" on deployed cluster fails on TASK "restart ceph osds daemon(s) - non container"
Issue
We were introducing some changes on our RHCS environment running "ansible-playbook site.yml" and it keeps failing on TASK "restart ceph osds daemon(s) - non container" without obvious reason:
failed: [osd1 -> osd1] (item=osd1) => {"changed": true, "cmd": ["/usr/bin/env", "bash", "/tmp/restart_osd_daemon.sh"], "delta": "0:20:53.733572", "end": "2018-09-07 13:24:09.880074", "item": "osd1", "msg": "non-zero return code", "rc": 1, "start": "2018-09-07 13:03:16.146502", "stderr": "", "stderr_lines": [], "stdout": "Error while running 'ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --cluster ceph -s', PGs were not reported as active+clean\nIt is possible that the cluster has less OSDs than the replica configuration\nWill refuse to continue\n cluster:\n id: XXX\n health: HEALTH_WARN\n 1 MDSs have many clients failing to respond to cache pressure\n \n services:\n mon: 3 daemons, quorum mon1,mon2,mon3\n mgr: mon1(active), standbys: mon2, mon3\n mds: cephfs-1/1/1 up {0=mon1=up:active}, 1 up:standby\n osd: 188 osds: 188 up, 188 in\n \n data:\n pools: 3 pools, 12288 pgs\n objects: 154M objects, 218 TB\n usage: 656 TB used, 366 TB / 1023 TB avail\n pgs: 12277 active+clean\n 6 active+clean+scrubbing+deep\n 5 active+clean+scrubbing\n \n io:\n client: 148 MB/s rd, 38526 kB/s wr, 6597 op/s rd, 175 op/s wr\n ", "stdout_lines": ["Error while running 'ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --cluster ceph -s', PGs were not reported as active+clean", "It is possible that the cluster has less OSDs than the replica configuration", "Will refuse to continue", " cluster:", " id: XXX", " health: HEALTH_WARN", " 1 MDSs have many clients failing to respond to cache pressure", " ", " services:", " mon: 3 daemons, quorum mon1,mon2,mon3", " mgr: mon1(active), standbys: mon2, mon3", " mds: cephfs-1/1/1 up {0=mon1=up:active}, 1 up:standby", " osd: 188 osds: 188 up, 188 in", " ", " data:", " pools: 3 pools, 12288 pgs", " objects: 154M objects, 218 TB", " usage: 656 TB used, 366 TB / 1023 TB avail", " pgs: 12277 active+clean", " 6 active+clean+scrubbing+deep", " 5 active+clean+scrubbing", " ", " io:", " client: 148 MB/s rd, 38526 kB/s wr, 6597 op/s rd, 175 op/s wr", " "]}
Environment
Red Hat Ceph Storage 2
Red Hat Ceph Storage 3
ceph-ansible versions prior to version "ceph-ansible-3.1.8"
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.