Redeployment with Ceph fails with director on task [ceph-mon : copy ceph mgr key(s) to the ansible server]

Issue

After deleting an overcloud which uses Ceph with "openstack stack delete overcloud" and then redpeloying the overcloud, the deployment fails during Ceph configuration with a checksum mismatch of the ceph.mgr.controller-1.keyring between the undercloud and overcloud.

The ceph-ansible error log shows the following:

2019-05-09 11:21:14,375 p=248301 u=root |  TASK [ceph-mon : copy ceph mgr key(s) to the ansible server] *******************
2019-05-09 11:21:14,375 p=248301 u=root |  task path: /usr/share/ceph-ansible/roles/ceph-mon/tasks/ceph_keys.yml:56
2019-05-09 11:21:14,375 p=248301 u=root |  Thursday 09 May 2019  11:21:14 +0000 (0:00:06.029)       0:02:50.826 **********
2019-05-09 11:21:14,397 p=248301 u=root |  [DEPRECATION WARNING]: evaluating cephx as a bare variable, this behaviour will
 go away and you might need to add |bool to the expression in the future. Also
see CONDITIONAL_BARE_VARS configuration toggle.. This feature will be removed
in version 2.12. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
2019-05-09 11:21:14,496 p=248301 u=root |  [DEPRECATION WARNING]: evaluating cephx as a bare variable, this behaviour will
 go away and you might need to add |bool to the expression in the future. Also
see CONDITIONAL_BARE_VARS configuration toggle.. This feature will be removed
in version 2.12. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
2019-05-09 11:21:14,611 p=248301 u=root |  [DEPRECATION WARNING]: evaluating cephx as a bare variable, this behaviour will
 go away and you might need to add |bool to the expression in the future. Also
see CONDITIONAL_BARE_VARS configuration toggle.. This feature will be removed
in version 2.12. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
2019-05-09 11:21:14,970 p=248301 u=root |  failed: [controller-1] (item=controller-0) => changed=false
  ansible_loop_var: item
  checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709
  dest: /var/lib/mistral/overcloud/ceph-ansible/fetch_dir/9ee7e954-71ca-11e9-954c-52540090cf22/etc/ceph/ceph.mgr.controller-0.keyring
  file: /etc/ceph/ceph.mgr.controller-0.keyring
  item: controller-0
  md5sum: 82686cfa5c69a8b0cd998dc0fe7472b3
  msg: **checksum mismatch**
  remote_checksum: f62278db35e0285d76e10e614ac98a7c0ca7772e
  remote_md5sum: null
2019-05-09 11:21:14,974 p=248301 u=root |  changed: [controller-2] => (item=controller-0) => changed=true
  ansible_loop_var: item
  checksum: f62278db35e0285d76e10e614ac98a7c0ca7772e
  dest: /var/lib/mistral/overcloud/ceph-ansible/fetch_dir/9ee7e954-71ca-11e9-954c-52540090cf22/etc/ceph/ceph.mgr.controller-0.keyring
  item: controller-0
  md5sum: 82686cfa5c69a8b0cd998dc0fe7472b3
  remote_checksum: f62278db35e0285d76e10e614ac98a7c0ca7772e
  remote_md5sum: null
2019-05-09 11:21:14,977 p=248301 u=root |  changed: [controller-0] => (item=controller-0) => changed=true
  ansible_loop_var: item
  checksum: f62278db35e0285d76e10e614ac98a7c0ca7772e
  dest: /var/lib/mistral/overcloud/ceph-ansible/fetch_dir/9ee7e954-71ca-11e9-954c-52540090cf22/etc/ceph/ceph.mgr.controller-0.keyring
  item: controller-0
  md5sum: d41d8cd98f00b204e9800998ecf8427e
  remote_checksum: f62278db35e0285d76e10e614ac98a7c0ca7772e
  remote_md5sum: null
2019-05-09 11:21:15,298 p=248301 u=root |  **failed**: [controller-1] (item=controller-1) => changed=false
  ansible_loop_var: item
  checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709
  dest: /var/lib/mistral/overcloud/ceph-ansible/fetch_dir/9ee7e954-71ca-11e9-954c-52540090cf22/etc/ceph/ceph.mgr.controller-1.keyring
  file: /etc/ceph/ceph.mgr.controller-1.keyring
  item: controller-1
  md5sum: 15ca008c99bd13347e64c214be8911ea
  msg: checksum mismatch
  remote_checksum: 41d1d87c63ed4499a122f658070bd78d74700bb6
  remote_md5sum: null
2019-05-09 11:21:15,300 p=248301 u=root |  changed: [controller-2] => (item=controller-1) => changed=true
  ansible_loop_var: item
  checksum: 41d1d87c63ed4499a122f658070bd78d74700bb6
  dest: /var/lib/mistral/overcloud/ceph-ansible/fetch_dir/9ee7e954-71ca-11e9-954c-52540090cf22/etc/ceph/ceph.mgr.controller-1.keyring
  item: controller-1
  md5sum: 15ca008c99bd13347e64c214be8911ea
  remote_checksum: 41d1d87c63ed4499a122f658070bd78d74700bb6
  remote_md5sum: null
2019-05-09 11:21:15,302 p=248301 u=root |  **failed**: [controller-0] (item=controller-1) => changed=false
  ansible_loop_var: item
  checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709
  dest: /var/lib/mistral/overcloud/ceph-ansible/fetch_dir/9ee7e954-71ca-11e9-954c-52540090cf22/etc/ceph/ceph.mgr.controller-1.keyring
  file: /etc/ceph/ceph.mgr.controller-1.keyring
  item: controller-1
  md5sum: 15ca008c99bd13347e64c214be8911ea
  msg: checksum mismatch
  remote_md5sum: null
2019-05-09 11:21:15,621 p=248301 u=root |  changed: [controller-1] => (item=controller-2) => changed=true
  ansible_loop_var: item
  checksum: b53e8c63523da07cea8bbc170a243643f2b5c9a7
  dest: /var/lib/mistral/overcloud/ceph-ansible/fetch_dir/9ee7e954-71ca-11e9-954c-52540090cf22/etc/ceph/ceph.mgr.controller-2.keyring
  item: controller-2
  md5sum: cab5cd93457c327cfed132d99084dfb8
  remote_checksum: b53e8c63523da07cea8bbc170a243643f2b5c9a7
  remote_md5sum: null
2019-05-09 11:21:15,623 p=248301 u=root |  ok: [controller-2] => (item=controller-2) => changed=false
  ansible_loop_var: item
  checksum: b53e8c63523da07cea8bbc170a243643f2b5c9a7
  dest: /var/lib/mistral/overcloud/ceph-ansible/fetch_dir/9ee7e954-71ca-11e9-954c-52540090cf22/etc/ceph/ceph.mgr.controller-2.keyring
  file: /etc/ceph/ceph.mgr.controller-2.keyring
  item: controller-2
  md5sum: cab5cd93457c327cfed132d99084dfb8
2019-05-09 11:21:15,624 p=248301 u=root |  changed: [controller-0] => (item=controller-2) => changed=true
  ansible_loop_var: item
  checksum: b53e8c63523da07cea8bbc170a243643f2b5c9a7
  dest: /var/lib/mistral/overcloud/ceph-ansible/fetch_dir/9ee7e954-71ca-11e9-954c-52540090cf22/etc/ceph/ceph.mgr.controller-2.keyring
  item: controller-2
  md5sum: cab5cd93457c327cfed132d99084dfb8
  remote_checksum: b53e8c63523da07cea8bbc170a243643f2b5c9a7
  remote_md5sum: null
2019-05-09 11:21:15,628 p=248301 u=root |  NO MORE HOSTS LEFT *************************************************************
2019-05-09 11:21:15,630 p=248301 u=root |  PLAY RECAP *********************************************************************
2019-05-09 11:21:15,630 p=248301 u=root |  ceph-0                     : ok=37   changed=0    unreachable=0    failed=0    skipped=82   rescued=0    ignored=0                                                                                                                 
2019-05-09 11:21:15,630 p=248301 u=root |  ceph-1                     : ok=37   changed=0    unreachable=0    failed=0    skipped=82   rescued=0    ignored=0                                                                                                                 
2019-05-09 11:21:15,630 p=248301 u=root |  ceph-2                     : ok=37   changed=0    unreachable=0    failed=0    skipped=82   rescued=0    ignored=0                                                                                                                 
2019-05-09 11:21:15,630 p=248301 u=root |  compute-0                  : ok=31   changed=0    unreachable=0    failed=0    skipped=86   rescued=0    ignored=0                                                                                                                 
2019-05-09 11:21:15,630 p=248301 u=root |  controller-0               : ok=98   changed=13   unreachable=0    **failed=1**    skipped=179  rescued=0    ignored=0                                                                                                                 
2019-05-09 11:21:15,630 p=248301 u=root |  controller-1               : ok=90   changed=11   unreachable=0    **failed=1**    skipped=173  rescued=0    ignored=0                                                                                                                 
2019-05-09 11:21:15,631 p=248301 u=root |  controller-2               : ok=91   changed=12   unreachable=0    failed=0    skipped=173  rescued=0    ignored=0                                                                                                                 
2019-05-09 11:21:15,631 p=248301 u=root |  INSTALLER STATUS ***************************************************************
2019-05-09 11:21:15,634 p=248301 u=root |  Install Ceph Monitor        : In Progress (0:01:34)