Upgrade from OSP8 to OSP9 fails on Installing Aodh

Issue

• Upgrade fails from RHOSP8 to RHOSP9 on section 3.4.2. Installing Aodh from the Upgrade guide
• This happens after first running the upgrade deploy, it failed because a pacemaker resource was not in a stopped state. Fix that in pacemaker then re-deploy. This time the update script fails because ceilometer alarm evaluator is not present in pcs.

[stack@cctg-aci-cloud-host1 ~]$ heat resource-list a5a8f420-86e9-4e81-8c11-4226917d5c77
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+---------------+--------------------------------------+------------------------------+-----------------+---------------------+
| resource_name | physical_resource_id                 | resource_type                | resource_status | updated_time        |
+---------------+--------------------------------------+------------------------------+-----------------+---------------------+
| 1             | e2b66dc0-ddd4-4401-96d1-e6fe22a563f9 | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-10-05T19:14:59 |
| 2             | dfc72353-aa10-4f13-b7a0-e9531d6fae59 | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-10-05T19:14:59 |
| 0             | fa29ee25-457c-44ff-89ba-29c9b82c17fa | OS::Heat::SoftwareDeployment | CREATE_FAILED   | 2016-10-05T20:09:48 |
+---------------+--------------------------------------+------------------------------+-----------------+---------------------+
[stack@cctg-aci-cloud-host1 ~]$ heat deployment-show  fa29ee25-457c-44ff-89ba-29c9b82c17fa
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
{
  "status": "FAILED", 
  "server_id": "e393527a-042b-4cac-ae3f-660595551211", 
  "config_id": "9d949012-129a-4372-964c-3a54e18fd3ed", 
  "output_values": {
    "deploy_stdout": " Clone Set: openstack-ceilometer-alarm-notifier-clone [openstack-ceilometer-alarm-notifier] (unmanaged)\n", 
    "deploy_stderr": "Error: unable to find a resource/clone/master/group: openstack-ceilometer-alarm-evaluator\n", 
    "deploy_status_code": 1
  }, 
  "creation_time": "2016-10-05T20:09:48", 
  "updated_time": "2016-10-05T20:10:41", 
  "input_values": {}, 
  "action": "CREATE", 
  "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", 
  "id": "fa29ee25-457c-44ff-89ba-29c9b82c17fa"