Overcloud major upgrade from OSP 8 to OSP 9 fails at Installing Aodh step with error message "Could not find declared class ::aodh" in Red Hat OpenStack Platform

Solution In Progress - Updated -

Issue

While following the upgrade documentation from OSP 8 to OSP 9 to upgrade a Red Hat OpenStack Platform environment which was successfully upgraded from 7 to 8 and now needs to be upgraded to 9.

Upgrade fails at step 3.4.2. Installing Aodh, with the following error:

[stack@undercloud tmp]$  heat deployment-list | grep -v COMPLETE
WARNING (shell) "heat deployment-list" is deprecated, please use "openstack software deployment list" instead
+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------+---------------------+---------------------------------------------------------------------+
| id                                   | config_id                            | server_id                            | action | status   | creation_time       | status_reason                                                       |
+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------+---------------------+---------------------------------------------------------------------+
| dcce2f94-d195-48fa-aa3b-9cbadc5b0036 | f54073ec-c8e0-4564-8db9-10f1c75c9861 | f9d0b61e-9576-452c-a666-d281de1db418 | CREATE | FAILED   | 2017-01-13T20:34:57 | deploy_status_code : Deployment exited with non-zero status code: 1 |
| aea2d4ee-26ff-41da-93d1-2d51ea70cc1d | 12ae463d-166b-42ce-925f-169b78a31cd5 | 952ff3eb-2ce6-4381-9622-86c370f11f51 | CREATE | FAILED   | 2017-01-13T20:34:58 | deploy_status_code : Deployment exited with non-zero status code: 1 |
| e7282760-351d-446f-a65c-1bb274de2eea | d9acd13f-a6ab-4688-8af4-95a8bb105d1e | d397aa94-f6f1-4cf8-b43a-f6f6e6ad6e15 | CREATE | FAILED   | 2017-01-13T20:34:59 | deploy_status_code : Deployment exited with non-zero status code: 1 |
+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------+---------------------+---------------------------------------------------------------------+

And a more detailed look at the deployments reveals:

[stack@undercloud tmp]$  heat deployment-show dcce2f94-d195-48fa-aa3b-9cbadc5b0036
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
{
  "status": "FAILED",
  "server_id": "f9d0b61e-9576-452c-a666-d281de1db418",
  "config_id": "f54073ec-c8e0-4564-8db9-10f1c75c9861",
  "output_values": {
    "deploy_stdout": "",
    "deploy_stderr": "Could not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass\nCould not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass\n\u001b[1;31mError: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class ::aodh at /var/lib/heat-config/heat-config-puppet/f54073ec-c8e0-4564-8db9-10f1c75c9861.pp:30 on node controller1.example.com\nWrapped exception:\nCould not find declared class ::aodh\u001b[0m\n\u001b[1;31mError: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class ::aodh at /var/lib/heat-config/heat-config-puppet/f54073ec-c8e0-4564-8db9-10f1c75c9861.pp:30 on node controller1.example.com\u001b[0m\n",
    "deploy_status_code": 1
  },
  "creation_time": "2017-01-13T20:34:57",
  "updated_time": "2017-01-13T20:35:36",
  "input_values": {},
  "action": "CREATE",
  "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1",
  "id": "dcce2f94-d195-48fa-aa3b-9cbadc5b0036"
}

This can be fixed by applying the workaround mentioned in AODH migration fails because puppet-aodh module cannot be found by Puppet, and made the following softlink on our controller nodes, and reran this step, it worked.

# ln -f  -s /usr/share/openstack-puppet/modules/* /etc/puppet/modules/

The upgrade reports completed:

Stack overcloud UPDATE_COMPLETE
Overcloud Endpoint: https://cloud.example.com:13000/v2.0
Overcloud Deployed

But aodh service is not up, even after a restart of the pcs cluster:

Failed Actions:
* openstack-heat-engine_start_0 on controller1 'not running' (7): call=279, status=complete, exitreason='none',
    last-rc-change='Fri Jan 13 21:21:01 2017', queued=0ms, exec=2082ms
* openstack-aodh-evaluator_start_0 on controller1 'not running' (7): call=238, status=complete, exitreason='none',
    last-rc-change='Fri Jan 13 21:20:28 2017', queued=0ms, exec=2084ms
* openstack-heat-engine_start_0 on controller0 'not running' (7): call=282, status=complete, exitreason='none',
    last-rc-change='Fri Jan 13 21:21:01 2017', queued=0ms, exec=2083ms
* openstack-aodh-evaluator_start_0 on controller0 'not running' (7): call=239, status=complete, exitreason='none',
    last-rc-change='Fri Jan 13 21:20:28 2017', queued=0ms, exec=2091ms
* openstack-heat-engine_start_0 on controller2 'not running' (7): call=278, status=complete, exitreason='none',
    last-rc-change='Fri Jan 13 21:21:01 2017', queued=0ms, exec=2081ms
* openstack-aodh-evaluator_start_0 on controller2 'not running' (7): call=241, status=complete, exitreason='none',
    last-rc-change='Fri Jan 13 21:20:28 2017', queued=0ms, exec=2084ms

Logs for aodh services on all overcloud controllers will show:

Jan 13 21:02:39 controller0.example.com aodh-notifier[14182]: AttributeError: 'Opt' object has no attribute 'group'

And the service status reveals:

[root@controller2 aodh]# systemctl status openstack-aodh-evaluator -l
● openstack-aodh-evaluator.service - OpenStack Alarm evaluator service
   Loaded: loaded (/usr/lib/systemd/system/openstack-aodh-evaluator.service; disabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Tue 2017-01-17 21:22:03 UTC; 13min ago
  Process: 31553 ExecStart=/usr/bin/aodh-evaluator --logfile /var/log/aodh/evaluator.log (code=exited, status=1/FAILURE)
 Main PID: 31553 (code=exited, status=1/FAILURE)

Jan 17 21:22:03 controller2.example.com systemd[1]: openstack-aodh-evaluator.service: main process exited, code=exited, status=1/FAILURE
Jan 17 21:22:03 controller2.example.com systemd[1]: Unit openstack-aodh-evaluator.service entered failed state.
Jan 17 21:22:03 controller2.example.com systemd[1]: openstack-aodh-evaluator.service failed.
Jan 17 21:22:03 controller2.example.com systemd[1]: openstack-aodh-evaluator.service holdoff time over, scheduling restart.
Jan 17 21:22:03 controller2.example.com systemd[1]: start request repeated too quickly for openstack-aodh-evaluator.service
Jan 17 21:22:03 controller2.example.com systemd[1]: Failed to start OpenStack Alarm evaluator service.
Jan 17 21:22:03 controller2.example.com systemd[1]: Unit openstack-aodh-evaluator.service entered failed state.
Jan 17 21:22:03 controller2.example.com systemd[1]: openstack-aodh-evaluator.service failed.

[root@controller2 aodh]# /usr/bin/aodh-evaluator --logfile /var/log/aodh/evaluator.log
No handlers could be found for logger "oslo_config.cfg"
Traceback (most recent call last):
  File "/usr/bin/aodh-evaluator", line 10, in <module>
    sys.exit(evaluator())
  File "/usr/lib/python2.7/site-packages/aodh/cmd/alarm.py", line 32, in evaluator
    conf = service.prepare_service()
  File "/usr/lib/python2.7/site-packages/aodh/service.py", line 70, in prepare_service
    keystone_client.setup_keystoneauth(conf)
  File "/usr/lib/python2.7/site-packages/aodh/keystone_client.py", line 145, in setup_keystoneauth
    if conf[CFG_GROUP].auth_type == "password-aodh-legacy":
  File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2950, in __getattr__
    return self._conf._get(name, self._group)
  File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2571, in _get
    value = self._do_get(name, group, namespace)
  File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2608, in _do_get
    return convert(opt._get_from_namespace(namespace, group_name))
  File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 811, in _get_from_namespace
    dname, dgroup = opt.name, opt.group
AttributeError: 'Opt' object has no attribute 'group'

Environment

Red Hat OpenStack Platform 8.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content