Overcloud major upgrade from OSP 8 to OSP 9 fails at Installing Aodh step with error message "Could not find declared class ::aodh" in Red Hat OpenStack Platform
Issue
While following the upgrade documentation from OSP 8 to OSP 9 to upgrade a Red Hat OpenStack Platform environment which was successfully upgraded from 7 to 8 and now needs to be upgraded to 9.
Upgrade fails at step 3.4.2. Installing Aodh, with the following error:
[stack@undercloud tmp]$ heat deployment-list | grep -v COMPLETE
WARNING (shell) "heat deployment-list" is deprecated, please use "openstack software deployment list" instead
+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------+---------------------+---------------------------------------------------------------------+
| id | config_id | server_id | action | status | creation_time | status_reason |
+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------+---------------------+---------------------------------------------------------------------+
| dcce2f94-d195-48fa-aa3b-9cbadc5b0036 | f54073ec-c8e0-4564-8db9-10f1c75c9861 | f9d0b61e-9576-452c-a666-d281de1db418 | CREATE | FAILED | 2017-01-13T20:34:57 | deploy_status_code : Deployment exited with non-zero status code: 1 |
| aea2d4ee-26ff-41da-93d1-2d51ea70cc1d | 12ae463d-166b-42ce-925f-169b78a31cd5 | 952ff3eb-2ce6-4381-9622-86c370f11f51 | CREATE | FAILED | 2017-01-13T20:34:58 | deploy_status_code : Deployment exited with non-zero status code: 1 |
| e7282760-351d-446f-a65c-1bb274de2eea | d9acd13f-a6ab-4688-8af4-95a8bb105d1e | d397aa94-f6f1-4cf8-b43a-f6f6e6ad6e15 | CREATE | FAILED | 2017-01-13T20:34:59 | deploy_status_code : Deployment exited with non-zero status code: 1 |
+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------+---------------------+---------------------------------------------------------------------+
And a more detailed look at the deployments reveals:
[stack@undercloud tmp]$ heat deployment-show dcce2f94-d195-48fa-aa3b-9cbadc5b0036
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
{
"status": "FAILED",
"server_id": "f9d0b61e-9576-452c-a666-d281de1db418",
"config_id": "f54073ec-c8e0-4564-8db9-10f1c75c9861",
"output_values": {
"deploy_stdout": "",
"deploy_stderr": "Could not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass\nCould not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass\n\u001b[1;31mError: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class ::aodh at /var/lib/heat-config/heat-config-puppet/f54073ec-c8e0-4564-8db9-10f1c75c9861.pp:30 on node controller1.example.com\nWrapped exception:\nCould not find declared class ::aodh\u001b[0m\n\u001b[1;31mError: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class ::aodh at /var/lib/heat-config/heat-config-puppet/f54073ec-c8e0-4564-8db9-10f1c75c9861.pp:30 on node controller1.example.com\u001b[0m\n",
"deploy_status_code": 1
},
"creation_time": "2017-01-13T20:34:57",
"updated_time": "2017-01-13T20:35:36",
"input_values": {},
"action": "CREATE",
"status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1",
"id": "dcce2f94-d195-48fa-aa3b-9cbadc5b0036"
}
This can be fixed by applying the workaround mentioned in AODH migration fails because puppet-aodh module cannot be found by Puppet, and made the following softlink on our controller nodes, and reran this step, it worked.
# ln -f -s /usr/share/openstack-puppet/modules/* /etc/puppet/modules/
The upgrade reports completed:
Stack overcloud UPDATE_COMPLETE
Overcloud Endpoint: https://cloud.example.com:13000/v2.0
Overcloud Deployed
But aodh service is not up, even after a restart of the pcs cluster:
Failed Actions:
* openstack-heat-engine_start_0 on controller1 'not running' (7): call=279, status=complete, exitreason='none',
last-rc-change='Fri Jan 13 21:21:01 2017', queued=0ms, exec=2082ms
* openstack-aodh-evaluator_start_0 on controller1 'not running' (7): call=238, status=complete, exitreason='none',
last-rc-change='Fri Jan 13 21:20:28 2017', queued=0ms, exec=2084ms
* openstack-heat-engine_start_0 on controller0 'not running' (7): call=282, status=complete, exitreason='none',
last-rc-change='Fri Jan 13 21:21:01 2017', queued=0ms, exec=2083ms
* openstack-aodh-evaluator_start_0 on controller0 'not running' (7): call=239, status=complete, exitreason='none',
last-rc-change='Fri Jan 13 21:20:28 2017', queued=0ms, exec=2091ms
* openstack-heat-engine_start_0 on controller2 'not running' (7): call=278, status=complete, exitreason='none',
last-rc-change='Fri Jan 13 21:21:01 2017', queued=0ms, exec=2081ms
* openstack-aodh-evaluator_start_0 on controller2 'not running' (7): call=241, status=complete, exitreason='none',
last-rc-change='Fri Jan 13 21:20:28 2017', queued=0ms, exec=2084ms
Logs for aodh services on all overcloud controllers will show:
Jan 13 21:02:39 controller0.example.com aodh-notifier[14182]: AttributeError: 'Opt' object has no attribute 'group'
And the service status reveals:
[root@controller2 aodh]# systemctl status openstack-aodh-evaluator -l
● openstack-aodh-evaluator.service - OpenStack Alarm evaluator service
Loaded: loaded (/usr/lib/systemd/system/openstack-aodh-evaluator.service; disabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Tue 2017-01-17 21:22:03 UTC; 13min ago
Process: 31553 ExecStart=/usr/bin/aodh-evaluator --logfile /var/log/aodh/evaluator.log (code=exited, status=1/FAILURE)
Main PID: 31553 (code=exited, status=1/FAILURE)
Jan 17 21:22:03 controller2.example.com systemd[1]: openstack-aodh-evaluator.service: main process exited, code=exited, status=1/FAILURE
Jan 17 21:22:03 controller2.example.com systemd[1]: Unit openstack-aodh-evaluator.service entered failed state.
Jan 17 21:22:03 controller2.example.com systemd[1]: openstack-aodh-evaluator.service failed.
Jan 17 21:22:03 controller2.example.com systemd[1]: openstack-aodh-evaluator.service holdoff time over, scheduling restart.
Jan 17 21:22:03 controller2.example.com systemd[1]: start request repeated too quickly for openstack-aodh-evaluator.service
Jan 17 21:22:03 controller2.example.com systemd[1]: Failed to start OpenStack Alarm evaluator service.
Jan 17 21:22:03 controller2.example.com systemd[1]: Unit openstack-aodh-evaluator.service entered failed state.
Jan 17 21:22:03 controller2.example.com systemd[1]: openstack-aodh-evaluator.service failed.
[root@controller2 aodh]# /usr/bin/aodh-evaluator --logfile /var/log/aodh/evaluator.log
No handlers could be found for logger "oslo_config.cfg"
Traceback (most recent call last):
File "/usr/bin/aodh-evaluator", line 10, in <module>
sys.exit(evaluator())
File "/usr/lib/python2.7/site-packages/aodh/cmd/alarm.py", line 32, in evaluator
conf = service.prepare_service()
File "/usr/lib/python2.7/site-packages/aodh/service.py", line 70, in prepare_service
keystone_client.setup_keystoneauth(conf)
File "/usr/lib/python2.7/site-packages/aodh/keystone_client.py", line 145, in setup_keystoneauth
if conf[CFG_GROUP].auth_type == "password-aodh-legacy":
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2950, in __getattr__
return self._conf._get(name, self._group)
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2571, in _get
value = self._do_get(name, group, namespace)
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2608, in _do_get
return convert(opt._get_from_namespace(namespace, group_name))
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 811, in _get_from_namespace
dname, dgroup = opt.name, opt.group
AttributeError: 'Opt' object has no attribute 'group'
Environment
Red Hat OpenStack Platform 8.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
