Overcloud major upgrade from OSP 8 to OSP 9 fails at Installing Aodh step with error message "Could not find declared class ::aodh" in Red Hat OpenStack Platform
Issue
While following the upgrade documentation from OSP 8 to OSP 9 to upgrade a Red Hat OpenStack Platform environment which was successfully upgraded from 7 to 8 and now needs to be upgraded to 9.
Upgrade fails at step 3.4.2. Installing Aodh
, with the following error:
[stack@undercloud tmp]$ heat deployment-list | grep -v COMPLETE
WARNING (shell) "heat deployment-list" is deprecated, please use "openstack software deployment list" instead
+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------+---------------------+---------------------------------------------------------------------+
| id | config_id | server_id | action | status | creation_time | status_reason |
+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------+---------------------+---------------------------------------------------------------------+
| dcce2f94-d195-48fa-aa3b-9cbadc5b0036 | f54073ec-c8e0-4564-8db9-10f1c75c9861 | f9d0b61e-9576-452c-a666-d281de1db418 | CREATE | FAILED | 2017-01-13T20:34:57 | deploy_status_code : Deployment exited with non-zero status code: 1 |
| aea2d4ee-26ff-41da-93d1-2d51ea70cc1d | 12ae463d-166b-42ce-925f-169b78a31cd5 | 952ff3eb-2ce6-4381-9622-86c370f11f51 | CREATE | FAILED | 2017-01-13T20:34:58 | deploy_status_code : Deployment exited with non-zero status code: 1 |
| e7282760-351d-446f-a65c-1bb274de2eea | d9acd13f-a6ab-4688-8af4-95a8bb105d1e | d397aa94-f6f1-4cf8-b43a-f6f6e6ad6e15 | CREATE | FAILED | 2017-01-13T20:34:59 | deploy_status_code : Deployment exited with non-zero status code: 1 |
+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------+---------------------+---------------------------------------------------------------------+
And a more detailed look at the deployments reveals:
[stack@undercloud tmp]$ heat deployment-show dcce2f94-d195-48fa-aa3b-9cbadc5b0036
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
{
"status": "FAILED",
"server_id": "f9d0b61e-9576-452c-a666-d281de1db418",
"config_id": "f54073ec-c8e0-4564-8db9-10f1c75c9861",
"output_values": {
"deploy_stdout": "",
"deploy_stderr": "Could not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass\nCould not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass\n\u001b[1;31mError: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class ::aodh at /var/lib/heat-config/heat-config-puppet/f54073ec-c8e0-4564-8db9-10f1c75c9861.pp:30 on node controller1.example.com\nWrapped exception:\nCould not find declared class ::aodh\u001b[0m\n\u001b[1;31mError: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class ::aodh at /var/lib/heat-config/heat-config-puppet/f54073ec-c8e0-4564-8db9-10f1c75c9861.pp:30 on node controller1.example.com\u001b[0m\n",
"deploy_status_code": 1
},
"creation_time": "2017-01-13T20:34:57",
"updated_time": "2017-01-13T20:35:36",
"input_values": {},
"action": "CREATE",
"status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1",
"id": "dcce2f94-d195-48fa-aa3b-9cbadc5b0036"
}
This can be fixed by applying the workaround mentioned in AODH migration fails because puppet-aodh module cannot be found by Puppet, and made the following softlink on our controller nodes, and reran this step, it worked.
# ln -f -s /usr/share/openstack-puppet/modules/* /etc/puppet/modules/
The upgrade reports completed:
Stack overcloud UPDATE_COMPLETE
Overcloud Endpoint: https://cloud.example.com:13000/v2.0
Overcloud Deployed
But aodh
service is not up, even after a restart of the pcs cluster:
Failed Actions:
* openstack-heat-engine_start_0 on controller1 'not running' (7): call=279, status=complete, exitreason='none',
last-rc-change='Fri Jan 13 21:21:01 2017', queued=0ms, exec=2082ms
* openstack-aodh-evaluator_start_0 on controller1 'not running' (7): call=238, status=complete, exitreason='none',
last-rc-change='Fri Jan 13 21:20:28 2017', queued=0ms, exec=2084ms
* openstack-heat-engine_start_0 on controller0 'not running' (7): call=282, status=complete, exitreason='none',
last-rc-change='Fri Jan 13 21:21:01 2017', queued=0ms, exec=2083ms
* openstack-aodh-evaluator_start_0 on controller0 'not running' (7): call=239, status=complete, exitreason='none',
last-rc-change='Fri Jan 13 21:20:28 2017', queued=0ms, exec=2091ms
* openstack-heat-engine_start_0 on controller2 'not running' (7): call=278, status=complete, exitreason='none',
last-rc-change='Fri Jan 13 21:21:01 2017', queued=0ms, exec=2081ms
* openstack-aodh-evaluator_start_0 on controller2 'not running' (7): call=241, status=complete, exitreason='none',
last-rc-change='Fri Jan 13 21:20:28 2017', queued=0ms, exec=2084ms
Logs for aodh services on all overcloud controllers will show:
Jan 13 21:02:39 controller0.example.com aodh-notifier[14182]: AttributeError: 'Opt' object has no attribute 'group'
And the service status reveals:
[root@controller2 aodh]# systemctl status openstack-aodh-evaluator -l
● openstack-aodh-evaluator.service - OpenStack Alarm evaluator service
Loaded: loaded (/usr/lib/systemd/system/openstack-aodh-evaluator.service; disabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Tue 2017-01-17 21:22:03 UTC; 13min ago
Process: 31553 ExecStart=/usr/bin/aodh-evaluator --logfile /var/log/aodh/evaluator.log (code=exited, status=1/FAILURE)
Main PID: 31553 (code=exited, status=1/FAILURE)
Jan 17 21:22:03 controller2.example.com systemd[1]: openstack-aodh-evaluator.service: main process exited, code=exited, status=1/FAILURE
Jan 17 21:22:03 controller2.example.com systemd[1]: Unit openstack-aodh-evaluator.service entered failed state.
Jan 17 21:22:03 controller2.example.com systemd[1]: openstack-aodh-evaluator.service failed.
Jan 17 21:22:03 controller2.example.com systemd[1]: openstack-aodh-evaluator.service holdoff time over, scheduling restart.
Jan 17 21:22:03 controller2.example.com systemd[1]: start request repeated too quickly for openstack-aodh-evaluator.service
Jan 17 21:22:03 controller2.example.com systemd[1]: Failed to start OpenStack Alarm evaluator service.
Jan 17 21:22:03 controller2.example.com systemd[1]: Unit openstack-aodh-evaluator.service entered failed state.
Jan 17 21:22:03 controller2.example.com systemd[1]: openstack-aodh-evaluator.service failed.
[root@controller2 aodh]# /usr/bin/aodh-evaluator --logfile /var/log/aodh/evaluator.log
No handlers could be found for logger "oslo_config.cfg"
Traceback (most recent call last):
File "/usr/bin/aodh-evaluator", line 10, in <module>
sys.exit(evaluator())
File "/usr/lib/python2.7/site-packages/aodh/cmd/alarm.py", line 32, in evaluator
conf = service.prepare_service()
File "/usr/lib/python2.7/site-packages/aodh/service.py", line 70, in prepare_service
keystone_client.setup_keystoneauth(conf)
File "/usr/lib/python2.7/site-packages/aodh/keystone_client.py", line 145, in setup_keystoneauth
if conf[CFG_GROUP].auth_type == "password-aodh-legacy":
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2950, in __getattr__
return self._conf._get(name, self._group)
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2571, in _get
value = self._do_get(name, group, namespace)
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2608, in _do_get
return convert(opt._get_from_namespace(namespace, group_name))
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 811, in _get_from_namespace
dname, dgroup = opt.name, opt.group
AttributeError: 'Opt' object has no attribute 'group'
Environment
Red Hat OpenStack Platform 8.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.