Ironic queries IPMI too often and exhausts sessions
Issue
- Ironic, using the ipmitool driver, queries IPMI so often that IPMI runs out of it's (16) sessions
- Ironic timeouts were tweaked, which have improved things, but the following error is still ocurring:
"Unable to establish IPMI v2 / RMCP+ session"
- The following error message will be displayed in
/var/log/ironic/ironic-conductor.log:
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.817 1451 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:223
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.828 1451 DEBUG oslo_concurrency.processutils [-] CMD "ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status" returned: 1 in 0.011s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:254
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.829 1451 DEBUG oslo_concurrency.processutils [-] u'ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:291
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.829 1451 WARNING ironic.drivers.modules.ipmitool [-] IPMI Error encountered, retrying "ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status" for node 74a06170-4375-48a4-b9ac-5c194ab35e57. Error: Unexpected error while running command.
Sep 25 17:08:01 hostname01 ironic-conductor: Command: ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status
Sep 25 17:08:01 hostname01 ironic-conductor: Exit code: 1
Sep 25 17:08:01 hostname01 ironic-conductor: Stdout: u''
Sep 25 17:08:01 hostname01 ironic-conductor: Stderr: u'Error in open session response message : insufficient resources for session\n\nError: Unable to establish IPMI v2 / RMCP+ session\nUnable to get Chassis Power Status\n'
-
The following changes were applied:
sudo openstack-config --set /etc/nova/nova.conf DEFAULT rpc_response_timeout 600
sudo openstack-config --set /etc/ironic/ironic.conf DEFAULT rpc_response_timeout 600
sudo openstack-config --set /etc/ironic/ironic.conf conductor sync_power_state_interval 90
sudo openstack-config --set /etc/ironic/ironic.conf conductor power_state_sync_max_retries 3
sudo openstack-config --set /etc/ironic/ironic.conf ilo power_wait 15
sudo openstack-config --set /etc/ironic/ironic.conf ipmi retry_timeout 90
sudo openstack-config --set /etc/ironic/ironic.conf ipmi min_command_interval 15
sudo openstack-service restart nova
sudo openstack-service restart ironic. -
When deploying an overcloud, hosts are often in "wait / callback" state
Environment
- Red Hat OpenStack 7.X (RHOS)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
