Ironic queries IPMI too often and exhausts sessions
Issue
- Ironic, using the ipmitool driver, queries IPMI so often that IPMI runs out of it's (16) sessions
- Ironic timeouts were tweaked, which have improved things, but the following error is still ocurring:
"Unable to establish IPMI v2 / RMCP+ session"
- The following error message will be displayed in
/var/log/ironic/ironic-conductor.log
:
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.817 1451 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:223
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.828 1451 DEBUG oslo_concurrency.processutils [-] CMD "ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status" returned: 1 in 0.011s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:254
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.829 1451 DEBUG oslo_concurrency.processutils [-] u'ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:291
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.829 1451 WARNING ironic.drivers.modules.ipmitool [-] IPMI Error encountered, retrying "ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status" for node 74a06170-4375-48a4-b9ac-5c194ab35e57. Error: Unexpected error while running command.
Sep 25 17:08:01 hostname01 ironic-conductor: Command: ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status
Sep 25 17:08:01 hostname01 ironic-conductor: Exit code: 1
Sep 25 17:08:01 hostname01 ironic-conductor: Stdout: u''
Sep 25 17:08:01 hostname01 ironic-conductor: Stderr: u'Error in open session response message : insufficient resources for session\n\nError: Unable to establish IPMI v2 / RMCP+ session\nUnable to get Chassis Power Status\n'
-
The following changes were applied:
sudo openstack-config --set /etc/nova/nova.conf DEFAULT rpc_response_timeout 600
sudo openstack-config --set /etc/ironic/ironic.conf DEFAULT rpc_response_timeout 600
sudo openstack-config --set /etc/ironic/ironic.conf conductor sync_power_state_interval 90
sudo openstack-config --set /etc/ironic/ironic.conf conductor power_state_sync_max_retries 3
sudo openstack-config --set /etc/ironic/ironic.conf ilo power_wait 15
sudo openstack-config --set /etc/ironic/ironic.conf ipmi retry_timeout 90
sudo openstack-config --set /etc/ironic/ironic.conf ipmi min_command_interval 15
sudo openstack-service restart nova
sudo openstack-service restart ironic. -
When deploying an overcloud, hosts are often in "wait / callback" state
Environment
- Red Hat OpenStack 7.X (RHOS)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.