Ironic queries IPMI too often and exhausts sessions

Solution In Progress - Updated -

Issue

  • Ironic, using the ipmitool driver, queries IPMI so often that IPMI runs out of it's (16) sessions
  • Ironic timeouts were tweaked, which have improved things, but the following error is still ocurring:
"Unable to establish IPMI v2 / RMCP+ session" 
  • The following error message will be displayed in /var/log/ironic/ironic-conductor.log:
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.817 1451 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:223
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.828 1451 DEBUG oslo_concurrency.processutils [-] CMD "ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status" returned: 1 in 0.011s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:254
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.829 1451 DEBUG oslo_concurrency.processutils [-] u'ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:291
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.829 1451 WARNING ironic.drivers.modules.ipmitool [-] IPMI Error encountered, retrying "ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status" for node 74a06170-4375-48a4-b9ac-5c194ab35e57. Error: Unexpected error while running command.
Sep 25 17:08:01 hostname01 ironic-conductor: Command: ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status
Sep 25 17:08:01 hostname01 ironic-conductor: Exit code: 1
Sep 25 17:08:01 hostname01 ironic-conductor: Stdout: u''
Sep 25 17:08:01 hostname01 ironic-conductor: Stderr: u'Error in open session response message : insufficient resources for session\n\nError: Unable to establish IPMI v2 / RMCP+ session\nUnable to get Chassis Power Status\n'
  • The following changes were applied:
    sudo openstack-config --set /etc/nova/nova.conf DEFAULT rpc_response_timeout 600
    sudo openstack-config --set /etc/ironic/ironic.conf DEFAULT rpc_response_timeout 600
    sudo openstack-config --set /etc/ironic/ironic.conf conductor sync_power_state_interval 90
    sudo openstack-config --set /etc/ironic/ironic.conf conductor power_state_sync_max_retries 3
    sudo openstack-config --set /etc/ironic/ironic.conf ilo power_wait 15
    sudo openstack-config --set /etc/ironic/ironic.conf ipmi retry_timeout 90
    sudo openstack-config --set /etc/ironic/ironic.conf ipmi min_command_interval 15
    sudo openstack-service restart nova
    sudo openstack-service restart ironic.

  • When deploying an overcloud, hosts are often in "wait / callback" state

Environment

  • Red Hat OpenStack 7.X (RHOS)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In