Ironic queries IPMI too often and exhausts sessions

Solution In Progress - Updated -

Issue

  • Ironic, using the ipmitool driver, queries IPMI so often that IPMI runs out of it's (16) sessions
  • Ironic timeouts were tweaked, which have improved things, but the following error is still ocurring:
"Unable to establish IPMI v2 / RMCP+ session" 
  • The following error message will be displayed in /var/log/ironic/ironic-conductor.log:
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.817 1451 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:223
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.828 1451 DEBUG oslo_concurrency.processutils [-] CMD "ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status" returned: 1 in 0.011s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:254
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.829 1451 DEBUG oslo_concurrency.processutils [-] u'ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:291
Sep 25 17:08:01 hostname01 ironic-conductor: 2015-09-25 17:08:01.829 1451 WARNING ironic.drivers.modules.ipmitool [-] IPMI Error encountered, retrying "ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status" for node 74a06170-4375-48a4-b9ac-5c194ab35e57. Error: Unexpected error while running command.
Sep 25 17:08:01 hostname01 ironic-conductor: Command: ipmitool -I lanplus -H 10.10.10.10 -L ADMINISTRATOR -U ADMIN -R 6 -N 15 -f /tmp/tmpCeEmxP power status
Sep 25 17:08:01 hostname01 ironic-conductor: Exit code: 1
Sep 25 17:08:01 hostname01 ironic-conductor: Stdout: u''
Sep 25 17:08:01 hostname01 ironic-conductor: Stderr: u'Error in open session response message : insufficient resources for session\n\nError: Unable to establish IPMI v2 / RMCP+ session\nUnable to get Chassis Power Status\n'
  • The following changes were applied:
    sudo openstack-config --set /etc/nova/nova.conf DEFAULT rpc_response_timeout 600
    sudo openstack-config --set /etc/ironic/ironic.conf DEFAULT rpc_response_timeout 600
    sudo openstack-config --set /etc/ironic/ironic.conf conductor sync_power_state_interval 90
    sudo openstack-config --set /etc/ironic/ironic.conf conductor power_state_sync_max_retries 3
    sudo openstack-config --set /etc/ironic/ironic.conf ilo power_wait 15
    sudo openstack-config --set /etc/ironic/ironic.conf ipmi retry_timeout 90
    sudo openstack-config --set /etc/ironic/ironic.conf ipmi min_command_interval 15
    sudo openstack-service restart nova
    sudo openstack-service restart ironic.

  • When deploying an overcloud, hosts are often in "wait / callback" state

Environment

  • Red Hat OpenStack 7.X (RHOS)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content