After patching cinder, volumes are not coming back
Environment
Red Hat OpenStack Platform 5.0 on Red Hat Enterprise Linux 7
Issue
- After an update and reboot of a Cinder node, volumes are no longer accessible.
Updated: openstack-utils-2014.2-1.el7ost.noarch
Updated: openstack-cinder-2014.1.5-1.el7ost.noarch
Updated: openstack-selinux-0.6.43-1.el7ost.noarch
Updated: openstack-swift-1.13.1-7.el7ost.noarch
- Error message seen:
systemd: Startup finished in 1.729s (kernel) + 7.622s (initrd) + 16.275s (userspace) = 25.626s.
cinder-volume: 2015-12-10 16:45:03.214 3736 ERROR cinder.volume.manager [req-bc164ca7-f4ee-42fd-9428-7cd7c434e1ac - - - - -] Failed to re-export volume 2ec61633-af88-4e8c-95d0-c19edbd91074: setting to error state
cinder-volume: 2015-12-10 16:45:03.217 3736 ERROR cinder.volume.manager [req-bc164ca7-f4ee-42fd-9428-7cd7c434e1ac - - - - -] local variable 'volume_info' referenced before assignment
cinder-volume: 2015-12-10 16:45:03.217 3736 TRACE cinder.volume.manager Traceback (most recent call last):
cinder-volume: 2015-12-10 16:45:03.217 3736 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 265, in init_host
cinder-volume: 2015-12-10 16:45:03.217 3736 TRACE cinder.volume.manager self.driver.ensure_export(ctxt, volume)
cinder-volume: 2015-12-10 16:45:03.217 3736 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 522, in ensure_export
cinder-volume: 2015-12-10 16:45:03.217 3736 TRACE cinder.volume.manager volume_path)
cinder-volume: 2015-12-10 16:45:03.217 3736 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/iscsi.py", line 245, in ensure_export
cinder-volume: 2015-12-10 16:45:03.217 3736 TRACE cinder.volume.manager LOG.debug(_("volume_info:%s"), volume_info)
cinder-volume: 2015-12-10 16:45:03.217 3736 TRACE cinder.volume.manager UnboundLocalError: local variable 'volume_info' referenced before assignment
cinder-volume: 2015-12-10 16:45:03.217 3736 TRACE cinder.volume.manager
cinder-volume: 2015-12-10 16:45:03.299 3736 ERROR cinder.volume.manager [req-bc164ca7-f4ee-42fd-9428-7cd7c434e1ac - - - - -] Failed to re-export volume 5c55ffc1-dbf6-4ef1-9d9e-7c9c616bb1b5: setting to error state
cinder-volume: 2015-12-10 16:45:03.299 3736 ERROR cinder.volume.manager [req-bc164ca7-f4ee-42fd-9428-7cd7c434e1ac - - - - -] local variable 'volume_info' referenced before assignment
cinder-volume: 2015-12-10 16:45:03.299 3736 TRACE cinder.volume.manager Traceback (most recent call last):
cinder-volume: 2015-12-10 16:45:03.299 3736 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 265, in init_host
cinder-volume: 2015-12-10 16:45:03.299 3736 TRACE cinder.volume.manager self.driver.ensure_export(ctxt, volume)
cinder-volume: 2015-12-10 16:45:03.299 3736 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 522, in ensure_export
cinder-volume: 2015-12-10 16:45:03.299 3736 TRACE cinder.volume.manager volume_path)
cinder-volume: 2015-12-10 16:45:03.299 3736 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/iscsi.py", line 245, in ensure_export
cinder-volume: 2015-12-10 16:45:03.299 3736 TRACE cinder.volume.manager LOG.debug(_("volume_info:%s"), volume_info)
cinder-volume: 2015-12-10 16:45:03.299 3736 TRACE cinder.volume.manager UnboundLocalError: local variable 'volume_info' referenced before assignment
cinder-volume: 2015-12-10 16:45:03.299 3736 TRACE cinder.volume.manager
cinder-volume: 2015-12-10 16:45:03.375 3736 ERROR cinder.volume.manager [req-bc164ca7-f4ee-42fd-9428-7cd7c434e1ac - - - - -] Failed to re-export volume 6e4bc7d4-73d3-4648-938f-5a62b8d5af6f: setting to error state
-
Volumes are not accessible from the compute nodes
-
iSCSI Target is not running on the faulty node. However iSCSI services are running on the non-patched server
Resolution
- To recover this specific node manually create the exports:
1) Create targets:
- Look up target information in db (retrieve volumes that were set to error by cinder on restart):
$ sudo mysql -e 'select provider_location, provider_auth from cinder.volumes where deleted=0 and status="error" and provider_location is not NULL;'
That will output something like:
- With that information, construct the commands to create the targets following this pattern:
$ cinder-rtstool create /dev/$VOLUME_GROUP/$Volume $IQN $User $Password
For this example:
$ cinder-rtstool create /dev/cinder-volumes/volume-11040865-e428-45ca-a7ff-be99461b19c1 iqn.2010-10.org.openstack:volume-11040865-e428-45ca-a7ff-be99461b19c1 zvACAevwzD5mTDsrrFiC ZhCqwZ4CnxPHRWJY9P3g
$ cinder-rtstool create /dev/cinder-volumes/volume-59f17e88-4851-40d3-bd5d-5eb0f7987b4a iqn.2010-10.org.openstack:volume-59f17e88-4851-40d3-bd5d-5eb0f7987b4a 6gHs6rPmFfUrspRyEqbB wW5sg948Ro4JHNqH7KPc
2) Change status in cinder, as volumes will be in error status:
$ cinder reset-state --state in-use 11040865-e428-45ca-a7ff-be99461b19c1
$ cinder reset-state --state in-use 59f17e88-4851-40d3-bd5d-5eb0f7987b4a
3) Save configuration (in case there's a reboot before there are any changes in the targets):
$ targetcli saveconfig
- With that, volumes will be properly exported.
Root Cause
- Error in description was due to bug https://bugs.launchpad.net/cinder/1387866 that is already fixed in the next release, but in this case that was just masking another issue, that the cinder volume node was not finding on initialization the volumes to create the volume exports when using LIO.
- That issue has been found, it was only creating problems from updates from cinder versions that didn't include LIO persistence and systems that were not using target.service, and resolved.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments