Crerating a new iSCSI storage domain with RHEV 3.2 and storage live migration
Hello -
This makes me a little bit apprehensive. I have a RHEV VM filling up its virtual disk. The iSCSI storage domain where it lives is also filling up. The SAN has plenty of room, so I made another iSCSI volume and set up another iSCSI storage domain in my data center. This all went smoothly. With the new storage domain in place, I want to live migrate the storage for my full VM to this new storage domain. Then I'll create another virtual disk and move some of the data from the full disk to the new disk. The VM is running Windows Server 2008R2.
But now I'm afraid to pull the trigger. I have 2 hosts, named rheva and rhevb. Here's the sequence of events and why I am now afraid:
Add a new iSCSI storage domain - so now the datacenter has 2 storage domains.
Host rhevb is SPM and handles it. The process goes smoothly.
But host rheva declares itself non operational after it can't see the new storage domain.
The VM running on rheva live migrates to host rhevb.
I try to activate rheva - this fails with an error because it can't see the new iSCSI storage domain.
I put rheva into maintenance, then activate it.
This works.
OK, no disasters and no application downtime. And I think I know what's going on. Host rhevb had the SPM role and did the work, so it "knew" about the new iSCSI volume. But how to tell host rheva about the new storage? That was ugly. I have to put rheva into maintenance mode and take it out again so that RHEV-M can feed it the right commands to connect up to its new iSCSI world.
In my case, I got away with it because my surviving host has enough capacity to run all the VMs here. Barely.
What would happen if this were a large datacenter with lots of hosts? Would all the non SPM hosts have gone non-operational, unable to find this new storage? And would they all try cram their VMs onto a now overburdened SPM host?
So before I try a storage live migration on this company's email and application server, what is everyone else's experience with storage live migrations? The current virtual disk is thin provisioned - I would like to make it thick provisioned after migrating it.
thoughts?
thanks
- Greg Scott
Responses
Guru
2354 points
Greg
To be honest, my experience of storage live-migration was not a happy one. I used this feature to migrate VMs between storage solutions at the back end of 2013 and we're still clearing up resultant mess today.
Unbeknown to us at the time, storage live-migration involves the auto-generation of VM snapshots, one snapshot per vDisk assigned to your VM. As such, your newly migrated pre-allocated VM now sits on thin-provisioned storage.
Moreover, snapshots cannot be removed with the VM online. Downtime is required in order to merge snaps.
And that's where the fun began ! Too much to go into detail here but its fair to say that RH GSS did a great job of understanding the issue and the resultant fixes are in the latest release.
There is still an outstanding BZ to allow the removal/merge of snapshots with VMs online which upstream Engineering are working on.
I wouldn't want to put anyone off of using storage live-migration, as its a fantastic feature but I would advise that you upgrade to latest RHEV before attempting.
Oh, and ensure that your vDisks are configured with "Wipe after Delete" otherwise the resultant snapshot will leave metadata all over your storage domain when removed/merged, waiting to trip you up when it gets re-allocated. Subject of another BZ
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.