thin provisioning in RHEV or on SAN ?

Latest response

I'm considering finally starting to use thin provisioning somewhere, but am uncertain about where. Ideally I'd want to do it on the SAN only, so that we have one place to monitor for never running out-of-storage.

So, if I do thin provisioning on my SAN, how will this work with RHEV images? At which point does it get allocated?

  1. On storage domain creation?
  2. When we add a disk to a VM ?
  3. When VM creates a logical volume in it's VG?
  4. When VM creates a filesystem on top of the logical volume?

 

(btw: storage domain is accessed over iSCSI, VMs are mainly RHEL5/RHEL6 -- in case that matters)

 

  -jf

ja
Log in to join the conversation

Responses

Henry Hutton's picture Red Hat Guru 4309 points
11 December 2012 4:43 PM

Thanks for your post, Jan-Frode. This is taking a bit of investigation so I wanted you to know that we're looking into it and should have a response soon.

ja Guru 1340 points
11 December 2012 7:33 PM

Good to hear it wasn't a lazy trivial question :-)  Thanks for looking into it!

Sadique Puthen's picture Red Hat Guru 4674 points
12 December 2012 7:31 AM

You can see some details about it at https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Virtualization/3.1/html-single/Technical_Reference_Guide/index.html#sect-Technical_Reference_Guide-Storage_Functions-Provisioning_Storage

Especially section "3.3.2.2. Over-commitment"

DY Guru 5340 points
17 December 2012 11:54 AM

Underlying storage is transparent to RHEV, so if you create a 100Gb LUN, that takes up 1Kb on the SAN, RHEV will see 100Gb, and use them normally. I would suggest combining sparse storage with online deduplication, and then using RHEV with fully provisioned images only. This way, sparseness and deduplication are offloaded to the array, and you do not need to worry about anything on the RHEV side, just monitor the SAN.

 

Without deduplication, thick provisioned VM disks will potentially get thick provisioned on the SAN, which pretty much beats the purpose of having the LUN sparse

ja Guru 1340 points
17 December 2012 12:58 PM

 

Without deduplication, thick provisioned VM disks will potentially get thick provisioned on the SAN, which pretty much beats the purpose of having the LUN sparse

Dan, we agree about offloading to SAN, but we're using IBM Storwize V7000, which doesn't support dedup. It does support compression, if we invest in the license for it, which should give the same effect. But you didn't really answer my question :-) What makes it thick?  SD format, VM add disk, lvcreate, mkfs.ext4 or file-write ?

 

  -jf

DY Guru 5340 points
17 December 2012 1:17 PM

It's a good question, but I'm afraid the answer is a bit complex, and it lies with what RHEV does under the hood for every action. Moreover, depending on the storage type, the action might be different. For example, provisioning a thick provisioned disk on block storage (FC/iSCSI/etc) means lvcreate, how a thin provisioned LUN responds to that is a question for the SAN vendor. With NFS, thick provisioned virtual disk image is created with qemu-img, and that fills the disk with zeroes. Zerofilling is guerranteed to remain thin on deduplicated storage, but what your specific storage will do, I, again, do not know - it might detect incoming zeroes and disregard them, or simply write them, effectively wasting space.

So if you really want to get to the bottom of this, you have two options - gather a list of all the common storage commands RHEV issues, contact the SAN vendor and inquire about the predicted behavior of a thin provisioned LUN when these commands are issued, or simply set up a demo setup, and see what happens for yourself

ja Guru 1340 points
4 January 2013 9:39 AM

I now checked this.

Created a 2 TB thin provisioned volume on the Storwize. 

Created a SD on it, connected over iSCSI.

Added a 1 TB RAW/preallocated/virt-io disk on a VM.

Created a new LVM volumegroup on it. Created a 1 TB LV.

Formatted a 1 TB ext4 filesystem inside a VM, and it's still thin.

 

Perfect!

Sadique Puthen's picture Red Hat Guru 4674 points
4 January 2013 10:49 AM

What if you create a 500G file inside the filesystem and remove it? Does the storage reclaim the space, may be running fstrim? Just curious to know. Note that you have to mount the filesystem with -o discard option.

james.radtke@siriusxm.com's picture Guru 6863 points
4 January 2013 5:36 PM

There is no single, or straight-forward answer (at this time) regarding how Thin Provisioning and storage reclamation works.  This is not specific to RHEV, you will find the same dependencies on AIX, Solaris, etc...

There are numerous factors to consider when deploying Thin provisioning and the implications of the different components.  I believe that the arrays need to work in conjunction with the Volume Manager to effectively "recover" the files which are released by the hosts.

If you create a file and popluate the file with data and then delete the file, we all know that it doesn't literally remove the data, it simply removes the pointers to the data.  The array would not know that was the case unless there was some interaction with the host.  HDS and Veritas have an ASL and you need to have a particular version installed for the Thin Provisioning and Storage Reclamation to work correctly.  EMC needs a diferent ASL.

Like Dan mentioned, you would need to work with the storage Vendor to determine how their hardware attempts to do reclamation.