RedHat, Storage and Operating as a VM

Latest response

The organization I work for deploys RedHat primarily as virtualized instances. These instances are deployed primarily from standard templates. These templates, obviously, are created with specific sizes of virtual disk for the root drive and specific partition layouts.

 

Ocasionally, these RedHat-containing VMs are not properly sized for their ultimate use. Tools like VMware make this not too much of a problem: you can add additional virtual disks or up the size of  existing virtual disks. Even better, you can do it "on-the-fly" (without having to restart the VM). RedHat handles on-the-fly adding and removing of disks decently well (a little more manual than is ideal, but still workable).

 

Unfortunately, I've not really found a good way for handling on-the-fly changes to the size of existing virtual disks. I can change the size of a disk all day and RedHat won't notice it. I can force RedHat to notice the geometry change if I use the `blockdev` command. Unfortunately, if there are active filesystems on the disk or active LVMs on the disk, `blockdev`  errors out with a "Device Busy" message. If `blockdev` can't function, RedHat won't see the geometry change. If RedHat doesn't see the geometry change, I can't modify the disk's partition table to make the grown disk's storage available.

 

It would be nice to be able to resize a disk, on-the-fly, and have RedHat be able to see the new geometry. In the case of boot disks, it would save having to reboot the system. In the case of application data disks, it would (potentially) save having to take the application offline.

 

Am I missing some bit of magic in RHEL that would allow me to do things fully on-the-fly?

TJ
Log in to join the conversation

Responses

Sadique Puthen's picture Red Hat Guru 4679 points
4 May 2011 6:06 AM

It's recommended to add additional disks and expand the VG and LV inside the guest to resize storage. RHEL6 KVM and RHEL5 Xen supports hot adding disks. So no need to reboot the guest.

 

Getting the guest recognize the geometry change after resizing the disk without rebooting the guest is tricky and may not work.

TJ Guru 1883 points
4 May 2011 11:38 AM

Getting the guest to recognize geometry changes, if you're able to take the LVs and/or mounted filesystems offline, is trivial and has worked 100% of the time I've had to do it. Getting the guest to recognize geometry change of disks with active LVs or mounted filesystems is the challenge. Given the storage management policies I'm working under in our VMware environment, expansion of existing VMDKs is preferred to presenting additional VMDKs to the system. Thus, wanting to see if there's any magic I'm missing.

LA Active Contributor 105 points
5 May 2011 10:01 AM

I'm just a Linux techie but our company has a network group which sets up RHEL5 on VMware. Recently, I was aksed if an existing RHEL5 server which lived on VMware can be moved via image to HyperV environment. I don't know VMware or HyperV, so I don't know if this move will cause any issues. I was told that apart from the video driver which seems to have re-jiggered itself somehow and now works, everything seems OK. I was a bit concerned about RHN registration and wondered if this may cause YUM to download the wrong patches to a now migrated HyperV environment? The network guy is adamant that the move by image works OK. Personally I'd have installed a clean OS on HyperV and move the apps and data across but was told that that's too slow. If anybody had tried this method of migrating an existing RHEL5 OS, I'd be interested to hear about their experiences of any problems etc. Thanks. Lina

TJ Guru 1883 points
5 May 2011 12:18 PM

At any rate, the value to moving your VM from one environment to another is that you shouldn't have to re-register all the software. When you build a new system and move data, that new system will usually have a new system ID (there's ways of forcing that not to be the case, but, it's frequently frowned upon by the various vendors). Any node-locked licensing or features you have will use that system ID and have to be re-licensed to use the new system ID. Depending on your licensing agreements and the vendor that has sold you your RTUs,  re-licensing may not be free. If your systems are in RHN, you'll probably have to de-register the migrated (rebuilt) system and re-register it.

 

Drivers should take care of themselves. For the most part, when you export your VM's files from VMware and import them into another virtualization solution, the import utilities should resolve most hypervisor differences. Depending on how tied your device configurations are to the system (e.g., whether you use things like NPIV or setting HWADDR values in your network scripts and whether you use generic device drivers or the VMware-optimized drivers), you might have to reconfigure some of your boot scripts and other configuration items. But, again, this is what a testing environment is for.

Phil Jensen's picture Guru 3262 points
19 May 2011 2:30 AM

After adjusting the geometry of the VM's disk, consider using 'partprobe' to rescan the partition table on the disk.

 

Phil Jensen, RHCA

Sr. Operations Engineer

Proofpoint, Inc.

TJ Guru 1883 points
19 May 2011 12:07 PM

I've already presented an 8GB VMDK to the system. Just setting it up, as follows:

# fdisk /dev/sdd
Device contains neither a valid DOS partition table, nor Sun, SGI or
OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 1044.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4):
Value out of range.
Partition number (1-4): 1
First cylinder (1-1044, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1044, default 1044):
Using default value 1044

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
# mkdir /mnt/TEST
# mkfs -t ext3 /dev/sdd1
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1048576 inodes, 2096474 blocks
104823 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2147483648
64 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 22 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
# mount -t ext3 /dev/sdd1 /mnt/TEST

At this point, the disk is only mounted but there are no I/Os to the device. I go into vCenter and grow the disk (doubling the size from 8GB to 16GB. I then rescan the bus just to be super-sure that the kernel has a chance to notice any changes...

# echo "- - -" >  /sys/class/scsi_host/host2/scan
# echo "- - -" >  /sys/class/scsi_host/host1/scan
# echo "- - -" >  /sys/class/scsi_host/host0/scan

I then run fdisk to ensure that the original geometry (1044 cylinders) is still seen by the kernel

# fdisk -l /dev/sdd

Disk /dev/sdd: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1        1044     8385898+  83  Linux

I then attempt to tell the kernel, "look for geometry changes"

# blockdev --rereadpt /dev/sdd
BLKRRPART: Device or resource busy

Per usual, `blockdev` fails with a "busy" error. So, I try the suggestion of using `partprobe`

# partprobe /dev/sdd

From the standpoint of not getting errors, `partprobe` looks good ...until you use fdisk to look at the disk geometry:

# fdisk -l /dev/sdd

Disk /dev/sdd: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1        1044     8385898+  83  Linux
#

Oops: no change (still 1044 cylinders). So, I figure, "maybe partprobe fails silently: I'll dismount the filesystem and rerun..."

# umount /mnt/TEST
# fdisk -l /dev/sdd

Disk /dev/sdd: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1        1044     8385898+  83  Linux
# partprobe /dev/sdd
# fdisk -l /dev/sdd

Disk /dev/sdd: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1        1044     8385898+  83  Linux

Nope: partprobe still fails to update the geometry (still 1044 cylinders), even though the filesystem is now dismounted. So, I go to what I know works, `blockdev`:

# blockdev --rereadpt /dev/sdd
# fdisk -l /dev/sdd

Disk /dev/sdd: 17.1 GB, 17179869184 bytes
255 heads, 63 sectors/track, 2088 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1        1044     8385898+  83  Linux
#

As you can see, after running `blockdev`, the OS now sees that the disk has doubled in size (cylinder count goes from 1044 to 2088).

UG Community Member 23 points
20 May 2011 2:18 PM

Thomas,

 

This is a subject I've struggled with myself ever since we started deploying RHEL on VMware a year ago. I've yet to find a workable solution to extending the vmdk and having linux pick it up (without the need to umount, or reboot).  For now, the only enterprise solution I have for my clients is to add more and more vmdk's when more storage is needed. I have no problem extending the lv and fs when needed in this way. Its rather embarrasing though because our Windows admins ARE able to extend the vmdk and Windows is able to extend on the fly with no reboots.  Any advice here, or improvements on this subject would be great!

TJ Guru 1883 points
20 May 2011 10:10 PM

Figured that, since this forum operates off RedHat's support portal, that  maybe someone at RedHat could tell me if there's any voodoo I've yet to find.

 

Thus far, it just seems that that bit of enterprise-grade that you find in commercial UNIX releases and Windows hasn't quite made it into Linux. Oh well: one can home for "someday".