Red Hat Training

A Red Hat training course is available for Red Hat Enterprise Linux

Chapter 24. Using shared storage with virtual disk images

This chapter covers the use of shared and network storage devices for virtual disks.

24.1. Using iSCSI for storing virtual disk images

This section demonstrates how to set up an iSCSI target on Red Hat Enterprise Linux and how to configure iSCSI on a libvirt KVM host using virsh, and finally how to provision a guest on iSCSI using virt-install.

Important

Setting up a Red Hat Enterprise Linux server as an iSCSI target is not recommended. The example used in this section should not be used in production, and is provided as an example which should only be referred to for basic shared storage testing and educational purposes.

24.1.1. How to set up an iSCSI target on Red Hat Enterprise Linux

  1. Install and enable the iSCSI target service

    Install and enable the iSCSI target service with the following commands:
    # yum install scsi-target-utils
    # chkconfig tgtd on
    # service tgtd start
    

    Important

    The scsi-target-utils package required for this example is provided only in the Cluster Storage add-on for Red Hat Enterprise Linux 5, and may not be available for your system. Contact your support agent to activate this add-on if you are currently unable to install this package.
  2. Allocate storage for the LUNs

    The iSCSI target service is not dependent on a particular type of exported LUN. The LUNs can be plain files, LVM volumes, or block devices. There is however a performance overhead if using the LVM and/or file system layers as compared to block devices. The guest providing the iSCSI service in this example has no spare block or LVM devices, so raw files will be used.
    The following commands demonstrate the creation of two LUNs; one is 10GB (sparse file), the other is 500MB (fully-allocated). Be aware that your environment will be different and this is provided only as an example:
    # mkdir -p /var/lib/tgtd/kvmguest
    # dd if=/dev/zero of=/var/lib/tgtd/kvmguest/rhelx86_64.img bs=1M seek=10240 count=0
    # dd if=/dev/zero of=/var/lib/tgtd/kvmguest/shareddata.img bs=1M count=512
    # restorecon -R /var/lib/tgtd
    
  3. Export the iSCSI target and LUNs

    For Red Hat Enterprise Linux 5, several tgtadm commands are required to create a target and associate the storage volumes created earlier. First, the following command adds a target using an iSCSI Qualified Name (IQN):
    # tgtadm --lld iscsi --op new --mode target --tid 1 --targetname \ 
    iqn.2004-04.rhel:rhel5:iscsi.kvmguest
    
    Now the storage volumes must be associated with LUNs in the iSCSI target with these two commands:
    # tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 \
    --backing-store /var/lib/tgtd/kvmguest/rhelx86_64.img
    
    # tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 2 \
    --backing-store /var/lib/tgtd/kvmguest/shareddata.img
    

    Important

    Add the previous 3 tgtadm commands to the end of the /etc/rc.local file to ensure normal operation upon restarting of the system.
    To confirm the successful operation of the previous commands, query the iSCSI target setup:
    tgtadm --lld iscsi --op show --mode target
    Target 1: iqn.2004-04.rhel:rhel5:iscsi.kvmguest
        System information:
            Driver: iscsi
            State: ready
        I_T nexus information:
        LUN information:
            LUN: 0
                Type: controller
                SCSI ID: IET     00010000
                SCSI SN: beaf10
                Size: 0 MB, Block size: 1
                Online: Yes
                Removable media: No
                Readonly: No
                Backing store type: null
                Backing store path: None
                Backing store flags: 
            LUN: 1
                Type: disk
                SCSI ID: IET     00010001
                SCSI SN: beaf11
                Size: 10737 MB, Block size: 1024
                Online: Yes
                Removable media: No
                Readonly: No
                Backing store type: rdwr
                Backing store path: /var/lib/tgtd/kvmguest/rhelx86_64.img
                Backing store flags: 
            LUN: 2
                Type: disk
                SCSI ID: IET     00010002
                SCSI SN: beaf12
                Size: 537 MB, Block size: 512
                Online: Yes
                Removable media: No
                Readonly: No
                Backing store type: rdwr
                Backing store path: /var/lib/tgtd/kvmguest/shareddata.img
                Backing store flags: 
        Account information:
        ACL information:
    
  4. Allow client access to the target

    Finally, this example command allows access to all clients without any authentication:
    # tgtadm --lld iscsi --op bind --mode target --tid 1 --initiator-address ALL
    

Note

The two most common problems encountered when configuring the iSCSI target are related to SELinux and iptables. If adding plain files as LUNs in an iSCSI target, ensure the files are labeled system_u:object_r:tgtd_var_lib_t:s0. TCP port 3260 must be open in your iptables configuration.

24.1.2. How to configure iSCSI on a libvirt KVM host and provision a guest using virt-install

The previous section described how to set up an iSCSI target. This section demonstrates how to configure iSCSI on a libvirt KVM instance using virsh, and then how to provision a guest using virt-install.
  1. Defining the storage pool

    All libvirt storage is managed through storage pools, of which there are many possible types: SCSI, NFS, ext4, plain directory and iSCSI. All libvirt objects are configured via XML description files, and storage pools are no different in this regard. For an iSCSI storage pool there are three important pieces of information to provide:
    • The target path - this determines how libvirt will expose device paths for the pool. Paths like /dev/sda and /dev/sdb are not an ideal choice as they can change between reboots, and can change across machines within a cluster; in other words, the names are assigned on a first come, first served basis by the kernel. It is strongly recommended to use the /dev/disk/by-path format. This results in a consistent naming scheme across all machines.
    • The host name - this is the fully-qualified DNS name of the iSCSI server.
    • The source path - this is the iSCSI qualified name (IQN) seen in the previous setup procedure (iqn.2004-04.rhel:rhel5:iscsi.kvmguest).
    Although your environment will likely be different, the following text is what an iSCSI configuration will look like:
    <pool type='iscsi'>
        <name>kvmguest</name>
        <source>
            <host name='myiscsi.example.com'/>
            <device path='iqn.2004-04.rhel:rhel5:iscsi.kvmguest'/>
        </source>
        <target>
            <path>/dev/disk/by-path</path>
        </target>
    </pool>
    
    Save this XML code to a file named iscsirhel5.xml and load it into libvirt using the pool-define command:
    # virsh pool-define iscsirhel5.xml
    Pool kvmguest defined from iscsirhel5.xml
    
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    kvmguest             inactive   no
    
  2. Starting the storage pool

    The configuration is saved, but the iSCSI target has not yet been logged into, so no LUNs will be visible on the host at this point. Use the pool-start command to make the LUNs visible with the vol-list command.
    # virsh pool-start kvmguest
    Pool kvmguest started
    
    # virsh vol-list kvmguest
    Name                 Path
    -----------------------------------------
    10.0.0.1             /dev/disk/by-path/ip-192.168.122.2:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-1
    10.0.0.2             /dev/disk/by-path/ip-192.168.122.2:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-2
    
  3. Querying LUN information

    Further information about each LUN can be obtained using the vol-info and vol-dumpxml commands:
    # virsh vol-info /dev/disk/by-path/ip-192.168.122.2:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-1
    
    Name:           10.0.0.1
    Type:           block
    Capacity:       10.00 GB
    Allocation:     10.00 GB
    
    
    # virsh vol-dumpxml /dev/disk/by-path/ip-192.168.122.2:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-1
    <volume>
      <name>10.0.0.1</name>
      <key>/dev/disk/by-path/ip-192.168.122.2:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-1</key>
      <source>
      </source>
      <capacity>10737418240</capacity>
      <allocation>10737418240</allocation>
      <target>
        <path>/dev/disk/by-path/ip-192.168.122.2:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-2</path>
        <format type='unknown'/>
        <permissions>
          <mode>0660</mode>
          <owner>0</owner>
          <group>6</group>
          <label>system_u:object_r:fixed_disk_device_t:s0</label>
        </permissions>
      </target>
    </volume>
    
  4. Activating the storage at boot time

    Once correctly configured, the pool can be set to start automatically upon booting of the host:
    # virsh pool-autostart kvmguest
    Pool kvmguest marked as autostarted
    
  5. Provisioning a guest on iSCSI

    The virt-install command can be used to install new guests from the command line. The --disk argument can take the name of a storage pool, followed by the name of any contained volumes. Continuing this example, the following command will begin the installation of a guest with two disks; the first disk is the root file system, the second disk can be shared between multiple guests for common data:
    # virt-install --accelerate --name rhelx86_64 --ram 800 --vnc --disk \ vol=kvmguest/10.0.0.1 --disk vol=kvmguest/10.0.0.2,perms=sh --pxe
    
    Once this rhelx86_64 guest is installed, the following command and output shows the XML that virt-install used to associate the guest with the iSCSI LUNs:
    # virsh dumpxml rhelx86_64
    
    <domain type='kvm' id='4'>
      <name>rhelx86_64</name>
      <uuid>ad8961e9-156f-746f-5a4e-f220bfafd08d</uuid>
      <memory>819200</memory>
      <currentMemory>819200</currentMemory>
      <vcpu>1</vcpu>
      <os>
        <type arch='x86_64' machine='rhel'>hvm</type>
        <boot dev='network'/>
      </os>
      <features>
        <acpi/>
        <apic/>
        <pae/>
      </features>
      <clock offset='utc'/>
      <on_poweroff>destroy</on_poweroff>
      <on_reboot>destroy</on_reboot>
      <on_crash>destroy</on_crash>
      <devices>
        <emulator>/usr/libexec/qemu-kvm</emulator>
        <disk type='block' device='disk'>
          <driver name='qemu' type='raw'/>
          <source dev='/dev/disk/by-path/ip-192.168.122.170:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-1'/>
          <target dev='hda' bus='ide'/>
          <alias name='ide0-0-0'/>
          <address type='drive' controller='0' bus='0' unit='0'/>
        </disk>
        <disk type='block' device='disk'>
          <driver name='qemu' type='raw'/>
          <source dev='/dev/disk/by-path/ip-192.168.122.170:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-2'/>
          <target dev='hdb' bus='ide'/>
          <shareable/>
          <alias name='ide0-0-1'/>
          <address type='drive' controller='0' bus='0' unit='1'/>
        </disk>
        <controller type='ide' index='0'>
          <alias name='ide0'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
        </controller>
        <interface type='network'>
          <mac address='52:54:00:0a:ca:84'/>
          <source network='default'/>
          <target dev='vnet1'/>
          <alias name='net0'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
        </interface>
        <serial type='pty'>
          <source path='/dev/pts/28'/>
          <target port='0'/>
          <alias name='serial0'/>
        </serial>
        <console type='pty' tty='/dev/pts/28'>
          <source path='/dev/pts/28'/>
          <target port='0'/>
          <alias name='serial0'/>
        </console>
        <input type='mouse' bus='ps2'/>
        <graphics type='vnc' port='5901' autoport='yes' keymap='en-us'/>
        <video>
          <model type='cirrus' vram='9216' heads='1'/>
          <alias name='video0'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
        </video>
      </devices>
    </domain>
    
    There are two important items of note in this output:
    • The guest uses the large /dev/disk/by-path paths to reference the LUNs. As described earlier, this is so that file names and paths will remain constant.
    • The second disk has the <shareable/> flag set. Critical for the disk to be safely shared between guests, this ensures that the SELinux labelling will be appropriate for multiple guests to access the disk and that all I/O caching is disabled on the host.
For migration of guests between hosts to succeed, some form of shared storage is required. Although NFS is often used for this purpose, the lack of SELinux labelling for NFS means there is limited sVirt protection between guests. This lack of sVirt support could allow one guest to use another guest's disks, which is usually undesirable.
Using iSCSI provides full sVirt isolation between guests to the same degree of non-shared storage.