Virtualization Deployment and Administration Guide

Procedure 3.1. Creating a Red Hat Enterprise Linux 7 guest virtual machine with virt-manager using local installation media

1. Optional: Preparation

   Prepare the storage environment for the virtual machine. For more information on preparing storage, see Chapter 13, Managing Storage for Virtual Machines.

   Important

   Various storage types may be used for storing guest virtual machines. However, for a virtual machine to be able to use migration features, the virtual machine must be created on networked storage.
   Red Hat Enterprise Linux 7 requires at least 1 GB of storage space. However, Red Hat recommends at least 5 GB of storage space for a Red Hat Enterprise Linux 7 installation and for the procedures in this guide.

2. Open virt-manager and start the wizard

   Open virt-manager by executing the virt-manager command as root or opening Applications → System Tools → Virtual Machine Manager.
   

   Figure 3.1. The Virtual Machine Manager window

   Optionally, open a remote hypervisor by selecting the hypervisor and clicking the Connect button.
   Click to start the new virtualized guest wizard.
   The New VM window opens.

3. Specify installation type

   Select the installation type:

   Local install media (ISO image or CDROM)

   This method uses an image of an installation disk (for example, .iso). However, using a host CD-ROM or a DVD-ROM device is not possible.

   Network Install (HTTP, FTP, or NFS)

   This method involves the use of a mirrored Red Hat Enterprise Linux or Fedora installation tree to install a guest. The installation tree must be accessible through either HTTP, FTP, or NFS.

   If you select Network Install, provide the installation URL and also Kernel options, if required.

   Network Boot (PXE)

   This method uses a Preboot eXecution Environment (PXE) server to install the guest virtual machine. Setting up a PXE server is covered in the Red Hat Enterprise Linux 7 Installation Guide. To install using network boot, the guest must have a routable IP address or shared network device.

   If you select Network Boot, continue to STEP 5. After all steps are completed, a DHCP request is sent and if a valid PXE server is found the guest virtual machine's installation processes will start.

   Import existing disk image

   This method allows you to create a new guest virtual machine and import a disk image (containing a pre-installed, bootable operating system) to it.

   

   Figure 3.2. Virtual machine installation method

   Click Forward to continue.

4. Select the installation source

   1. If you selected Local install media (ISO image or CDROM), specify your intended local installation media.
      

      Figure 3.3. Local ISO image installation

      Warning

      Even though the option is currently present in the GUI, installing from a physical CD-ROM or DVD device on the host is not possible. Therefore, selecting the Use CDROM or DVD option will cause the VM installation to fail. For details, see the Red Hat Knowledge Base.
      To install from an ISO image, select Use ISO image and click the Browse... button to open the Locate media volume window.
      Select the installation image you wish to use, and click Choose Volume.
      If no images are displayed in the Locate media volume window, click the Browse Local button to browse the host machine for the installation image or DVD drive containing the installation disk. Select the installation image or DVD drive containing the installation disk and click Open; the volume is selected for use and you are returned to the Create a new virtual machine wizard.

      Important

      For ISO image files and guest storage images, the recommended location to use is /var/lib/libvirt/images/. Any other location may require additional configuration by SELinux. See the Red Hat Enterprise Linux Virtualization Security Guide or the Red Hat Enterprise Linux SELinux User's and Administrator's Guide for more details on configuring SELinux.
   2. If you selected Network Install, input the URL of the installation source and also the required Kernel options, if any. The URL must point to the root directory of an installation tree, which must be accessible through either HTTP, FTP, or NFS.
      To perform a kickstart installation, specify the URL of a kickstart file in Kernel options, starting with ks=
      

      Figure 3.4. Network kickstart installation

      Note

      For a complete list of kernel options, see the Red Hat Enterprise Linux 7 Installation Guide.
   Next, configure the OS type and Version of the installation. Ensure that you select the appropriate operating system type for your virtual machine. This can be specified manually or by selecting the Automatically detect operating system based on install media check box.
   Click Forward to continue.

5. Configure memory (RAM) and virtual CPUs

   Specify the number of CPUs and amount of memory (RAM) to allocate to the virtual machine. The wizard shows the number of CPUs and amount of memory you can allocate; these values affect the host's and guest's performance.
   Virtual machines require sufficient physical memory (RAM) to run efficiently and effectively. Red Hat supports a minimum of 512MB of RAM for a virtual machine. Red Hat recommends at least 1024MB of RAM for each logical core.
   Assign sufficient virtual CPUs for the virtual machine. If the virtual machine runs a multi-threaded application, assign the number of virtual CPUs the guest virtual machine will require to run efficiently.
   You cannot assign more virtual CPUs than there are physical processors (or hyper-threads) available on the host system. The number of virtual CPUs available is noted in the Up to X available field.
   

   Figure 3.5. Configuring Memory and CPU

   After you have configured the memory and CPU settings, click Forward to continue.

   Note

   Memory and virtual CPUs can be overcommitted. For more information on overcommitting, see Chapter 7, Overcommitting with KVM.

6. Configure storage

   Enable and assign sufficient space for your virtual machine and any applications it requires. Assign at least 5 GB for a desktop installation or at least 1 GB for a minimal installation.
   

   Figure 3.6. Configuring virtual storage

   Note

   Live and offline migrations require virtual machines to be installed on shared network storage. For information on setting up shared storage for virtual machines, see Section 15.4, “Shared Storage Example: NFS for a Simple Migration”.

   1. With the default local storage

      Select the Create a disk image on the computer's hard drive radio button to create a file-based image in the default storage pool, the /var/lib/libvirt/images/ directory. Enter the size of the disk image to be created. If the Allocate entire disk now check box is selected, a disk image of the size specified will be created immediately. If not, the disk image will grow as it becomes filled.

      Note

      Although the storage pool is a virtual container it is limited by two factors: maximum size allowed to it by qemu-kvm and the size of the disk on the host physical machine. Storage pools may not exceed the size of the disk on the host physical machine. The maximum sizes are as follows:

      • virtio-blk = 2^63 bytes or 8 Exabytes(using raw files or disk)
      • Ext4 = ~ 16 TB (using 4 KB block size)
      • XFS = ~8 Exabytes
      • qcow2 and host file systems keep their own metadata and scalability should be evaluated/tuned when trying very large image sizes. Using raw disks means fewer layers that could affect scalability or max size.
      Click Forward to create a disk image on the local hard drive. Alternatively, select Select managed or other existing storage, then select Browse to configure managed storage.

   2. With a storage pool

      If you select Select managed or other existing storage to use a storage pool, click Browse to open the Locate or create storage volume window.
      

      Figure 3.7. The Choose Storage Volume window

      1. Select a storage pool from the Storage Pools list.
      2. Optional: Click to create a new storage volume. The Add a Storage Volume screen will appear. Enter the name of the new storage volume.
         Choose a format option from the Format drop-down menu. Format options include raw, qcow2, and qed. Adjust other fields as needed. Note that the qcow2 version used here is version 3. To change the qcow version see Section 23.19.2, “Setting Target Elements”
         

         Figure 3.8. The Add a Storage Volume window

   Select the new volume and click Choose volume. Next, click Finish to return to the New VM wizard. Click Forward to continue.

7. Name and final configuration

   Name the virtual machine. Virtual machine names can contain letters, numbers and the following characters: underscores (_), periods (.), and hyphens (-). Virtual machine names must be unique for migration and cannot consist only of numbers.
   By default, the virtual machine will be created with network address translation (NAT) for a network called 'default' . To change the network selection, click Network selection and select a host device and source mode.
   Verify the settings of the virtual machine and click Finish when you are satisfied; this will create the virtual machine with specified networking settings, virtualization type, and architecture.
   

   Figure 3.9. Verifying the configuration

   Or, to further configure the virtual machine's hardware, check the Customize configuration before install check box to change the guest's storage or network devices, to use the paravirtualized (virtio) drivers or to add additional devices. This opens another wizard that will allow you to add, remove, and configure the virtual machine's hardware settings.

   Note

   Red Hat Enterprise Linux 4 or Red Hat Enterprise Linux 5 guest virtual machines cannot be installed using graphical mode. As such, you must select "Cirrus" instead of "QXL" as a video card.
   After configuring the virtual machine's hardware, click Apply. virt-manager will then create the virtual machine with your specified hardware settings.

   Warning

   When installing a Red Hat Enterprise Linux 7 guest virtual machine from a remote medium but without a configured TCP/IP connection, the installation fails. However, when installing a guest virtual machine of Red Hat Enterprise Linux 5 or 6 in such circumstances, the installer opens a "Configure TCP/IP" interface.

   For further information about this difference, see the related knowledgebase article.
   Click Finish to continue into the Red Hat Enterprise Linux installation sequence. For more information on installing Red Hat Enterprise Linux 7, see the Red Hat Enterprise Linux 7 Installation Guide.

Procedure 4.2. Cloning a Virtual Machine with virt-manager

1. Open virt-manager

   Start virt-manager. Launch the Virtual Machine Manager application from the Applications menu and System Tools submenu. Alternatively, run the virt-manager command as root.
   Select the guest virtual machine you want to clone from the list of guest virtual machines in Virtual Machine Manager.
   Right-click the guest virtual machine you want to clone and select Clone. The Clone Virtual Machine window opens.
   

   Figure 4.1. Clone Virtual Machine window

2. Configure the clone

   • To change the name of the clone, enter a new name for the clone.
   • To change the networking configuration, click Details.
     Enter a new MAC address for the clone.
     Click OK.
     

     Figure 4.2. Change MAC Address window

   • For each disk in the cloned guest virtual machine, select one of the following options:
     • Clone this disk - The disk will be cloned for the cloned guest virtual machine
     • Share disk with guest virtual machine name - The disk will be shared by the guest virtual machine that will be cloned and its clone
     • Details - Opens the Change storage path window, which enables selecting a new path for the disk
       

       Figure 4.3. Change storage path window

3. Clone the guest virtual machine

   Click Clone.

Procedure 6.1. Creating a bridge with virt-manager

1. From the virt-manager main menu, click Edit ⇒ Connection Details to open the Connection Details window.
2. Click the Network Interfaces tab.
3. Click the + at the bottom of the window to configure a new network interface.
4. In the Interface type drop-down menu, select Bridge, and then click Forward to continue.
   

   Figure 6.1. Adding a bridge

5. 1. In the Name field, enter a name for the bridge, such as br0.
   2. Select a Start mode from the drop-down menu. Choose from one of the following:
      • none - deactivates the bridge
      • onboot - activates the bridge on the next guest virtual machine reboot
      • hotplug - activates the bridge even if the guest virtual machine is running
   3. Check the Activate now check box to activate the bridge immediately.
   4. To configure either the IP settings or Bridge settings, click the appropriate Configure button. A separate window will open to specify the required settings. Make any necessary changes and click OK when done.
   5. Select the physical interface to connect to your virtual machines. If the interface is currently in use by another guest virtual machine, you will receive a warning message.
6. Click Finish and the wizard closes, taking you back to the Connections menu.
   

   Figure 6.2. Adding a bridge

Procedure 9.1. Configuring the PXE boot server

1. Place the PXE boot images and configuration in /var/lib/tftpboot.
2. enter the following commands:

   # virsh net-destroy default
   # virsh net-edit default

3. Edit the <ip> element in the configuration file for the default network to include the appropriate address, network mask, DHCP address range, and boot file, where BOOT_FILENAME represents the file name you are using to boot the guest virtual machine.

   <ip address='192.168.122.1' netmask='255.255.255.0'>
      <tftp root='/var/lib/tftpboot' />
      <dhcp>
         <range start='192.168.122.2' end='192.168.122.254' />
         <bootp file='BOOT_FILENAME' />
      </dhcp>
   </ip>

4. Run:

   # virsh net-start default

5. Boot the guest using PXE (refer to Section 9.2, “Booting a Guest Using PXE”).

Procedure 9.2. Booting a guest using PXE and bridged networking

1. Ensure bridging is enabled such that the PXE boot server is available on the network.
2. Boot a guest virtual machine with PXE booting enabled. You can use the virt-install command to create a new virtual machine with PXE booting enabled, as shown in the following example command:

   virt-install --pxe --network bridge=breth0 --prompt

   Alternatively, ensure that the guest network is configured to use your bridged network, and that the XML guest configuration file has a <boot order='1'/> element inside the network's <interface> element, as shown in the following example:

   <interface type='bridge'>
      <mac address='52:54:00:5a:ad:cb'/>
      <source bridge='breth0'/>
      <target dev='vnet0'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      <boot order='1'/>
   </interface>

Procedure 9.3. Using a private libvirt network

1. Configure PXE booting on libvirt as shown in Section 9.1.1, “Setting up a PXE Boot Server on a Private libvirt Network”.
2. Boot a guest virtual machine using libvirt with PXE booting enabled. You can use the virt-install command to create/install a new virtual machine using PXE:

   virt-install --pxe --network network=default --prompt

   Alternatively, ensure that the guest network is configured to use your private libvirt network, and that the XML guest configuration file has a <boot order='1'/> element inside the network's <interface> element. In addition, ensure that the guest virtual machine is connected to the private network:

Procedure 11.4. Setting up communication between guest agent and host with virsh on a Linux guest

1. Shut down the virtual machine

   Ensure the virtual machine (named rhel7 in this example) is shut down before configuring the SPICE agent:

   # virsh shutdown rhel7 

2. Add the SPICE agent channel to the guest XML configuration

   Edit the guest's XML file to add the SPICE agent details:

   # virsh edit rhel7

   Add the following to the guest's XML file and save the changes:

   <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
   </channel>

3. Start the virtual machine

   # virsh start rhel7

4. Install the SPICE agent on the guest

   Install the SPICE agent if not yet installed in the guest virtual machine:

   # yum install spice-vdagent

5. Start the SPICE agent in the guest

   Start the SPICE agent service in the guest:

   # systemctl start spice-vdagent

Procedure 11.5. Setting up communication between SPICE agent and host on a running Linux guest

1. Create an XML file for the SPICE agent

   # cat agent.xml
   <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
   </channel>

2. Attach the SPICE agent to the virtual machine

   Attach the SPICE agent to the running virtual machine (named rhel7 in this example) with this command:

   # virsh attach-device rhel7 agent.xml

3. Install the SPICE agent on the guest

   Install the SPICE agent if not yet installed in the guest virtual machine:

   # yum install spice-vdagent

4. Start the SPICE agent in the guest

   Start the SPICE agent service in the guest:

   # systemctl start spice-vdagent

Procedure 11.6. Setting up communication between the SPICE agent and host with virt-manager

1. Shut down the virtual machine

   Ensure the virtual machine is shut down before configuring the SPICE agent.
   To shut down the virtual machine, select it from the list of virtual machines in Virtual Machine Manager, then click the light switch icon from the menu bar.

2. Add the SPICE agent channel to the guest

   Open the virtual machine's hardware details by clicking the lightbulb icon at the top of the guest window.
   Click the Add Hardware button to open the Add New Virtual Hardware window, and select Channel.
   Select the SPICE agent from the Name drop-down list, edit the channel address, and click Finish:
   

   Figure 11.2. Selecting the SPICE agent channel device

3. Start the virtual machine

   To start the virtual machine, select it from the list of virtual machines in Virtual Machine Manager, then click on the menu bar.

4. Install the SPICE agent on the guest

   Open the guest with virt-manager and install the SPICE agent if not yet installed in the guest virtual machine:

   # yum install spice-vdagent

5. Start the SPICE agent in the guest

   Start the SPICE agent service in the guest:

   # systemctl start spice-vdagent

1. Open the libvirtd.conf file as described in Procedure 15.1, “Configuring libvirtd.conf”.
2. Look for the Processing controls section.

   #################################################################
   #
   # Processing controls
   #
   
   # The maximum number of concurrent client connections to allow
   # over all sockets combined.
   #max_clients = 5000
   
   # The maximum length of queue of connections waiting to be
   # accepted by the daemon. Note, that some protocols supporting
   # retransmission may obey this so that a later reattempt at
   # connection succeeds.
   #max_queued_clients = 1000
   
   # The minimum limit sets the number of workers to start up
   # initially. If the number of active clients exceeds this,
   # then more threads are spawned, upto max_workers limit.
   # Typically you'd want max_workers to equal maximum number
   # of clients allowed
   #min_workers = 5
   #max_workers = 20
   
   
   # The number of priority workers. If all workers from above
   # pool will stuck, some calls marked as high priority
   # (notably domainDestroy) can be executed in this pool.
   #prio_workers = 5
   
   # Total global limit on concurrent RPC calls. Should be
   # at least as large as max_workers. Beyond this, RPC requests
   # will be read into memory and queued. This directly impact
   # memory usage, currently each request requires 256 KB of
   # memory. So by default upto 5 MB of memory is used
   #
   # XXX this isn't actually enforced yet, only the per-client
   # limit is used so far
   #max_requests = 20
   
   # Limit on concurrent requests from a single client
   # connection. To avoid one client monopolizing the server
   # this should be a small fraction of the global max_requests
   # and max_workers parameter
   #max_client_requests = 5
   
   #################################################################
   

3. Change the max_clients and max_workers parameters settings. It is recommended that the number be the same in both parameters. The max_clients will use 2 clients per migration (one per side) and max_workers will use 1 worker on the source and 0 workers on the destination during the perform phase and 1 worker on the destination during the finish phase.

   Important

   The max_clients and max_workers parameters settings are affected by all guest virtual machine connections to the libvirtd service. This means that any user that is using the same guest virtual machine and is performing a migration at the same time will also obey the limits set in the max_clients and max_workers parameters settings. This is why the maximum value needs to be considered carefully before performing a concurrent live migration.

   Important

   The max_clients parameter controls how many clients are allowed to connect to libvirt. When a large number of containers are started at once, this limit can be easily reached and exceeded. The value of the max_clients parameter could be increased to avoid this, but doing so can leave the system more vulnerable to denial of service (DoS) attacks against instances. To alleviate this problem, a new max_anonymous_clients setting has been introduced in Red Hat Enterprise Linux 7.0 that specifies a limit of connections which are accepted but not yet authenticated. You can implement a combination of max_clients and max_anonymous_clients to suit your workload.
4. Save the file and restart the service.

   Note

   There may be cases where a migration connection drops because there are too many ssh sessions that have been started, but not yet authenticated. By default, sshd allows only 10 sessions to be in a "pre-authenticated state" at any time. This setting is controlled by the MaxStartups parameter in the sshd configuration file (located here: /etc/ssh/sshd_config), which may require some adjustment. Adjusting this parameter should be done with caution as the limitation is put in place to prevent DoS attacks (and over-use of resources in general). Setting this value too high will negate its purpose. To change this parameter, edit the file /etc/ssh/sshd_config, remove the # from the beginning of the MaxStartups line, and change the 10 (default value) to a higher number. Remember to save the file and restart the sshd service. For more information, see the sshd_config man page.

Procedure 16.3. Assigning a PCI device to a guest virtual machine with virsh

1. Identify the device

   First, identify the PCI device designated for device assignment to the virtual machine. Use the lspci command to list the available PCI devices. You can refine the output of lspci with grep.
   This example uses the Ethernet controller highlighted in the following output:

   # lspci | grep Ethernet
   00:19.0 Ethernet controller: Intel Corporation 82567LM-2 Gigabit Network Connection
   01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

   This Ethernet controller is shown with the short identifier 00:19.0. We need to find out the full identifier used by virsh in order to assign this PCI device to a virtual machine.
   To do so, use the virsh nodedev-list command to list all devices of a particular type (pci) that are attached to the host machine. Then look at the output for the string that maps to the short identifier of the device you wish to use.
   This example shows the string that maps to the Ethernet controller with the short identifier 00:19.0. Note that the : and . characters are replaced with underscores in the full identifier.

   # virsh nodedev-list --cap pci
   pci_0000_00_00_0
   pci_0000_00_01_0
   pci_0000_00_03_0
   pci_0000_00_07_0
   pci_0000_00_10_0
   pci_0000_00_10_1
   pci_0000_00_14_0
   pci_0000_00_14_1
   pci_0000_00_14_2
   pci_0000_00_14_3
   pci_0000_00_19_0
   pci_0000_00_1a_0
   pci_0000_00_1a_1
   pci_0000_00_1a_2
   pci_0000_00_1a_7
   pci_0000_00_1b_0
   pci_0000_00_1c_0
   pci_0000_00_1c_1
   pci_0000_00_1c_4
   pci_0000_00_1d_0
   pci_0000_00_1d_1
   pci_0000_00_1d_2
   pci_0000_00_1d_7
   pci_0000_00_1e_0
   pci_0000_00_1f_0
   pci_0000_00_1f_2
   pci_0000_00_1f_3
   pci_0000_01_00_0
   pci_0000_01_00_1
   pci_0000_02_00_0
   pci_0000_02_00_1
   pci_0000_06_00_0
   pci_0000_07_02_0
   pci_0000_07_03_0

   Record the PCI device number that maps to the device you want to use; this is required in other steps.

2. Review device information

   Information on the domain, bus, and function are available from output of the virsh nodedev-dumpxml command:

   # virsh nodedev-dumpxml pci_0000_00_19_0
   <device>
     <name>pci_0000_00_19_0</name>
     <parent>computer</parent>
     <driver>
       <name>e1000e</name>
     </driver>
     <capability type='pci'>
       <domain>0</domain>
       <bus>0</bus>
       <slot>25</slot>
       <function>0</function>
       <product id='0x1502'>82579LM Gigabit Network Connection</product>
       <vendor id='0x8086'>Intel Corporation</vendor>
       <iommuGroup number='7'>
         <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
       </iommuGroup>
     </capability>
   </device>
   

   Figure 16.1. Dump contents

   Note

   An IOMMU group is determined based on the visibility and isolation of devices from the perspective of the IOMMU. Each IOMMU group may contain one or more devices. When multiple devices are present, all endpoints within the IOMMU group must be claimed for any device within the group to be assigned to a guest. This can be accomplished either by also assigning the extra endpoints to the guest or by detaching them from the host driver using virsh nodedev-detach. Devices contained within a single group may not be split between multiple guests or split between host and guest. Non-endpoint devices such as PCIe root ports, switch ports, and bridges should not be detached from the host drivers and will not interfere with assignment of endpoints.

   Devices within an IOMMU group can be determined using the iommuGroup section of the virsh nodedev-dumpxml output. Each member of the group is provided in a separate "address" field. This information may also be found in sysfs using the following:

   $ ls /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/

   An example of the output from this would be:

   0000:01:00.0  0000:01:00.1

   To assign only 0000.01.00.0 to the guest, the unused endpoint should be detached from the host before starting the guest:

   $ virsh nodedev-detach pci_0000_01_00_1

3. Determine required configuration details

   See the output from the virsh nodedev-dumpxml pci_0000_00_19_0 command for the values required for the configuration file.
   The example device has the following values: bus = 0, slot = 25 and function = 0. The decimal configuration uses those three values:

   bus='0'
   slot='25'
   function='0'

4. Add configuration details

   Run virsh edit, specifying the virtual machine name, and add a device entry in the <devices> section to assign the PCI device to the guest virtual machine. For example:

   # virsh edit guest1-rhel7-64

   <devices>
   	[...]
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
         <address domain='0' bus='0' slot='25' function='0'/>
      </source>
    </hostdev>
    [...]
   </devices>
   

   Figure 16.2. Add PCI device

   Alternately, run virsh attach-device, specifying the virtual machine name and the guest's XML file:

   virsh attach-device guest1-rhel7-64 file.xml

   Note

   PCI devices may include an optional read-only memory (ROM) module, also known as an option ROM or expansion ROM, for delivering device firmware or pre-boot drivers (such as PXE) for the device. Generally, these option ROMs also work in a virtualized environment when using PCI device assignment to attach a physical PCI device to a VM.

   However, in some cases, the option ROM can be unnecessary, which may cause the VM to boot more slowly, or the pre-boot driver delivered by the device can be incompatible with virtualization, which may cause the guest OS boot to fail. In such cases, Red Hat recommends masking the option ROM from the VM. To do so:

   1. On the host, verify that the device to assign has an expansion ROM base address register (BAR). To do so, use the lspci -v command for the device, and check the output for a line that includes the following:

      Expansion ROM at

   2. Add the <rom bar='off'/> element as a child of the <hostdev> element in the guest's XML configuration:

      <hostdev mode='subsystem' type='pci' managed='yes'>
        <source>
           <address domain='0' bus='0' slot='25' function='0'/>
        </source>
        <rom bar='off'/>
      </hostdev>
      

5. Start the virtual machine

   # virsh start guest1-rhel7-64

Procedure 16.4. Assigning a PCI device to a guest virtual machine using virt-manager

1. Open the hardware settings

   Open the guest virtual machine and click the Add Hardware button to add a new device to the virtual machine.
   

   Figure 16.3. The virtual machine hardware information window

2. Select a PCI device

   Select PCI Host Device from the Hardware list on the left.
   Select an unused PCI device. Note that selecting PCI devices presently in use by another guest causes errors. In this example, a spare audio controller is used. Click Finish to complete setup.
   

   Figure 16.4. The Add new virtual hardware wizard

3. Add the new device

   The setup is complete and the guest virtual machine now has direct access to the PCI device.
   

   Figure 16.5. The virtual machine hardware information window

Procedure 16.5. Assigning a PCI device to a virtual machine with virt-install

1. Identify the device

   Identify the PCI device designated for device assignment to the guest virtual machine.

   # lspci | grep Ethernet
   00:19.0 Ethernet controller: Intel Corporation 82567LM-2 Gigabit Network Connection
   01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

   The virsh nodedev-list command lists all devices attached to the system, and identifies each PCI device with a string. To limit output to only PCI devices, enter the following command:

   # virsh nodedev-list --cap pci
   pci_0000_00_00_0
   pci_0000_00_01_0
   pci_0000_00_03_0
   pci_0000_00_07_0
   pci_0000_00_10_0
   pci_0000_00_10_1
   pci_0000_00_14_0
   pci_0000_00_14_1
   pci_0000_00_14_2
   pci_0000_00_14_3
   pci_0000_00_19_0
   pci_0000_00_1a_0
   pci_0000_00_1a_1
   pci_0000_00_1a_2
   pci_0000_00_1a_7
   pci_0000_00_1b_0
   pci_0000_00_1c_0
   pci_0000_00_1c_1
   pci_0000_00_1c_4
   pci_0000_00_1d_0
   pci_0000_00_1d_1
   pci_0000_00_1d_2
   pci_0000_00_1d_7
   pci_0000_00_1e_0
   pci_0000_00_1f_0
   pci_0000_00_1f_2
   pci_0000_00_1f_3
   pci_0000_01_00_0
   pci_0000_01_00_1
   pci_0000_02_00_0
   pci_0000_02_00_1
   pci_0000_06_00_0
   pci_0000_07_02_0
   pci_0000_07_03_0

   Record the PCI device number; the number is needed in other steps.
   Information on the domain, bus and function are available from output of the virsh nodedev-dumpxml command:

   # virsh nodedev-dumpxml pci_0000_01_00_0

   <device>
     <name>pci_0000_01_00_0</name>
     <parent>pci_0000_00_01_0</parent>
     <driver>
       <name>igb</name>
     </driver>
     <capability type='pci'>
       <domain>0</domain>
       <bus>1</bus>
       <slot>0</slot>
       <function>0</function>
       <product id='0x10c9'>82576 Gigabit Network Connection</product>
       <vendor id='0x8086'>Intel Corporation</vendor>
       <iommuGroup number='7'>
         <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
       </iommuGroup>
     </capability>
   </device>
   

   Figure 16.6. PCI device file contents

   Note

   If there are multiple endpoints in the IOMMU group and not all of them are assigned to the guest, you will need to manually detach the other endpoint(s) from the host by running the following command before you start the guest:

   $ virsh nodedev-detach pci_0000_00_19_1

   See the Note in Section 16.1.1, “Assigning a PCI Device with virsh” for more information on IOMMU groups.

2. Add the device

   Use the PCI identifier output from the virsh nodedev command as the value for the --host-device parameter.

   virt-install \
   --name=guest1-rhel7-64 \
   --disk path=/var/lib/libvirt/images/guest1-rhel7-64.img,size=8 \
   --vcpus=2 --ram=2048 \
   --location=http://example1.com/installation_tree/RHEL7.0-Server-x86_64/os \
   --nonetworks \
   --os-type=linux \
   --os-variant=rhel7
   --host-device=pci_0000_01_00_0

3. Complete the installation

   Complete the guest installation. The PCI device should be attached to the guest.

Procedure 16.6. Detaching a PCI device from a guest with virsh

1. Detach the device

   Use the following command to detach the PCI device from the guest by removing it in the guest's XML file:

   # virsh detach-device name_of_guest file.xml

2. Re-attach the device to the host (optional)

   If the device is in managed mode, skip this step. The device will be returned to the host automatically.
   If the device is not using managed mode, use the following command to re-attach the PCI device to the host machine:

   # virsh nodedev-reattach device

   For example, to re-attach the pci_0000_01_00_0 device to the host:

   # virsh nodedev-reattach pci_0000_01_00_0

   The device is now available for host use.

Procedure 16.7. Detaching a PCI Device from a guest with virt-manager

1. Open the virtual hardware details screen

   In virt-manager, double-click the virtual machine that contains the device. Select the Show virtual hardware details button to display a list of virtual hardware.
   

   Figure 16.7. The virtual hardware details button

2. Select and remove the device

   Select the PCI device to be detached from the list of virtual devices in the left panel.
   

   Figure 16.8. Selecting the PCI device to be detached

   Click the Remove button to confirm. The device is now available for host use.

Procedure 16.8. Attach an SR-IOV network device on an Intel or AMD system

1. Enable Intel VT-d or the AMD IOMMU specifications in the BIOS and kernel

   On an Intel system, enable Intel VT-d in the BIOS if it is not enabled already. See Procedure 16.1, “Preparing an Intel system for PCI device assignment” for procedural help on enabling Intel VT-d in the BIOS and kernel.
   Skip this step if Intel VT-d is already enabled and working.
   On an AMD system, enable the AMD IOMMU specifications in the BIOS if they are not enabled already. See Procedure 16.2, “Preparing an AMD system for PCI device assignment” for procedural help on enabling IOMMU in the BIOS.

2. Verify support

   Verify if the PCI device with SR-IOV capabilities is detected. This example lists an Intel 82576 network interface card which supports SR-IOV. Use the lspci command to verify whether the device was detected.

   # lspci
   03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

   Note that the output has been modified to remove all other devices.

3. Activate Virtual Functions

   Run the following command:

   # echo ${num_vfs} > /sys/class/net/enp14s0f0/device/sriov_numvfs

4. Make the Virtual Functions persistent

   To make the Virtual Functions persistent across reboots, use the editor of your choice to create an udev rule similar to the following, where you specify the intended number of VFs (in this example, 2), up to the limit supported by the network interface card. In the following example, replace enp14s0f0 with the PF network device name(s) and adjust the value of ENV{ID_NET_DRIVER} to match the driver in use:

   # vim /etc/udev/rules.d/enp14s0f0.rules

   ACTION=="add", SUBSYSTEM=="net", ENV{ID_NET_DRIVER}=="ixgbe",
   ATTR{device/sriov_numvfs}="2"
   

   This will ensure the feature is enabled at boot-time.

5. Inspect the new Virtual Functions

   Using the lspci command, list the newly added Virtual Functions attached to the Intel 82576 network device. (Alternatively, use grep to search for Virtual Function, to search for devices that support Virtual Functions.)

   # lspci | grep 82576
   0b:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   0b:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   0b:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   0b:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)

   The identifier for the PCI device is found with the -n parameter of the lspci command. The Physical Functions correspond to 0b:00.0 and 0b:00.1. All Virtual Functions have Virtual Function in the description.

6. Verify devices exist with virsh

   The libvirt service must recognize the device before adding a device to a virtual machine. libvirt uses a similar notation to the lspci output. All punctuation characters, : and ., in lspci output are changed to underscores (_).
   Use the virsh nodedev-list command and the grep command to filter the Intel 82576 network device from the list of available host devices. 0b is the filter for the Intel 82576 network devices in this example. This may vary for your system and may result in additional devices.

   # virsh nodedev-list | grep 0b
   pci_0000_0b_00_0
   pci_0000_0b_00_1
   pci_0000_0b_10_0
   pci_0000_0b_10_1
   pci_0000_0b_10_2
   pci_0000_0b_10_3
   pci_0000_0b_10_4
   pci_0000_0b_10_5
   pci_0000_0b_10_6
   pci_0000_0b_11_7
   pci_0000_0b_11_1
   pci_0000_0b_11_2
   pci_0000_0b_11_3
   pci_0000_0b_11_4
   pci_0000_0b_11_5

   The PCI addresses for the Virtual Functions and Physical Functions should be in the list.

7. Get device details with virsh

   The pci_0000_0b_00_0 is one of the Physical Functions and pci_0000_0b_10_0 is the first corresponding Virtual Function for that Physical Function. Use the virsh nodedev-dumpxml command to get device details for both devices.

   # virsh nodedev-dumpxml pci_0000_03_00_0
   <device>
     <name>pci_0000_03_00_0</name>
     <path>/sys/devices/pci0000:00/0000:00:01.0/0000:03:00.0</path>
     <parent>pci_0000_00_01_0</parent>
     <driver>
       <name>igb</name>
     </driver>
     <capability type='pci'>
       <domain>0</domain>
       <bus>3</bus>
       <slot>0</slot>
       <function>0</function>
       <product id='0x10c9'>82576 Gigabit Network Connection</product>
       <vendor id='0x8086'>Intel Corporation</vendor>
       <capability type='virt_functions'>
         <address domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
         <address domain='0x0000' bus='0x03' slot='0x10' function='0x2'/>
         <address domain='0x0000' bus='0x03' slot='0x10' function='0x4'/>
         <address domain='0x0000' bus='0x03' slot='0x10' function='0x6'/>
         <address domain='0x0000' bus='0x03' slot='0x11' function='0x0'/>
         <address domain='0x0000' bus='0x03' slot='0x11' function='0x2'/>
         <address domain='0x0000' bus='0x03' slot='0x11' function='0x4'/>
       </capability>
       <iommuGroup number='14'>
         <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
         <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
       </iommuGroup>
     </capability>
   </device>

   # virsh nodedev-dumpxml pci_0000_03_11_5
   <device>
     <name>pci_0000_03_11_5</name>
     <path>/sys/devices/pci0000:00/0000:00:01.0/0000:03:11.5</path>
     <parent>pci_0000_00_01_0</parent>
     <driver>
       <name>igbvf</name>
     </driver>
     <capability type='pci'>
       <domain>0</domain>
       <bus>3</bus>
       <slot>17</slot>
       <function>5</function>
       <product id='0x10ca'>82576 Virtual Function</product>
       <vendor id='0x8086'>Intel Corporation</vendor>
       <capability type='phys_function'>
         <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
       </capability>
       <iommuGroup number='35'>
         <address domain='0x0000' bus='0x03' slot='0x11' function='0x5'/>
       </iommuGroup>
     </capability>
   </device>

   This example adds the Virtual Function pci_0000_03_10_2 to the virtual machine in Step 8. Note the bus, slot and function parameters of the Virtual Function: these are required for adding the device.
   Copy these parameters into a temporary XML file, such as /tmp/new-interface.xml for example.

      <interface type='hostdev' managed='yes'>
        <source>
          <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x2'/>
        </source>
      </interface>

   Note

   When the virtual machine starts, it should see a network device of the type provided by the physical adapter, with the configured MAC address. This MAC address will remain unchanged across host and guest reboots.

   The following <interface> example shows the syntax for the optional <mac address>, <virtualport>, and <vlan> elements. In practice, use either the <vlan> or <virtualport> element, not both simultaneously as shown in the example:

   ...
    <devices>
      ...
      <interface type='hostdev' managed='yes'>
        <source>
          <address type='pci' domain='0' bus='11' slot='16' function='0'/>
        </source>
        <mac address='52:54:00:6d:90:02'>
        <vlan>
           <tag id='42'/>
        </vlan>
        <virtualport type='802.1Qbh'>
          <parameters profileid='finance'/>
        </virtualport>
      </interface>
      ...
    </devices>

   If you do not specify a MAC address, one will be automatically generated. The <virtualport> element is only used when connecting to an 802.11Qbh hardware switch. The <vlan> element will transparently put the guest's device on the VLAN tagged 42.

8. Add the Virtual Function to the virtual machine

   Add the Virtual Function to the virtual machine using the following command with the temporary file created in the previous step. This attaches the new device immediately and saves it for subsequent guest restarts.

   virsh attach-device MyGuest /tmp/new-interface.xml --live --config
   

   Specifying the --live option with virsh attach-device attaches the new device to the running guest. Using the --config option ensures the new device is available after future guest restarts.

   Note

   The --live option is only accepted when the guest is running. virsh will return an error if the --live option is used on a non-running guest.

Procedure 16.9. Configuring MAC addresses, vLAN, and virtual ports for assigning PCI devices on SR-IOV

1. Gather information

   In order to use <interface type='hostdev'>, you must have an SR-IOV-capable network card, host physical machine hardware that supports either the Intel VT-d or AMD IOMMU extensions, and you must know the PCI address of the VF that you wish to assign.

2. Shut down the guest virtual machine

   Using virsh shutdown command, shut down the guest virtual machine (here named guestVM).

   # virsh shutdown guestVM

3. Open the XML file for editing

   # virsh edit guestVM.xml

   Optional: For the XML configuration file that was created by the virsh save command, run:

   # virsh save-image-edit guestVM.xml --running 

   The configuration file, in this example guestVM.xml, opens in your default editor. For more information, see Section 20.7.5, “Editing the Guest Virtual Machine Configuration”

4. Edit the XML file

   Update the configuration file (guestVM.xml) to have a <devices> entry similar to the following:

    <devices>
      ...
      <interface type='hostdev' managed='yes'>
        <source>
          <address type='pci' domain='0x0' bus='0x00' slot='0x07' function='0x0'/> <!--these values can be decimal as well-->
        </source>
        <mac address='52:54:00:6d:90:02'/>                                         <!--sets the mac address-->
        <virtualport type='802.1Qbh'>                                              <!--sets the virtual port for the 802.1Qbh switch-->
          <parameters profileid='finance'/>
        </virtualport>
        <vlan>                                                                     <!--sets the vlan tag-->
         <tag id='42'/>
        </vlan>
      </interface>
      ...
    </devices>
   
   

   Figure 16.11. Sample domain XML for hostdev interface type

   Note

   If you do not provide a MAC address, one will be automatically generated, just as with any other type of interface device. In addition, the <virtualport> element is only used if you are connecting to an 802.11Qgh hardware switch. 802.11Qbg (also known as "VEPA") switches are currently not supported.

5. Restart the guest virtual machine

   Run the virsh start command to restart the guest virtual machine you shut down in step 2. See Section 20.6, “Starting, Resuming, and Restoring a Virtual Machine” for more information.

    # virsh start guestVM 

   When the guest virtual machine starts, it sees the network device provided to it by the physical host machine's adapter, with the configured MAC address. This MAC address remains unchanged across guest virtual machine and host physical machine reboots.

Procedure 16.10. Creating a device pool

1. Shut down the guest virtual machine

   Using virsh shutdown command, shut down the guest virtual machine, here named guestVM.

   # virsh shutdown guestVM

2. Create a configuration file

   Using your editor of choice, create an XML file (named passthrough.xml, for example) in the /tmp directory. Make sure to replace pf dev='eth3' with the netdev name of your own SR-IOV device's Physical Function (PF).
   The following is an example network definition that will make available a pool of all VFs for the SR-IOV adapter with its PF at "eth3' on the host physical machine:

         
   <network>
      <name>passthrough</name> <!-- This is the name of the file you created -->
      <forward mode='hostdev' managed='yes'>
        <pf dev='myNetDevName'/>  <!-- Use the netdev name of your SR-IOV devices PF here -->
      </forward>
   </network>
         
   
   

   Figure 16.12. Sample network definition domain XML

3. Load the new XML file

   Enter the following command, replacing /tmp/passthrough.xml with the name and location of your XML file you created in the previous step:

   # virsh net-define /tmp/passthrough.xml

4. Restarting the guest

   Run the following, replacing passthrough.xml with the name of your XML file you created in the previous step:

    # virsh net-autostart passthrough # virsh net-start passthrough 

5. Re-start the guest virtual machine

   Run the virsh start command to restart the guest virtual machine you shutdown in the first step (example uses guestVM as the guest virtual machine's domain name). See Section 20.6, “Starting, Resuming, and Restoring a Virtual Machine” for more information.

    # virsh start guestVM 

6. Initiating passthrough for devices

   Although only a single device is shown, libvirt will automatically derive the list of all VFs associated with that PF the first time a guest virtual machine is started with an interface definition in its domain XML like the following:

            
   <interface type='network'>
      <source network='passthrough'>
   </interface>
         
   
   

   Figure 16.13. Sample domain XML for interface network definition

7. Verification

   You can verify this by running virsh net-dumpxml passthrough command after starting the first guest that uses the network; you will get output similar to the following:

         
   <network connections='1'>
      <name>passthrough</name>
      <uuid>a6b49429-d353-d7ad-3185-4451cc786437</uuid>
      <forward mode='hostdev' managed='yes'>
        <pf dev='eth3'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x1'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x3'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x5'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x7'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x1'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x3'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x5'/>
      </forward>
   </network>
         
   
   

   Figure 16.14. XML dump file passthrough contents