Red Hat Enterprise Linux 7

Virtualization Deployment and Administration Guide

Installing, configuring, and managing virtual machines on a Red Hat Enterprise Linux physical machine

Jiri Herrmann

Red Hat Customer Content Services

Yehuda Zimmerman

Red Hat Customer Content Services

Laura Novich

Red Hat Customer Content Services

Dayle Parker

Red Hat Customer Content Services

Scott Radvan

Red Hat Customer Content Services

Tahlia Richardson

Red Hat Customer Content Services

Legal Notice

Copyright © 2016 Red Hat, Inc.
This document is licensed by Red Hat under the Creative Commons Attribution-ShareAlike 3.0 Unported License. If you distribute this document, or a modified version of it, you must provide attribution to Red Hat, Inc. and provide a link to the original. If the document is modified, all Red Hat trademarks must be removed.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.

Abstract

This guide covers how to configure a Red Hat Enterprise Linux 7 host physical machine and how to install and configure guest virtual machines using the KVM hypervisor. Other topics include PCI device configuration, SR-IOV, networking, storage, device and guest virtual machine management, as well as troubleshooting, compatibility and restrictions.

Important

All procedures described in this guide are intended to be performed on the host machine, unless otherwise stated. Procedures that need to be run on the guest virtual machine are clearly marked as such.
For a more general introduction into virtualization solutions provided by Red Hat, see the Red Hat Enterprise Linux 7 Virtualization Getting Started Guide.
Note: This document is under development, is subject to substantial change, and is provided only as a preview. The included information and instructions should not be considered complete, and should be used with caution.

Note

To expand your expertise, you might also be interested in the Red Hat Enterprise Virtualization (RH318) training course.
I. Deployment
1. System Requirements
1.1. Host System Requirements
1.2. KVM Hypervisor Requirements
1.3. KVM Guest Virtual Machine Compatibility
1.4. Supported Guest CPU Models
1.4.1. Listing the Guest CPU Models
2. Installing the Virtualization Packages
2.1. Configuring a Virtualization Host During a New Red Hat Enterprise Linux 7 Installation
2.1.1. Installing KVM Packages with Kickstart Files
2.2. Installing Virtualization Packages on an Existing Red Hat Enterprise Linux System
2.2.1. Installing the Virtualization Packages with yum
2.2.2. Installing Virtualization Package Groups
3. Installing a Virtual Machine
3.1. Guest Virtual Machine Prerequisites and Considerations
3.2. Creating Guests with virt-install
3.2.1. Network Installation with virt-install
3.2.2. PXE Installation with virt-install
3.2.3. Kickstart Installation with virt-install
3.2.4. Guest Installation with virt-install and Text-based Anaconda
3.2.5. Guest Installation with virt-install Using NAT
3.2.6. Guest Installation with virt-install Using an External DHCP Server
3.2.7. Guest Installation with virt-install Using an External Static IPv4 Address
3.2.8. Optional Workaround to Allow for Graceful Shutdown
3.3. Creating Guests with virt-manager
3.3.1. virt-manager Installation Overview
3.3.2. Creating a Red Hat Enterprise Linux 7 Guest with virt-manager
3.4. Comparison of virt-install and virt-manager Installation options
4. Cloning Virtual Machines
4.1. Cloning Guests with virt-clone
4.2. Cloning Guests with virt-manager
5. KVM Paravirtualized (virtio) Drivers
5.1. Using KVM virtio Drivers for Existing Devices
5.2. Using KVM virtio Drivers for New Devices
6. Network Configuration
6.1. Network Address Translation (NAT) with libvirt
6.2. Disabling vhost-net
6.3. Enabling vhost-net zero-copy
6.4. Bridged Networking
6.4.1. Configuring Bridged Networking on a Red Hat Enterprise Linux 7 Host
6.4.2. Bridged Networking with Virtual Machine Manager
6.4.3. Bridged Networking with libvirt
7. Overcommitting with KVM
7.1. Introduction
7.2. Overcommitting Memory
7.3. Overcommitting Virtualized CPUs (vCPUs)
8. KVM Guest Timing Management
8.1. Required Parameters for Red Hat Enterprise Linux Guests
8.2. Steal Time Accounting
9. Network Booting with libvirt
9.1. Preparing the Boot Server
9.1.1. Setting up a PXE Boot Server on a Private libvirt Network
9.2. Booting a Guest Using PXE
9.2.1. Using bridged networking
9.2.2. Using a Private libvirt Network
10. Registering the Hypervisor and Virtual Machine
10.1. Installing virt-who on the Host Physical Machine
10.2. Registering a New Guest Virtual Machine
10.3. Removing a Guest Virtual Machine Entry
10.4. Installing virt-who Manually
10.5. Troubleshooting virt-who
10.5.1. Why is the hypervisor status red?
10.5.2. I have subscription status errors, what do I do?
11. Enhancing Virtualization with the QEMU Guest Agent and SPICE Agent
11.1. QEMU Guest Agent
11.1.1. Setting up Communication between the QEMU Guest Agent and Host
11.2. Using the QEMU Guest Agent with libvirt
11.2.1. Creating a Guest Disk Backup
11.3. SPICE Agent
11.3.1. Setting up Communication between the SPICE Agent and Host
12. Nested Virtualization
12.1. Overview
12.2. Setup
12.3. Restrictions and Limitations
II. Administration
13. Securing the Host Physical Machine and Improving Performance
13.1. Security Deployment Plan
13.2. Client Access Control
13.2.1. Access Control Drivers
13.2.2. Objects and Permissions
13.2.3. Security Concerns when Adding Block Devices to a Guest
14. Storage Pools
14.1. Disk-based Storage Pools
14.1.1. Creating a Disk-based Storage Pool Using virsh
14.1.2. Deleting a Storage Pool Using virsh
14.2. Partition-based Storage Pools
14.2.1. Creating a Partition-based Storage Pool Using virt-manager
14.2.2. Deleting a Storage Pool Using virt-manager
14.2.3. Creating a Partition-based Storage Pool Using virsh
14.2.4. Deleting a Storage Pool Using virsh
14.3. Directory-based Storage Pools
14.3.1. Creating a Directory-based Storage Pool with virt-manager
14.3.2. Deleting a Storage Pool Using virt-manager
14.3.3. Creating a Directory-based Storage Pool with virsh
14.3.4. Deleting a Storage Pool Using virsh
14.4. LVM-based Storage Pools
14.4.1. Creating an LVM-based Storage Pool with virt-manager
14.4.2. Deleting a Storage Pool Using virt-manager
14.4.3. Creating an LVM-based Storage Pool with virsh
14.4.4. Deleting a Storage Pool Using virsh
14.5. iSCSI-based Storage Pools
14.5.1. Configuring a Software iSCSI Target
14.5.2. Creating an iSCSI Storage Pool in virt-manager
14.5.3. Deleting a Storage Pool Using virt-manager
14.5.4. Creating an iSCSI-based Storage Pool with virsh
14.5.5. Securing an iSCSI Storage Pool
14.5.6. Deleting a Storage Pool Using virsh
14.6. NFS-based Storage Pools
14.6.1. Creating an NFS-based Storage Pool with virt-manager
14.6.2. Deleting a Storage Pool Using virt-manager
14.7. Using an NPIV Virtual Adapter (vHBA) with SCSI Devices
14.7.1. Creating a vHBA
14.7.2. Creating a Storage Pool Using the vHBA
14.7.3. Configuring the Virtual Machine to Use a vHBA LUN
14.7.4. Destroying the vHBA Storage Pool
14.8. GlusterFS Storage Pools
14.8.1. Creating a GlusterFS Storage Pool Using virsh
14.8.2. Deleting a GlusterFS Storage Pool Using virsh
15. Storage Volumes
15.1. Introduction
15.1.1. Referencing Volumes
15.2. Creating Volumes
15.3. Cloning Volumes
15.4. Deleting and Removing Volumes
15.5. Adding Storage Devices to Guests
15.5.1. Adding File-based Storage to a Guest
15.5.2. Adding Hard Drives and Other Block Devices to a Guest
15.5.3. Adding SCSI LUN-based Storage to a Guest
15.5.4. Managing Storage Controllers in a Guest Virtual Machine
16. Using qemu-img
16.1. Checking the Disk Image
16.2. Committing Changes to an Image
16.3. Comparing Images
16.4. Mapping an Image
16.4.1. The human Format
16.4.2. The json Format
16.5. Amending an Image
16.6. Converting an Existing Image to Another Format
16.7. Creating and Formatting New Images or Devices
16.8. Displaying Image Information
16.9. Rebasing a Backing File of an Image
16.10. Re-sizing the Disk Image
16.11. Listing, Creating, Applying, and Deleting a Snapshot
16.12. Supported qemu-img Formats
17. KVM Migration
17.1. Migration Definition and Benefits
17.2. Migration Requirements and Limitations
17.3. Live Migration and Red Hat Enterprise Linux Version Compatibility
17.4. Shared Storage Example: NFS for a Simple Migration
17.5. Live KVM Migration with virsh
17.5.1. Additional Tips for Migration with virsh
17.5.2. Additional Options for the virsh migrate Command
17.6. Migrating with virt-manager
18. Guest Virtual Machine Device Configuration
18.1. PCI Devices
18.1.1. Assigning a PCI Device with virsh
18.1.2. Assigning a PCI Device with virt-manager
18.1.3. PCI Device Assignment with virt-install
18.1.4. Detaching an Assigned PCI Device
18.1.5. Creating PCI Bridges
18.1.6. PCI Device Assignment with SR-IOV Devices
18.2. USB Devices
18.2.1. Assigning USB Devices to Guest Virtual Machines
18.2.2. Setting a Limit on USB Device Redirection
18.3. Configuring Device Controllers
18.4. Setting Addresses for Devices
18.5. Random Number Generator Device
18.6. Assigning GPU Devices
19. SR-IOV
19.1. Advantages of SR-IOV
19.2. Using SR-IOV
19.3. Troubleshooting SR-IOV
20. Virtual Networking
20.1. Virtual Network Switches
20.2. Bridged Mode
20.3. Network Address Translation
20.4. DNS and DHCP
20.5. Routed Mode
20.6. Isolated Mode
20.7. The Default Configuration
20.8. Examples of Common Scenarios
20.8.1. Bridged Mode
20.8.2. Routed Mode
20.8.3. NAT Mode
20.8.4. Isolated Mode
20.9. Managing a Virtual Network
20.10. Creating a Virtual Network
20.11. Attaching a Virtual Network to a Guest
20.12. Directly Attaching to Physical Interface
20.13. Dynamically Changing a Host Physical Machine or a Network Bridge that is Attached to a Virtual NIC
20.14. Applying Network Filtering
20.14.1. Introduction
20.14.2. Filtering Chains
20.14.3. Filtering Chain Priorities
20.14.4. Usage of Variables in Filters
20.14.5. Automatic IP Address Detection and DHCP Snooping
20.14.6. Reserved Variables
20.14.7. Element and Attribute Overview
20.14.8. References to Other Filters
20.14.9. Filter Rules
20.14.10. Supported Protocols
20.14.11. Advanced Filter Configuration Topics
20.14.12. Limitations
20.15. Creating Tunnels
20.15.1. Creating Multicast Tunnels
20.15.2. Creating TCP Tunnels
20.16. Setting vLAN Tags
20.17. Applying QoS to Your Virtual Network
21. Remote Management of Guests
21.1. Transport Modes
21.2. Remote Management with SSH
21.3. Remote Management over TLS and SSL
21.4. Configuring a VNC Server
21.5. Enhancing Remote Management of Virtual Machines with NSS
22. Managing Guests with the Virtual Machine Manager (virt-manager)
22.1. Starting virt-manager
22.2. The Virtual Machine Manager Main Window
22.3. The Virtual Hardware Details Window
22.3.1. Applying Boot Options to Guest Virtual Machines
22.3.2. Attaching USB Devices to a Guest Virtual Machine
22.3.3. USB Redirection
22.4. Virtual Machine Graphical Console
22.5. Adding a Remote Connection
22.6. Displaying Guest Details
22.7. Managing Snapshots
22.8. Performance Monitoring
22.9. Displaying CPU Usage for Guests
22.10. Displaying CPU Usage for Hosts
22.11. Displaying Disk I/O
22.12. Displaying Network I/O
22.13. Displaying Memory Usage
23. Managing Guest Virtual Machines with virsh
23.1. Guest Virtual Machine States and Types
23.2. Running the virsh Program
23.3. Interactive Mode Commands
23.3.1. Quitting and Exiting from the virsh Interactive Terminal
23.3.2. Changing and Displaying Directories in a Virtual Machine
23.4. Displaying the virsh Version
23.5. Getting Help
23.6. Sending Commands with echo
23.7. Connecting to the Hypervisor with virsh Connect
23.8. Displaying Information about a Guest Virtual Machine
23.9. Starting, Resuming, and Restoring a Virtual Machine
23.9.1. Starting a Guest Virtual Machine
23.9.2. Configuring a Virtual Machine to be Started Automatically at Boot
23.9.3. Rebooting a Guest Virtual Machine
23.9.4. Restoring a Guest Virtual Machine
23.9.5. Resuming a Guest Virtual Machine
23.9.6. Restoring a Guest Virtual Machine from a Suspended State
23.10. Managing a Virtual Machine Configuration
23.10.1. Saving a Guest Virtual Machine's Configuration
23.10.2. Defining a Guest Virtual Machine with an XML File
23.10.3. Updating the XML File That will be Used for Restoring a Guest Virtual Machine
23.10.4. Extracting the Guest Virtual Machine XML File
23.10.5. Editing the Guest Virtual Machine Configuration
23.11. Shutting off, Shutting down, Rebooting, and Forcing a Shutdown of a Guest Virtual Machine
23.11.1. Suspending a Running Guest Virtual Machine Using Power Management
23.11.2. Shutting down a Guest Virtual Machine
23.11.3. Suspending a Guest Virtual Machine
23.11.4. Resetting a Virtual Machine
23.11.5. Stopping a Running Guest Virtual Machine in Order to Restart It Later
23.12. Removing and Deleting a Virtual Machine
23.12.1. Undefining a Virtual Machine
23.12.2. Forcing a Guest Virtual Machine to Stop
23.13. Connecting the Serial Console for the Guest Virtual Machine
23.14. Injecting NMI
23.15. Retrieving Information about Your Virtual Machine
23.15.1. Displaying Device Block Statistics
23.15.2. Retrieving Network Statistics
23.15.3. Modifying the Link State of a Guest Virtual Machine's Virtual Interface
23.15.4. Listing the Link State of a Guest Virtual Machine's Virtual Interface
23.15.5. Setting Network Interface Bandwidth Parameters
23.15.6. Retrieving Memory Statistics
23.15.7. Displaying Errors on Block Devices
23.15.8. Displaying the Block Device Size
23.15.9. Displaying the Block Devices Associated with a Guest Virtual Machine
23.15.10. Displaying Virtual Interfaces Associated with a Guest Virtual Machine
23.16. Working with Snapshots
23.16.1. Shortening a Backing Chain by Coping the Data
23.16.2. Shortening a Backing Chain by Flattening the Image
23.16.3. Changing the Size of a Guest Virtual Machine's Block Device
23.17. Displaying a URI for Connection to a Graphical Display
23.18. Discarding Blocks Not in Use
23.19. Guest Virtual Machine Retrieval Commands
23.19.1. Displaying the Host Physical Machine Name
23.19.2. Displaying General Information about a Virtual Machine
23.19.3. Displaying a Virtual Machine's ID Number
23.19.4. Aborting Running Jobs on a Guest Virtual Machine
23.19.5. Displaying Information about Jobs Running on the Guest Virtual Machine
23.19.6. Displaying the Guest Virtual Machine's Name
23.19.7. Displaying the Virtual Machine's State
23.19.8. Displaying the Connection State to the Virtual Machine
23.20. Converting QEMU Arguments to Domain XML
23.21. Creating a Dump File of a Guest Virtual Machine's Core
23.22. Creating a Virtual Machine XML Dump (Configuration File)
23.23. Creating a Guest Virtual Machine from a Configuration File
23.24. Editing a Guest Virtual Machine's Configuration Settings
23.25. Adding Multifunction PCI Devices to KVM Guest Virtual Machines
23.26. Displaying CPU Statistics for a Specified Guest Virtual Machine
23.27. Saving a Screenshot
23.28. Sending a Keystroke Combination to a Specified Guest Virtual machine
23.29. Sending Process Signal Names to Virtual Processes
23.30. Displaying the IP Address and Port Number for the VNC Display
23.31. NUMA Node Management
23.31.1. Displaying Node Information
23.31.2. Setting NUMA Parameters
23.31.3. Displaying the Amount of Free Memory in a NUMA Cell
23.31.4. Displaying a CPU List
23.31.5. Displaying CPU Statistics
23.31.6. Managing Devices
23.31.7. Suspending the Host Physical Machine
23.31.8. Setting and Displaying the Node Memory Parameters
23.31.9. Listing Devices on a Node
23.31.10. Creating Devices on Host Nodes
23.31.11. Removing a Device
23.31.12. Collect a Device Configuration Settings
23.31.13. Triggering a Reset for a Device
23.32. Retrieving Guest Virtual Machine Information
23.32.1. Getting the Domain ID of a Guest Virtual Machine
23.32.2. Getting the Domain Name of a Guest Virtual Machine
23.32.3. Getting the UUID of a Guest Virtual Machine
23.32.4. Displaying Guest Virtual Machine Information
23.33. Storage Pool Commands
23.33.1. Searching for a Storage Pool XML
23.33.2. Finding a storage Pool
23.33.3. Listing Storage Pool Information
23.33.4. Listing the Available Storage Pools
23.33.5. Refreshing a Storage Pool List
23.33.6. Creating, Defining, and Starting Storage Pools
23.33.7. Stopping and Deleting Storage Pools
23.33.8. Creating an XML Dump File for a Pool
23.33.9. Editing the Storage Pool's Configuration File
23.34. Storage Volume Commands
23.34.1. Creating Storage Volumes
23.34.2. Creating a Storage Volume from Parameters
23.34.3. Creating a Storage Volume from an XML File
23.34.4. Cloning a Storage Volume
23.35. Deleting Storage Volumes
23.36. Deleting a Storage Volume's Contents
23.37. Dumping Storage Volume Information to an XML File
23.38. Listing Volume Information
23.39. Retrieving Storage Volume Information
23.40. Uploading and Downloading Storage Volumes
23.41. Resizing Storage Volumes
23.42. Displaying Per-guest Virtual Machine Information
23.42.1. Displaying the Guest Virtual Machines
23.42.2. Displaying Virtual CPU Information
23.42.3. Pinning vCPU to a Host Physical Machine's CPU
23.42.4. Displaying Information about the Virtual CPU Counts of a Given Domain
23.42.5. Configuring Virtual CPU Affinity
23.42.6. Configuring Virtual CPU Count
23.42.7. Configuring Memory Allocation
23.42.8. Changing the Memory Allocation for the Domain
23.42.9. Displaying Guest Virtual Machine Block Device Information
23.42.10. Displaying Guest Virtual Machine Network Device Information
23.43. Managing Virtual Networks
23.43.1. Autostarting a Virtual Network
23.43.2. Creating a Virtual Network from an XML File
23.43.3. Defining a Virtual Network from an XML File
23.43.4. Stopping a Virtual Network
23.43.5. Creating a Dump File
23.43.6. Editing a Virtual Network's XML Configuration File
23.43.7. Getting Information about a Virtual Network
23.43.8. Listing Information about a Virtual Network
23.43.9. Converting a Network UUID to Network Name
23.43.10. Starting a (previously defined) Inactive Network
23.43.11. Undefining the Configuration for an Inactive Network
23.43.12. Converting a Network Name to Network UUID
23.43.13. Updating an Existing Network Definition File
23.43.14. Migrating Guest Virtual Machines with virsh
23.43.15. Setting a Static IP Address for the Guest Virtual Machine
23.44. Interface Commands
23.44.1. Defining and Starting a Host Physical Machine Interface via an XML File
23.44.2. Editing the XML Configuration File for the Host Interface
23.44.3. Listing Active Host Interfaces
23.44.4. Converting a MAC Address into an Interface Name
23.44.5. Stopping a Specific Host Physical Machine Interface
23.44.6. Displaying the Host Configuration File
23.44.7. Creating Bridge Devices
23.44.8. Tearing Down a Bridge Device
23.44.9. Manipulating Interface Snapshots
23.45. Managing Snapshots
23.45.1. Creating Snapshots
23.45.2. Creating a Snapshot for the Current Guest Virtual Machine
23.45.3. Displaying the Current Snapshot of a Guest Virtual Machine
23.45.4. snapshot-edit
23.45.5. snapshot-info
23.45.6. snapshot-list
23.45.7. snapshot-dumpxml
23.45.8. snapshot-parent
23.45.9. snapshot-revert
23.45.10. snapshot-delete
23.46. Guest Virtual Machine CPU Model Configuration
23.46.1. Introduction
23.46.2. Learning about the Host Physical Machine CPU Model
23.46.3. Determining Support for VFIO IOMMU Devices
23.46.4. Determining a Compatible CPU Model to Suit a Pool of Host Physical Machines
23.47. Configuring the Guest Virtual Machine CPU Model
23.48. Managing Resources for Guest Virtual Machines
23.49. Setting Schedule Parameters
23.50. Disk I/O Throttling
23.51. Display or Set Block I/O Parameters
23.52. Configuring Memory Tuning
24. Guest Virtual Machine Disk Access with Offline Tools
24.1. Introduction
24.1.1. Caution about Using Remote Connections
24.2. Terminology
24.3. Installation
24.4. The guestfish Shell
24.4.1. Viewing File Systems with guestfish
24.4.2. Adding Files with guestfish
24.4.3. Modifying Files with guestfish
24.4.4. Other Actions with guestfish
24.4.5. Shell Scripting with guestfish
24.4.6. Augeas and libguestfs Scripting
24.5. Other Commands
24.6. virt-rescue: The Rescue Shell
24.6.1. Introduction
24.6.2. Running virt-rescue
24.7. virt-df: Monitoring Disk Usage
24.7.1. Introduction
24.7.2. Running virt-df
24.8. virt-resize: Resizing Guest Virtual Machines Offline
24.8.1. Introduction
24.8.2. Expanding a Disk Image
24.9. virt-inspector: Inspecting Guest Virtual Machines
24.9.1. Introduction
24.9.2. Installation
24.9.3. Running virt-inspector
24.10. Using the API from Programming Languages
24.10.1. Interaction with the API via a C program
24.11. virt-sysprep: Resetting Virtual Machine Settings
24.12. virt-customize: Customizing Virtual Machine Settings
24.13. virt-diff: Listing the Differences between Virtual Machine Files
25. Graphic User Interface Tools for Guest Virtual Machine Management
25.1. Using virt-viewer Command Line
25.2. remote-viewer
25.3. GNOME Boxes
26. Manipulating the Domain XML
26.1. General Information and Meta-data
26.2. Operating System Booting
26.2.1. BIOS Boot Loader
26.2.2. Host Physical Machine Boot Loader
26.2.3. Direct Kernel Boot
26.2.4. Container Boot
26.3. SMBIOS System Information
26.4. CPU allocation
26.5. CPU tuning
26.6. Memory Backing
26.7. Memory Tuning
26.8. Memory Allocation
26.9. NUMA Node Tuning
26.10. Block I/O Tuning
26.11. Resource Partitioning
26.12. CPU Models and Topology
26.12.1. Changing the Feature Set for a Specified CPU
26.12.2. Guest Virtual Machine NUMA Topology
26.13. Events Configuration
26.14. Power Management
26.15. Hypervisor Features
26.16. Timekeeping
26.17. Timer Element Attributes
26.18. Devices
26.18.1. Hard Drives, Floppy Disks, and CD-ROMs
26.18.2. File Systems
26.18.3. Device Addresses
26.18.4. Controllers
26.18.5. Device Leases
26.18.6. Host Physical Machine Device Assignment
26.18.7. Redirected devices
26.18.8. Smartcard Devices
26.18.9. Network Interfaces
26.18.10. Input Devices
26.18.11. Hub Devices
26.18.12. Graphical Framebuffers
26.18.13. Video Devices
26.18.14. Consoles, Serial, and Channel Devices
26.18.15. Guest Virtual Machine Interfaces
26.18.16. Channel
26.18.17. Host Physical Machine Interface
26.18.18. Sound Devices
26.18.19. Watchdog Device
26.18.20. Setting a Panic Device
26.18.21. Memory Balloon Device
26.19. Storage Pools
26.19.1. Providing Metadata for the Storage Pool
26.19.2. Source Elements
26.19.3. Creating Target Elements
26.19.4. Setting Device Extents
26.20. Storage Volumes
26.20.1. General Metadata
26.20.2. Setting Target Elements
26.20.3. Setting Backing Store Elements
26.21. Security Label
26.22. A Sample Configuration File
III. Appendices
A. Troubleshooting
A.1. Debugging and Troubleshooting Tools
A.2. Preparing for Disaster Recovery
A.3. Creating virsh Dump Files
A.4. Capturing Trace Data on a Constant Basis Using the Systemtap Flight Recorder
A.5. kvm_stat
A.6. Troubleshooting with Serial Consoles
A.7. Virtualization Log Files
A.8. Loop Device Errors
A.9. Live Migration Errors
A.10. Enabling Intel VT-x and AMD-V Virtualization Hardware Extensions in BIOS
A.11. Shutting down Red Hat Enterprise Linux 6 Guests on a Red Hat Enterprise Linux 7 Host
A.12. KVM Networking Performance
A.13. Workaround for Creating External Snapshots with libvirt
A.14. Missing Characters on Guest Console with Japanese Keyboard
A.15. Guest Virtual Machine Fails to Shutdown
A.16. Disable SMART Disk Monitoring for Guest Virtual Machines
A.17. libguestfs Troubleshooting
A.18. Common libvirt Errors and Troubleshooting
A.18.1. libvirtd failed to start
A.18.2. The URI Failed to Connect to the Hypervisor
A.18.3. The Guest Virtual Machine Cannot be Started: internal error guest CPU is not compatible with host CPU
A.18.4. Guest Starting Fails with Error: monitor socket did not show up
A.18.5. internal error cannot find character device (null)
A.18.6. Guest Virtual Machine Booting Stalls with Error: No boot device
A.18.7. Virtual network default has not been started
A.18.8. PXE Boot (or DHCP) on Guest Failed
A.18.9. Guest Can Reach Outside Network, but Cannot Reach Host When Using macvtap interface
A.18.10. Could not add rule to fixup DHCP response checksums on network 'default'
A.18.11. Unable to add bridge br0 port vnet0: No such device
A.18.12. Guest is Unable to Start with Error: warning: could not open /dev/net/tun
A.18.13. Migration Fails with error: unable to resolve address
A.18.14. Migration Fails with Unable to allow access for disk path: No such file or directory
A.18.15. No Guest Virtual Machines are Present when libvirtd is Started
A.18.16. unable to connect to server at 'host:16509': Connection refused ... error: failed to connect to the hypervisor
A.18.17. Common XML Errors
B. Virtualization Restrictions
B.1. Support Restrictions
B.2. KVM Restrictions
B.3. Application Restrictions
B.4. Other Restrictions
B.5. Storage Support
B.6. USB 3 / xHCI Support
C. Additional Resources
C.1. Online Resources
C.2. Installed Documentation
D. Working with IOMMU Groups
D.1. IOMMU Overview
D.2. A Deep-dive into IOMMU Groups
D.3. How to Identify and Assign IOMMU Groups
D.4. IOMMU Strategies and Use Cases
E. Revision History

Part I. Deployment

Table of Contents

1. System Requirements
1.1. Host System Requirements
1.2. KVM Hypervisor Requirements
1.3. KVM Guest Virtual Machine Compatibility
1.4. Supported Guest CPU Models
1.4.1. Listing the Guest CPU Models
2. Installing the Virtualization Packages
2.1. Configuring a Virtualization Host During a New Red Hat Enterprise Linux 7 Installation
2.1.1. Installing KVM Packages with Kickstart Files
2.2. Installing Virtualization Packages on an Existing Red Hat Enterprise Linux System
2.2.1. Installing the Virtualization Packages with yum
2.2.2. Installing Virtualization Package Groups
3. Installing a Virtual Machine
3.1. Guest Virtual Machine Prerequisites and Considerations
3.2. Creating Guests with virt-install
3.2.1. Network Installation with virt-install
3.2.2. PXE Installation with virt-install
3.2.3. Kickstart Installation with virt-install
3.2.4. Guest Installation with virt-install and Text-based Anaconda
3.2.5. Guest Installation with virt-install Using NAT
3.2.6. Guest Installation with virt-install Using an External DHCP Server
3.2.7. Guest Installation with virt-install Using an External Static IPv4 Address
3.2.8. Optional Workaround to Allow for Graceful Shutdown
3.3. Creating Guests with virt-manager
3.3.1. virt-manager Installation Overview
3.3.2. Creating a Red Hat Enterprise Linux 7 Guest with virt-manager
3.4. Comparison of virt-install and virt-manager Installation options
4. Cloning Virtual Machines
4.1. Cloning Guests with virt-clone
4.2. Cloning Guests with virt-manager
5. KVM Paravirtualized (virtio) Drivers
5.1. Using KVM virtio Drivers for Existing Devices
5.2. Using KVM virtio Drivers for New Devices
6. Network Configuration
6.1. Network Address Translation (NAT) with libvirt
6.2. Disabling vhost-net
6.3. Enabling vhost-net zero-copy
6.4. Bridged Networking
6.4.1. Configuring Bridged Networking on a Red Hat Enterprise Linux 7 Host
6.4.2. Bridged Networking with Virtual Machine Manager
6.4.3. Bridged Networking with libvirt
7. Overcommitting with KVM
7.1. Introduction
7.2. Overcommitting Memory
7.3. Overcommitting Virtualized CPUs (vCPUs)
8. KVM Guest Timing Management
8.1. Required Parameters for Red Hat Enterprise Linux Guests
8.2. Steal Time Accounting
9. Network Booting with libvirt
9.1. Preparing the Boot Server
9.1.1. Setting up a PXE Boot Server on a Private libvirt Network
9.2. Booting a Guest Using PXE
9.2.1. Using bridged networking
9.2.2. Using a Private libvirt Network
10. Registering the Hypervisor and Virtual Machine
10.1. Installing virt-who on the Host Physical Machine
10.2. Registering a New Guest Virtual Machine
10.3. Removing a Guest Virtual Machine Entry
10.4. Installing virt-who Manually
10.5. Troubleshooting virt-who
10.5.1. Why is the hypervisor status red?
10.5.2. I have subscription status errors, what do I do?
11. Enhancing Virtualization with the QEMU Guest Agent and SPICE Agent
11.1. QEMU Guest Agent
11.1.1. Setting up Communication between the QEMU Guest Agent and Host
11.2. Using the QEMU Guest Agent with libvirt
11.2.1. Creating a Guest Disk Backup
11.3. SPICE Agent
11.3.1. Setting up Communication between the SPICE Agent and Host
12. Nested Virtualization
12.1. Overview
12.2. Setup
12.3. Restrictions and Limitations

Chapter 1. System Requirements

Virtualization is available with the KVM hypervisor for Red Hat Enterprise Linux 7 on the Intel 64 and AMD64 architectures. This chapter lists system requirements for running virtual machines, also referred to as VMs.
For information on installing the virtualization packages, see Chapter 2, Installing the Virtualization Packages.

1.1. Host System Requirements

Minimum host system requirements

  • 6 GB free disk space.
  • 2 GB RAM.

Recommended system requirements

  • One core or thread for each virtualized CPU and one for the host.
  • 2 GB of RAM, plus additional RAM for virtual machines.
  • 6 GB disk space for the host, plus the required disk space for the virtual machine(s).
    Most guest operating systems require at least 6 GB of disk space. Additional storage space for each guest depends on their workload.

    Swap space

    Swap space in Linux is used when the amount of physical memory (RAM) is full. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space. While swap space can help machines with a small amount of RAM, it should not be considered a replacement for more RAM. Swap space is located on hard drives, which have a slower access time than physical memory. The size of your swap partition can be calculated from the physical RAM of the host. The Red Hat Customer Portal contains an article on safely and efficiently determining the size of the swap partition: https://access.redhat.com/site/solutions/15244.
    • When using raw image files, the total disk space required is equal to or greater than the sum of the space required by the image files, the 6 GB of space required by the host operating system, and the swap space for the guest.

      Equation 1.1. Calculating required space for guest virtual machines using raw images

      total for raw format = images + hostspace + swap
      For qcow images, you must also calculate the expected maximum storage requirements of the guest (total for qcow format), as qcow and qcow2 images are able to grow as required. To allow for this expansion, first multiply the expected maximum storage requirements of the guest (expected maximum guest storage) by 1.01, and add to this the space required by the host (host), and the necessary swap space (swap).

      Equation 1.2. Calculating required space for guest virtual machines using qcow images

      total for qcow format = (expected maximum guest storage * 1.01) + host + swap
Guest virtual machine requirements are further outlined in Chapter 7, Overcommitting with KVM.

1.2. KVM Hypervisor Requirements

The KVM hypervisor requires:
  • an Intel processor with the Intel VT-x and Intel 64 virtualization extensions for x86-based systems; or
  • an AMD processor with the AMD-V and the AMD64 virtualization extensions.
Virtualization extensions (Intel VT-x or AMD-V) are required for full virtualization. enter the following commands to determine whether your system has the hardware virtualization extensions, and that they are enabled.

Procedure 1.1. Verifying virtualization extensions

  1. Verify the CPU virtualization extensions are available

    enter the following command to verify the CPU virtualization extensions are available:
    $ grep -E 'svm|vmx' /proc/cpuinfo
  2. Analyze the output

    • The following example output contains a vmx entry, indicating an Intel processor with the Intel VT-x extension:
      flags   : fpu tsc msr pae mce cx8 vmx apic mtrr mca cmov pat pse36 clflush
      dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor ds_cpl
      vmx est tm2 cx16 xtpr lahf_lm
      
    • The following example output contains an svm entry, indicating an AMD processor with the AMD-V extensions:
      flags   :  fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush
      mmx fxsr sse sse2 ht syscall nx mmxext svm fxsr_opt lm 3dnowext 3dnow pni cx16
      lahf_lm cmp_legacy svm cr8legacy ts fid vid ttp tm stc
      
    If the grep -E 'svm|vmx' /proc/cpuinfo command returns any output, the processor contains the hardware virtualization extensions. In some circumstances, manufacturers disable the virtualization extensions in the BIOS. If the extensions do not appear, or full virtualization does not work, see Procedure A.4, “Enabling virtualization extensions in BIOS” for instructions on enabling the extensions in your BIOS configuration utility.
  3. Ensure the KVM kernel modules are loaded

    As an additional check, verify that the kvm modules are loaded in the kernel with the following command:
    # lsmod | grep kvm
    If the output includes kvm_intel or kvm_amd, the kvm hardware virtualization modules are loaded.

Note

The virsh utility (provided by the libvirt-client package) can output a full list of your system's virtualization capabilities with the following command:
# virsh capabilities

1.3. KVM Guest Virtual Machine Compatibility

Red Hat Enterprise Linux 7 servers have certain support limits.
The following URLs explain the processor and memory amount limitations for Red Hat Enterprise Linux:
The following URL lists guest operating systems certified to run on a Red Hat Enterprise Linux KVM host:

Note

For additional information on the KVM hypervisor's restrictions and support limits, see Appendix B, Virtualization Restrictions.

1.4. Supported Guest CPU Models

Every hypervisor has its own policy for which CPU features the guest will see by default. The set of CPU features presented to the guest by the hypervisor depends on the CPU model chosen in the guest virtual machine configuration.

1.4.1. Listing the Guest CPU Models

To view a full list of the CPU models supported for an architecture type, run the virsh cpu-models <arch> command. For example:
$ virsh cpu-models x86_64
486
pentium
pentium2
pentium3
pentiumpro
coreduo
n270
core2duo
qemu32
kvm32
cpu64-rhel5
cpu64-rhel6
kvm64
qemu64
Conroe
Penryn
Nehalem
Westmere
SandyBridge
Haswell
athlon
phenom
Opteron_G1
Opteron_G2
Opteron_G3
Opteron_G4
Opteron_G5
$ virsh cpu-models ppc64
POWER7
POWER7_v2.1
POWER7_v2.3
POWER7+_v2.1
POWER8_v1.0
The full list of supported CPU models and features is contained in the cpu_map.xml file, located in /usr/share/libvirt/:
# cat /usr/share/libvirt/cpu_map.xml
A guest's CPU model and features can be changed in the <cpu> section of the domain XML file. See Section 26.12, “CPU Models and Topology” for more information.
The host model can be configured to use a specified feature set as needed. For more information, see Section 26.12.1, “Changing the Feature Set for a Specified CPU”.

Chapter 2. Installing the Virtualization Packages

To use virtualization, the virtualization packages must be installed on your computer. Virtualization packages can be installed either during the host installation sequence or after host installation using the yum command and Subscription Manager.
The KVM hypervisor uses the default Red Hat Enterprise Linux kernel with the kvm kernel module.

2.1. Configuring a Virtualization Host During a New Red Hat Enterprise Linux 7 Installation

This section covers installing virtualization tools and virtualization packages as part of a fresh Red Hat Enterprise Linux installation.

Note

The Red Hat Enterprise Linux 7 Installation Guide covers installing Red Hat Enterprise Linux in detail.

Procedure 2.1. Installing the virtualization package group

  1. Launch the Red Hat Enterprise Linux 7 installation program

    Start an interactive Red Hat Enterprise Linux 7 installation from the Red Hat Enterprise Linux Installation CD-ROM, DVD or PXE.
  2. Continue installation up to software selection

    Complete the other steps up to the software selection step. The Installation Summary screen prompts the user to complete any steps still requiring attention.
    Image shows the Installation Summary screen, which lists configurable options under group headings. Under the Localization heading : Date & Time, Language Support, and Keyboard. Under the Software heading: Installation Source and Software Selection. Under the System heading: Installation Destination and Network & Hostname.

    Figure 2.1. The Installation Summary screen

    Software Selection defaults to Minimal Install. Open the Software Selection screen to select the virtualization packages instead.
  3. Select the server type and package groups

    Red Hat Enterprise Linux 7 has two available options for installing a virtualization host: a minimal virtualization host with only the basic packages installed (step 3.a), or a virtualization host with packages installed to allow management of guests through a graphical user interface (step 3.b).
    1. Selecting a minimal virtualization host

      Select the Virtualization Host radio button under Base Environment, and the Virtualization Platform check box under Add-Ons for Selected Environment. This installs a basic virtualization environment which can be run with virsh, or remotely over the network.
      Image shows the Software Selection screen, which lists options under two headings: Base Environment and Add-Ons for Selected Environment. Virtualization Host is highlighted from the options under Base Environment, and Virtualization Platform is highlighted from the options under Add-Ons for Selected Environment.

      Figure 2.2. Virtualization Host selected in the software selection screen

    2. Selecting a virtualization host with a graphical user interface

      Select the Server with GUI radio button under Base Environment, and the check boxes for Virtualization Client, Virtualization Hypervisor, and Virtualization Tools under Add-Ons for Selected Environment. This installs a virtualization environment along with graphical tools for installing and managing guest virtual machines.
      Image shows the Software Selection screen, which lists options under two headings: Base Environment and Add-Ons for Selected Environment. Server with GUI is highlighted from the options under Base Environment, and Virtualization Client, Virtualization Hypervisor, and Virtualization Tools are highlighted from the options under Add-Ons for Selected Environment.

      Figure 2.3. Server with GUI selected in the software selection screen

  4. Finalize installation

    On the Installation Summary screen, complete the steps as necessary and click Begin Installation.
When the installation is complete, reboot the system.

Important

You require a valid virtualization entitlement to receive updates for the virtualization packages.

2.1.1. Installing KVM Packages with Kickstart Files

Kickstart files allow for large, automated installations without a user manually installing each individual host system.
To use a Kickstart file to install Red Hat Enterprise Linux with the virtualization packages, append the following package groups in the %packages section of your Kickstart file:
@virtualization-hypervisor
@virtualization-client
@virtualization-platform
@virtualization-tools
For more information about installing with Kickstart files, see the Red Hat Enterprise Linux 7 Installation Guide.

2.2. Installing Virtualization Packages on an Existing Red Hat Enterprise Linux System

This section describes the steps for installing the KVM hypervisor on an existing Red Hat Enterprise Linux 7 system.
To install the packages, your machine must be registered and subscribed to the Red Hat Customer Portal. To register via Red Hat Subscription Manager, run the subscription-manager register command and follow the prompts. Alternatively, run the Red Hat Subscription Manager application from ApplicationsSystem Tools on the desktop to register.
If you do not have a valid Red Hat subscription, visit the Red Hat online store to obtain one. For more information on registering and subscribing a system to the Red Hat Customer Portal, see https://access.redhat.com/solutions/253273.

2.2.1. Installing the Virtualization Packages with yum

To use virtualization on Red Hat Enterprise Linux, you require at minimum the qemu-kvm, qemu-img, and libvirt packages. These packages provide the user-level KVM emulator, disk image manager, and virtualization management tools on the host Red Hat Enterprise Linux system.
The libvirt package provides the server and host-side libraries for interacting with hypervisors and host systems, and the libvirtd daemon that handles the library calls, manages virtual machines and controls the hypervisor.
To install these packages, enter the following command:
# yum install qemu-kvm qemu-img libvirt
Several additional virtualization management packages are also available and are recommended when using virtualization:

Recommended virtualization packages

virt-install
This package provides the virt-install command for creating virtual machines from the command line.
libvirt-python
The libvirt-python package contains a module that permits applications written in the Python programming language to use the interface supplied by the libvirt API.
virt-manager
virt-manager, also known as Virtual Machine Manager, provides a graphical tool for administering virtual machines. It uses the libvirt-client library as the management API.
libvirt-client
The libvirt-client package provides the client-side APIs and libraries for accessing libvirt servers. The libvirt-client package includes the virsh command-line tool to manage and control virtual machines and hypervisors from the command line or a special virtualization shell.
Install all of these recommended virtualization packages with the following command:
# yum install virt-install libvirt-python virt-manager virt-install libvirt-client

2.2.2. Installing Virtualization Package Groups

The virtualization packages can also be installed from package groups. The following table describes the virtualization package groups and what they provide.

Note

Note that the qemu-img package is installed as a dependency of the Virtualization package group if it is not already installed on the system. It can also be installed manually with the yum install qemu-img command as described previously.

Table 2.1. Virtualization Package Groups

Package Group Description Mandatory Packages Optional Packages
Virtualization Hypervisor Smallest possible virtualization host installation libvirt, qemu-kvm qemu-kvm-tools
Virtualization Client Clients for installing and managing virtualization instances gnome-boxes, virt-install, virt-manager, virt-viewer virt-top, libguestfs-tools, libguestfs-tools-c
Virtualization Platform Provides an interface for accessing and controlling virtual machines and containers libvirt, libvirt-client, virt-who fence-virtd-libvirt, fence-virtd-multicast, fence-virtd-serial, libvirt-cim, libvirt-java, libvirt-snmp, perl-Sys-Virt
Virtualization Tools Tools for offline virtual image management libguestfs libguestfs-java, libguestfs-tools, libguestfs-tools-c
To install a package group, run the yum groupinstall <package group> command. For instance, to install the Virtualization Tools package group, run:
# yum groupinstall "Virtualization Tools"

Chapter 3. Installing a Virtual Machine

After you have installed the virtualization packages on your Red Hat Enterprise Linux 7 host system, you can create guest operating systems. You can create guest virtual machines using the New button in virt-manager or use the virt-install command-line interface to install virtual machines by a list of parameters or with a script. Both methods are covered by this chapter.
Detailed installation instructions are available in the following chapters for specific versions of Red Hat Enterprise Linux.

3.1. Guest Virtual Machine Prerequisites and Considerations

Various factors should be considered before creating any guest virtual machines. Not only should the role of a virtual machine be considered before deployment, but regular ongoing monitoring and assessment based on variable factors (load, amount of clients) should be performed. Some factors include:
Performance
Guest virtual machines should be deployed and configured based on their intended tasks. Some guest systems (for instance, guests running a database server) may require special performance considerations. Guests may require more assigned CPUs or memory based on their role and projected system load.
Input/Output requirements and types of Input/Output
Some guest virtual machines may have a particularly high I/O requirement or may require further considerations or projections based on the type of I/O (for instance, typical disk block size access, or the amount of clients).
Storage
Some guest virtual machines may require higher priority access to storage or faster disk types, or may require exclusive access to areas of storage. The amount of storage used by guests should also be regularly monitored and taken into account when deploying and maintaining storage. Make sure to read all the considerations outlined in Chapter 13, Securing the Host Physical Machine and Improving Performance. It is also important to understand that your physical storage may limit your options in your virtual storage.
Networking and network infrastructure
Depending upon your environment, some guest virtual machines could require faster network links than other guests. Bandwidth or latency are often factors when deploying and maintaining guests, especially as requirements or load changes.
Request requirements
SCSI requests can only be issued to guest virtual machines on virtio drives if the virtio drives are backed by whole disks, and the disk device parameter is set to lun, as shown in the following example:
<devices>
   <emulator>/usr/libexec/qemu-kvm</emulator>
   <disk type='block' device='lun'>

3.2. Creating Guests with virt-install

You can use the virt-install command to create virtual machines from the command line. virt-install is used either interactively (with a graphical application such as virt-viewer) or as part of a script to automate the creation of virtual machines. Using virt-install with kickstart files allows for unattended installation of virtual machines.
Note that you need root privileges in order for virt-install commands to complete successfully.
The virt-install tool provides a number of options that can be passed on the command line. Most virt-install options are not required. Minimum requirements are --name, --ram, guest storage (--disk, --filesystem or --nodisks), and an install option. To see a complete list of options, enter the following command:
# virt-install --help
The virt-install man page also documents each command option, important variables, and examples.
Prior to running virt-install, qemu-img is a related command which can be used to configure storage options. See refer to Chapter 16, Using qemu-img for instructions on using qemu-img.

3.2.1. Network Installation with virt-install

The following examples use a network bridge (named br0 in these examples), which must be created separately, prior to running virt-install. For details on creating a network bridge, see Section 6.4.1, “Configuring Bridged Networking on a Red Hat Enterprise Linux 7 Host”.

Example 3.1. Using virt-install to install a Red Hat Enterprise Linux 6 virtual machine

This example creates a Red Hat Enterprise Linux 6 guest:
# virt-install \
   --name=guest1-rhel7-64 \
   --disk path=/var/lib/libvirt/images/guest1-rhel7-64.dsk,size=8,sparse=false,cache=none \
   --graphics spice \
   --vcpus=2 --ram=2048 \
   --location=http://example1.com/installation_tree/RHEL7.2-Server-x86_64/os \
   --network bridge=br0 \
   --os-type=linux \
   --os-variant=rhel7
The options used in this example are as follows:
--name
The name of the virtual machine.
--disk
Specifies storage configuration details for the virtual machine.
--graphics
An important option which allows graphical installation of a virtual machine. It specifies the type of graphical tool to use for interactive installation, or can be set to none for a fully automated installation by script.
--vcpus
The number of vCPUs to allocate to the guest.
--ram
The amount of memory (RAM) to allocate to the guest, in MiB.
--location
The location of the installation media. The above example uses an http:// network installation, but several other protocols can be used.
--network bridge
Specifies the network bridge to use for installation, which must be configured before running virt-install. For details on creating a network bridge, see Section 6.4.1, “Configuring Bridged Networking on a Red Hat Enterprise Linux 7 Host”.
--os-type
The guest operating system type.
--os-variant
Another important option in virt-install, used to further optimize the guest configuration. Using this option can reduce installation time and improve performance.
Running the osinfo-query os command returns a complete list of operating system variants identified by a short ID to use with the --os-variant option. For example, --os-variant=rhel7.0 configures a Red Hat Enterprise Linux 7.0 virtual machine.
In Red Hat Enterprise Linux 7, the virtio-scsi controller is available for use in guests. If both the host and guest support virtio-scsi, you can use it as follows:

Example 3.2. Using virt-install to install a guest virtual machine with the virtio-scsi controller

The items in bold are required on top of a standard installation in order to use the virtio-scsi controller.
# virt-install \
   --name=guest1-rhel7 \
   --controller type=scsi,model=virtio-scsi \
   --disk path=/var/lib/libvirt/images/guest1-rhel7.dsk,size=8,sparse=false,cache=none,bus=scsi \
   --graphics spice \
   --vcpus=2 --ram=2048 \
   --location=http://example1.com/installation_tree/RHEL7.1-Server-x86_64/os \
   --network bridge=br0 \
   --os-type=linux \
   --os-variant=rhel7

Note

Ensure that you select the correct --os-type for your operating system when running this command. This option prevents the installation disk from disconnecting when rebooting during the installation procedure. The --os-variant option further optimizes the configuration for a specific guest operating system.

3.2.2. PXE Installation with virt-install

virt-install PXE installations require both the --network=bridge:bridge_name parameter, where bridge_name is the name of the bridge, and the --pxe parameter.
By default, if no network is found, the guest virtual machine will attempt to boot from alternative bootable devices. If there is no other bootable device found, the guest virtual machine will pause. You can use the qemu-kvm boot parameter reboot-timeout to allow the guest to retry booting if no bootable device is found, like so:
# qemu-kvm -boot reboot-timeout=1000

Example 3.3. Fully-virtualized PXE installation with virt-install

# virt-install --hvm --connect qemu:///system \
--network=bridge:br0 --pxe --graphics spice \
--name=rhel7-machine --ram=756 --vcpus=4 \
--os-type=linux --os-variant=rhel7 \
--disk path=/var/lib/libvirt/images/rhel7-machine.img,size=10
Note that the command above cannot be executed in a text-only environment. A fully-virtualized (--hvm) guest can only be installed in a text-only environment if the --location and --extra-args "console=console_type" are provided instead of the --graphics spice parameter.

3.2.3. Kickstart Installation with virt-install

The following example shows using a kickstart file with virt-install:

Example 3.4. Kickstart Installation with virt-install

# virt-install -n rhel7ks-guest -r 1024 --file=/var/lib/libvirt/images/rhel7ks-guest.img --file-size=10 \
 --location /var/lib/libvirt/images/rhel-server-7.1-x86_64-dvd.iso --nographics \
 --extra-args="ks=http://192.168.122.1/ks.cfg ip=dhcp \
console=tty0 console=ttyS0,115200n8” --os-variant=rhel7.0

3.2.4. Guest Installation with virt-install and Text-based Anaconda

The following example shows using virt-install with text-based Anaconda installation:

Example 3.5. Guest installation with virt-install and text-based Anaconda

# virt-install -n rhel7anaconda-guest -r 1024 --disk=path=/path/to/rhel7anaconda-guest.img,size=10 \
--location /mnt/RHEL7DVD --nographics \
--extra-args=”console=tty0 console=ttyS0,115200n8” \
--disk=path=/path/to/rhel7-dvd.iso,device=cdrom
Refer to the Red Hat Enterprise Linux 7 Installation Guide for further instructions for installations using text-based Anaconda.

3.2.5. Guest Installation with virt-install Using NAT

The following example shows using virt-install with libvirtd's network address translation (NAT) virtual network switch:

Example 3.6. Guest installation with virt-install using NAT

# virt-install --network=default \
      --name=rhel7-machine --ram=756 --vcpus=4 \
      --os-type=linux --os-variant=rhel7

3.2.6. Guest Installation with virt-install Using an External DHCP Server

The following example shows using virt-install to install a guest from an external dynamic host configuration protocol (DHCP) server. Before this installation, a bridge br0 interface must be set up, with the host machine's eth0 interface as the slave.

Example 3.7. Guest installation with virt-install using NAT using an external DHCP server

# virt-install --network=br0 \
        --name=rhel7-machine --ram=756 --vcpus=4 \
        --os-type=linux --os-variant=rhel7
Refer to the Red Hat Enterprise Linux 7 Networking Guide for details about setting up an external DHCP server.

3.2.7. Guest Installation with virt-install Using an External Static IPv4 Address

The following example shows using virt-install to install a guest from an external static IPv4 address. Before this installation, a bridge br0 interface must be set up, with the host machine's eth0 interface as the slave.

Example 3.8. Guest installation with virt-install using an external static IPv4 address

# virt-install
        --network=br0 \
        --name=rhel7-machine --ram=756 --vcpus=4 \
        --os-type=linux --os-variant=rhel7 \
        --extra-args=”ip=192.168.1.2::192.168.1.1:255.255.255.0:test.example.com:eth0:none”
The arguments of the --extra-args parameter specify the static network setting, using to the following structure:
ip=[ip]::[gateway]:[netmask]:[hostname]:[interface]:[autoconf]

3.2.8. Optional Workaround to Allow for Graceful Shutdown

The libvirt-guests service has parameter settings that can be configured to assure that the guest can shutdown properly. It is a package that is a part of the libvirt installation and is installed by default. This service automatically saves guests to the disk when the host shuts down, and restores them to their pre-shutdown state when the host reboots. By default, this setting is set to suspend the guest. If you want the guest to be gracefully shutdown, you will need to change one of the parameters of the libvirt-guests configuration file.

Procedure 3.1. Changing the libvirt-guests service parameters to allow for the graceful shutdown of guests

The procedure described here allows for the graceful shutdown of guest virtual machines when the host physical machine is stuck, powered off, or needs to be restarted.
  1. Open the configuration file

    The configuration file is located in /etc/sysconfig/libvirt-guests. Edit the file, remove the comment mark (#) and change the ON_SHUTDOWN=suspend to ON_SHUTDOWN=shutdown. Remember to save the change.
    $ vi /etc/sysconfig/libvirt-guests
    
    # URIs to check for running guests
    # example: URIS='default xen:/// vbox+tcp://host/system lxc:///'                   1
    #URIS=default
    
    # action taken on host boot
    # - start   all guests which were running on shutdown are started on boot
    #           regardless on their autostart settings
    # - ignore  libvirt-guests init script won't start any guest on boot, however,
    #           guests marked as autostart will still be automatically started by
    #           libvirtd                                                               2
    #ON_BOOT=start
    
    # Number of seconds to wait between each guest start. Set to 0 to allow            3
    # parallel startup.
    #START_DELAY=0
    
    # action taken on host shutdown
    # - suspend   all running guests are suspended using virsh managedsave
    # - shutdown  all running guests are asked to shutdown. Please be careful with
    #             this settings since there is no way to distinguish between a
    #             guest which is stuck or ignores shutdown requests and a guest
    #             which just needs a long time to shutdown. When setting
    #             ON_SHUTDOWN=shutdown, you must also set SHUTDOWN_TIMEOUT to a
    #             value suitable for your guests.                                      4
    ON_SHUTDOWN=shutdown
    
    # If set to non-zero, shutdown will suspend guests concurrently. Number of
    # guests on shutdown at any time will not exceed number set in this variable.      5
    #PARALLEL_SHUTDOWN=0
    
    # Number of seconds we're willing to wait for a guest to shut down. If parallel
    # shutdown is enabled, this timeout applies as a timeout for shutting down all
    # guests on a single URI defined in the variable URIS. If this is 0, then there
    # is no time out (use with caution, as guests might not respond to a shutdown
    # request). The default value is 300 seconds (5 minutes).                          6
    #SHUTDOWN_TIMEOUT=300
    
    # If non-zero, try to bypass the file system cache when saving and
    # restoring guests, even though this may give slower operation for
    # some file systems.                                                               7
    #BYPASS_CACHE=0

    1

    URIS - checks the specified connections for a running guest. The Default setting functions in the same manner as virsh does when no explicit URI is set In addition, one can explicitly set the URI from /etc/libvirt/libvirt.conf. It should be noted that when using the libvirt configuration file default setting, no probing will be used.

    2

    ON_BOOT - specifies the action to be done to / on the guests when the host boots. The start option starts all guests that were running prior to shutdown regardless on their autostart settings. The ignore option will not start the formally running guest on boot, however, any guest marked as autostart will still be automatically started by libvirtd.

    3

    The START_DELAY - sets a delay interval in between starting up the guests. This time period is set in seconds. Use the 0 time setting to make sure there is no delay and that all guests are started simultaneously.

    4

    ON_SHUTDOWN - specifies the action taken when a host shuts down. Options that can be set include: suspend which suspends all running guests using virsh managedsave and shutdown which shuts down all running guests. It is best to be careful with using the shutdown option as there is no way to distinguish between a guest which is stuck or ignores shutdown requests and a guest that just needs a longer time to shutdown. When setting the ON_SHUTDOWN=shutdown, you must also set SHUTDOWN_TIMEOUT to a value suitable for the guests.

    5

    PARALLEL_SHUTDOWN Dictates that the number of guests on shutdown at any time will not exceed number set in this variable and the guests will be suspended concurrently. If set to 0, then guests are not shutdown concurrently.

    6

    Number of seconds to wait for a guest to shut down. If SHUTDOWN_TIMEOUT is enabled, this timeout applies as a timeout for shutting down all guests on a single URI defined in the variable URIS. If SHUTDOWN_TIMEOUT is set to 0, then there is no time out (use with caution, as guests might not respond to a shutdown request). The default value is 300 seconds (5 minutes).

    7

    BYPASS_CACHE can have 2 values, 0 to disable and 1 to enable. If enabled it will by-pass the file system cache when guests are restored. Note that setting this may effect performance and may cause slower operation for some file systems.
  2. Start libvirt-guests service

    If you have not started the service, start the libvirt-guests service. Do not restart the service as this will cause all running guest virtual machines to shutdown.

3.3. Creating Guests with virt-manager

virt-manager, also known as Virtual Machine Manager, is a graphical tool for creating and managing guest virtual machines.
This section covers how to install a Red Hat Enterprise Linux 7 guest virtual machine on a Red Hat Enterprise Linux 7 host using virt-manager.
These procedures assume that the KVM hypervisor and all other required packages are installed and the host is configured for virtualization. For more information on installing the virtualization packages, refer to Chapter 2, Installing the Virtualization Packages.

3.3.1. virt-manager Installation Overview

The New VM wizard breaks down the virtual machine creation process into five steps:
  1. Choosing the hypervisor and installation type
  2. Locating and configuring the installation media
  3. Configuring memory and CPU options
  4. Configuring the virtual machine's storage
  5. Configuring virtual machine name, networking, architecture, and other hardware settings
Ensure that virt-manager can access the installation media (whether locally or over the network) before you continue.

3.3.2. Creating a Red Hat Enterprise Linux 7 Guest with virt-manager

This procedure covers creating a Red Hat Enterprise Linux 7 guest virtual machine with a locally stored installation DVD or DVD image. Red Hat Enterprise Linux 7 DVD images are available from the Red Hat Customer Portal.

Procedure 3.2. Creating a Red Hat Enterprise Linux 7 guest virtual machine with virt-manager using local installation media

  1. Optional: Preparation

    Prepare the storage environment for the virtual machine. For more information on preparing storage, refer to Chapter 14, Storage Pools.

    Important

    Various storage types may be used for storing guest virtual machines. However, for a virtual machine to be able to use migration features, the virtual machine must be created on networked storage.
    Red Hat Enterprise Linux 7 requires at least 1 GB of storage space. However, Red Hat recommends at least 5 GB of storage space for a Red Hat Enterprise Linux 7 installation and for the procedures in this guide.
  2. Open virt-manager and start the wizard

    Open virt-manager by executing the virt-manager command as root or opening ApplicationsSystem ToolsVirtual Machine Manager. Alternatively, run the virt-manager command as root.
    The Virtual Machine Manager window

    Figure 3.1. The Virtual Machine Manager window

    Optionally, open a remote hypervisor by selecting the hypervisor and clicking the Connect button.
    Click to start the new virtualized guest wizard.
    The New VM window opens.
  3. Specify installation type

    Select the installation type:
    Local install media (ISO image or CDROM)
    This method uses a CD-ROM, DVD, or image of an installation disk (for example, .iso).
    Network Install (HTTP, FTP, or NFS)
    This method involves the use of a mirrored Red Hat Enterprise Linux or Fedora installation tree to install a guest. The installation tree must be accessible through either HTTP, FTP, or NFS.
    If you select Network Install, provide the installation URL and also Kernel options, if required.
    Network Boot (PXE)
    This method uses a Preboot eXecution Environment (PXE) server to install the guest virtual machine. Setting up a PXE server is covered in the Red Hat Enterprise Linux 7 Installation Guide. To install via network boot, the guest must have a routable IP address or shared network device.
    If you select Network Boot, continue to STEP 5. After all steps are completed, a DHCP request is sent and if a valid PXE server is found the guest virtual machine's installation processes will start.
    Import existing disk image
    This method allows you to create a new guest virtual machine and import a disk image (containing a pre-installed, bootable operating system) to it.
    Virtual machine installation method

    Figure 3.2. Virtual machine installation method

    Click Forward to continue.
  4. Select the installation source

    1. If you selected Local install media (ISO image or CDROM), specify your desired local installation media.
      Local ISO image installation

      Figure 3.3. Local ISO image installation

      • If you wish to install from a CD-ROM or DVD, select the Use CDROM or DVD radio button, and select the appropriate disk drive from the drop-down list of drives available.
      • If you wish to install from an ISO image, select Use ISO image, and then click the Browse... button to open the Locate media volume window.
        Select the installation image you wish to use, and click Choose Volume.
        If no images are displayed in the Locate media volume window, click the Browse Local button to browse the host machine for the installation image or DVD drive containing the installation disk. Select the installation image or DVD drive containing the installation disk and click Open; the volume is selected for use and you are returned to the Create a new virtual machine wizard.

        Important

        For ISO image files and guest storage images, the recommended location to use is /var/lib/libvirt/images/. Any other location may require additional configuration by SELinux. Refer to the Red Hat Enterprise Linux Virtualization Security Guide or the Red Hat Enterprise Linux SELinux User's and Administrator's Guide for more details on configuring SELinux.
    2. If you selected Network Install, input the URL of the installation source and also the required Kernel options, if any. The URL must point to the root directory of an installation tree, which must be accessible through either HTTP, FTP, or NFS.
      To perform a kickstart installation, specify the URL of a kickstart file in Kernel options, starting with ks=
      Network kickstart installation

      Figure 3.4. Network kickstart installation

      Note

      For a complete list of kernel options, see the Red Hat Enterprise Linux 7 Installation Guide.
    Next, configure the OS type and Version of the installation. Ensure that you select the appropriate operating system type for your virtual machine. This can be specified manually or by selecting the Automatically detect operating system based on install media check box.
    Click Forward to continue.
  5. Configure memory (RAM) and virtual CPUs

    Specify the number of CPUs and amount of memory (RAM) to allocate to the virtual machine. The wizard shows the number of CPUs and amount of memory you can allocate; these values affect the host's and guest's performance.
    Virtual machines require sufficient physical memory (RAM) to run efficiently and effectively. Red Hat supports a minimum of 512MB of RAM for a virtual machine. Red Hat recommends at least 1024MB of RAM for each logical core.
    Assign sufficient virtual CPUs for the virtual machine. If the virtual machine runs a multi-threaded application, assign the number of virtual CPUs the guest virtual machine will require to run efficiently.
    You cannot assign more virtual CPUs than there are physical processors (or hyper-threads) available on the host system. The number of virtual CPUs available is noted in the Up to X available field.
    Configuring Memory and CPU

    Figure 3.5. Configuring Memory and CPU

    After you have configured the memory and CPU settings, click Forward to continue.

    Note

    Memory and virtual CPUs can be overcommitted. For more information on overcommitting, refer to Chapter 7, Overcommitting with KVM.
  6. Configure storage

    Enable and assign sufficient space for your virtual machine and any applications it requires. Assign at least 5 GB for a desktop installation or at least 1 GB for a minimal installation.
    Configuring virtual storage

    Figure 3.6. Configuring virtual storage

    Note

    Live and offline migrations require virtual machines to be installed on shared network storage. For information on setting up shared storage for virtual machines, refer to Section 17.4, “Shared Storage Example: NFS for a Simple Migration”.
    1. With the default local storage

      Select the Create a disk image on the computer's hard drive radio button to create a file-based image in the default storage pool, the /var/lib/libvirt/images/ directory. Enter the size of the disk image to be created. If the Allocate entire disk now check box is selected, a disk image of the size specified will be created immediately. If not, the disk image will grow as it becomes filled.

      Note

      Although the storage pool is a virtual container it is limited by two factors: maximum size allowed to it by qemu-kvm and the size of the disk on the host physical machine. Storage pools may not exceed the size of the disk on the host physical machine. The maximum sizes are as follows:
      • virtio-blk = 2^63 bytes or 8 Exabytes(using raw files or disk)
      • Ext4 = ~ 16 TB (using 4 KB block size)
      • XFS = ~8 Exabytes
      • qcow2 and host file systems keep their own metadata and scalability should be evaluated/tuned when trying very large image sizes. Using raw disks means fewer layers that could affect scalability or max size.
      Click Forward to create a disk image on the local hard drive. Alternatively, select Select managed or other existing storage, then select Browse to configure managed storage.
    2. With a storage pool

      If you select Select managed or other existing storage to use a storage pool, click Browse to open the Locate or create storage volume window.
      The Choose Storage Volume window

      Figure 3.7. The Choose Storage Volume window

      1. Select a storage pool from the Storage Pools list.
      2. Optional: Click to create a new storage volume. The Add a Storage Volume screen will appear. Enter the name of the new storage volume.
        Choose a format option from the Format drop-down menu. Format options include raw, qcow2, and qed. Adjust other fields as desired. Note that the qcow2 version used here is version 3. To change the qcow version refer to Section 26.20.2, “Setting Target Elements”
        The Add a Storage Volume window

        Figure 3.8. The Add a Storage Volume window

    Select the new volume and click Choose volume. Next, click Finish to return to the New VM wizard. Click Forward to continue.
  7. Name and final configuration

    Name the virtual machine. Virtual machine names can contain letters, numbers and the following characters: underscores (_), periods (.), and hyphens (-). Virtual machine names must be unique for migration and cannot consist only of numbers.
    By default, the virtual machine will be created with network address translation (NAT) for a network called 'default' . To change the network selection, click Network selection and select a host device and source mode.
    Verify the settings of the virtual machine and click Finish when you are satisfied; this will create the virtual machine with specified networking settings, virtualization type, and architecture.
    Verifying the configuration

    Figure 3.9. Verifying the configuration

    Or, to further configure the virtual machine's hardware, check the Customize configuration before install check box to change the guest's storage or network devices, to use the paravirtualized (virtio) drivers or to add additional devices. This opens another wizard that will allow you to add, remove, and configure the virtual machine's hardware settings.

    Note

    Red Hat Enterprise Linux 4 or Red Hat Enterprise Linux 5 guest virtual machines cannot be installed using graphical mode. As such, you must select "Cirrus" instead of "QXL" as a video card.
    After configuring the virtual machine's hardware, click Apply. virt-manager will then create the virtual machine with your specified hardware settings.

    Warning

    When installing a Red Hat Enterprise Linux 7 guest virtual machine from a remote medium but without a configured TCP/IP connection, the installation fails. However, when installing a guest virtual machine of Red Hat Enterprise Linux 5 or 6 in such circumstances, the installer opens a "Configure TCP/IP" interface.
    For further information about this difference, see the related knowledgebase article.
    Click Finish to continue into the Red Hat Enterprise Linux installation sequence. For more information on installing Red Hat Enterprise Linux 7, refer to the Red Hat Enterprise Linux 7 Installation Guide.
A Red Hat Enterprise Linux 7 guest virtual machine is now created from an ISO installation disk image.

3.4. Comparison of virt-install and virt-manager Installation options

This table provides a quick reference to compare equivalent virt-install and virt-manager installation options for when installing a virtual machine.
Most virt-install options are not required. Minimum requirements are --name, --ram, guest storage (--disk, --filesystem or --nodisks), and an install option. These options are further specified with arguments; to see a complete list of command options and related arguments, enter the following command:
# virt-install --help
In virt-manager, at minimum, a name, installation method, memory (RAM), vCPUs, storage are required.

Table 3.1. virt-install and virt-manager configuration comparison for guest installations

Configuration on virtual machine virt-install option virt-manager installation wizard label and step number
Virtual machine name --name, -n Name (step 5)
RAM to allocate (MiB) --ram, -r Memory (RAM) (step 3)
Storage - specify storage media --disk Enable storage for this virtual machine → Create a disk image on the computer's hard drive, or Select managed or other existing storage (step 4)
Storage - export a host directory to the guest --filesystem Enable storage for this virtual machine → Select managed or other existing storage (step 4)
Storage - configure no local disk storage on the guest --nodisks Deselect the Enable storage for this virtual machine check box (step 4)
Installation media location (local install) --file Local install media → Locate your install media (steps 1-2)
Installation via distribution tree (network install) --location Network install → URL (steps 1-2)
Install guest with PXE --pxe Network boot (step 1)
Number of vCPUs --vcpus CPUs (step 3)
Host network --network Advanced options drop-down menu (step 5)
Operating system --os-type OS type (step 2)
Operating system variant/version --os-variant Version (step 2)
Graphical display method --graphics, --nographics * virt-manager provides GUI installation only

Chapter 4. Cloning Virtual Machines

Cloning virtual machines provides a quick and easy method for creating guest virtual machines that are identical to an existing virtual machine. Cloning can be used to set up a network of identical virtual machines. You can also use cloning by making minor changes to otherwise identical virtual machines, to see the effect of the changes on performance.
After cloning a machine, there may be parameters that need to be configured manually, for example, the MAC address of the cloned virtual machine.
You can clone guest virtual machines using the Clone menu option in virt-manager or use the virt-clone command. Both methods are described in this chapter.

Note

For information on cloning volumes, see Section 15.3, “Cloning Volumes”.

4.1. Cloning Guests with virt-clone

You can use virt-clone to clone virtual machines from the command line.
Note that you need root privileges for virt-clone to complete successfully.
The virt-clone command provides a number of options that can be passed on the command line. These include general options, storage configuration options, networking configuration options, and miscellaneous options. Only the --original is required. To see a complete list of options, enter the following command:
# virt-clone --help
The virt-clone man page also documents each command option, important variables, and examples.
The following example shows how to clone a guest virtual machine called "demo" on the default connection, automatically generating a new name and disk clone path.

Example 4.1. Using virt-clone to clone a guest

# virt-clone --original demo --auto-clone
The following example shows how to clone a QEMU guest virtual machine called "demo" with multiple disks.

Example 4.2. Using virt-clone to clone a guest

# virt-clone --connect qemu:///system --original demo --name newdemo --file /var/lib/xen/images/newdemo.img --file /var/lib/xen/images/newdata.img

4.2. Cloning Guests with virt-manager

This procedure describes cloning a guest virtual machine using the virt-manager utility.

Procedure 4.1. Cloning a Virtual Machine with virt-manager

  1. Open virt-manager

    Start virt-manager. Launch the Virtual Machine Manager application from the Applications menu and System Tools submenu. Alternatively, run the virt-manager command as root.
    Select the guest virtual machine you want to clone from the list of guest virtual machines in Virtual Machine Manager.
    Right-click the guest virtual machine you want to clone and select Clone. The Clone Virtual Machine window opens.
    Clone Virtual Machine window

    Figure 4.1. Clone Virtual Machine window

  2. Configure the clone

    • To change the name of the clone, enter a new name for the clone.
    • To change the networking configuration, click Details.
      Enter a new MAC address for the clone.
      Click OK.
      Change MAC Address window

      Figure 4.2. Change MAC Address window

    • For each disk in the cloned guest virtual machine, select one of the following options:
      • Clone this disk - The disk will be cloned for the cloned guest virtual machine
      • Share disk with guest virtual machine name - The disk will be shared by the guest virtual machine that will be cloned and its clone
      • Details - Opens the Change storage path window, which enables selecting a new path for the disk
        Change storage path window

        Figure 4.3. Change storage path window

  3. Clone the guest virtual machine

    Click Clone.

Chapter 5. KVM Paravirtualized (virtio) Drivers

Paravirtualized drivers enhance the performance of guests, decreasing guest I/O latency and increasing throughput to near bare-metal levels. It is recommended to use the paravirtualized drivers for fully virtualized guests running I/O heavy tasks and applications.
Virtio drivers are KVM's paravirtualized device drivers, available for guest virtual machines running on KVM hosts. These drivers are included in the virtio package. The virtio package supports block (storage) devices and network interface controllers.
The KVM virtio drivers are automatically loaded and installed on the following:
  • Red Hat Enterprise Linux 4.8 and newer
  • Red Hat Enterprise Linux 5.3 and newer
  • Red Hat Enterprise Linux 6 and newer
  • Red Hat Enterprise Linux 7 and newer
  • Some versions of Linux based on the 2.6.27 kernel or newer kernel versions.
Versions of Red Hat Enterprise Linux in the list above detect and install the drivers; additional installation steps are not required.
In Red Hat Enterprise Linux 3 (3.9 and above), manual installation is required.

Note

PCI devices are limited by the virtualized system architecture. Refer to Chapter 18, Guest Virtual Machine Device Configuration for additional limitations when using assigned devices.

Note

Network connectivity issues sometimes arise when attempting to use older virtio drivers with newer versions of QEMU. Keeping the drivers up to date is therefore recommended.

5.1. Using KVM virtio Drivers for Existing Devices

You can modify an existing hard disk device attached to the guest to use the virtio driver instead of the virtualized IDE driver. The example shown in this section edits libvirt configuration files. Note that the guest virtual machine does not need to be shut down to perform these steps, however the change will not be applied until the guest is completely shut down and rebooted.

Procedure 5.1. Using KVM virtio drivers for existing devices

  1. Ensure that you have installed the appropriate driver (viostor), before continuing with this procedure.
  2. Run the virsh edit <guestname> command as root to edit the XML configuration file for your device. For example, virsh edit guest1. The configuration files are located in /etc/libvirt/qemu.
  3. Below is a file-based block device using the virtualized IDE driver. This is a typical entry for a virtual machine not using the virtio drivers.
    <disk type='file' device='disk'>
       <source file='/var/lib/libvirt/images/disk1.img'/>
       <target dev='hda' bus='ide'/>
    </disk>
  4. Change the entry to use the virtio device by modifying the bus= entry to virtio. Note that if the disk was previously IDE it will have a target similar to hda, hdb, or hdc and so on. When changing to bus=virtio the target needs to be changed to vda, vdb, or vdc accordingly.
    <disk type='file' device='disk'>
       <source file='/var/lib/libvirt/images/disk1.img'/>
       <target dev='vda' bus='virtio'/>
    </disk>
  5. Remove the address tag inside the disk tags. This must be done for this procedure to work. Libvirt will regenerate the address tag appropriately the next time the virtual machine is started.
Alternatively, virt-manager, virsh attach-disk or virsh attach-interface can add a new device using the virtio drivers.
Refer to the libvirt website for more details on using Virtio: http://www.linux-kvm.org/page/Virtio

5.2. Using KVM virtio Drivers for New Devices

This procedure covers creating new devices using the KVM virtio drivers with virt-manager.
Alternatively, the virsh attach-disk or virsh attach-interface commands can be used to attach devices using the virtio drivers.

Important

Ensure the drivers have been installed on the guest before proceeding to install new devices. If the drivers are unavailable the device will not be recognized and will not work.

Procedure 5.2. Adding a storage device using the virtio storage driver

  1. Open the guest virtual machine by double clicking on the name of the guest in virt-manager.
  2. Open the Show virtual hardware details tab by clicking the lightbulb button.
    The Show virtual hardware details tab

    Figure 5.1. The Show virtual hardware details tab

  3. In the Show virtual hardware details tab, click on the Add Hardware button.
  4. Select hardware type

    Select Storage as the Hardware type.
    The Add new virtual hardware wizard with Storage selected as the hardware type.

    Figure 5.2. The Add new virtual hardware wizard

  5. Select the storage device and driver

    Create a new disk image or select a storage pool volume.
    Set the Device type to Disk device and the Bus type to VirtIO to use the virtio drivers.
    The Add new virtual hardware wizard Storage window, with "Create a disk image on the computer's hard drive" selected.

    Figure 5.3. The Add new virtual hardware wizard

    Click Finish to complete the procedure.

Procedure 5.3. Adding a network device using the virtio network driver

  1. Open the guest virtual machine by double clicking on the name of the guest in virt-manager.
  2. Open the Show virtual hardware details tab by clicking the lightbulb button.
    The Show virtual hardware details tab

    Figure 5.4. The Show virtual hardware details tab

  3. In the Show virtual hardware details tab, click on the Add Hardware button.
  4. Select hardware type

    Select Network as the Hardware type.
    The Add new virtual hardware wizard with Network selected as the hardware type.

    Figure 5.5. The Add new virtual hardware wizard

  5. Select the network device and driver

    Set the Device model to virtio to use the virtio drivers. Choose the desired Host device.
    The Add new virtual hardware wizard Network window, with Device model set to virtio.

    Figure 5.6. The Add new virtual hardware wizard

    Click Finish to complete the procedure.
Once all new devices are added, reboot the virtual machine. Virtual machines may not recognize the devices until the guest is rebooted.

Chapter 6. Network Configuration

This chapter provides an introduction to the common networking configurations used by libvirt based guest virtual machines. For additional information, consult the libvirt network architecture documentation: http://libvirt.org/archnetwork.html.
Red Hat Enterprise Linux 7 supports the following networking setups for virtualization:
  • virtual networks using Network Address Translation (NAT)
  • directly allocated physical devices using PCI device assignment
  • directly allocated virtual functions using PCIe SR-IOV
  • bridged networks
You must enable NAT, network bridging or directly assign a PCI device to allow external hosts access to network services on guest virtual machines.

6.1. Network Address Translation (NAT) with libvirt

One of the most common methods for sharing network connections is to use Network Address Translation (NAT) forwarding (also known as virtual networks).
Host Configuration
Every standard libvirt installation provides NAT-based connectivity to virtual machines as the default virtual network. Verify that it is available with the virsh net-list --all command.
# virsh net-list --all
Name                 State      Autostart
-----------------------------------------
default              active     yes
If it is missing the following could be used in the XML configuration file (such as /etc/libvirtd/qemu/myguest.xml) for the guest:
# ll /etc/libvirt/qemu/
total 12
drwx------. 3 root root 4096 Nov  7 23:02 networks
-rw-------. 1 root root 2205 Nov 20 01:20 r6.4.xml
-rw-------. 1 root root 2208 Nov  8 03:19 r6.xml
The default network is defined from /etc/libvirt/qemu/networks/default.xml
Mark the default network to automatically start:
# virsh net-autostart default
Network default marked as autostarted
Start the default network:
# virsh net-start default
Network default started
Once the libvirt default network is running, you will see an isolated bridge device. This device does not have any physical interfaces added. The new device uses NAT and IP forwarding to connect to the physical network. Do not add new interfaces.
# brctl show
bridge name     bridge id               STP enabled     interfaces
virbr0          8000.000000000000       yes
libvirt adds iptables rules which allow traffic to and from guest virtual machines attached to the virbr0 device in the INPUT, FORWARD, OUTPUT and POSTROUTING chains. libvirt then attempts to enable the ip_forward parameter. Some other applications may disable ip_forward, so the best option is to add the following to /etc/sysctl.conf.
 net.ipv4.ip_forward = 1
Guest Virtual Machine Configuration
Once the host configuration is complete, a guest virtual machine can be connected to the virtual network based on its name. To connect a guest to the 'default' virtual network, the following could be used in the XML configuration file (such as /etc/libvirtd/qemu/myguest.xml) for the guest:
<interface type='network'>
   <source network='default'/>
</interface>

Note

Defining a MAC address is optional. If you do not define one, a MAC address is automatically generated and used as the MAC address of the bridge device used by the network. Manually setting the MAC address may be useful to maintain consistency or easy reference throughout your environment, or to avoid the very small chance of a conflict.
<interface type='network'>
  <source network='default'/>
  <mac address='00:16:3e:1a:b3:4a'/>
</interface>

6.2. Disabling vhost-net

The vhost-net module is a kernel-level back end for virtio networking that reduces virtualization overhead by moving virtio packet processing tasks out of user space (the QEMU process) and into the kernel (the vhost-net driver). vhost-net is only available for virtio network interfaces. If the vhost-net kernel module is loaded, it is enabled by default for all virtio interfaces, but can be disabled in the interface configuration in the case that a particular workload experiences a degradation in performance when vhost-net is in use.
Specifically, when UDP traffic is sent from a host machine to a guest virtual machine on that host, performance degradation can occur if the guest virtual machine processes incoming data at a rate slower than the host machine sends it. In this situation, enabling vhost-net causes the UDP socket's receive buffer to overflow more quickly, which results in greater packet loss. It is therefore better to disable vhost-net in this situation to slow the traffic, and improve overall performance.
To disable vhost-net, edit the <interface> sub-element in the guest virtual machine's XML configuration file and define the network as follows:
<interface type="network">
   ...
   <model type="virtio"/>
   <driver name="qemu"/>
   ...
</interface>
Setting the driver name to qemu forces packet processing into QEMU user space, effectively disabling vhost-net for that interface.

6.3. Enabling vhost-net zero-copy

In Red Hat Enterprise Linux 7, vhost-net zero-copy is disabled by default. To enable this action on a permanent basis, add a new file vhost-net.conf to /etc/modprobe.d with the following content:
options vhost_net  experimental_zcopytx=1
If you want to disable this again, you can run the following:
modprobe -r vhost_net
modprobe vhost_net experimental_zcopytx=0
The first command removes the old file, the second one makes a new file (like above) and disables zero-copy. You can use this to enable as well but the change will not be permanent.
To confirm that this has taken effect, check the output of cat /sys/module/vhost_net/parameters/experimental_zcopytx. It should show:
$ cat /sys/module/vhost_net/parameters/experimental_zcopytx
0

6.4. Bridged Networking

Bridged networking (also known as virtual network switching) is used to place virtual machine network interfaces on the same network as the physical interface. Bridges require minimal configuration and make a virtual machine appear on an existing network, which reduces management overhead and network complexity. As bridges contain few components and configuration variables, they provide a transparent set-up which is straightforward to understand and troubleshoot, if required.
Bridging can be configured in a virtualized environment using standard Red Hat Enterprise Linux tools, virt-manager, or libvirt, and is described in the following sections.
However, even in a virtualized environment, bridges may be more easily created using the host operating system's networking tools. More information about this bridge creation method can be found in the Red Hat Enterprise Linux 7 Networking Guide.

6.4.1. Configuring Bridged Networking on a Red Hat Enterprise Linux 7 Host

Bridged networking can be configured for virtual machines on a Red Hat Enterprise Linux host, independent to the virtualization management tools. This configuration may be more appropriate when the virtualization bridge is the host's only network interface, or is the host's management network interface.
The Red Hat Enterprise Linux 7 Networking Guide contains detailed instructions on configuring bridged networking. See the Red Hat Enterprise Linux 7 Networking Guide - Configure Network Bridging for instructions on configuring network bridging outside of the virtualization tools.

6.4.2. Bridged Networking with Virtual Machine Manager

This section provides instructions on creating a bridge from a host machine's interface to a guest virtual machine using virt-manager.

Note

Depending on your environment, setting up a bridge with libvirt tools in Red Hat Enterprise Linux 7 may require disabling Network Manager, which is not recommended by Red Hat. A bridge created with libvirt also requires libvirtd to be running for the bridge to maintain network connectivity.
It is recommended to configure bridged networking on the physical Red Hat Enterprise Linux host as described in the Red Hat Enterprise Linux 7 Networking Guide, while using libvirt after bridge creation to add virtual machine interfaces to the bridges.

Procedure 6.1. Creating a bridge with virt-manager

  1. From the virt-manager main menu, click Edit > Connection Details to open the Connection Details window.
  2. Click the Network Interfaces tab.
  3. Click the + at the bottom of the window to configure a new network interface.
  4. In the Interface type drop-down menu, select Bridge, and then click Forward to continue.
    Adding a bridge

    Figure 6.1. Adding a bridge

    1. In the Name field, enter a name for the bridge, such as br0.
    2. Select a Start mode from the drop-down menu. Choose from one of the following:
      • none - deactivates the bridge
      • onboot - activates the bridge on the next guest virtual machine reboot
      • hotplug - activates the bridge even if the guest virtual machine is running
    3. Check the Activate now check box to activate the bridge immediately.
    4. To configure either the IP settings or Bridge settings, click the appropriate Configure button. A separate window will open to specify the desired settings. Make any necessary changes and click OK when done.
    5. Select the physical interface to connect to your virtual machines. If the interface is currently in use by another guest virtual machine, you will receive a warning message.
  5. Click Finish and the wizard closes, taking you back to the Connections menu.
    Adding a bridge

    Figure 6.2. Adding a bridge

Select the bridge to use, and click Apply to exit the wizard.
To stop the interface, click the Stop Interface key. Once the bridge is stopped, to delete the interface, click the Delete Interface key.

6.4.3. Bridged Networking with libvirt

Depending on your environment, setting up a bridge with libvirt in Red Hat Enterprise Linux 7 may require disabling Network Manager, which is not recommended by Red Hat. This also requires libvirtd to be running for the bridge to operate.
It is recommended to configure bridged networking on the physical Red Hat Enterprise Linux host as described in the Red Hat Enterprise Linux 7 Networking Guide.

Important

libvirt is now able to take advantage of new kernel tunable parameters to manage host bridge forwarding database (FDB) entries, thus potentially improving system network performance when bridging multiple virtual machines. Set the macTableManager attribute of a network's <bridge> element to 'libvirt' in the host's XML configuration file:
<bridge name='br0' macTableManager='libvirt'/>
This will turn off learning (flood) mode on all bridge ports, and libvirt will add or remove entries to the FDB as necessary. Along with removing the overhead of learning the proper forwarding ports for MAC addresses, this also allows the kernel to disable promiscuous mode on the physical device that connects the bridge to the network, which further reduces overhead.

Chapter 7. Overcommitting with KVM

7.1. Introduction

The KVM hypervisor supports overcommitting CPUs and overcommitting memory. Overcommitting is allocating more virtualized CPUs (vCPUs) or memory than there are physical resources on the system. With CPU overcommit, under-utilized virtualized servers or desktops can run on fewer servers which saves a number of system resources, with the net effect of less power, cooling, and investment in server hardware.
As most processes do not access 100% of their allocated memory all the time, KVM can use this behavior to its advantage and allocate more memory for guest virtual machines than the host machine actually has available, in a process called overcommitting of resources.

7.2. Overcommitting Memory

Important

Overcommitting is not an ideal solution for all memory issues, as the recommended methods to deal with memory shortage are to allocate less memory per guest, add more physical memory to the host, or utilize swap space. If you decide to overcommit memory, ensure sufficient testing is performed. Contact Red Hat support for assistance with overcommitting.
Virtual machines running on a KVM hypervisor do not have dedicated blocks of physical RAM assigned to them. Instead, each guest functions as a Linux process where the host machine's Linux kernel allocates memory only when requested. In addition, the host's memory manager can move the guest's memory between its own physical memory and swap space. This is why overcommitting requires allotting sufficient swap space on the host physical machine to accommodate all guests, as well as enough memory for the host's processes. As a basic rule, the host's operating system requires a maximum of 4GB of memory along with a minimum of 4GB of swap space. For more information, refer to Example 7.1, “Memory overcommit example”.
Red Hat Knowledgebase has an article on safely and efficiently determining the size of the swap partition.

Note

The example below is provided as a guide for configuring swap only. The settings listed may not be appropriate for your environment.

Example 7.1. Memory overcommit example

ExampleServer1 has 32GB of physical RAM. The system is being configured to run 50 guest virtual machines, each requiring 1GB of virtualized memory. As mentioned above, the host machine's system itself needs a maximum of 4GB (apart from the guests) as well as an additional 4GB as a swap space minimum.
The swap space is calculated as follows:
  • Calculate the amount of memory needed for the sum of all the virtual machines - In this example: (50 virtual machines * 1GB of memory per virtual machine) = 50GB
  • Add the virtual machine's memory amount to the amount needed for the host's operating system and for the host machine's minimum swap space - In this example: 50GB guest memory + 4GB host's OS + 4GB minimal swap = 58GB
  • Subtract this amount from the amount of physical RAM there is on the system - In this example 58GB - 32GB = 26GB
  • The answer is the amount of swap space that needs to be allocated - In this example 26GB.

Note

Overcommitting does not work with all virtual machines, but has been found to work in a desktop virtualization setup with minimal intensive usage or running several identical guests with KSM. Proceed with caution before changing these settings.
For more information on KSM and overcommitting, see the Red Hat Enterprise Linux 7 Virtualization Tuning and Optimization Guide.

7.3. Overcommitting Virtualized CPUs (vCPUs)

The KVM hypervisor supports overcommitting virtualized CPUs (vCPUs). Virtualized CPUs can be overcommitted as far as load limits of guest virtual machines allow. Use caution when overcommitting vCPUs, as loads near 100% may cause dropped requests or unusable response times.
In Red Hat Enterprise Linux 7, it is possible to overcommit guests with more than one vCPU (known as symmetric multiprocessing virtual machines), however, you may experience performance deterioration when running more cores on the virtual machine than are present on your physical CPU. For example, a virtual machine with four vCPUs should not be run on a host machine with a dual core processor. Overcommitting symmetric multiprocessing (SMP) virtual machines beyond the physical number of processing cores will cause significant performance degradation, due to programs getting less CPU time than required.
With SMP guests, some processing overhead is inherent. CPU overcommitting can increase the SMP overhead, because time-slicing guests can make inter-CPU communication inside a guest slower. This overhead increases with guests that have a larger number of vCPUs, or a larger overcommit ratio.
Virtualized CPUs are overcommitted best when when a single host has multiple guests, and each guest has a small number of vCPUs, compared to the number of host CPUs. The Linux scheduler is very efficient with this type of load. KVM should safely support guests with loads under 100% at a ratio of five vCPUs (on 5 virtual machines) to one physical CPU on one single host. The KVM hypervisor will switch between all of the virtual machines, making sure that the load is balanced.
For best performance, Red Hat recommends assigning guests only as many vCPUs as are required to run the programs that are inside each guest.

Important

Applications which use 100% of memory or processing resources may become unstable in overcommitted environments. Do not overcommit memory or CPUs in a production environment without extensive testing, as the amount of SMP and the CPU overcommit ratio is workload dependent.

Chapter 8. KVM Guest Timing Management

Virtualization involves several intrinsic challenges for time keeping in guest virtual machines. Interrupts cannot always be delivered simultaneously and instantaneously to all guest virtual machines, because interrupts in virtual machines are not true interrupts; they are injected into the guest virtual machine by the host machine. The host may be running another guest virtual machine, or a different process, meaning that the precise timing typically required by interrupts may not always be possible.
Guest virtual machines without accurate time keeping may experience issues with network applications and processes, as session validity, migration, and other network activities rely on timestamps to remain correct.
KVM avoids these issues by providing guest virtual machines with a paravirtualized clock (kvm-clock). However, it is still vital to test timing before attempting activities that may be affected by time keeping inaccuracies.

Note

Red Hat Enterprise Linux 5.5 and later, Red Hat Enterprise Linux 6.0 and later, and Red Hat Enterprise Linux 7 use kvm-clock as their default clock source. Running without kvm-clock requires special configuration, and is not recommended.

Important

The Network Time Protocol (NTP) should be configured on the host and the guest virtual machines. On guests using Red Hat Enterprise Linux 6 and earlier, NTP is implemented by the ntpd service. On guests using Red Hat Enterprise Linux 7, NTP time synchronization service can be provided by ntpd, but also by the chronyd service.
For enabling ntpd, adding it to the default startup sequence, and starting it, use the following commands:
# systemctl enable ntpd
# systemctl start ntpd
For enabling chronyd, adding it to the default startup sequence, and starting it, use the following commands:
# systemctl enable chronyd
# systemctl start chronyd
ntpd and chronyd will correct the effects of clock skew if the clock runs no more than 0.05% faster or slower than the reference time source. The ntp startup script adjusts the clock offset from the reference time by adjusting the system clock at startup time, if required.
The mechanics of guest virtual machine time synchronization
By default, the guest synchronizes its time with the hypervisor as follows:
  • When the guest system boots, the guest reads the time from the emulated Real Time Clock (RTC).
  • When the NTP protocol is initiated, it automatically synchronizes the guest clock. Afterwards, during normal guest operation, NTP performs clock adjustments in the guest.
  • When a guest is resumed after a pause or a restoration process, a command to synchronize the guest clock to a specified value should be issued by the management software (such as virt-manager). This synchronization works only if the QEMU guest agent is installed in the guest and supports the feature. The value to which the guest clock synchronizes is usually the host clock value.
Constant Time Stamp Counter (TSC)
Modern Intel and AMD CPUs provide a constant Time Stamp Counter (TSC). The count frequency of the constant TSC does not vary when the CPU core itself changes frequency, for example to comply with a power saving policy. A CPU with a constant TSC frequency is necessary in order to use the TSC as a clock source for KVM guests.
Your CPU has a constant Time Stamp Counter if the constant_tsc flag is present. To determine if your CPU has the constant_tsc flag enter the following command:
$ cat /proc/cpuinfo | grep constant_tsc
If any output is given your CPU has the constant_tsc bit. If no output is given follow the instructions below.
Configuring Hosts without a Constant Time Stamp Counter
Systems without a constant TSC frequency cannot use the TSC as a clock source for virtual machines, and require additional configuration. Power management features interfere with accurate time keeping and must be disabled for guest virtual machines to accurately keep time with KVM.

Important

These instructions are for AMD revision F CPUs only.
If the CPU lacks the constant_tsc bit, disable all power management features . Each system has several timers it uses to keep time. The TSC is not stable on the host, which is sometimes caused by cpufreq changes, deep C state, or migration to a host with a faster TSC. Deep C sleep states can stop the TSC. To prevent the kernel using deep C states append processor.max_cstate=1 to the kernel boot. To make this change persistent, edit values of the GRUB_CMDLINE_LINUX key in the /etc/default/grubfile. For example. if you want to enable emergency mode for each boot, edit the entry as follows:
GRUB_CMDLINE_LINUX="emergency"
Note that you can specify multiple parameters for the GRUB_CMDLINE_LINUX key, similarly to adding the parameters in the GRUB 2 boot menu.
To disable cpufreq (only necessary on hosts without the constant_tsc), install kernel-tools and enable the cpupower.service (systemctl disable cpupower.service). If you want to disable this service every time the guest virtual machine boots, change the configuration file in /etc/sysconfig/cpupower and change the CPUPOWER_START_OPTS and CPUPOWER_STOP_OPTS. Valid limits can be found in the /sys/devices/system/cpu/[cpuid]/cpufreq/scaling_available_governors files. For more information on this package or on power management and governors, refer to the Red Hat Enterprise Linux 7 Power Management Guide.

8.1. Required Parameters for Red Hat Enterprise Linux Guests

For certain Red Hat Enterprise Linux guest virtual machines, additional kernel parameters are required. These parameters can be set by appending them to the end of the /kernel line in the /etc/grub2.cfg file of the guest virtual machine.
The table below lists versions of Red Hat Enterprise Linux and the parameters required on the specified systems.

Table 8.1. Kernel parameter requirements

Red Hat Enterprise Linux version Additional guest kernel parameters
7.0 AMD64/Intel 64 with the paravirtualized clock Additional parameters are not required
6.1 and higher AMD64/Intel 64 with the paravirtualized clock Additional parameters are not required
6.0 AMD64/Intel 64 with the paravirtualized clock Additional parameters are not required
6.0 AMD64/Intel 64 without the paravirtualized clock notsc lpj=n
5.5 AMD64/Intel 64 with the paravirtualized clock Additional parameters are not required
5.5 AMD64/Intel 64 without the paravirtualized clock notsc lpj=n
5.5 x86 with the paravirtualized clock Additional parameters are not required
5.5 x86 without the paravirtualized clock clocksource=acpi_pm lpj=n
5.4 AMD64/Intel 64 notsc
5.4 x86 clocksource=acpi_pm
5.3 AMD64/Intel 64 notsc
5.3 x86 clocksource=acpi_pm
4.8 AMD64/Intel 64 notsc
4.8 x86 clock=pmtmr
3.9 AMD64/Intel 64 Additional parameters are not required

Note

The lpj parameter requires a numeric value equal to the loops per jiffy value of the specific CPU on which the guest virtual machine runs. If you do not know this value, do not set the lpj parameter.

Warning

The divider kernel parameter was previously recommended for Red Hat Enterprise Linux 4 and 5 guest virtual machines that did not have high responsiveness requirements, or exist on systems with high guest density. It is no longer recommended for use with guests running Red Hat Enterprise Linux 4, or Red Hat Enterprise Linux 5 versions prior to version 5.8.
divider can improve throughput on Red Hat Enterprise Linux 5 versions equal to or later than 5.8 by lowering the frequency of timer interrupts. For example, if HZ=1000, and divider is set to 10 (that is, divider=10), the number of timer interrupts per period changes from the default value (1000) to 100 (the default value, 1000, divided by the divider value, 10).
BZ#698842 details a bug in the way that the divider parameter interacts with interrupt and tick recording. This bug is fixed as of Red Hat Enterprise Linux 5.8. However, the divider parameter can still cause kernel panic in guests using Red Hat Enterprise Linux 4, or Red Hat Enterprise Linux 5 versions prior to version 5.8.
Red Hat Enterprise Linux 6 and later does not have a fixed-frequency clock interrupt; it operates in tickless mode and uses the timer dynamically as required. The divider parameter is therefore not useful for Red Hat Enterprise Linux 6 and Red Hat Enterprise Linux 7, and guests on these systems are not affected by this bug.

8.2. Steal Time Accounting

Steal time is the amount of CPU time desired by a guest virtual machine that is not provided by the host. Steal time occurs when the host allocates these resources elsewhere: for example, to another guest.
Steal time is reported in the CPU time fields in /proc/stat. It is automatically reported by utilities such as top and vmstat. It is displayed as "%st", or in the "st" column. Note that it cannot be switched off.
Large amounts of steal time indicate CPU contention, which can reduce guest performance. To relieve CPU contention, increase the guest's CPU priority or CPU quota, or run fewer guests on the host.

Chapter 9. Network Booting with libvirt

Guest virtual machines can be booted with PXE enabled. PXE allows guest virtual machines to boot and load their configuration off the network itself. This section demonstrates some basic configuration steps to configure PXE guests with libvirt.
This section does not cover the creation of boot images or PXE servers. It is used to explain how to configure libvirt, in a private or bridged network, to boot a guest virtual machine with PXE booting enabled.

Warning

These procedures are provided only as an example. Ensure that you have sufficient backups before proceeding.

9.1. Preparing the Boot Server

To perform the steps in this chapter you will need:
  • A PXE Server (DHCP and TFTP) - This can be a libvirt internal server, manually-configured dhcpd and tftpd, dnsmasq, a server configured by Cobbler, or some other server.
  • Boot images - for example, PXELINUX configured manually or by Cobbler.

9.1.1. Setting up a PXE Boot Server on a Private libvirt Network

This example uses the default network. Perform the following steps:

Procedure 9.1. Configuring the PXE boot server

  1. Place the PXE boot images and configuration in /var/lib/tftpboot.
  2. enter the following commands:
    # virsh net-destroy default
    # virsh net-edit default
  3. Edit the <ip> element in the configuration file for the default network to include the appropriate address, network mask, DHCP address range, and boot file, where BOOT_FILENAME represents the file name you are using to boot the guest virtual machine.
    <ip address='192.168.122.1' netmask='255.255.255.0'>
       <tftp root='/var/lib/tftpboot' />
       <dhcp>
          <range start='192.168.122.2' end='192.168.122.254' />
          <bootp file='BOOT_FILENAME' />
       </dhcp>
    </ip>
  4. Run:
    # virsh net-start default
  5. Boot the guest using PXE (refer to Section 9.2, “Booting a Guest Using PXE”).

9.2. Booting a Guest Using PXE

This section demonstrates how to boot a guest virtual machine with PXE.

9.2.1. Using bridged networking

Procedure 9.2. Booting a guest using PXE and bridged networking

  1. Ensure bridging is enabled such that the PXE boot server is available on the network.
  2. Boot a guest virtual machine with PXE booting enabled. You can use the virt-install command to create a new virtual machine with PXE booting enabled, as shown in the following example command:
    virt-install --pxe --network bridge=breth0 --prompt
    Alternatively, ensure that the guest network is configured to use your bridged network, and that the XML guest configuration file has a <boot dev='network'/> element inside the <os> element, as shown in the following example:
    <os>
       <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
       <boot dev='network'/>
       <boot dev='hd'/>
    </os>
    <interface type='bridge'>
       <mac address='52:54:00:5a:ad:cb'/>
       <source bridge='breth0'/>
       <target dev='vnet0'/>
       <alias name='net0'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

9.2.2. Using a Private libvirt Network

Procedure 9.3. Using a private libvirt network

  1. Boot a guest virtual machine using libvirt with PXE booting enabled. You can use the virt-install command to create/install a new virtual machine using PXE:
    virt-install --pxe --network network=default --prompt
Alternatively, ensure that the guest network is configured to use your private libvirt network, and that the XML guest configuration file has a <boot dev='network'/> element inside the <os> element, as shown in the following example:
<os>
   <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
   <boot dev='network'/>
   <boot dev='hd'/>
</os>
Also ensure that the guest virtual machine is connected to the private network:
<interface type='network'>
   <mac address='52:54:00:66:79:14'/>
   <source network='default'/>
   <target dev='vnet0'/>
   <alias name='net0'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

Chapter 10. Registering the Hypervisor and Virtual Machine

Red Hat Enterprise Linux 6 and 7 require that every guest virtual machine is mapped to a specific hypervisor in order to ensure that every guest is allocated the same level of subscription service. To do this you need to install a subscription agent that automatically detects all guest Virtual Machines (VMs) on each KVM hypervisor that is installed and registered, which in turn will create a mapping file that sits on the host. This mapping file ensures that all guest VMs receive the following benefits:
  • Subscriptions specific to virtual systems are readily available and can be applied to all of the associated guest VMs.
  • All subscription benefits that can be inherited from the hypervisor are readily available and can be applied to all of the associated guest VMs.

Note

The information provided in this chapter is specific to Red Hat Enterprise Linux subscriptions only. If you also have a Red Hat Enterprise Virtualization subscription, or a Red Hat Satellite subscription, you should also consult the virt-who information provided with those subscriptions. More information on Red Hat Subscription Management can also be found in the Red Hat Subscription Management Guide found on the customer portal.

10.1. Installing virt-who on the Host Physical Machine

  1. Register the KVM hypervisor

    Register the KVM Hypervisor by running the subscription-manager register [options] command in a terminal as the root user on the host physical machine. More options are available using the # subscription-manager register --help menu. In cases where you are using a user name and password, use the credentials that are known to the subscription manager. If this is your very first time subscribing and you do not have a user account, contact customer support. For example to register the VM as 'admin' with 'secret' as a password, you would send the following command:
    [root@rhel-server ~]# subscription-manager register --username=admin --password=secret --auto-attach --type=hypervisor
  2. Install the virt-who packages

    Install the virt-who packages, by running the following command in a terminal as root on the host physical machine:
    [root@rhel-server ~]# yum install virt-who
  3. Create a virt-who configuration file

    Add a configuration file in the /etc/virt-who.d/ directory. It does not matter what the name of the file is, but you should give it a name that makes sense and the file must be located in the /etc/virt-who.d/ directory. Inside that file add the following snippet and remember to save the file before closing it.
    [libvirt]
    type=libvirt
    
  4. Start the virt-who service

    Start the virt-who service by running the following command in a terminal as root on the host physical machine:
    [root@virt-who ~]# systemctl start virt-who.service
    [root@virt-who ~]# systemctl enable virt-who.service
    
  5. Confirm virt-who service is receiving guest information

    At this point, the virt-who service will start collecting a list of domains from the host. Check the /var/log/rhsm/rhsm.log file on the host physical machine to confirm that the file contains a list of the guest VMs. For example:
    2015-05-28 12:33:31,424 DEBUG: Libvirt domains found: [{'guestId': '58d59128-cfbb-4f2c-93de-230307db2ce0', 'attributes': {'active': 0, 'virtWhoType': 'libvirt', 'hypervisorType': 'QEMU'}, 'state': 5}]
    

Procedure 10.1. Managing the subscription on the customer portal

  1. Subscribing the hypervisor

    As the virtual machines will be receiving the same subscription benefits as the hypervisor, it is important that the hypervisor has a valid subscription and that the subscription is available for the VMs to use.
    1. Login to the customer portal

      Login to the Red Hat customer portal https://access.redhat.com/ and click the Subscriptions button at the top of the page.
    2. Click the Systems link

      In the Subscriber Inventory section (towards the bottom of the page), click Systems link.
    3. Select the hypervisor

      On the Systems page, there is a table of all subscribed systems. Click on the name of the hypervisor (localhost.localdomain for example). In the details page that opens, click Attach a subscription and select all the subscriptions listed. Click Attach Selected. This will attach the host's physical subscription to the hypervisor so that the guests can benefit from the subscription.
  2. Subscribing the guest virtual machines - first time use

    This step is for those who have a new subscription and have never subscribed a guest virtual machine before. If you are adding virtual machines, skip this step. To consume the subscription assigned to the hypervisor profile on the machine running the virt-who service, auto subscribe by running the following command in a terminal, on the guest virtual machine as root.
    [root@virt-who ~]# subscription-manager attach --auto
    
  3. Subscribing additional guest virtual machines

    If you just subscribed a for the first time, skip this step. If you are adding additional virtual machines, it should be noted that running this command will not necessarily re-attach the same subscriptions to the guest virtual machine. This is because removing all subscriptions then allowing auto attach to resolve what is necessary for a given guest virtual machine may result in different subscriptions consumed than before. This may not have any effect on your system, but it is something you should be aware about. If you used a manual attachment procedure to attach the virtual machine, which is not described below, you will need to re-attach those virtual machines manually as the auto-attach will not work. Use the following command as root in a terminal to first remove the subscriptions for the old guests and then use the auto-attach to attach subscriptions to all the guests. Run these commands on the guest virtual machine.
    [root@virt-who ~]# subscription-manager remove --all
    [root@virt-who ~]# subscription-manager attach --auto
    
  4. Confirm subscriptions are attached

    Confirm that the subscription is attached to the hypervisor by running the following command as root in a terminal on the guest virtual machine:
    [root@virt-who ~]# subscription-manager list --consumed
    Output similar to the following will be displayed. Pay attention to the Subscription Details. It should say 'Subscription is current'.
    [root@virt-who ~]#  subscription-manager list --consumed
    +-------------------------------------------+
       Consumed Subscriptions
    +-------------------------------------------+
    Subscription Name:	Awesome OS with unlimited virtual guests
    Provides: 		Awesome OS Server Bits
    SKU: 			awesomeos-virt-unlimited
    Contract: 		0
    Account: 		######### Your account number #####
    Serial: 		######### Your serial number ######
    Pool ID: 		XYZ123                                             1
    Provides Management: 	No
    Active: 		True
    Quantity Used: 		1
    Service Level:
    Service Type:
    Status Details:		Subscription is current                       2
    Subscription Type:
    Starts: 		01/01/2015
    Ends: 			12/31/2015
    System Type: 		Virtual
    

    1

    The ID for the subscription to attach to the system is displayed here. You will need this ID if you need to attach the subscription manually.

    2

    Indicates if your subscription is current. If your subscription is not current, an error message appears. One example is Guest has not been reported on any host and is using a temporary unmapped guest subscription. In this case the guest needs to be subscribed. In other cases, use the information as indicated in Section 10.5.2, “I have subscription status errors, what do I do?”.
  5. Register additional guests

    When you install new guest VMs on the hypervisor, you must register the new VM and use the subscription attached to the hypervisor, by running the following commands in a terminal as root on the guest virtual machine:
    [root@server1 ~]# subscription-manager register
    [root@server1 ~]# subscription-manager attach --auto
    [root@server1 ~]# subscription-manager list --consumed
    

10.2. Registering a New Guest Virtual Machine

In cases where a new guest virtual machine is to be created on a host that is already registered and running, the virt-who service must also be running. This ensures that the virt-who service maps the guest to a hypervisor, so the system is properly registered as a virtual system. To register the virtual machine, enter the following command as root in a terminal:
[root@virt-server ~]# subscription-manager register --username=admin --password=secret --auto-attach

10.3. Removing a Guest Virtual Machine Entry

If the guest virtual machine is running, unregister the system, by running the following command in a terminal window as root on the guest:
[root@virt-guest ~]# subscription-manager unregister
If the system has been deleted, however, the virtual service cannot tell whether the service is deleted or paused. In that case, you must manually remove the system from the server side, using the following steps:
  1. Login to the Subscription Manager

    The Subscription Manager is located on the Red Hat Customer Portal. Login to the Customer Portal using your user name and password, by clicking the login icon at the top of the screen.
  2. Click the Subscriptions tab

    Click the Subscriptions tab.
  3. Click the Systems link

    Scroll down the page and click the Systems link.
  4. Delete the system

    To delete the system profile, locate the specified system's profile in the table, select the check box beside its name and click Delete.

10.4. Installing virt-who Manually

This section will describe how to manually attach the subscription provided by the hypervisor.

Procedure 10.2. How to attach a subscription manually

  1. List subscription information and find the Pool ID

    First you need to list the available subscriptions which are of the virtual type. enter the following command in a terminal as root:
    [root@server1 ~]# subscription-manager list --avail --match-installed | grep 'Virtual' -B12
    Subscription Name: Red Hat Enterprise Linux ES (Basic for Virtualization)
    Provides:          Red Hat Beta
                       Oracle Java (for RHEL Server)
                       Red Hat Enterprise Linux Server
    SKU:               -------
    Pool ID:           XYZ123
    Available:         40
    Suggested:         1
    Service Level:     Basic
    Service Type:      L1-L3
    Multi-Entitlement: No
    Ends:              01/02/2017
    System Type:       Virtual
    
    Note the Pool ID displayed. Copy this ID as you will need it in the next step.
  2. Attach the subscription with the Pool ID

    Using the Pool ID you copied in the previous step run the attach command. Replace the Pool ID XYZ123 with the Pool ID you retrieved. enter the following command in a terminal as root:
    [root@server1 ~]# subscription-manager attach --pool=XYZ123
    
    Successfully attached a subscription for: Red Hat Enterprise Linux ES (Basic for Virtualization)
    

10.5. Troubleshooting virt-who

10.5.1. Why is the hypervisor status red?

Scenario: On the server side, you deploy a guest on a hypervisor that does not have a subscription. 24 hours later, the hypervisor displays its status as red. To remedy this situation you must get a subscription for that hypervisor. Or, permanently migrate the guest to a hypervisor with a subscription.

10.5.2. I have subscription status errors, what do I do?

Scenario: Any of the following error messages display:
  • System not properly subscribed
  • Status unknown
  • Late binding of a guest to a hypervisor through virt-who (host/guest mapping)
To find the reason for the error open the virt-who log file, named rhsm.log, located in the /var/log/rhsm/ directory.

Chapter 11. Enhancing Virtualization with the QEMU Guest Agent and SPICE Agent

Agents in Red Hat Enterprise Linux such as the QEMU guest agent and the SPICE agent can be deployed to help the virtualization tools run more optimally on your system. These agents are described in this chapter.

Note

To further optimize and tune host and guest performance, see the Red Hat Enterprise Linux 7 Virtualization Tuning and Optimization Guide.

11.1. QEMU Guest Agent

The QEMU guest agent runs inside the guest and allows the host machine to issue commands to the guest operating system using libvirt, helping with functions such as freezing and thawing filesystems. The guest operating system then responds to those commands asynchronously. The QEMU guest agent package, qemu-guest-agent, is installed by default in Red Hat Enterprise Linux 7.
This section covers the libvirt commands and options available to the guest agent.

Important

Note that it is only safe to rely on the QEMU guest agent when run by trusted guests. An untrusted guest may maliciously ignore or abuse the guest agent protocol, and although built-in safeguards exist to prevent a denial of service attack on the host, the host requires guest co-operation for operations to run as expected.
Note that QEMU guest agent can be used to enable and disable virtual CPUs (vCPUs) while the guest is running, thus adjusting the number of vCPUs without using the hot plug and hot unplug features. For more information, refer to Section 23.42.6, “Configuring Virtual CPU Count”.

11.1.1. Setting up Communication between the QEMU Guest Agent and Host

The host machine communicates with the QEMU guest agent through a VirtIO serial connection between the host and guest machines. A VirtIO serial channel is connected to the host via a character device driver (typically a Unix socket), and the guest listens on this serial channel.

Note

The qemu-guest-agent does not detect if the host is listening to the VirtIO serial channel. However, as the current use for this channel is to listen for host-to-guest events, the probability of a guest virtual machine running into problems by writing to the channel with no listener is very low. Additionally, the qemu-guest-agent protocol includes synchronization markers which allow the host physical machine to force a guest virtual machine back into sync when issuing a command, and libvirt already uses these markers, so that guest virtual machines are able to safely discard any earlier pending undelivered responses.

11.1.1.1. Configuring the QEMU Guest Agent on a Linux Guest

The QEMU guest agent can be configured on a running or shut down virtual machine. If configured on a running guest, the guest will start using the guest agent immediately. If the guest is shut down, the QEMU guest agent will be enabled at next boot.
Either virsh or virt-manager can be used to configure communication between the guest and the QEMU guest agent. The following instructions describe how to configure the QEMU guest agent on a Linux guest.

Procedure 11.1. Setting up communication between guest agent and host with virsh on a shut down Linux guest

  1. Shut down the virtual machine

    Ensure the virtual machine (named rhel7 in this example) is shut down before configuring the QEMU guest agent:
    # virsh shutdown rhel7 
  2. Add the QEMU guest agent channel to the guest XML configuration

    Edit the guest's XML file to add the QEMU guest agent details:
    # virsh edit rhel7
    Add the following to the guest's XML file and save the changes:
    <channel type='unix'>
       <target type='virtio' name='org.qemu.guest_agent.0'/>
    </channel>
  3. Start the virtual machine

    # virsh start rhel7
  4. Install the QEMU guest agent on the guest

    Install the QEMU guest agent if not yet installed in the guest virtual machine:
    # yum install qemu-guest-agent
  5. Start the QEMU guest agent in the guest

    Start the QEMU guest agent service in the guest:
    # systemctl start qemu-guest-agent
Alternatively, the QEMU guest agent can be configured on a running guest with the following steps:

Procedure 11.2. Setting up communication between guest agent and host on a running Linux guest

  1. Create an XML file for the QEMU guest agent

    # cat agent.xml
    <channel type='unix'>
       <target type='virtio' name='org.qemu.guest_agent.0'/>
    </channel>
  2. Attach the QEMU guest agent to the virtual machine

    Attach the QEMU guest agent to the running virtual machine (named rhel7 in this example) with this command:
    # virsh attach-device rhel7 agent.xml
  3. Install the QEMU guest agent on the guest

    Install the QEMU guest agent if not yet installed in the guest virtual machine:
    # yum install qemu-guest-agent
  4. Start the QEMU guest agent in the guest

    Start the QEMU guest agent service in the guest:
    # systemctl start qemu-guest-agent

Procedure 11.3. Setting up communication between the QEMU guest agent and host with virt-manager

  1. Shut down the virtual machine

    Ensure the virtual machine is shut down before configuring the QEMU guest agent.
    To shut down the virtual machine, select it from the list of virtual machines in Virtual Machine Manager, then click the light switch icon from the menu bar.
  2. Add the QEMU guest agent channel to the guest

    Open the virtual machine's hardware details by clicking the lightbulb icon at the top of the guest window.
    Click the Add Hardware button to open the Add New Virtual Hardware window, and select Channel.
    Select the QEMU guest agent from the Name drop-down list and click Finish:
    Selecting the QEMU guest agent channel device

    Figure 11.1. Selecting the QEMU guest agent channel device

  3. Start the virtual machine

    To start the virtual machine, select it from the list of virtual machines in Virtual Machine Manager, then click the triangle (play) icon from the menu bar.
  4. Install the QEMU guest agent on the guest

    Open the guest with virt-manager and install the QEMU guest agent if not yet installed in the guest virtual machine:
    # yum install qemu-guest-agent
  5. Start the QEMU guest agent in the guest

    Start the QEMU guest agent service in the guest:
    # systemctl start qemu-guest-agent
The QEMU guest agent is now configured on the rhel7 virtual machine.

11.2. Using the QEMU Guest Agent with libvirt

Installing the QEMU guest agent allows various libvirt commands to become more powerful. The guest agent enhances the following virsh commands:
  • virsh shutdown --mode=agent - This shutdown method is more reliable than virsh shutdown --mode=acpi, as virsh shutdown used with the QEMU guest agent is guaranteed to shut down a cooperative guest in a clean state. If the agent is not present, libvirt must instead rely on injecting an ACPI shutdown event, but some guests ignore that event and thus will not shut down.
    Can be used with the same syntax for virsh reboot.
  • virsh snapshot-create --quiesce - Allows the guest to flush its I/O into a stable state before the snapshot is created, which allows use of the snapshot without having to perform a fsck or losing partial database transactions. The guest agent allows a high level of disk contents stability by providing guest co-operation.
  • virsh domfsfreeze and virsh domfsthaw - Quiesces the guest filesystem in isolation.
  • virsh domfstrim - Instructs the guest to trim its filesystem.
  • virsh domtime - Queries or sets the guest's clock.
  • virsh setvcpus --guest - Instructs the guest to take CPUs offline.
  • virsh domifaddr --source agent - Queries the guest operating system's IP address via the guest agent.
  • virsh domfsinfo - Shows a list of mounted filesystems within the running guest.
  • virsh set-user-password - Sets the password for a user account in the guest.

11.2.1. Creating a Guest Disk Backup

libvirt can communicate with qemu-guest-agent to ensure that snapshots of guest virtual machine file systems are consistent internally and ready to use as needed. Guest system administrators can write and install application-specific freeze/thaw hook scripts. Before freezing the filesystems, the qemu-guest-agent invokes the main hook script (included in the qemu-guest-agent package). The freezing process temporarily deactivates all guest virtual machine applications.
The snapshot process is comprised of the following steps:
  • File system applications / databases flush working buffers to the virtual disk and stop accepting client connections
  • Applications bring their data files into a consistent state
  • Main hook script returns
  • qemu-guest-agent freezes the filesystems and the management stack takes a snapshot
  • Snapshot is confirmed
  • Filesystem function resumes
Thawing happens in reverse order.
To create a snapshot of the guest's file system, run the virsh snapshot-create --quiesce --disk-only command (alternatively, run virsh snapshot-create-as guest_name --quiesce --disk-only, explained in further detail in Section 23.45.2, “Creating a Snapshot for the Current Guest Virtual Machine”).

Note

An application-specific hook script might need various SELinux permissions in order to run correctly, as is done when the script needs to connect to a socket in order to talk to a database. In general, local SELinux policies should be developed and installed for such purposes. Accessing file system nodes should work out of the box, after issuing the restorecon -FvvR command listed in Table 11.1, “QEMU guest agent package contents” in the table row labeled /etc/qemu-ga/fsfreeze-hook.d/.
The qemu-guest-agent binary RPM includes the following files:

Table 11.1. QEMU guest agent package contents

File name Description
/usr/lib/systemd/system/qemu-guest-agent.service Service control script (start/stop) for the QEMU guest agent.
/etc/sysconfig/qemu-ga Configuration file for the QEMU guest agent, as it is read by the /usr/lib/systemd/system/qemu-guest-agent.service control script. The settings are documented in the file with shell script comments.
/usr/bin/qemu-ga QEMU guest agent binary file.
/etc/qemu-ga Root directory for hook scripts.
/etc/qemu-ga/fsfreeze-hook Main hook script. No modifications are needed here.
/etc/qemu-ga/fsfreeze-hook.d Directory for individual, application-specific hook scripts. The guest system administrator should copy hook scripts manually into this directory, ensure proper file mode bits for them, and then run restorecon -FvvR on this directory.
/usr/share/qemu-kvm/qemu-ga/ Directory with sample scripts (for example purposes only). The scripts contained here are not executed.
The main hook script, /etc/qemu-ga/fsfreeze-hook logs its own messages, as well as the application-specific script's standard output and error messages, in the following log file: /var/log/qemu-ga/fsfreeze-hook.log. For more information, refer to the qemu-guest-agent wiki page at wiki.qemu.org or libvirt.org.

11.3. SPICE Agent

The SPICE agent helps run graphical applications such as virt-manager more smoothly, by helping integrate the guest operating system with the SPICE client.
For example, when resizing a window in virt-manager, the SPICE agent allows for automatic X session resolution adjustment to the client resolution. The SPICE agent also provides support for copy and paste between the host and guest, and prevents mouse cursor lag.
For system-specific information on the SPICE agent's capabilities, see the spice-vdagent package's README file.

11.3.1. Setting up Communication between the SPICE Agent and Host

The SPICE agent can be configured on a running or shut down virtual machine. If configured on a running guest, the guest will start using the guest agent immediately. If the guest is shut down, the SPICE agent will be enabled at next boot.
Either virsh or virt-manager can be used to configure communication between the guest and the SPICE agent. The following instructions describe how to configure the SPICE agent on a Linux guest.

Procedure 11.4. Setting up communication between guest agent and host with virsh on a Linux guest

  1. Shut down the virtual machine

    Ensure the virtual machine (named rhel7 in this example) is shut down before configuring the SPICE agent:
    # virsh shutdown rhel7 
  2. Add the SPICE agent channel to the guest XML configuration

    Edit the guest's XML file to add the SPICE agent details:
    # virsh edit rhel7
    Add the following to the guest's XML file and save the changes:
    <channel type='spicevmc'>
       <target type='virtio' name='com.redhat.spice.0'/>
    </channel>
  3. Start the virtual machine

    # virsh start rhel7
  4. Install the SPICE agent on the guest

    Install the SPICE agent if not yet installed in the guest virtual machine:
    # yum install spice-vdagent
  5. Start the SPICE agent in the guest

    Start the SPICE agent service in the guest:
    # systemctl start spice-vdagent
Alternatively, the SPICE agent can be configured on a running guest with the following steps:

Procedure 11.5. Setting up communication between SPICE agent and host on a running Linux guest

  1. Create an XML file for the SPICE agent

    # cat agent.xml
    <channel type='spicevmc'>
       <target type='virtio' name='com.redhat.spice.0'/>
    </channel>
  2. Attach the SPICE agent to the virtual machine

    Attach the SPICE agent to the running virtual machine (named rhel7 in this example) with this command:
    # virsh attach-device rhel7 agent.xml
  3. Install the SPICE agent on the guest

    Install the SPICE agent if not yet installed in the guest virtual machine:
    # yum install spice-vdagent
  4. Start the SPICE agent in the guest

    Start the SPICE agent service in the guest:
    # systemctl start spice-vdagent

Procedure 11.6. Setting up communication between the SPICE agent and host with virt-manager

  1. Shut down the virtual machine

    Ensure the virtual machine is shut down before configuring the SPICE agent.
    To shut down the virtual machine, select it from the list of virtual machines in Virtual Machine Manager, then click the light switch icon from the menu bar.
  2. Add the SPICE agent channel to the guest

    Open the virtual machine's hardware details by clicking the lightbulb icon at the top of the guest window.
    Click the Add Hardware button to open the Add New Virtual Hardware window, and select Channel.
    Select the SPICE agent from the Name drop-down list and click Finish:
    Selecting the SPICE agent channel device

    Figure 11.2. Selecting the SPICE agent channel device

  3. Start the virtual machine

    To start the virtual machine, select it from the list of virtual machines in Virtual Machine Manager, then click the triangle (play) icon from the menu bar.
  4. Install the SPICE agent on the guest

    Open the guest with virt-manager and install the SPICE agent if not yet installed in the guest virtual machine:
    # yum install spice-vdagent
  5. Start the SPICE agent in the guest

    Start the SPICE agent service in the guest:
    # systemctl start spice-vdagent
The SPICE agent is now configured on the rhel7 virtual machine.

Chapter 12. Nested Virtualization

12.1. Overview

As of Red Hat Enterprise Linux 7.2, nested virtualization is available as a Technology Preview for KVM guest virtual machines. With this feature, a guest virtual machine (also referred to as level 1 or L1) running on a physical host (level 0 or L0) can act as a hypervisor, and create its own (L2) guest virtual machines.
Nested virtualization is useful in a variety of scenarios, such as debugging hypervisors in a constrained environment and testing larger virtual deployments on a limited amount of physical resources. However, note that nested virtualization is not supported or recommended in production user environments, and is primarily intended for development and testing.
Nested virtualization is a Technology Preview only on Intel and AMD processors. On any other processors, it is not likely to function and not recommended to use.
Nested virtualization relies on host virtualization extensions to function as is not to be confused with running guests in a virtual environment using the QEMU Tiny Code Generator (TCG) emulation.

12.2. Setup

Follow these steps to enable, configure, and start using nested virtualization:
  1. The feature is disabled by default. To enable it, use the following procedure on the L0 host physical machine.
    For Intel:
    1. Check whether nested virtualization is available on your host system.
      $ cat /sys/module/kvm_intel/parameters/nested
      If this command returns Y or 1, the feature is enabled.
      If the command returns 0 or N, use steps 2 and 3.
    2. Unload the kvm_intel module:
      # modprobe -r kvm_intel
    3. Activate the nesting feature:
      # modprobe kvm_intel nested=1
    4. The nesting feature is now enabled only until the next reboot of the L0 host. To enable it permanently, add the following line to the /etc/modprobe.d/kvm.conf file:
      options kvm_intel nested=1
    For AMD:
    1. Check whether nested virtualization is available on your system:
      $ cat /sys/module/kvm_amd/parameters/nested
      If this command returns "Y" or "1", the feature is enabled.
      If the command returns "0" or "N", use steps 2 and 3.
    2. Unload the kvm_amd module
      # modprobe -r kvm_amd
    3. Activate the nesting feature
      # modprobe kvm_amd nested=1
    4. The nesting feature is now enabled only until the next reboot of the L0 host. To enable it permanently, add the following line to the /etc/modprobe.d/kvm.conf file:
      options kvm_amd nested=1
  2. Configure your L1 virtual machine for nested virtualization using one of the following methods:
    virt-manager
    1. Open the GUI of the desired guest and click the Show Virtual Hardware Details icon.
    2. Select the Processor menu, and in the Configuration section, type host-passthrough in the Model field (do not use the drop-down selection), and click Apply.
    Domain XML
    Add the following line to the domain XML file of the guest:
    <cpu mode='host-passthrough'/>
    If the XML file already contains a <cpu> element, rewrite it.
  3. To start using nested virtualization, install an L2 guest within the L1 guest. To do this, follow the same procedure as when installing the L1 guest - see Chapter 3, Installing a Virtual Machine for more information.

12.3. Restrictions and Limitations

As of Red Hat Enterprise Linux 7.2, it is strongly recommended to run Red Hat Enterprise Linux 7.2 or later in the L0 host and the L1 guests. L2 guests can contain any of the supported systems.
It is not supported to migrate L1 or L2 guests.
Use of L2 guests as hypervisors and creating L3 guests is not supported.
Not all features available on the host are available to be utilized by the L1 hypervisor. For instance, IOMMU/VT-d or APICv cannot be used by the L1 hypervisor.
To use nested virtualization, the host CPU must have the necessary feature flags. To determine if the L0 and L1 hypervisors are set up correctly, use the $ cat /proc/cpuinfo command on both L0 and L1, and make sure that the following flags are listed for the respective CPUs on both hypervisors:
  • For Intel - vmx (Hardware Virtualization) and ept (Extended Page Tables)
  • For AMD - svm (equivalent to vmx) and npt (equivalent to ept)

Part II. Administration

Table of Contents

13. Securing the Host Physical Machine and Improving Performance
13.1. Security Deployment Plan
13.2. Client Access Control
13.2.1. Access Control Drivers
13.2.2. Objects and Permissions
13.2.3. Security Concerns when Adding Block Devices to a Guest
14. Storage Pools
14.1. Disk-based Storage Pools
14.1.1. Creating a Disk-based Storage Pool Using virsh
14.1.2. Deleting a Storage Pool Using virsh
14.2. Partition-based Storage Pools
14.2.1. Creating a Partition-based Storage Pool Using virt-manager
14.2.2. Deleting a Storage Pool Using virt-manager
14.2.3. Creating a Partition-based Storage Pool Using virsh
14.2.4. Deleting a Storage Pool Using virsh
14.3. Directory-based Storage Pools
14.3.1. Creating a Directory-based Storage Pool with virt-manager
14.3.2. Deleting a Storage Pool Using virt-manager
14.3.3. Creating a Directory-based Storage Pool with virsh
14.3.4. Deleting a Storage Pool Using virsh
14.4. LVM-based Storage Pools
14.4.1. Creating an LVM-based Storage Pool with virt-manager
14.4.2. Deleting a Storage Pool Using virt-manager
14.4.3. Creating an LVM-based Storage Pool with virsh
14.4.4. Deleting a Storage Pool Using virsh
14.5. iSCSI-based Storage Pools
14.5.1. Configuring a Software iSCSI Target
14.5.2. Creating an iSCSI Storage Pool in virt-manager
14.5.3. Deleting a Storage Pool Using virt-manager
14.5.4. Creating an iSCSI-based Storage Pool with virsh
14.5.5. Securing an iSCSI Storage Pool
14.5.6. Deleting a Storage Pool Using virsh
14.6. NFS-based Storage Pools
14.6.1. Creating an NFS-based Storage Pool with virt-manager
14.6.2. Deleting a Storage Pool Using virt-manager
14.7. Using an NPIV Virtual Adapter (vHBA) with SCSI Devices
14.7.1. Creating a vHBA
14.7.2. Creating a Storage Pool Using the vHBA
14.7.3. Configuring the Virtual Machine to Use a vHBA LUN
14.7.4. Destroying the vHBA Storage Pool
14.8. GlusterFS Storage Pools
14.8.1. Creating a GlusterFS Storage Pool Using virsh
14.8.2. Deleting a GlusterFS Storage Pool Using virsh
15. Storage Volumes
15.1. Introduction
15.1.1. Referencing Volumes
15.2. Creating Volumes
15.3. Cloning Volumes
15.4. Deleting and Removing Volumes
15.5. Adding Storage Devices to Guests
15.5.1. Adding File-based Storage to a Guest
15.5.2. Adding Hard Drives and Other Block Devices to a Guest
15.5.3. Adding SCSI LUN-based Storage to a Guest
15.5.4. Managing Storage Controllers in a Guest Virtual Machine
16. Using qemu-img
16.1. Checking the Disk Image
16.2. Committing Changes to an Image
16.3. Comparing Images
16.4. Mapping an Image
16.4.1. The human Format
16.4.2. The json Format
16.5. Amending an Image
16.6. Converting an Existing Image to Another Format
16.7. Creating and Formatting New Images or Devices
16.8. Displaying Image Information
16.9. Rebasing a Backing File of an Image
16.10. Re-sizing the Disk Image
16.11. Listing, Creating, Applying, and Deleting a Snapshot
16.12. Supported qemu-img Formats
17. KVM Migration
17.1. Migration Definition and Benefits
17.2. Migration Requirements and Limitations
17.3. Live Migration and Red Hat Enterprise Linux Version Compatibility
17.4. Shared Storage Example: NFS for a Simple Migration
17.5. Live KVM Migration with virsh
17.5.1. Additional Tips for Migration with virsh
17.5.2. Additional Options for the virsh migrate Command
17.6. Migrating with virt-manager
18. Guest Virtual Machine Device Configuration
18.1. PCI Devices
18.1.1. Assigning a PCI Device with virsh
18.1.2. Assigning a PCI Device with virt-manager
18.1.3. PCI Device Assignment with virt-install
18.1.4. Detaching an Assigned PCI Device
18.1.5. Creating PCI Bridges
18.1.6. PCI Device Assignment with SR-IOV Devices
18.2. USB Devices
18.2.1. Assigning USB Devices to Guest Virtual Machines
18.2.2. Setting a Limit on USB Device Redirection
18.3. Configuring Device Controllers
18.4. Setting Addresses for Devices
18.5. Random Number Generator Device
18.6. Assigning GPU Devices
19. SR-IOV
19.1. Advantages of SR-IOV
19.2. Using SR-IOV
19.3. Troubleshooting SR-IOV
20. Virtual Networking
20.1. Virtual Network Switches
20.2. Bridged Mode
20.3. Network Address Translation
20.4. DNS and DHCP
20.5. Routed Mode
20.6. Isolated Mode
20.7. The Default Configuration
20.8. Examples of Common Scenarios
20.8.1. Bridged Mode
20.8.2. Routed Mode
20.8.3. NAT Mode
20.8.4. Isolated Mode
20.9. Managing a Virtual Network
20.10. Creating a Virtual Network
20.11. Attaching a Virtual Network to a Guest
20.12. Directly Attaching to Physical Interface
20.13. Dynamically Changing a Host Physical Machine or a Network Bridge that is Attached to a Virtual NIC
20.14. Applying Network Filtering
20.14.1. Introduction
20.14.2. Filtering Chains
20.14.3. Filtering Chain Priorities
20.14.4. Usage of Variables in Filters
20.14.5. Automatic IP Address Detection and DHCP Snooping
20.14.6. Reserved Variables
20.14.7. Element and Attribute Overview
20.14.8. References to Other Filters
20.14.9. Filter Rules
20.14.10. Supported Protocols
20.14.11. Advanced Filter Configuration Topics
20.14.12. Limitations
20.15. Creating Tunnels
20.15.1. Creating Multicast Tunnels
20.15.2. Creating TCP Tunnels
20.16. Setting vLAN Tags
20.17. Applying QoS to Your Virtual Network
21. Remote Management of Guests
21.1. Transport Modes
21.2. Remote Management with SSH
21.3. Remote Management over TLS and SSL
21.4. Configuring a VNC Server
21.5. Enhancing Remote Management of Virtual Machines with NSS
22. Managing Guests with the Virtual Machine Manager (virt-manager)
22.1. Starting virt-manager
22.2. The Virtual Machine Manager Main Window
22.3. The Virtual Hardware Details Window
22.3.1. Applying Boot Options to Guest Virtual Machines
22.3.2. Attaching USB Devices to a Guest Virtual Machine
22.3.3. USB Redirection
22.4. Virtual Machine Graphical Console
22.5. Adding a Remote Connection
22.6. Displaying Guest Details
22.7. Managing Snapshots
22.8. Performance Monitoring
22.9. Displaying CPU Usage for Guests
22.10. Displaying CPU Usage for Hosts
22.11. Displaying Disk I/O
22.12. Displaying Network I/O
22.13. Displaying Memory Usage
23. Managing Guest Virtual Machines with virsh
23.1. Guest Virtual Machine States and Types
23.2. Running the virsh Program
23.3. Interactive Mode Commands
23.3.1. Quitting and Exiting from the virsh Interactive Terminal
23.3.2. Changing and Displaying Directories in a Virtual Machine
23.4. Displaying the virsh Version
23.5. Getting Help
23.6. Sending Commands with echo
23.7. Connecting to the Hypervisor with virsh Connect
23.8. Displaying Information about a Guest Virtual Machine
23.9. Starting, Resuming, and Restoring a Virtual Machine
23.9.1. Starting a Guest Virtual Machine
23.9.2. Configuring a Virtual Machine to be Started Automatically at Boot
23.9.3. Rebooting a Guest Virtual Machine
23.9.4. Restoring a Guest Virtual Machine
23.9.5. Resuming a Guest Virtual Machine
23.9.6. Restoring a Guest Virtual Machine from a Suspended State
23.10. Managing a Virtual Machine Configuration
23.10.1. Saving a Guest Virtual Machine's Configuration
23.10.2. Defining a Guest Virtual Machine with an XML File
23.10.3. Updating the XML File That will be Used for Restoring a Guest Virtual Machine
23.10.4. Extracting the Guest Virtual Machine XML File
23.10.5. Editing the Guest Virtual Machine Configuration
23.11. Shutting off, Shutting down, Rebooting, and Forcing a Shutdown of a Guest Virtual Machine
23.11.1. Suspending a Running Guest Virtual Machine Using Power Management
23.11.2. Shutting down a Guest Virtual Machine
23.11.3. Suspending a Guest Virtual Machine
23.11.4. Resetting a Virtual Machine
23.11.5. Stopping a Running Guest Virtual Machine in Order to Restart It Later
23.12. Removing and Deleting a Virtual Machine
23.12.1. Undefining a Virtual Machine
23.12.2. Forcing a Guest Virtual Machine to Stop
23.13. Connecting the Serial Console for the Guest Virtual Machine
23.14. Injecting NMI
23.15. Retrieving Information about Your Virtual Machine
23.15.1. Displaying Device Block Statistics
23.15.2. Retrieving Network Statistics
23.15.3. Modifying the Link State of a Guest Virtual Machine's Virtual Interface
23.15.4. Listing the Link State of a Guest Virtual Machine's Virtual Interface
23.15.5. Setting Network Interface Bandwidth Parameters
23.15.6. Retrieving Memory Statistics
23.15.7. Displaying Errors on Block Devices
23.15.8. Displaying the Block Device Size
23.15.9. Displaying the Block Devices Associated with a Guest Virtual Machine
23.15.10. Displaying Virtual Interfaces Associated with a Guest Virtual Machine
23.16. Working with Snapshots
23.16.1. Shortening a Backing Chain by Coping the Data
23.16.2. Shortening a Backing Chain by Flattening the Image
23.16.3. Changing the Size of a Guest Virtual Machine's Block Device
23.17. Displaying a URI for Connection to a Graphical Display
23.18. Discarding Blocks Not in Use
23.19. Guest Virtual Machine Retrieval Commands
23.19.1. Displaying the Host Physical Machine Name
23.19.2. Displaying General Information about a Virtual Machine
23.19.3. Displaying a Virtual Machine's ID Number
23.19.4. Aborting Running Jobs on a Guest Virtual Machine
23.19.5. Displaying Information about Jobs Running on the Guest Virtual Machine
23.19.6. Displaying the Guest Virtual Machine's Name
23.19.7. Displaying the Virtual Machine's State
23.19.8. Displaying the Connection State to the Virtual Machine
23.20. Converting QEMU Arguments to Domain XML
23.21. Creating a Dump File of a Guest Virtual Machine's Core
23.22. Creating a Virtual Machine XML Dump (Configuration File)
23.23. Creating a Guest Virtual Machine from a Configuration File
23.24. Editing a Guest Virtual Machine's Configuration Settings
23.25. Adding Multifunction PCI Devices to KVM Guest Virtual Machines
23.26. Displaying CPU Statistics for a Specified Guest Virtual Machine
23.27. Saving a Screenshot
23.28. Sending a Keystroke Combination to a Specified Guest Virtual machine
23.29. Sending Process Signal Names to Virtual Processes
23.30. Displaying the IP Address and Port Number for the VNC Display
23.31. NUMA Node Management
23.31.1. Displaying Node Information
23.31.2. Setting NUMA Parameters
23.31.3. Displaying the Amount of Free Memory in a NUMA Cell
23.31.4. Displaying a CPU List
23.31.5. Displaying CPU Statistics
23.31.6. Managing Devices
23.31.7. Suspending the Host Physical Machine
23.31.8. Setting and Displaying the Node Memory Parameters
23.31.9. Listing Devices on a Node
23.31.10. Creating Devices on Host Nodes
23.31.11. Removing a Device
23.31.12. Collect a Device Configuration Settings
23.31.13. Triggering a Reset for a Device
23.32. Retrieving Guest Virtual Machine Information
23.32.1. Getting the Domain ID of a Guest Virtual Machine
23.32.2. Getting the Domain Name of a Guest Virtual Machine
23.32.3. Getting the UUID of a Guest Virtual Machine
23.32.4. Displaying Guest Virtual Machine Information
23.33. Storage Pool Commands
23.33.1. Searching for a Storage Pool XML
23.33.2. Finding a storage Pool
23.33.3. Listing Storage Pool Information
23.33.4. Listing the Available Storage Pools
23.33.5. Refreshing a Storage Pool List
23.33.6. Creating, Defining, and Starting Storage Pools
23.33.7. Stopping and Deleting Storage Pools
23.33.8. Creating an XML Dump File for a Pool
23.33.9. Editing the Storage Pool's Configuration File
23.34. Storage Volume Commands
23.34.1. Creating Storage Volumes
23.34.2. Creating a Storage Volume from Parameters
23.34.3. Creating a Storage Volume from an XML File
23.34.4. Cloning a Storage Volume
23.35. Deleting Storage Volumes
23.36. Deleting a Storage Volume's Contents
23.37. Dumping Storage Volume Information to an XML File
23.38. Listing Volume Information
23.39. Retrieving Storage Volume Information
23.40. Uploading and Downloading Storage Volumes
23.41. Resizing Storage Volumes
23.42. Displaying Per-guest Virtual Machine Information
23.42.1. Displaying the Guest Virtual Machines
23.42.2. Displaying Virtual CPU Information
23.42.3. Pinning vCPU to a Host Physical Machine's CPU
23.42.4. Displaying Information about the Virtual CPU Counts of a Given Domain
23.42.5. Configuring Virtual CPU Affinity
23.42.6. Configuring Virtual CPU Count
23.42.7. Configuring Memory Allocation
23.42.8. Changing the Memory Allocation for the Domain
23.42.9. Displaying Guest Virtual Machine Block Device Information
23.42.10. Displaying Guest Virtual Machine Network Device Information
23.43. Managing Virtual Networks
23.43.1. Autostarting a Virtual Network
23.43.2. Creating a Virtual Network from an XML File
23.43.3. Defining a Virtual Network from an XML File
23.43.4. Stopping a Virtual Network
23.43.5. Creating a Dump File
23.43.6. Editing a Virtual Network's XML Configuration File
23.43.7. Getting Information about a Virtual Network
23.43.8. Listing Information about a Virtual Network
23.43.9. Converting a Network UUID to Network Name
23.43.10. Starting a (previously defined) Inactive Network
23.43.11. Undefining the Configuration for an Inactive Network
23.43.12. Converting a Network Name to Network UUID
23.43.13. Updating an Existing Network Definition File
23.43.14. Migrating Guest Virtual Machines with virsh
23.43.15. Setting a Static IP Address for the Guest Virtual Machine
23.44. Interface Commands
23.44.1. Defining and Starting a Host Physical Machine Interface via an XML File
23.44.2. Editing the XML Configuration File for the Host Interface
23.44.3. Listing Active Host Interfaces
23.44.4. Converting a MAC Address into an Interface Name
23.44.5. Stopping a Specific Host Physical Machine Interface
23.44.6. Displaying the Host Configuration File
23.44.7. Creating Bridge Devices
23.44.8. Tearing Down a Bridge Device
23.44.9. Manipulating Interface Snapshots
23.45. Managing Snapshots
23.45.1. Creating Snapshots
23.45.2. Creating a Snapshot for the Current Guest Virtual Machine
23.45.3. Displaying the Current Snapshot of a Guest Virtual Machine
23.45.4. snapshot-edit
23.45.5. snapshot-info
23.45.6. snapshot-list
23.45.7. snapshot-dumpxml
23.45.8. snapshot-parent
23.45.9. snapshot-revert
23.45.10. snapshot-delete
23.46. Guest Virtual Machine CPU Model Configuration
23.46.1. Introduction
23.46.2. Learning about the Host Physical Machine CPU Model
23.46.3. Determining Support for VFIO IOMMU Devices
23.46.4. Determining a Compatible CPU Model to Suit a Pool of Host Physical Machines
23.47. Configuring the Guest Virtual Machine CPU Model
23.48. Managing Resources for Guest Virtual Machines
23.49. Setting Schedule Parameters
23.50. Disk I/O Throttling
23.51. Display or Set Block I/O Parameters
23.52. Configuring Memory Tuning
24. Guest Virtual Machine Disk Access with Offline Tools
24.1. Introduction
24.1.1. Caution about Using Remote Connections
24.2. Terminology
24.3. Installation
24.4. The guestfish Shell
24.4.1. Viewing File Systems with guestfish
24.4.2. Adding Files with guestfish
24.4.3. Modifying Files with guestfish
24.4.4. Other Actions with guestfish
24.4.5. Shell Scripting with guestfish
24.4.6. Augeas and libguestfs Scripting
24.5. Other Commands
24.6. virt-rescue: The Rescue Shell
24.6.1. Introduction
24.6.2. Running virt-rescue
24.7. virt-df: Monitoring Disk Usage
24.7.1. Introduction
24.7.2. Running virt-df
24.8. virt-resize: Resizing Guest Virtual Machines Offline
24.8.1. Introduction
24.8.2. Expanding a Disk Image
24.9. virt-inspector: Inspecting Guest Virtual Machines
24.9.1. Introduction
24.9.2. Installation
24.9.3. Running virt-inspector
24.10. Using the API from Programming Languages
24.10.1. Interaction with the API via a C program
24.11. virt-sysprep: Resetting Virtual Machine Settings
24.12. virt-customize: Customizing Virtual Machine Settings
24.13. virt-diff: Listing the Differences between Virtual Machine Files
25. Graphic User Interface Tools for Guest Virtual Machine Management
25.1. Using virt-viewer Command Line
25.2. remote-viewer
25.3. GNOME Boxes
26. Manipulating the Domain XML
26.1. General Information and Meta-data
26.2. Operating System Booting
26.2.1. BIOS Boot Loader
26.2.2. Host Physical Machine Boot Loader
26.2.3. Direct Kernel Boot
26.2.4. Container Boot
26.3. SMBIOS System Information
26.4. CPU allocation
26.5. CPU tuning
26.6. Memory Backing
26.7. Memory Tuning
26.8. Memory Allocation
26.9. NUMA Node Tuning
26.10. Block I/O Tuning
26.11. Resource Partitioning
26.12. CPU Models and Topology
26.12.1. Changing the Feature Set for a Specified CPU
26.12.2. Guest Virtual Machine NUMA Topology
26.13. Events Configuration
26.14. Power Management
26.15. Hypervisor Features
26.16. Timekeeping
26.17. Timer Element Attributes
26.18. Devices
26.18.1. Hard Drives, Floppy Disks, and CD-ROMs
26.18.2. File Systems
26.18.3. Device Addresses
26.18.4. Controllers
26.18.5. Device Leases
26.18.6. Host Physical Machine Device Assignment
26.18.7. Redirected devices
26.18.8. Smartcard Devices
26.18.9. Network Interfaces
26.18.10. Input Devices
26.18.11. Hub Devices
26.18.12. Graphical Framebuffers
26.18.13. Video Devices
26.18.14. Consoles, Serial, and Channel Devices
26.18.15. Guest Virtual Machine Interfaces
26.18.16. Channel
26.18.17. Host Physical Machine Interface
26.18.18. Sound Devices
26.18.19. Watchdog Device
26.18.20. Setting a Panic Device
26.18.21. Memory Balloon Device
26.19. Storage Pools
26.19.1. Providing Metadata for the Storage Pool
26.19.2. Source Elements
26.19.3. Creating Target Elements
26.19.4. Setting Device Extents
26.20. Storage Volumes
26.20.1. General Metadata
26.20.2. Setting Target Elements
26.20.3. Setting Backing Store Elements
26.21. Security Label
26.22. A Sample Configuration File

Chapter 13. Securing the Host Physical Machine and Improving Performance

The following tasks and tips can assist you with increasing the performance of your Red Hat Enterprise Linux host.
  • Run SELinux in enforcing mode. Set SELinux to run in enforcing mode with the setenforce command.
    # setenforce 1
  • Remove or disable any unnecessary services such as AutoFS, NFS, FTP, HTTP, NIS, telnetd, sendmail and so on.
  • Only add the minimum number of user accounts needed for platform management on the server and remove unnecessary user accounts.
  • Avoid running any unessential applications on your host. Running applications on the host may impact virtual machine performance and can affect server stability. Any application which may crash the server will also cause all virtual machines on the server to go down.
  • Use a central location for virtual machine installations and images. Virtual machine images should be stored under /var/lib/libvirt/images/. If you are using a different directory for your virtual machine images make sure you add the directory to your SELinux policy and relabel it before starting the installation. Use of shareable, network storage in a central location is highly recommended.

Note

Additional performance tips can be found in the Red Hat Enterprise Linux Virtualization Tuning and Optimization Guide.
Additional security tips can be found in the Red Hat Enterprise Linux Virtualization Security Guide.

13.1. Security Deployment Plan

When deploying virtualization technologies, you must ensure that the host physical machine and its operating system cannot be compromised. In this case the host physical machine is a Red Hat Enterprise Linux system that manages the system, devices, memory and networks as well as all guest virtual machines. If the host physical machine is insecure, all guest virtual machines in the system are vulnerable. There are several ways to enhance security on systems using virtualization. You or your organization should create a Deployment Plan. This plan needs to contain the following:
  • Operating specifications
  • Specifies which services are needed on your guest virtual machines
  • Specifies the host physical servers as well as what support is required for these services
Here are a few security issues to consider while developing a deployment plan:
  • Run only necessary services on host physical machines. The fewer processes and services running on the host physical machine, the higher the level of security and performance.
  • Enable SELinux on the hypervisor. Refer to the Red Hat Enterprise Linux Virtualization Security Guide for more information on using SELinux and virtualization.
  • Use a firewall to restrict traffic to the host physical machine. You can setup a firewall with default-reject rules that will help secure the host physical machine from attacks. It is also important to limit network-facing services.
  • Do not allow normal users to access the host operating system. If the host operating system is privileged, granting access to unprivileged accounts may compromise the level of security.

13.2. Client Access Control

libvirt's client access control framework allows system administrators to setup fine grained permission rules across client users, managed objects, and API operations. This allows client connections to be locked down to a minimal set of privileges.
In a default configuration, the libvirtd daemon has three levels of access control. All connections start off in an unauthenticated state, where the only API operations allowed are those required to complete authentication. After successful authentication, a connection either has full, unrestricted access to all libvirt API calls, or is locked down to only "read only" operations, according to what socket the client connection originated on. The access control framework allows authenticated connections to have fine grained permission rules to be defined by the administrator. Every API call in libvirt has a set of permissions that will be validated against the object being used. Further permissions will also be checked if certain flags are set in the API call. In addition to checks on the object passed in to an API call, some methods will filter their results.

13.2.1. Access Control Drivers

The access control framework is designed as a pluggable system to enable future integration with arbitrary access control technologies. By default, the none driver is used, which does no access control checks at all. At this time, libvirt provides support for using polkit as a real access control driver. To learn how to use the polkit access driver refer to the configuration documentation.
The access driver is configured in the libvirtd.conf configuration file, using the access_drivers parameter. This parameter accepts an array of access control driver names. If more than one access driver is requested, then all must succeed in order for access to be granted. To enable 'polkit' as the driver run the command:
# augtool -s set '/files/etc/libvirt/libvirtd.conf/access_drivers[1]' polkit
To set the driver back to the default (no access control), enter the following command:
# augtool -s rm /files/etc/libvirt/libvirtd.conf/access_drivers
It should be noted that changes made to libvirtd.conf require that the libvirtd daemon be restarted.

13.2.2. Objects and Permissions

libvirt applies access control to all the main object types in its API. Each object type, in turn, has a set of permissions defined. To determine what permissions are checked for specific API call, consult the API reference manual documentation for the API in question. For the complete list of objects and permissions, refer to libvirt.org.

13.2.3. Security Concerns when Adding Block Devices to a Guest

  • The host physical machine should not use filesystem labels to identify file systems in the fstab file, the initrd file or on the kernel command line. Doing so presents a security risk if guest virtual machines have write access to whole partitions or LVM volumes, because a guest virtual machine could potentially write a filesystem label belonging to the host physical machine, to its own block device storage. Upon reboot of the host physical machine, the host physical machine could then mistakenly use the guest virtual machine's disk as a system disk, which would compromise the host physical machine system.
    It is preferable to use the UUID of a device to identify it in the fstab file, the initrd file or on the kernel command line. While using UUIDs is still not completely secure on certain file systems, a similar compromise with UUID is significantly less feasible.
  • Guest virtual machines should not be given write access to whole disks or block devices (for example, /dev/sdb). Guest virtual machines with access to whole block devices may be able to modify volume labels, which can be used to compromise the host physical machine system. Use partitions (for example, /dev/sdb1) or LVM volumes to prevent this problem. Refer to LVM Administration with CLI Commands or LVM Configuration Examples for information on LVM administration and configuration examples.
    If you are using raw access to partitions, for example /dev/sdb1 or raw disks such as /dev/sdb, you should configure LVM to only scan disks that are safe, using the global_filter setting. Refer to the Sample lvm.conf File for an example of an LVM configuration script using the global_filter command.

Chapter 14. Storage Pools

14.1. Disk-based Storage Pools
14.1.1. Creating a Disk-based Storage Pool Using virsh
14.1.2. Deleting a Storage Pool Using virsh
14.2. Partition-based Storage Pools
14.2.1. Creating a Partition-based Storage Pool Using virt-manager
14.2.2. Deleting a Storage Pool Using virt-manager
14.2.3. Creating a Partition-based Storage Pool Using virsh
14.2.4. Deleting a Storage Pool Using virsh
14.3. Directory-based Storage Pools
14.3.1. Creating a Directory-based Storage Pool with virt-manager
14.3.2. Deleting a Storage Pool Using virt-manager
14.3.3. Creating a Directory-based Storage Pool with virsh
14.3.4. Deleting a Storage Pool Using virsh
14.4. LVM-based Storage Pools
14.4.1. Creating an LVM-based Storage Pool with virt-manager
14.4.2. Deleting a Storage Pool Using virt-manager
14.4.3. Creating an LVM-based Storage Pool with virsh
14.4.4. Deleting a Storage Pool Using virsh
14.5. iSCSI-based Storage Pools
14.5.1. Configuring a Software iSCSI Target
14.5.2. Creating an iSCSI Storage Pool in virt-manager
14.5.3. Deleting a Storage Pool Using virt-manager
14.5.4. Creating an iSCSI-based Storage Pool with virsh
14.5.5. Securing an iSCSI Storage Pool
14.5.6. Deleting a Storage Pool Using virsh
14.6. NFS-based Storage Pools
14.6.1. Creating an NFS-based Storage Pool with virt-manager
14.6.2. Deleting a Storage Pool Using virt-manager
14.7. Using an NPIV Virtual Adapter (vHBA) with SCSI Devices
14.7.1. Creating a vHBA
14.7.2. Creating a Storage Pool Using the vHBA
14.7.3. Configuring the Virtual Machine to Use a vHBA LUN
14.7.4. Destroying the vHBA Storage Pool
14.8. GlusterFS Storage Pools
14.8.1. Creating a GlusterFS Storage Pool Using virsh
14.8.2. Deleting a GlusterFS Storage Pool Using virsh
This chapter includes instructions on creating storage pools of assorted types. A storage pool is a quantity of storage set aside by an administrator, often a dedicated storage administrator, for use by guest virtual machines. Storage pools are divided into storage volumes either by the storage administrator or the system administrator, and the volumes are then assigned to guest virtual machines as block devices.
For example, the storage administrator responsible for an NFS server creates a shared disk to store all of the guest virtual machines' data. The system administrator would define a storage pool on the virtualization host using the details of the shared disk. In this example, the administrator may want nfs.example.com:/path/to/share to be mounted on /vm_data). When the storage pool is started, libvirt mounts the share on the specified directory, just as if the system administrator logged in and executed mount nfs.example.com:/path/to/share /vmdata. If the storage pool is configured to autostart, libvirt ensures that the NFS shared disk is mounted on the directory specified when libvirt is started.
Once the storage pool is started, the files in the NFS shared disk are reported as storage volumes, and the storage volumes' paths may be queried using the libvirt APIs. The storage volumes' paths can then be copied into the section of a guest virtual machine's XML definition describing the source storage for the guest virtual machine's block devices.In the case of NFS, an application using the libvirt APIs can create and delete storage volumes in the storage pool (files in the NFS share) up to the limit of the size of the pool (the storage capacity of the share). Not all storage pool types support creating and deleting volumes. Stopping the storage pool (pool-destroy) undoes the start operation, in this case, unmounting the NFS share. The data on the share is not modified by the destroy operation, despite what the name of the command suggests. For more details, see man virsh.
A second example is an iSCSI storage pool. A storage administrator provisions an iSCSI target to present a set of LUNs to the host running the virtual machines. When libvirt is configured to manage that iSCSI target as a storage pool, libvirt will ensure that the host logs into the iSCSI target and libvirt can then report the available LUNs as storage volumes. The storage volumes' paths can be queried and used in virtual machines' XML definitions as in the NFS example. In this case, the LUNs are defined on the iSCSI server, and libvirt cannot create and delete volumes.
Storage pools and volumes are not required for the proper operation of guest virtual machines. Storage pools and volumes provide a way for libvirt to ensure that a particular piece of storage will be available for a guest virtual machine. On systems that do not use storage pools, system administrators must ensure the availability of the guest virtual machine's storage. For example, adding the NFS share to the host physical machine's fstab is required so that the share is mounted at boot time.
One of the advantages of using libvirt to manage storage pools and volumes is libvirt's remote protocol, so it is possible to manage all aspects of a guest virtual machine's life cycle, as well as the configuration of the resources required by the guest virtual machine. These operations can be performed on a remote host entirely within the libvirt API. As a result, a management application using libvirt can enable a user to perform all the required tasks for configuring the host physical machine for a guest virtual machine such as: allocating resources, running the guest virtual machine, shutting it down and de-allocating the resources, without requiring shell access or any other control channel.
Although the storage pool is a virtual container it is limited by two factors: maximum size allowed to it by qemu-kvm and the size of the disk on the host machine. Storage pools may not exceed the size of the disk on the host machine. The maximum sizes are as follows:
  • virtio-blk = 2^63 bytes or 8 Exabytes(using raw files or disk)
  • Ext4 = ~ 16 TB (using 4 KB block size)
  • XFS = ~8 Exabytes
  • qcow2 and host file systems keep their own metadata and scalability should be evaluated/tuned when trying very large image sizes. Using raw disks means fewer layers that could affect scalability or max size.
libvirt uses a directory-based storage pool, the /var/lib/libvirt/images/ directory, as the default storage pool. The default storage pool can be changed to another storage pool.
  • Local storage pools - Local storage pools are directly attached to the host physical machine server. Local storage pools include: local directories, directly attached disks, physical partitions, and LVM volume groups. These storage volumes store guest virtual machine images or are attached to guest virtual machines as additional storage. As local storage pools are directly attached to the host physical machine server, they are useful for development, testing and small deployments that do not require migration or large numbers of guest virtual machines. Local storage pools are not suitable for many production environments as local storage pools do not support live migration.
  • Networked (shared) storage pools - Networked storage pools include storage devices shared over a network using standard protocols. Networked storage is required when migrating virtual machines between host physical machines with virt-manager, but is optional when migrating with virsh. Networked storage pools are managed by libvirt. Supported protocols for networked storage pools include:
    • Fibre Channel-based LUNs
    • iSCSI
    • NFS
    • GFS2
    • SCSI RDMA protocols (SCSI RCP), the block export protocol used in InfiniBand and 10GbE iWARP adapters.

Note

Multi-path storage pools should not be created or used as they are not fully supported.

Example 14.1. NFS storage pool

Suppose a storage administrator responsible for an NFS server creates a share to store guest virtual machines' data. The system administrator defines a pool on the host physical machine with the details of the share (nfs.example.com:/path/to/share should be mounted on /vm_data). When the pool is started, libvirt mounts the share on the specified directory, just as if the system administrator logged in and executed mount nfs.example.com:/path/to/share /vmdata. If the pool is configured to autostart, libvirt ensures that the NFS share is mounted on the directory specified when libvirt is started.
Once the pool starts, the files that the NFS share, are reported as volumes, and the storage volumes' paths are then queried using the libvirt APIs. The volumes' paths can then be copied into the section of a guest virtual machine's XML definition file describing the source storage for the guest virtual machine's block devices. With NFS, applications using the libvirt APIs can create and delete volumes in the pool (files within the NFS share) up to the limit of the size of the pool (the maximum storage capacity of the share). Not all pool types support creating and deleting volumes. Stopping the pool negates the start operation, in this case, unmounts the NFS share. The data on the share is not modified by the destroy operation, despite the name. See man virsh for more details.

Note

Storage pools and volumes are not required for the proper operation of guest virtual machines. Pools and volumes provide a way for libvirt to ensure that a particular piece of storage will be available for a guest virtual machine, but some administrators will prefer to manage their own storage and guest virtual machines will operate properly without any pools or volumes defined. On systems that do not use pools, system administrators must ensure the availability of the guest virtual machines' storage using whatever tools they prefer. For example, adding the NFS share to the host physical machine's fstab is required so that the share is mounted at boot time.

Warning

When creating storage pools on a guest, make sure to follow the related security considerations.

14.1. Disk-based Storage Pools

This section covers creating disk based storage devices for guest virtual machines.

Warning

Guests should not be given write access to whole disks or block devices (for example, /dev/sdb). Use partitions (for example, /dev/sdb1) or LVM volumes.
If you pass an entire block device to the guest, the guest will likely partition it or create its own LVM groups on it. This can cause the host physical machine to detect these partitions or LVM groups and cause errors.

14.1.1. Creating a Disk-based Storage Pool Using virsh

This procedure creates a new storage pool using a disk device with the virsh command.

Warning

Dedicating a disk to a storage pool will reformat and erase all data presently stored on the disk device. It is strongly recommended to back up the data on the storage device before commencing with the following procedure:
  1. Create a GPT disk label on the disk

    The disk must be relabeled with a GUID Partition Table (GPT) disk label. GPT disk labels allow for creating a large numbers of partitions, up to 128 partitions, on each device. GPT partition tables can store partition data for far more partitions than the MS-DOS partition table.
    # parted /dev/sdb
    GNU Parted 2.1
    Using /dev/sdb
    Welcome to GNU Parted! Type 'help' to view a list of commands.
    (parted) mklabel
    New disk label type? gpt
    (parted) quit
    Information: You may need to update /etc/fstab.
    #
    
  2. Create the storage pool configuration file

    Create a temporary XML text file containing the storage pool information required for the new device.
    The file must be in the format shown below, and contain the following fields:
    <name>guest_images_disk</name>
    The name parameter determines the name of the storage pool. This example uses the name guest_images_disk in the example below.
    <device path='/dev/sdb'/>
    The device parameter with the path attribute specifies the device path of the storage device. This example uses the device /dev/sdb.
    <target> <path>/dev</path></target>
    The file system target parameter with the path sub-parameter determines the location on the host physical machine file system to attach volumes created with this storage pool.
    For example, sdb1, sdb2, sdb3. Using /dev/, as in the example below, means volumes created from this storage pool can be accessed as /dev/sdb1, /dev/sdb2, /dev/sdb3.
    <format type='gpt'/>
    The format parameter specifies the partition table type. This example uses the gpt in the example below, to match the GPT disk label type created in the previous step.
    Create the XML file for the storage pool device with a text editor.

    Example 14.2. Disk based storage device storage pool

    <pool type='disk'>
      <name>guest_images_disk</name>
      <source>
        <device path='/dev/sdb'/>
        <format type='gpt'/>
      </source>
      <target>
        <path>/dev</path>
      </target>
    </pool>
    
  3. Start the storage pool

    Start the storage pool with the virsh pool-start command. Verify the pool is started with the virsh pool-list --all command.
    # virsh pool-start iscsirhel7guest
    Pool iscsirhel7guest started
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    guest_images_disk    active     no
    
  4. Attach the device

    Add the storage pool definition using the virsh pool-define command with the XML configuration file created in the previous step.
    # virsh pool-define ~/guest_images_disk.xml
    Pool guest_images_disk defined from /root/guest_images_disk.xml
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    guest_images_disk    inactive   no
    
  5. Turn on autostart

    Turn on autostart for the storage pool. Autostart configures the libvirtd service to start the storage pool when the service starts.
    # virsh pool-autostart guest_images_disk
    Pool guest_images_disk marked as autostarted
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    guest_images_disk    active     yes
    
  6. Verify the storage pool configuration

    Verify the storage pool was created correctly, the sizes reported correctly, and the state reports as running.
    # virsh pool-info guest_images_disk
    Name:           guest_images_disk
    UUID:           551a67c8-5f2a-012c-3844-df29b167431c
    State:          running
    Capacity:       465.76 GB
    Allocation:     0.00
    Available:      465.76 GB
    # ls -la /dev/sdb
    brw-rw----. 1 root disk 8, 16 May 30 14:08 /dev/sdb
    # virsh vol-list guest_images_disk
    Name                 Path
    -----------------------------------------
    
  7. Optional: Remove the temporary configuration file

    Remove the temporary storage pool XML configuration file if it is not needed anymore.
    # rm ~/guest_images_disk.xml
A disk based storage pool is now available.

14.1.2. Deleting a Storage Pool Using virsh

The following demonstrates how to delete a storage pool using virsh:
  1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it.
    # virsh pool-destroy guest_images_disk
  2. Remove the storage pool's definition
    # virsh pool-undefine guest_images_disk

14.2. Partition-based Storage Pools

This section covers using a pre-formatted block device, a partition, as a storage pool.
For the following examples, a host physical machine has a 500GB hard drive (/dev/sdc) partitioned into one 500GB partition (/dev/sdc1). We set up a storage pool for it using the procedure below.

14.2.1. Creating a Partition-based Storage Pool Using virt-manager

This procedure creates a new storage pool using a partition of a storage device.

Procedure 14.1. Creating a partition-based storage pool with virt-manager

  1. Set the File System to ext4

    From a command window, enter the following command to set the file system to ext4
    # mkfs.ext4 /dev/sdc1
  2. Open the storage pool settings

    1. In the virt-manager graphical interface, select the host physical machine from the main window.
      Open the Edit menu and select Connection Details
    2. Click on the Storage tab of the Connection Details window.
      Storage tab

      Figure 14.1. Storage tab

  3. Create the new storage pool

    1. Add a new pool (part 1)

      Press the + button (at the bottom of the window). The Add a New Storage Pool wizard appears.
      Choose a Name for the storage pool. This example uses the name guest_images_fs. Change the Type to fs: Pre-Formatted Block Device.
      Storage pool name and type

      Figure 14.2. Storage pool name and type

      Press the Forward button to continue.
    2. Add a new pool (part 2)

      Change the Target Path, Format, and Source Path fields.
      Storage pool path and format

      Figure 14.3. Storage pool path and format

      Target Path
      Enter the location to mount the source device for the storage pool in the Target Path field. If the location does not already exist, virt-manager will create the directory.
      Format
      Select a format from the Format list. The device is formatted with the selected format.
      This example uses the ext4 file system, the default Red Hat Enterprise Linux file system.
      Source Path
      Enter the device in the Source Path field.
      This example uses the /dev/sdc1 device.
      Verify the details and press the Finish button to create the storage pool.
  4. Verify the new storage pool

    The new storage pool appears in the storage list on the left after a few seconds. Verify the size is reported as expected, 458.20 GB Free in this example. Verify the State field reports the new storage pool as Active.
    Select the storage pool. In the Autostart field, click the On Boot check box. This will make sure the storage device starts whenever the libvirtd service starts.
    Storage list confirmation

    Figure 14.4. Storage list confirmation

    The storage pool is now created, close the Connection Details window.

14.2.2. Deleting a Storage Pool Using virt-manager

This procedure demonstrates how to delete a storage pool.
  1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it. To do this, select the storage pool you want to stop and click at the bottom of the Storage window.
  2. Delete the storage pool by clicking . This icon is only enabled if you stop the storage pool first.

14.2.3. Creating a Partition-based Storage Pool Using virsh

This section covers creating a partition-based storage pool with the virsh command.

Warning

Do not use this procedure to assign an entire disk as a storage pool (for example, /dev/sdb). Guests should not be given write access to whole disks or block devices. Only use this method to assign partitions (for example, /dev/sdb1) to storage pools.

Procedure 14.2. Creating pre-formatted block device storage pools using virsh

  1. Create the storage pool definition

    Use the virsh pool-define-as command to create a new storage pool definition. There are three options that must be provided to define a pre-formatted disk as a storage pool:
    Partition name
    The name parameter determines the name of the storage pool. This example uses the name guest_images_fs in the example below.
    device
    The device parameter with the path attribute specifies the device path of the storage device. This example uses the partition /dev/sdc1.
    mountpoint
    The mountpoint on the local file system where the formatted device will be mounted. If the mount point directory does not exist, the virsh command can create the directory.
    The directory /guest_images is used in this example.
    # virsh pool-define-as guest_images_fs fs - - /dev/sdc1 - "/guest_images"
    Pool guest_images_fs defined
    
    The new pool is now created.
  2. Verify the new pool

    List the present storage pools.
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    guest_images_fs      inactive   no
    
  3. Create the mount point

    Use the virsh pool-build command to create a mount point for a pre-formatted file system storage pool.
    # virsh pool-build guest_images_fs
    Pool guest_images_fs built
    # ls -la /guest_images
    total 8
    drwx------.  2 root root 4096 May 31 19:38 .
    dr-xr-xr-x. 25 root root 4096 May 31 19:38 ..
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    guest_images_fs      inactive   no
    
  4. Start the storage pool

    Use the virsh pool-start command to mount the file system onto the mount point and make the pool available for use.
    # virsh pool-start guest_images_fs
    Pool guest_images_fs started
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    guest_images_fs      active     no
    
  5. Turn on autostart

    By default, a storage pool is defined with virsh is not set to automatically start each time libvirtd starts. Turn on automatic start with the virsh pool-autostart command. The storage pool is now automatically started each time libvirtd starts.
    # virsh pool-autostart guest_images_fs
    Pool guest_images_fs marked as autostarted
    
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    guest_images_fs      active     yes
    
  6. Verify the storage pool

    Verify the storage pool was created correctly, the sizes reported are as expected, and the state is reported as running. Verify there is a "lost+found" directory in the mount point on the file system, indicating the device is mounted.
    # virsh pool-info guest_images_fs
    Name:           guest_images_fs
    UUID:           c7466869-e82a-a66c-2187-dc9d6f0877d0
    State:          running
    Persistent:     yes
    Autostart:      yes
    Capacity:       458.39 GB
    Allocation:     197.91 MB
    Available:      458.20 GB
    # mount | grep /guest_images
    /dev/sdc1 on /guest_images type ext4 (rw)
    # ls -la /guest_images
    total 24
    drwxr-xr-x.  3 root root  4096 May 31 19:47 .
    dr-xr-xr-x. 25 root root  4096 May 31 19:38 ..
    drwx------.  2 root root 16384 May 31 14:18 lost+found
    

14.2.4. Deleting a Storage Pool Using virsh

  1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it.
    # virsh pool-destroy guest_images_disk
  2. Optionally, if you want to remove the directory where the storage pool resides use the following command:
    # virsh pool-delete guest_images_disk
  3. Remove the storage pool's definition
    # virsh pool-undefine guest_images_disk

14.3. Directory-based Storage Pools

This section covers storing guest virtual machines in a directory on the host physical machine.
Directory-based storage pools can be created with virt-manager or the virsh command-line tools.

14.3.1. Creating a Directory-based Storage Pool with virt-manager

  1. Create the local directory

    1. Optional: Create a new directory for the storage pool

      Create the directory on the host physical machine for the storage pool. This example uses a directory named /guest_images.
      # mkdir /guest_images
    2. Set directory ownership

      Change the user and group ownership of the directory. The directory must be owned by the root user.
      # chown root:root /guest_images
    3. Set directory permissions

      Change the file permissions of the directory.
      # chmod 700 /guest_images
    4. Verify the changes

      Verify the permissions were modified. The output shows a correctly configured empty directory.
      # ls -la /guest_images
      total 8
      drwx------.  2 root root 4096 May 28 13:57 .
      dr-xr-xr-x. 26 root root 4096 May 28 13:57 ..
      
  2. Configure SELinux file contexts

    Configure the correct SELinux context for the new directory. Note that the name of the pool and the directory do not have to match. However, when you shutdown the guest virtual machine, libvirt has to set the context back to a default value. The context of the directory determines what this default value is. It is worth explicitly labeling the directory virt_image_t, so that when the guest virtual machine is shutdown, the images get labeled 'virt_image_t' and are thus isolated from other processes running on the host physical machine.
    # semanage fcontext -a -t virt_image_t '/guest_images(/.*)?'
    # restorecon -R /guest_images
    
  3. Open the storage pool settings

    1. In the virt-manager graphical interface, select the host physical machine from the main window.
      Open the Edit menu and select Connection Details
    2. Click on the Storage tab of the Connection Details window.
      Storage tab

      Figure 14.5. Storage tab

  4. Create the new storage pool

    1. Add a new pool (part 1)

      Press the + button (the add pool button). The Add a New Storage Pool wizard appears.
      Choose a Name for the storage pool. This example uses the name guest_images. Change the Type to dir: Filesystem Directory.
      Name the storage pool

      Figure 14.6. Name the storage pool

      Press the Forward button to continue.
    2. Add a new pool (part 2)

      Change the Target Path field. For example, /guest_images.
      Selecting a path for the storage pool

      Figure 14.7. Selecting a path for the storage pool

      Verify the details and press the Finish button to create the storage pool.
  5. Verify the new storage pool

    The new storage pool appears in the storage list on the left after a few seconds. Verify the size is reported as expected, 36.41 GB Free in this example. Verify the State field reports the new storage pool as Active.
    Select the storage pool. In the Autostart field, confirm that the On Boot check box is checked. This will make sure the storage pool starts whenever the libvirtd service starts.
    Verify the storage pool information

    Figure 14.8. Verify the storage pool information

    The storage pool is now created, close the Connection Details window.

14.3.2. Deleting a Storage Pool Using virt-manager

This procedure demonstrates how to delete a storage pool.
  1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it. To do this, select the storage pool you want to stop and click at the bottom of the Storage window.
  2. Delete the storage pool by clicking . This icon is only enabled if you stop the storage pool first.

14.3.3. Creating a Directory-based Storage Pool with virsh

  1. Create the storage pool definition

    Use the virsh pool-define-as command to define a new storage pool. There are two options required for creating directory-based storage pools:
    • The name of the storage pool.
      This example uses the name guest_images. All further virsh commands used in this example use this name.
    • The path to a file system directory for storing guest image files. If this directory does not exist, virsh will create it.
      This example uses the /guest_images directory.
     # virsh pool-define-as guest_images dir - - - - "/guest_images"
    Pool guest_images defined
  2. Verify the storage pool is listed

    Verify the storage pool object is created correctly and the state reports it as inactive.
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    guest_images     inactive   no
  3. Create the local directory

    Use the virsh pool-build command to build the directory-based storage pool for the directory guest_images (for example), as shown:
    # virsh pool-build guest_images
    Pool guest_images built
    # ls -la /guest_images
    total 8
    drwx------.  2 root root 4096 May 30 02:44 .
    dr-xr-xr-x. 26 root root 4096 May 30 02:44 ..
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    guest_images     inactive   no
  4. Start the storage pool

    Use the virsh command pool-start to enable a directory storage pool, thereby allowing volumes of the pool to be used as guest disk images.
    # virsh pool-start guest_images
    Pool guest_images started
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default             active     yes
    guest_images    active     no
    
  5. Turn on autostart

    Turn on autostart for the storage pool. Autostart configures the libvirtd service to start the storage pool when the service starts.
    # virsh pool-autostart guest_images
    Pool guest_images marked as autostarted
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    guest_images         active     yes
    
  6. Verify the storage pool configuration

    Verify the storage pool was created correctly, the size is reported correctly, and the state is reported as running. If you want the pool to be accessible even if the guest virtual machine is not running, make sure that Persistent is reported as yes. If you want the pool to start automatically when the service starts, make sure that Autostart is reported as yes.
    # virsh pool-info guest_images
    Name:           guest_images
    UUID:           779081bf-7a82-107b-2874-a19a9c51d24c
    State:          running
    Persistent:     yes
    Autostart:      yes
    Capacity:       49.22 GB
    Allocation:     12.80 GB
    Available:      36.41 GB
    
    # ls -la /guest_images
    total 8
    drwx------.  2 root root 4096 May 30 02:44 .
    dr-xr-xr-x. 26 root root 4096 May 30 02:44 ..
    #
    
A directory-based storage pool is now available.

14.3.4. Deleting a Storage Pool Using virsh

The following demonstrates how to delete a storage pool using virsh:
  1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it.
    # virsh pool-destroy guest_images_disk
  2. Optionally, if you want to remove the directory where the storage pool resides use the following command:
    # virsh pool-delete guest_images_disk
  3. Remove the storage pool's definition
    # virsh pool-undefine guest_images_disk

14.4. LVM-based Storage Pools

This chapter covers using LVM volume groups as storage pools.
LVM-based storage groups provide the full flexibility of LVM.

Note

Thin provisioning is currently not possible with LVM based storage pools.

Note

For more details on LVM, refer to the Red Hat Enterprise Linux Storage Administration Guide.

Warning

LVM-based storage pools require a full disk partition. If activating a new partition/device with these procedures, the partition will be formatted and all data will be erased. If using the host's existing Volume Group (VG) nothing will be erased. It is recommended to back up the storage device before commencing the following procedure.

14.4.1. Creating an LVM-based Storage Pool with virt-manager

LVM-based storage pools can use existing LVM volume groups or create new LVM volume groups on a blank partition.
  1. Optional: Create new partition for LVM volumes

    These steps describe how to create a new partition and LVM volume group on a new hard disk drive.

    Warning

    This procedure will remove all data from the selected storage device.
    1. Create a new partition

      Use the fdisk command to create a new disk partition from the command line. The following example creates a new partition that uses the entire disk on the storage device /dev/sdb.
      # fdisk /dev/sdb
      Command (m for help):
      
      Press n for a new partition.
    2. Press p for a primary partition.
      Command action
         e   extended
         p   primary partition (1-4)
      
    3. Choose an available partition number. In this example the first partition is chosen by entering 1.
      Partition number (1-4): 1
    4. Enter the default first cylinder by pressing Enter.
      First cylinder (1-400, default 1):
      
    5. Select the size of the partition. In this example the entire disk is allocated by pressing Enter.
      Last cylinder or +size or +sizeM or +sizeK (2-400, default 400):
      
    6. Set the type of partition by pressing t.
      Command (m for help): t
    7. Choose the partition you created in the previous steps. In this example, the partition number is 1.
      Partition number (1-4): 1
    8. Enter 8e for a Linux LVM partition.
      Hex code (type L to list codes): 8e
    9. write changes to disk and quit.
      Command (m for help): w
      Command (m for help): q
    10. Create a new LVM volume group

      Create a new LVM volume group with the vgcreate command. This example creates a volume group named guest_images_lvm.
      # vgcreate guest_images_lvm /dev/sdb1
        Physical volume "/dev/vdb1" successfully created
        Volume group "guest_images_lvm" successfully created
      
    The new LVM volume group, guest_images_lvm, can now be used for an LVM-based storage pool.
  2. Open the storage pool settings

    1. In the virt-manager graphical interface, select the host from the main window.
      Open the Edit menu and select Connection Details
    2. Click on the Storage tab.
      Storage tab

      Figure 14.9. Storage tab

  3. Create the new storage pool

    1. Start the Wizard

      Press the + button (the add pool button). The Add a New Storage Pool wizard appears.
      Choose a Name for the storage pool. We use guest_images_lvm for this example. Then change the Type to logical: LVM Volume Group, and
      Add LVM storage pool

      Figure 14.10. Add LVM storage pool

      Press Forward to continue.
    2. Add a new pool (part 2)

      Fill in the Target Path and Source Path fields, and check the Build Pool check box.
      • Use the Target Path field to either select an existing LVM volume group or as the name for a new volume group. The default format is storage_pool_name/lvm_Volume_Group_name.
        This example uses a new volume group named /dev/guest_images_lvm.
      • The Source Path field is optional if an existing LVM volume group is used in the Target Path.
        For new LVM volume groups, input the location of a storage device in the Source Path field. This example uses a blank partition /dev/sdc.
      • The Build Pool check box instructs virt-manager to create a new LVM volume group. If you are using an existing volume group you should not select the Build Pool check box.
        This example is using a blank partition to create a new volume group so the Build Pool check box must be selected.
      Add target and source

      Figure 14.11. Add target and source

      Verify the details and press the Finish button format the LVM volume group and create the storage pool.
    3. Confirm the device to be formatted

      A warning message appears.
      Warning message

      Figure 14.12. Warning message

      Press the Yes button to proceed to erase all data on the storage device and create the storage pool.
  4. Verify the new storage pool

    The new storage pool will appear in the list on the left after a few seconds. Verify the details are what you expect, 465.76 GB Free in our example. Also verify the State field reports the new storage pool as Active.
    It is generally a good idea to have the Autostart check box enabled, to ensure the storage pool starts automatically with libvirtd.
    Confirm LVM storage pool details

    Figure 14.13. Confirm LVM storage pool details

    Close the Host Details dialog, as the task is now complete.

14.4.2. Deleting a Storage Pool Using virt-manager

This procedure demonstrates how to delete a storage pool.
  1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it. To do this, select the storage pool you want to stop and click .
    Stop Icon

    Figure 14.14. Stop Icon

  2. Delete the storage pool by clicking . This icon is only enabled if you stop the storage pool first.

14.4.3. Creating an LVM-based Storage Pool with virsh

This section outlines the steps required to create an LVM-based storage pool with the virsh command. It uses the example of a pool named guest_images_lvm from a single drive (/dev/sdc). This is only an example and your settings should be substituted as appropriate.

Procedure 14.3. Creating an LVM-based storage pool with virsh

  1. Define the pool name guest_images_lvm.
    # virsh pool-define-as guest_images_lvm logical - - /dev/sdc libvirt_lvm \ /dev/libvirt_lvm
    Pool guest_images_lvm defined
    
  2. Build the pool according to the specified name. If you are using an already existing volume group, skip this step.
    # virsh pool-build guest_images_lvm
    
    Pool guest_images_lvm built
    
  3. Initialize the new pool.
    # virsh pool-start guest_images_lvm
    
    Pool guest_images_lvm started
    
  4. Show the volume group information with the vgs command.
    # vgs
    VG          #PV #LV #SN Attr   VSize   VFree
    libvirt_lvm   1   0   0 wz--n- 465.76g 465.76g
    
  5. Set the pool to start automatically.
    # virsh pool-autostart guest_images_lvm
    Pool guest_images_lvm marked as autostarted
    
  6. List the available pools with the virsh command.
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    guest_images_lvm     active     yes
    
  7. The following commands demonstrate the creation of three volumes (volume1, volume2 and volume3) within this pool.
    # virsh vol-create-as guest_images_lvm volume1 8G
    Vol volume1 created
    
    # virsh vol-create-as guest_images_lvm volume2 8G
    Vol volume2 created
    
    # virsh vol-create-as guest_images_lvm volume3 8G
    Vol volume3 created
    
  8. List the available volumes in this pool with the virsh command.
    # virsh vol-list guest_images_lvm
    Name                 Path
    -----------------------------------------
    volume1              /dev/libvirt_lvm/volume1
    volume2              /dev/libvirt_lvm/volume2
    volume3              /dev/libvirt_lvm/volume3
    
  9. The following two commands (lvscan and lvs) display further information about the newly created volumes.
    # lvscan
    ACTIVE            '/dev/libvirt_lvm/volume1' [8.00 GiB] inherit
    ACTIVE            '/dev/libvirt_lvm/volume2' [8.00 GiB] inherit
    ACTIVE            '/dev/libvirt_lvm/volume3' [8.00 GiB] inherit
    
    # lvs
    LV       VG            Attr     LSize   Pool Origin Data%  Move Log Copy%  Convert
    volume1  libvirt_lvm   -wi-a-   8.00g
    volume2  libvirt_lvm   -wi-a-   8.00g
    volume3  libvirt_lvm   -wi-a-   8.00g
    

14.4.4. Deleting a Storage Pool Using virsh

The following demonstrates how to delete a storage pool using virsh:
  1. To avoid any issues with other guests using the same pool, it is best to stop the storage pool and release any resources in use by it.
    # virsh pool-destroy guest_images_disk
  2. Optionally, if you want to remove the directory where the storage pool resides use the following command:
    # virsh pool-delete guest_images_disk
  3. Remove the storage pool's definition
    # virsh pool-undefine guest_images_disk

14.5. iSCSI-based Storage Pools

This section covers using iSCSI-based devices to store guest virtual machines. This allows for more flexible storage options such as using iSCSI as a block storage device. The iSCSI devices use an LIO target, which is a multi-protocol SCSI target for Linux. In addition to iSCSI, LIO also supports Fibre Channel and Fibre Channel over Ethernet (FCoE).
iSCSI (Internet Small Computer System Interface) is a network protocol for sharing storage devices. iSCSI connects initiators (storage clients) to targets (storage servers) using SCSI instructions over the IP layer.

14.5.1. Configuring a Software iSCSI Target

Introduced in Red Hat Enterprise Linux 7, iSCSI targets are created with the targetcli package, which provides a command set for creating software-backed iSCSI targets.

Procedure 14.4. Creating an iSCSI target

  1. Install the required package

    Install the targetcli package and all dependencies:
    # yum install targetcli
  2. Launch targetcli

    Launch the targetcli command set:
    # targetcli
  3. Create storage objects

    Create three storage objects as follows, using the device created in Section 14.4, “LVM-based Storage Pools”:
    1. Create a block storage object, by changing into the /backstores/block directory and running the following command:
      # create [block-name][filepath]
      For example:
       # create block1 dev=/dev/vdb1
    2. Create a fileio object, by changing into the fileio directory and running the following command:
      # create [fileioname] [imagename] [image-size]
      For example:
      # create fileio1 /foo.img 50M
    3. Create a ramdisk object by changing into the ramdisk directory, and running the following command:
      # create [ramdiskname] [size]
      For example:
      # create ramdisk1 1M
    4. Remember the names of the disks you created in this step, as you will need them later.
  4. Navigate to the /iscsi directory

    Change into the iscsi directory:
    #cd /iscsi
  5. Create iSCSI target

    Create an iSCSI target in one of two ways:
    1. create with no additional parameters, automatically generates the IQN.
    2. create iqn.2010-05.com.example.server1:iscsirhel7guest creates a specific IQN on a specific server.
  6. Define the target portal group (TPG)

    Each iSCSI target needs to have a target portal group (TPG) defined. In this example, the default tpg1 will be used, but you can add additional tpgs as well. As this is the most common configuration, the example will configure tpg1. To do this, make sure you are still in the /iscsi directory and change to the /tpg1 directory.
    # /iscsi>iqn.iqn.2010-05.com.example.server1:iscsirhel7guest/tpg1
  7. Define the portal IP address

    In order to export the block storage over iSCSI, the portals, LUNs, and ACLs must all be configured first.
    The portal includes the IP address and TCP port that the target will listen on, and the initiators will connect to. iSCSI uses port 3260, which is the port that will be configured by default. To connect to this port, enter the following command from the /tpg directory:
    # portals/ create
    This command will have all available IP addresses listening to this port. To specify that only one specific IP address will listen on the port, run portals/ create [ipaddress], and the specified IP address will be configured to listen to port 3260.
  8. Configure the LUNs and assign the storage objects to the fabric

    This step uses the storage devices created in Procedure 14.4, “Creating an iSCSI target”. Make sure you change into the luns directory for the TPG you created in step 6, or iscsi>iqn.iqn.2010-05.com.example.server1:iscsirhel7guest, for example.
    1. Assign the first LUN to the ramdisk as follows:
      # create /backstores/ramdisk/ramdisk1
    2. Assign the second LUN to the block disk as follows:
      # create /backstores/block/block1
    3. Assign the third LUN to the fileio disk as follows:
      # create /backstores/fileio/file1
    4. Listing the resulting LUNs should resemble this screen output:
      /iscsi/iqn.20...csirhel7guest/tpg1 ls
      
      o- tgp1 ............................................................................[enabled, auth]
         o- acls..................................................................................[0 ACL]
         o- luns.................................................................................[3 LUNs]
         | o- lun0.....................................................................[ramdisk/ramdisk1]
         | o- lun1..............................................................[block/block1 (dev/vdb1)]
         | o- lun2...............................................................[fileio/file1 (foo.img)]
         o- portals............................................................................[1 Portal]
           o- IP-ADDRESS:3260........................................................................[OK]
      
  9. Creating ACLs for each initiator

    This step allows for the creation of authentication when the initiator connects, and it also allows for restriction of specified LUNs to specified initiators. Both targets and initiators have unique names. iSCSI initiators use an IQN.
    1. To find the IQN of the iSCSI initiator, enter the following command, replacing the name of the initiator:
      # cat /etc/iscsi/initiatorname.iscsi
      Use this IQN to create the ACLs.
    2. Change to the acls directory.
    3. Run the command create [iqn], or to create specific ACLs, refer to the following example:
      # create iqn.2010-05.com.example.foo:888
      Alternatively, to configure the kernel target to use a single user ID and password for all initiators, and allow all initiators to log in with that user ID and password, use the following commands (replacing userid and password):
      # set auth userid=redhat
      # set auth password=password123
      # set attribute authentication=1
      # set attribute generate_node_acls=1
      
  10. Make the configuration persistent with the saveconfig command. This will overwrite the previous boot settings. Alternatively, running exit from the targetcli saves the target configuration by default.
  11. Enable the service with systemctl enable target.service to apply the saved settings on next boot.

Procedure 14.5. Optional steps

  1. Create LVM volumes

    LVM volumes are useful for iSCSI backing images. LVM snapshots and re-sizing can be beneficial for guest virtual machines. This example creates an LVM image named virtimage1 on a new volume group named virtstore on a RAID5 array for hosting guest virtual machines with iSCSI.
    1. Create the RAID array

      Creating software RAID5 arrays is covered by the Red Hat Enterprise Linux 7 Storage Administration Guide.
    2. Create the LVM volume group

      Create a logical volume group named virtstore with the vgcreate command.
      # vgcreate virtstore /dev/md1
    3. Create a LVM logical volume

      Create a logical volume named virtimage1 on the virtstore volume group with a size of 20GB using the lvcreate command.
      # lvcreate **size 20G -n virtimage1 virtstore
      The new logical volume, virtimage1, is ready to use for iSCSI.

      Important

      Using LVM volumes for kernel target backstores can cause issues if the initiator also partitions the exported volume with LVM. This can be solved by adding global_filter = ["r|^/dev/vg0|"] to /etc/lvm/lvm.conf
  2. Optional: Test discovery

    Test whether the new iSCSI device is discoverable.
    # iscsiadm --mode discovery --type sendtargets --portal server1.example.com
    127.0.0.1:3260,1 iqn.2010-05.com.example.server1:iscsirhel7guest
  3. Optional: Test attaching the device

    Attach the new device (iqn.2010-05.com.example.server1:iscsirhel7guest) to determine whether the device can be attached.
    # iscsiadm -d2 -m node --login
    scsiadm: Max file limits 1024 1024
    
    Logging in to [iface: default, target: iqn.2010-05.com.example.server1:iscsirhel7guest, portal: 10.0.0.1,3260]
    Login to [iface: default, target: iqn.2010-05.com.example.server1:iscsirhel7guest, portal: 10.0.0.1,3260] successful.
    
  4. Detach the device.
    # iscsiadm -d2 -m node --logout
    scsiadm: Max file limits 1024 1024
    
    Logging out of session [sid: 2, target: iqn.2010-05.com.example.server1:iscsirhel7guest, portal: 10.0.0.1,3260
    Logout of [sid: 2, target: iqn.2010-05.com.example.server1:iscsirhel7guest, portal: 10.0.0.1,3260] successful.
An iSCSI device is now ready to use for virtualization.

14.5.2. Creating an iSCSI Storage Pool in virt-manager

This procedure covers creating a storage pool with an iSCSI target in virt-manager.

Procedure 14.6. Adding an iSCSI device to virt-manager

  1. Open the host machine's storage details

    1. In virt-manager, click the Edit and select Connection Details from the drop-down menu.
    2. Click on the Storage tab.
      Storage menu

      Figure 14.15. Storage menu

  2. Add a new pool (Step 1 of 2)

    Press the + button (the add pool button). The Add a New Storage Pool wizard appears.
    Add an iSCSI storage pool name and type

    Figure 14.16. Add an iSCSI storage pool name and type

    Choose a name for the storage pool, change the Type to iSCSI, and press Forward to continue.
  3. Add a new pool (Step 2 of 2)

    You will need the information you used in Section 14.5, “iSCSI-based Storage Pools” to complete the fields in this menu.
    1. Enter the iSCSI source and target. The Format option is not available as formatting is handled by the guest virtual machines. It is not advised to edit the Target Path. The default target path value, /dev/disk/by-path/, adds the drive path to that directory. The target path should be the same on all host physical machines for migration.
    2. Enter the host name or IP address of the iSCSI target. This example uses host1.example.com.
    3. In the Source IQN field, enter the iSCSI target IQN. If you look in Section 14.5, “iSCSI-based Storage Pools”, this is the information you added in the /etc/target/targets.conf file. This example uses iqn.2010-05.com.test_example.server1:iscsirhel7guest.
    4. (Optional) Check the Initiator IQN check box to enter the IQN for the initiator. This example uses iqn.2010-05.com.example.host1:iscsirhel7.
    5. Click Finish to create the new storage pool.
    Create an iSCSI storage pool

    Figure 14.17. Create an iSCSI storage pool

14.5.3. Deleting a Storage Pool Using virt-manager

This procedure demonstrates how to delete a storage pool.
  1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it. To do this, select the storage pool you want to stop and click .
    Deleting a storage pool

    Figure 14.18. Deleting a storage pool

  2. Delete the storage pool by clicking . This icon is only enabled if you stop the storage pool first.

14.5.4. Creating an iSCSI-based Storage Pool with virsh

  1. Optional: Secure the storage pool

    If desired, set up authentication with the steps in Section 14.5.5, “Securing an iSCSI Storage Pool”.
  2. Define the storage pool

    Storage pool definitions can be created with the virsh command-line tool. Creating storage pools with virsh is useful for system administrators using scripts to create multiple storage pools.
    The virsh pool-define-as command has several parameters which are accepted in the following format:
    virsh pool-define-as name type source-host source-path source-dev source-name target
    The parameters are explained as follows:
    type
    defines this pool as a particular type, iSCSI for example
    name
    sets the name for the storage pool; must be unique
    source-host and source-path
    the host name and iSCSI IQN, respectively
    source-dev and source-name
    these parameters are not required for iSCSI-based pools; use a - character to leave the field blank.
    target
    defines the location for mounting the iSCSI device on the host machine
    The example below creates the same iSCSI-based storage pool as the virsh pool-define-as example above:
    # virsh pool-define-as --name iscsirhel7pool --type iscsi \
         --source-host server1.example.com \
         --source-dev iqn.2010-05.com.example.server1:iscsirhel7guest \
         --target /dev/disk/by-path
    Pool iscsirhel7pool defined
  3. Verify the storage pool is listed

    Verify the storage pool object is created correctly and the state is inactive.
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    iscsirhel7pool       inactive   no
  4. Optional: Establish a direct connection to the iSCSI storage pool

    This step is optional, but it allows you to establish a direct connection to the iSCSI storage pool. By default this is enabled, but if the connection is to the host machine (and not direct to the network) you can change it back by editing the domain XML for the virtual machine to reflect this example:
       ...
       <disk type='volume' device='disk'>
          <driver name='qemu'/>
          <source pool='iscsi' volume='unit:0:0:1' mode='direct'/>
          <target dev='vda' bus='virtio'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
       </disk>
          ... 
    
    

    Figure 14.19. Disk type element XML example

    Note

    The same iSCSI storage pool can be used for a LUN or a disk, by specifying the disk device as either a disk or lun. For XML configuration examples of adding SCSI LUN-based storage to a guest, see Section 15.5.3, “Adding SCSI LUN-based Storage to a Guest”.
    Additionally, the source mode can be specified as mode='host' for a connection to the host machine.
    If you have configured authentication on the iSCSI server as detailed in Procedure 14.4, “Creating an iSCSI target”, then the following XML used as a <disk> sub-element will provide the authentication credentials for the disk. Section 14.5.5, “Securing an iSCSI Storage Pool” describes how to configure the libvirt secret.
             <auth type='chap' username='redhat'>
                <secret usage='iscsirhel7secret'/>
             </auth>
    
  5. Start the storage pool

    Use the virsh pool-start to enable a directory storage pool. This allows the storage pool to be used for volumes and guest virtual machines.
    # virsh pool-start iscsirhel7pool
    Pool iscsirhel7pool started
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    iscsirhel7pool       active     no
    
  6. Turn on autostart

    Turn on autostart for the storage pool. Autostart configures the libvirtd service to start the storage pool when the service starts.
    # virsh pool-autostart iscsirhel7pool
    Pool iscsirhel7pool marked as autostarted
    Verify that the iscsirhel7pool pool has autostart enabled:
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    default              active     yes
    iscsirhel7pool       active     yes
    
  7. Verify the storage pool configuration

    Verify the storage pool was created correctly, the sizes report correctly, and the state reports as running.
    # virsh pool-info iscsirhel7pool
    Name:           iscsirhel7pool
    UUID:           afcc5367-6770-e151-bcb3-847bc36c5e28
    State:          running
    Persistent:     unknown
    Autostart:      yes
    Capacity:       100.31 GB
    Allocation:     0.00
    Available:      100.31 GB
    
An iSCSI-based storage pool called iscsirhel7pool is now available.

14.5.5. Securing an iSCSI Storage Pool

User name and password parameters can be configured with virsh to secure an iSCSI storage pool. This can be configured before or after the pool is defined, but the pool must be started for the authentication settings to take effect.

Procedure 14.7. Configuring authentication for a storage pool with virsh

  1. Create a libvirt secret file

    Create a libvirt secret XML file called secret.xml, using the following example:
    # cat secret.xml
    <secret ephemeral='no' private='yes'>
        <description>Passphrase for the iSCSI example.com server</description>
        <auth type='chap' username='redhat'/>
        <usage type='iscsi'>
            <target>iscsirhel7secret</target>
        </usage>
    </secret>
    
  2. Define the secret file

    Define the secret.xml file with virsh:
    # virsh secret-define secret.xml
  3. Verify the secret file's UUID

    Verify the UUID in secret.xml:
    # virsh secret-list
    
     UUID                                  Usage
    --------------------------------------------------------------------------------
     2d7891af-20be-4e5e-af83-190e8a922360  iscsi iscsirhel7secret
  4. Assign a secret to the UUID

    Assign a secret to that UUID, using the following command syntax as an example:
    # MYSECRET=`printf %s "password123" | base64`
    # virsh secret-set-value 2d7891af-20be-4e5e-af83-190e8a922360 $MYSECRET
    This ensures the CHAP username and password are set in a libvirt-controlled secret list.
  5. Add an authentication entry to the storage pool

    Modify the <source> entry in the storage pool's XML file using virsh edit and add an <auth> element, specifying authentication type, username, and secret usage.
    The following shows an example of a storage pool XML definition with authentication configured:
    # cat iscsirhel7pool.xml
      <pool type='iscsi'>
        <name>iscsirhel7pool</name>
          <source>
             <host name='192.168.122.1'/>
             <device path='iqn.2010-05.com.example.server1:iscsirhel7guest'/>
             <auth type='chap' username='redhat'>
                <secret usage='iscsirhel7secret'/>
             </auth>
          </source>
        <target>
          <path>/dev/disk/by-path</path>
        </target>
      </pool>
    

    Note

    The <auth> sub-element exists in different locations within the guest XML's <pool> and <disk> elements. For a <pool>, <auth> is specified within the <source> element, as this describes where to find the pool sources, since authentication is a property of some pool sources (iSCSI and RBD). For a <disk>, which is a sub-element of a domain, the authentication to the iSCSI or RBD disk is a property of the disk. For an example of <disk> configured in the guest XML, see Section 14.5.4, “Creating an iSCSI-based Storage Pool with virsh”.
  6. Activate the changes in the storage pool

    The storage pool must be started to activate these changes.
    If the storage pool has not yet been started, follow the steps in Section 14.5.4, “Creating an iSCSI-based Storage Pool with virsh” to define and start the storage pool.
    If the pool has already been started, enter the following commands to stop and restart the storage pool:
    # virsh pool-destroy iscsirhel7pool
    # virsh pool-start iscsirhel7pool

14.5.6. Deleting a Storage Pool Using virsh

The following demonstrates how to delete a storage pool using virsh:
  1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it.
    # virsh pool-destroy iscsirhel7pool
  2. Remove the storage pool's definition
    # virsh pool-undefine iscsirhel7pool

14.6. NFS-based Storage Pools

This procedure covers creating a storage pool with an NFS mount point in virt-manager.

14.6.1. Creating an NFS-based Storage Pool with virt-manager

  1. Open the host physical machine's storage tab

    Open the Storage tab in the Host Details window.
    1. Open virt-manager.
    2. Select a host physical machine from the main virt-manager window. Click Edit menu and select Connection Details.
    3. Click on the Storage tab.
      Storage tab

      Figure 14.20. Storage tab

  2. Create a new pool (part 1)

    Press the + button (the add pool button). The Add a New Storage Pool wizard appears.
    Add an NFS name and type

    Figure 14.21. Add an NFS name and type

    Choose a name for the storage pool and press Forward to continue.
  3. Create a new pool (part 2)

    Enter the target path for the device, the host name and the NFS share path. Set the Format option to NFS or auto (to detect the type). The target path must be identical on all host physical machines for migration.
    Enter the host name or IP address of the NFS server. This example uses server1.example.com.
    Enter the NFS path. This example uses /nfstrial.
    Create an NFS storage pool

    Figure 14.22. Create an NFS storage pool

    Press Finish to create the new storage pool.

14.6.2. Deleting a Storage Pool Using virt-manager

This procedure demonstrates how to delete a storage pool.
  1. To avoid any issues with other guests using the same pool, it is best to stop the storage pool and release any resources in use by it. To do this, select the storage pool you want to stop and click .
    Stop Icon

    Figure 14.23. Stop Icon

  2. Delete the storage pool by clicking the Trash can icon. This icon is only enabled if you stop the storage pool first.

14.7. Using an NPIV Virtual Adapter (vHBA) with SCSI Devices

NPIV (N_Port ID Virtualization) is a software technology that allows sharing of a single physical Fibre Channel host bus adapter (HBA).
This allows multiple guests to see the same storage from multiple physical hosts, and thus allows for easier migration paths for the storage. As a result, there is no need for the migration to create or copy storage, as long as the correct storage path is specified.
In virtualization, the virtual host bus adapter, or vHBA, controls the LUNs for virtual machines. For a host to share one Fibre Channel device path between multiple KVM guests, a vHBA must be created for each virtual machine. A single vHBA must not be used by multiple KVM guests.
Each vHBA is identified by its own WWNN (World Wide Node Name) and WWPN (World Wide Port Name). The path to the storage is determined by the WWNN and WWPN values.
This section provides instructions for configuring a vHBA persistently on a virtual machine.

Note

Before creating a vHBA, it is recommended to configure storage array (SAN)-side zoning in the host LUN to provide isolation between guests and prevent the possibility of data corruption.

14.7.1. Creating a vHBA

Procedure 14.8. Creating a vHBA

  1. Locate HBAs on the host system

    To locate the HBAs on your host system, use the virsh nodedev-list --cap vports command.
    For example, the following output shows a host that has two HBAs that support vHBA:
    # virsh nodedev-list --cap vports
    scsi_host3
    scsi_host4
  2. Check the HBA's details

    Use the virsh nodedev-dumpxml HBA_device command to see the HBA's details.
    The XML output from the virsh nodedev-dumpxml command will list the fields <name>, <wwnn>, and <wwpn>, which are used to create a vHBA. The <max_vports> value shows the maximum number of supported vHBAs.
     # virsh nodedev-dumpxml scsi_host3
    <device>
      <name>scsi_host3</name>
      <path>/sys/devices/pci0000:00/0000:00:04.0/0000:10:00.0/host3</path>
      <parent>pci_0000_10_00_0</parent>
      <capability type='scsi_host'>
        <host>3</host>
        <unique_id>0</unique_id>
        <capability type='fc_host'>
          <wwnn>20000000c9848140</wwnn>
          <wwpn>10000000c9848140</wwpn>
          <fabric_wwn>2002000573de9a81</fabric_wwn>
        </capability>
        <capability type='vport_ops'>
          <max_vports>127</max_vports>
          <vports>0</vports>
        </capability>
      </capability>
    </device>   
    In this example, the <max_vports> value shows there are a total 127 virtual ports available for use in the HBA configuration. The <vports> value shows the number of virtual ports currently being used. These values update after creating a vHBA.
  3. Create a vHBA host device

    Create an XML file similar to the following (in this example, named vhba_host3.xml) for the vHBA host.
    # cat vhba_host3.xml
       <device>
         <parent>scsi_host3</parent>
         <capability type='scsi_host'>
           <capability type='fc_host'>
           </capability>
         </capability>
       </device>   
    The <parent> field specifies the HBA device to associate with this vHBA device. The details in the <device> tag are used in the next step to create a new vHBA device for the host. For more information on the nodedev XML format, see http://libvirt.org/formatnode.html.
  4. Create a new vHBA on the vHBA host device

    To create a vHBA on vhba_host3, use the virsh nodedev-create command:
    # virsh nodedev-create vhba_host3.xml
    Node device scsi_host5 created from vhba_host3.xml
  5. Verify the vHBA

    Verify the new vHBA's details (scsi_host5) with the virsh nodedev-dumpxml command:
    # virsh nodedev-dumpxml scsi_host5
    <device>
      <name>scsi_host5</name>
      <path>/sys/devices/pci0000:00/0000:00:04.0/0000:10:00.0/host3/vport-3:0-0/host5</path>
      <parent>scsi_host3</parent>
      <capability type='scsi_host'>
        <host>5</host>
        <unique_id>2</unique_id>
        <capability type='fc_host'>
          <wwnn>5001a4a93526d0a1</wwnn>
          <wwpn>5001a4ace3ee047d</wwpn>
          <fabric_wwn>2002000573de9a81</fabric_wwn>
        </capability>
      </capability>
    </device>  

14.7.2. Creating a Storage Pool Using the vHBA

It is recommended to define a libvirt storage pool based on the vHBA in order to preserve the vHBA configuration.
Using a storage pool has two primary advantages:
  • the libvirt code can easily find the LUN's path via virsh command output, and
  • virtual machine migration requires only defining and starting a storage pool with the same vHBA name on the target machine. To do this, the vHBA LUN, libvirt storage pool and volume name must be specified in the virtual machine's XML configuration. Refer to Section 14.7.3, “Configuring the Virtual Machine to Use a vHBA LUN” for an example.
  1. Create a SCSI storage pool

    To create a persistent vHBA configuration, first create a libvirt 'scsi' storage pool XML file using the format below. When creating a single vHBA that uses a storage pool on the same physical HBA, it is recommended to use a stable location for the <path> value, such as one of the /dev/disk/by-{path|id|uuid|label} locations on your system.
    When creating multiple vHBAs that use storage pools on the same physical HBA, the value of the <path> field must be only /dev/, otherwise storage pool volumes are visible only to one of the vHBAs, and devices from the host cannot be exposed to multiple guests with the NPIV configuration.
    More information on <path> and the elements within <target> can be found at http://libvirt.org/formatstorage.html.

    Example 14.3. Sample SCSI storage pool SML syntax

    In the following example, the 'scsi' storage pool is named vhbapool_host3.xml:
      <pool type='scsi'>
          <name>vhbapool_host3</name>
          <source>
            <adapter type='fc_host' wwnn='5001a4a93526d0a1' wwpn='5001a4ace3ee047d'/>
          </source>
          <target>
            <path>/dev/disk/by-path</path>
            <permissions>
              <mode>0700</mode>
              <owner>0</owner>
              <group>0</group>
            </permissions>
          </target>
        </pool> 
    Alternatively, the following syntax is used when vhbapool_host3.xml is one of more vHBA storage pools on a single HBA:
      <pool type='scsi'>
          <name>vhbapool_host3</name>
          <source>
            <adapter type='fc_host' wwnn='5001a4a93526d0a1' wwpn='5001a4ace3ee047d'/>
          </source>
          <target>
            <path>/dev/</path>
            <permissions>
              <mode>0700</mode>
              <owner>0</owner>
              <group>0</group>
            </permissions>
          </target>
        </pool> 

    Important

    In both cases, the pool must be type='scsi' and the source adapter type must be 'fc_host'. For a persistent configuration across host reboots, the wwnn and wwpn attributes must be the values assigned to the vHBA (scsi_host5 in this example) by libvirt.
    Optionally, the 'parent' attribute can be used in the <adapter> field to identify the parent scsi_host device as the vHBA. Note, the value is not the scsi_host of the vHBA created by virsh nodedev-create, but it is the parent of that vHBA.
    Providing the 'parent' attribute is also useful for duplicate pool definition checks. This is more important in environments where both the 'fc_host' and 'scsi_host' source adapter pools are being used, to ensure a new definition does not duplicate using the same scsi_host of another existing storage pool.
    The following example shows the optional 'parent' attribute used in the <adapter> field in a storage pool configuration:
    <adapter type='fc_host' parent='scsi_host3' wwnn='5001a4a93526d0a1' wwpn='5001a4ace3ee047d'/>
  2. Define the pool

    To define the storage pool (named vhbapool_host3 in this example) persistently, use the virsh pool-define command:
    # virsh pool-define vhbapool_host3.xml
    Pool vhbapool_host3 defined from vhbapool_host3.xml
    
  3. Start the pool

    Start the storage pool with the following command:
    # virsh pool-start vhbapool_host3
    Pool vhbapool_host3 started
    

    Note

    When starting the pool, libvirt will check if the vHBA with same wwpn:wwnn already exists. If it does not yet exist, a new vHBA with the provided wwpn:wwnn will be created and the configuration will not be persistent. Correspondingly, when destroying the pool, libvirt will destroy the vHBA using the same wwpn:wwnn values as well.
  4. Enable autostart

    Finally, to ensure that subsequent host reboots will automatically define vHBAs for use in virtual machines, set the storage pool autostart feature (in this example, for a pool named vhbapool_host3):
    # virsh pool-autostart vhbapool_host3

14.7.3. Configuring the Virtual Machine to Use a vHBA LUN

After a storage pool is created for a vHBA, add the vHBA LUN to the virtual machine configuration by creating a disk volume on the virtual machine in the virtual machine's XML. Specify the storage pool and the volume in the <source> parameter, using the following as an example:
   <disk type='volume' device='disk'>
     <driver name='qemu' type='raw'/>
     <source pool='vhbapool_host3' volume='unit:0:4:0'/>
     <target dev='hda' bus='ide'/>
   </disk>    
To specify a lun device instead of a disk, refer to the following example:
    <disk type='volume' device='lun' sgio='unfiltered'>
      <driver name='qemu' type='raw'/>
      <source pool='vhbapool_host3' volume='unit:0:4:0' mode='host'/>
      <target dev='sda' bus='scsi'/>
      <shareable />
    </disk>
For XML configuration examples of adding SCSI LUN-based storage to a guest, see Section 15.5.3, “Adding SCSI LUN-based Storage to a Guest”.
Note that to ensure successful reconnection to a LUN in case of a hardware failure, it is recommended to edit the fast_io_fail_tmo and dev_loss_tmo options. For more information, see Reconnecting to an exposed LUN after a hardware failure.

14.7.4. Destroying the vHBA Storage Pool

A vHBA created by the storage pool can be destroyed by the virsh pool-destroy command:
# virsh pool-destroy vhbapool_host3
Note that executing the virsh pool-destroy command will also remove the vHBA that was created in Section 14.7.1, “Creating a vHBA”.
To verify the pool and vHBA have been destroyed, run:
# virsh nodedev-list --cap scsi_host
scsi_host5 will no longer appear in the list of results.

14.8. GlusterFS Storage Pools

This section covers enabling a GlusterFS based storage pool. Red Hat Enterprise Linux 6.5 includes native support for creating virtual machines with GlusterFS. GlusterFS is a user-space file system that uses FUSE. When enabled in a guest virtual machine it enables a KVM host physical machine to boot guest virtual machine images from one or more GlusterFS storage volumes, and to use images from a GlusterFS storage volume as data disks for guest virtual machines.

14.8.1. Creating a GlusterFS Storage Pool Using virsh

This section will demonstrate how to prepare a Gluster server and an active Gluster volume.

Procedure 14.9. Preparing a Gluster server and an active Gluster volume

  1. Obtain the IP address of the Gluster server by listing its status with the following command:
    # gluster volume status
    Status of volume: gluster-vol1
    Gluster process						Port	Online	Pid
    ------------------------------------------------------------------------------
    Brick 222.111.222.111:/gluster-vol1 			49155	Y	18634
    
    Task Status of Volume gluster-vol1
    ------------------------------------------------------------------------------
    There are no active volume tasks
    
  2. If you have not already done so, install glusterfs-fuse and enable virt_use_fusefs. Then prepare one host which will connect to the Gluster server by running the following commands:
    # setsebool virt_use_fusefs on
    # getsebool virt_use_fusefs
    virt_use_fusefs --> on
    
  3. Create a new XML file to configure a Gluster storage pool (named glusterfs-pool.xml in this example) specifying pool type as gluster, and add the following data:
    			
    <pool type='gluster'>
    	<name>glusterfs-pool</name>
    	<source>
    		<host name='111.222.111.222'/>
    		<dir path='/'/>
    		<name>gluster-vol1</name>
    	</source>
    </pool>
    			

    Figure 14.24. GlusterFS XML file contents

  4. Define and start the Gluster pool, using the following commands:
    # virsh pool-define glusterfs-pool.xml
    Pool gluster-pool defined from glusterfs-pool.xml
    
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    gluster-pool         inactive   no
    
    # virsh pool-start gluster-pool
    Pool gluster-pool started
    
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    gluster-pool         active     no
    
    # virsh vol-list gluster-pool
    Name                 Path
    -----------------------------------------
    qcow2.img            gluster://111.222.111.222/gluster-vol1/qcow2.img
    raw.img              gluster://111.222.111.222/gluster-vol1/raw.img
    

14.8.2. Deleting a GlusterFS Storage Pool Using virsh

This section details how to delete a storage pool using virsh.

Procedure 14.10. Deleting a GlusterFS storage pool

  1. Set the status of the storage pool to inactive, using the following command:
    
     # virsh pool-destroy gluster-pool
    Pool gluster-pool destroyed
    
  2. Confirm the pool is inactive, using the following command
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    gluster-pool         inactive   no
    
  3. Undefine the GlusterFS storage pool using the following command:
    # virsh pool-undefine gluster-pool
    Pool gluster-pool has been undefined
    
  4. Confirm the pool is undefined, using the following command:
    # virsh pool-list --all
    Name                 State      Autostart
    -----------------------------------------
    
    

Chapter 15.  Storage Volumes

15.1. Introduction

Storage pools are divided into storage volumes. Storage volumes are an abstraction of physical partitions, LVM logical volumes, file-based disk images and other storage types handled by libvirt. Storage volumes are presented to guest virtual machines as local storage devices regardless of the underlying hardware. Note the sections below do not contain all of the possible commands and arguments that virsh allows, refer to Section 23.34, “Storage Volume Commands” for more information.

15.1.1. Referencing Volumes

For more additional parameters and arguments, refer to Section 23.38, “Listing Volume Information”.
To reference a specific volume, three approaches are possible:
The name of the volume and the storage pool
A volume may be referred to by name, along with an identifier for the storage pool it belongs in. On the virsh command line, this takes the form --pool storage_pool volume_name.
For example, a volume named firstimage in the guest_images pool.
# virsh vol-info --pool guest_images firstimage
Name:           firstimage
Type:           block
Capacity:       20.00 GB
Allocation:     20.00 GB

virsh #
The full path to the storage on the host physical machine system
A volume may also be referred to by its full path on the file system. When using this approach, a pool identifier does not need to be included.
For example, a volume named secondimage.img, visible to the host physical machine system as /images/secondimage.img. The image can be referred to as /images/secondimage.img.
# virsh vol-info /images/secondimage.img
Name:           secondimage.img
Type:           file
Capacity:       20.00 GB
Allocation:     136.00 kB
The unique volume key
When a volume is first created in the virtualization system, a unique identifier is generated and assigned to it. The unique identifier is termed the volume key. The format of this volume key varies upon the storage used.
When used with block based storage such as LVM, the volume key may follow this format:
c3pKz4-qPVc-Xf7M-7WNM-WJc8-qSiz-mtvpGn
When used with file based storage, the volume key may instead be a copy of the full path to the volume storage.
/images/secondimage.img
For example, a volume with the volume key of Wlvnf7-a4a3-Tlje-lJDa-9eak-PZBv-LoZuUr:
# virsh vol-info Wlvnf7-a4a3-Tlje-lJDa-9eak-PZBv-LoZuUr
Name:           firstimage
Type:           block
Capacity:       20.00 GB
Allocation:     20.00 GB
virsh provides commands for converting between a volume name, volume path, or volume key:
vol-name
Returns the volume name when provided with a volume path or volume key.
# virsh vol-name /dev/guest_images/firstimage
firstimage
# virsh vol-name Wlvnf7-a4a3-Tlje-lJDa-9eak-PZBv-LoZuUr
vol-path
Returns the volume path when provided with a volume key, or a storage pool identifier and volume name.
# virsh vol-path Wlvnf7-a4a3-Tlje-lJDa-9eak-PZBv-LoZuUr
/dev/guest_images/firstimage
# virsh vol-path --pool guest_images firstimage
/dev/guest_images/firstimage
The vol-key command
Returns the volume key when provided with a volume path, or a storage pool identifier and volume name.
# virsh vol-key /dev/guest_images/firstimage
Wlvnf7-a4a3-Tlje-lJDa-9eak-PZBv-LoZuUr
# virsh vol-key --pool guest_images firstimage
Wlvnf7-a4a3-Tlje-lJDa-9eak-PZBv-LoZuUr
For more information, refer to Section 23.38, “Listing Volume Information”.

15.2. Creating Volumes

This section shows how to create disk volumes inside a block based storage pool. In the example below, the virsh vol-create-as command will create a storage volume with a specific size in GB within the guest_images_disk storage pool. As this command is repeated per volume needed, three volumes are created as shown in the example. For additional parameters and arguments refer to Section 23.34.1, “Creating Storage Volumes”
# virsh vol-create-as guest_images_disk volume1 8G
Vol volume1 created

# virsh vol-create-as guest_images_disk volume2 8G
Vol volume2 created

# virsh vol-create-as guest_images_disk volume3 8G
Vol volume3 created

# virsh vol-list guest_images_disk
Name                 Path
-----------------------------------------
volume1              /dev/sdb1
volume2              /dev/sdb2
volume3              /dev/sdb3

# parted -s /dev/sdb print
Model: ATA ST3500418AS (scsi)
Disk /dev/sdb: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
2      17.4kB  8590MB  8590MB               primary
3      8590MB  17.2GB  8590MB               primary
1      21.5GB  30.1GB  8590MB               primary

15.3. Cloning Volumes

The new volume will be allocated from storage in the same storage pool as the volume being cloned. The virsh vol-clone must have the --pool argument which dictates the name of the storage pool that contains the volume to be cloned. The rest of the command names the volume to be cloned (volume3) and the name of the new volume that was cloned (clone1). The virsh vol-list command lists the volumes that are present in the storage pool (guest_images_disk). For additional commands and arguments refer to Section 23.34.4, “Cloning a Storage Volume”
# virsh vol-clone --pool guest_images_disk volume3 clone1
Vol clone1 cloned from volume3

# virsh vol-list guest_images_disk
Name                 Path
-----------------------------------------
volume1              /dev/sdb1
volume2              /dev/sdb2
volume3              /dev/sdb3
clone1               /dev/sdb4


# parted -s /dev/sdb print
Model: ATA ST3500418AS (scsi)
Disk /dev/sdb: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    File system  Name     Flags
1      4211MB  12.8GB  8595MB  primary
2      12.8GB  21.4GB  8595MB  primary
3      21.4GB  30.0GB  8595MB  primary
4      30.0GB  38.6GB  8595MB  primary

15.4. Deleting and Removing Volumes

For the virsh commands you need to delete and remove a volume, refer to Section 23.35, “Deleting Storage Volumes”.

15.5. Adding Storage Devices to Guests

This section covers adding storage devices to a guest. Additional storage can only be added as needed. The following types of storage is discussed in this section:

15.5.1. Adding File-based Storage to a Guest

File-based storage is a collection of files that are stored on the host physical machines file system that act as virtualized hard drives for guests. To add file-based storage, perform the following steps:

Procedure 15.1. Adding file-based storage

  1. Create a storage file or use an existing file (such as an IMG file). Note that both of the following commands create a 4GB file which can be used as additional storage for a guest:
    • Pre-allocated files are recommended for file-based storage images. Create a pre-allocated file using the following dd command as shown:
      # dd if=/dev/zero of=/var/lib/libvirt/images/FileName.img bs=1G count=4
    • Alternatively, create a sparse file instead of a pre-allocated file. Sparse files are created much faster and can be used for testing, but are not recommended for production environments due to data integrity and performance issues.
      # dd if=/dev/zero of=/var/lib/libvirt/images/FileName.img bs=1G seek=4096 count=4
  2. Create the additional storage by writing a <disk> element in a new file. In this example, this file will be known as NewStorage.xml.
    A <disk> element describes the source of the disk, and a device name for the virtual block device. The device name should be unique across all devices in the guest, and identifies the bus on which the guest will find the virtual block device. The following example defines a virtio block device whose source is a file-based storage container named FileName.img:
    <disk type='file' device='disk'>
       <driver name='qemu' type='raw' cache='none'/>
       <source file='/var/lib/libvirt/images/FileName.img'/>
       <target dev='vdb'/>
    </disk>
    
    Device names can also start with "hd" or "sd", identifying respectively an IDE and a SCSI disk. The configuration file can also contain an <address> sub-element that specifies the position on the bus for the new device. In the case of virtio block devices, this should be a PCI address. Omitting the <address> sub-element lets libvirt locate and assign the next available PCI slot.
  3. Attach the CD-ROM as follows:
    <disk type='file' device='cdrom'>
       <driver name='qemu' type='raw' cache='none'/>
       <source file='/var/lib/libvirt/images/FileName.img'/>
       <readonly/>
       <target dev='hdc'/>
    </disk >
    
  4. Add the device defined in NewStorage.xml with your guest (Guest1):
    # virsh attach-device --config Guest1 ~/NewStorage.xml

    Note

    This change will only apply after the guest has been destroyed and restarted. In addition, persistent devices can only be added to a persistent domain, that is a domain whose configuration has been saved with virsh define command.
    If the guest is running, and you want the new device to be added temporarily until the guest is destroyed, omit the --config option:
    # virsh attach-device Guest1 ~/NewStorage.xml

    Note

    The virsh command allows for an attach-disk command that can set a limited number of parameters with a simpler syntax and without the need to create an XML file. The attach-disk command is used in a similar manner to the attach-device command mentioned previously, as shown:
    # virsh attach-disk Guest1 /var/lib/libvirt/images/FileName.img vdb --cache none
    
    Note that the virsh attach-disk command also accepts the --config option.
  5. Start the guest machine (if it is currently not running):
    # virsh start Guest1

    Note

    The following steps are Linux guest specific. Other operating systems handle new storage devices in different ways. For other systems, refer to that operating system's documentation.
  6. Partitioning the disk drive

    The guest now has a hard disk device called /dev/vdb. If required, partition this disk drive and format the partitions. If you do not see the device that you added, then it indicates that there is an issue with the disk hotplug in your guest's operating system.
    1. Start fdisk for the new device:
      # fdisk /dev/vdb
      Command (m for help):
      
    2. Type n for a new partition.
    3. The following appears:
      Command action
      e   extended
      p   primary partition (1-4)
      
      Type p for a primary partition.
    4. Choose an available partition number. In this example, the first partition is chosen by entering 1.
      Partition number (1-4): 1
    5. Enter the default first cylinder by pressing Enter.
      First cylinder (1-400, default 1):
    6. Select the size of the partition. In this example the entire disk is allocated by pressing Enter.
      Last cylinder or +size or +sizeM or +sizeK (2-400, default 400):
    7. Enter t to configure the partition type.
      Command (m for help): t
    8. Select the partition you created in the previous steps. In this example, the partition number is 1 as there was only one partition created and fdisk automatically selected partition 1.
      Partition number (1-4): 1
    9. Enter 83 for a Linux partition.
      Hex code (type L to list codes): 83
    10. Enter w to write changes and quit.
      Command (m for help): w
      
    11. Format the new partition with the ext3 file system.
      # mke2fs -j /dev/vdb1
  7. Create a mount directory, and mount the disk on the guest. In this example, the directory is located in myfiles.
    # mkdir /myfiles
    # mount /dev/vdb1 /myfiles
    
    The guest now has an additional virtualized file-based storage device. Note however, that this storage will not mount persistently across reboot unless defined in the guest's /etc/fstab file:
    /dev/vdb1    /myfiles    ext3     defaults    0 0

15.5.2. Adding Hard Drives and Other Block Devices to a Guest

System administrators have the option to use additional hard drives to provide increased storage space for a guest, or to separate system data from user data.

Procedure 15.2. Adding physical block devices to guests

  1. This procedure describes how to add a hard drive on the host physical machine to a guest. It applies to all physical block devices, including CD-ROM, DVD and floppy devices.
    Physically attach the hard disk device to the host physical machine. Configure the host physical machine if the drive is not accessible by default.
  2. Do one of the following:
    1. Create the additional storage by writing a disk element in a new file. In this example, this file will be known as NewStorage.xml. The following example is a configuration file section which contains an additional device-based storage container for the host physical machine partition /dev/sr0:
      <disk type='block' device='disk'>
            <driver name='qemu' type='raw' cache='none'/>
            <source dev='/dev/sr0'/>
            <target dev='vdc' bus='virtio'/>
      </disk>
      
    2. Follow the instruction in the previous section to attach the device to the guest virtual machine. Alternatively, you can use the virsh attach-disk command, as shown:
      # virsh attach-disk Guest1 /dev/sr0 vdc
      
      Note that the following options are available:
      • The virsh attach-disk command also accepts the --config, --type, and --mode options, as shown:
        virsh attach-disk Guest1 /dev/sr0 vdc --config --type cdrom --mode readonly
      • Additionally, --type also accepts --type disk in cases where the device is a hard drive.
  3. The guest virtual machine now has a new hard disk device called /dev/vdc on Linux (or something similar, depending on what the guest virtual machine OS chooses) . You can now initialize the disk from the guest virtual machine, following the standard procedures for the guest virtual machine's operating system. Refer to Procedure 15.1, “Adding file-based storage” for an example.

Warning

When adding block devices to a guest, make sure to follow the related security considerations.

15.5.3. Adding SCSI LUN-based Storage to a Guest

A host SCSI LUN device can be exposed entirely to the guest using three mechanisms, depending on your host configuration. Exposing the SCSI LUN device in this way allows for SCSI commands to be executed directly to the LUN on the guest. This is useful as a means to share a LUN between guests, as well as to share Fibre Channel storage between hosts.

Important

The optional sgio attribute controls whether unprivileged SCSI Generical I/O (SG_IO) commands are filtered for a device='lun' disk. The sgio attribute can be specified as 'filtered' or 'unfiltered', but must be set to 'unfiltered' to allow SG_IO ioctl commands to be passed through on the guest in a persistent reservation.
In addition to setting sgio='unfiltered', the <shareable> element must be set to share a LUN between guests. The sgio attribute defaults to 'filtered' if not specified.
The <disk> XML attribute device='lun' is valid for the following guest disk configurations:
  • type='block' for <source dev='/dev/disk/by-{path|id|uuid|label}'/>
       <disk type='block' device='lun' sgio='unfiltered'>
          ​ <driver name='qemu' type='raw'/>
          ​ <source dev='/dev/disk/by-path/pci-0000\:04\:00.1-fc-0x203400a0b85ad1d7-lun-0'/>
          ​ <target dev='sda' bus='scsi'/>
          <shareable />
       ​</disk>

    Note

    The backslashes prior to the colons in the <source> device name are required.
  • type='network' for <source protocol='iscsi'... />
        <disk type='network' device='disk' sgio='unfiltered'>
          <driver name='qemu' type='raw'/>
          <source protocol='iscsi' name='iqn.2013-07.com.example:iscsi-net-pool/1'>
            <host name='example.com' port='3260'/>
          </source>
          <auth username='myuser'>
            <secret type='iscsi' usage='libvirtiscsi'/>
          </auth>
          <target dev='sda' bus='scsi'/>
          <shareable />
        </disk>
  • type='volume' when using an iSCSI or a NPIV/vHBA source pool as the SCSI source pool.
    The following example XML shows a guest using an iSCSI source pool (named iscsi-net-pool) as the SCSI source pool:
        <disk type='volume' device='lun' sgio='unfiltered'>
          <driver name='qemu' type='raw'/>
          <source pool='iscsi-net-pool' volume='unit:0:0:1' mode='host'/>
          <target dev='sda' bus='scsi'/>
          <shareable />
        </disk>

    Note

    The mode= option within the <source> tag is optional, but if used, it must be set to 'host' and not 'direct'. When set to 'host', libvirt will find the path to the device on the local host. When set to 'direct', libvirt will generate the path to the device using the source pool's source host data.
    The iSCSI pool (iscsi-net-pool) in the example above will have a similar configuration to the following:
    # virsh pool-dumpxml iscsi-net-pool
    <pool type='iscsi'>
      <name>iscsi-net-pool</name>
      <capacity unit='bytes'>11274289152</capacity>
      <allocation unit='bytes'>11274289152</allocation>
      <available unit='bytes'>0</available>
      <source>
        <host name='192.168.122.1' port='3260'/>
        <device path='iqn.2013-12.com.example:iscsi-chap-netpool'/>
        <auth type='chap' username='redhat'>
          <secret usage='libvirtiscsi'/>
        </auth>
      </source>
      <target>
        <path>/dev/disk/by-path</path>
        <permissions>
          <mode>0755</mode>
        </permissions>
      </target>
    </pool>
    To verify the details of the available LUNs in the iSCSI source pool, enter the following command:
    # virsh vol-list iscsi-net-pool
     Name                 Path
    ------------------------------------------------------------------------------
     unit:0:0:1           /dev/disk/by-path/ip-192.168.122.1:3260-iscsi-iqn.2013-12.com.example:iscsi-chap-netpool-lun-1
     unit:0:0:2           /dev/disk/by-path/ip-192.168.122.1:3260-iscsi-iqn.2013-12.com.example:iscsi-chap-netpool-lun-2
  • type='volume' when using a NPIV/vHBA source pool as the SCSI source pool.
    The following example XML shows a guest using a NPIV/vHBA source pool (named vhbapool_host3) as the SCSI source pool:
    <disk type='volume' device='lun' sgio='unfiltered'>
      <driver name='qemu' type='raw'/>
      <source pool='vhbapool_host3' volume='unit:0:1:0'/>
      <target dev='sda' bus='scsi'/>
      <shareable />
    </disk>    
    The NPIV/vHBA pool (vhbapool_host3) in the example above will have a similar configuration to:
    # virsh pool-dumpxml vhbapool_host3
    <pool type='scsi'>
      <name>vhbapool_host3</name>
      <capacity unit='bytes'>0</capacity>
      <allocation unit='bytes'>0</allocation>
      <available unit='bytes'>0</available>
      <source>
        <adapter type='fc_host' parent='scsi_host3' managed='yes' wwnn='5001a4a93526d0a1' wwpn='5001a4ace3ee045d'/>
      </source>
      <target>
        <path>/dev/disk/by-path</path>
        <permissions>
          <mode>0700</mode>
          <owner>0</owner>
          <group>0</group>
        </permissions>
      </target>
    </pool>
    To verify the details of the available LUNs on the vHBA, enter the following command:
    # virsh vol-list vhbapool_host3
     Name                 Path
    ------------------------------------------------------------------------------
     unit:0:0:0           /dev/disk/by-path/pci-0000:10:00.0-fc-0x5006016044602198-lun-0
     unit:0:1:0           /dev/disk/by-path/pci-0000:10:00.0-fc-0x5006016844602198-lun-0
    For more information on using a NPIV vHBA with SCSI devices, see Section 14.7.3, “Configuring the Virtual Machine to Use a vHBA LUN”.
The following procedure shows an example of adding a SCSI LUN-based storage device to a guest. Any of the above <disk device='lun'> guest disk configurations can be attached with this method. Substitute configurations according to your environment.

Procedure 15.3. Attaching SCSI LUN-based storage to a guest

  1. Create the device file by writing a <disk> element in a new file, and save this file with an XML extension (in this example, sda.xml):
    # cat sda.xml
    <disk type='volume' device='lun' sgio='unfiltered'>
      <driver name='qemu' type='raw'/>
      <source pool='vhbapool_host3' volume='unit:0:1:0'/>
      <target dev='sda' bus='scsi'/>
      <shareable />
    </disk>
  2. Associate the device created in sda.xml with your guest virtual machine (Guest1, for example):
    # virsh attach-device --config Guest1 ~/sda.xml

    Note

    Running the virsh attach-device command with the --config option requires a guest reboot to add the device permanently to the guest. Alternatively, the --persistent option can be used instead of --config, which can also be used to hotplug the device to a guest.
Alternatively, the SCSI LUN-based storage can be attached or configured on the guest using virt-manager. To configure this using virt-manager, click the Add Hardware button and add a virtual disk with the desired parameters, or change the settings of an existing SCSI LUN device from this window. In Red Hat Enterprise Linux 7.2 and above, the SGIO value can also be configured in virt-manager:
Configuring SCSI LUN storage with virt-manager

Figure 15.1. Configuring SCSI LUN storage with virt-manager

Reconnecting to an exposed LUN after a hardware failure

If the connection to an exposed Fiber Channel (FC) LUN is lost due to a failure of hardware (such as the host bus adapter), the exposed LUNs on the guest may continue to appear as failed even after the hardware failure is fixed. To prevent this, edit the dev_loss_tmo and fast_io_fail_tmo kernel options:
  • dev_loss_tmo controls how long the SCSI layer waits after a SCSI device fails before marking it as failed. To prevent a timeout, it is recommended to set the option to the maximum value, which is 2147483647.
  • fast_io_fail_tmo controls how long the SCSI layer waits after a SCSI device fails before failing back to the I/O. To ensure that dev_loss_tmo is not ignored by the kernel, set this option's value to any number lower than the value of dev_loss_tmo.
To modify the value of dev_loss_tmo and fast_io_fail, do one of the following:
  • Edit the the /etc/multipath.conf file, and set the values in the defaults section:
    defaults {
    ...
    fast_io_fail_tmo     20
    dev_loss_tmo    infinity
    }
    
  • Set dev_loss_tmo and fast_io_fail on the level of the FC host or remote port, for example as follows:
    # echo 20 > /sys/devices/pci0000:00/0000:00:06.0/0000:13:00.0/host1/rport-1:0-0/fc_remote_ports/rport-1:0-0/fast_io_fail_tmo
    # echo 2147483647 > /sys/devices/pci0000:00/0000:00:06.0/0000:13:00.0/host1/rport-1:0-0/fc_remote_ports/rport-1:0-0/dev_loss_tmo
To verify that the new values of dev_loss_tmo and fast_io_fail are active, use the following command:
# find /sys -name dev_loss_tmo -print -exec cat {} \;
If the parameters have been set correctly, the output will look similar to this, with the appropriate device or devices instead of pci0000:00/0000:00:06.0/0000:13:00.0/host1/rport-1:0-0/fc_remote_ports/rport-1:0-0:
# find /sys -name dev_loss_tmo -print -exec cat {} \;
/sys/devices/pci0000:00/0000:00:06.0/0000:13:00.0/host1/rport-1:0-0/fc_remote_ports/rport-1:0-0/dev_loss_tmo
2147483647
...

15.5.4. Managing Storage Controllers in a Guest Virtual Machine

Unlike virtio disks, SCSI devices require the presence of a controller in the guest virtual machine. This section details the necessary steps to create a virtual SCSI controller (also known as "Host Bus Adapter", or HBA), and to add SCSI storage to the guest virtual machine.

Procedure 15.4. Creating a virtual SCSI controller

  1. Display the configuration of the guest virtual machine (Guest1) and look for a pre-existing SCSI controller:
    # virsh dumpxml Guest1 | grep controller.*scsi
    
    If a device controller is present, the command will output one or more lines similar to the following:
    <controller type='scsi' model='virtio-scsi' index='0'/>
    
  2. If the previous step did not show a device controller, create the description for one in a new file and add it to the virtual machine, using the following steps:
    1. Create the device controller by writing a <controller> element in a new file and save this file with an XML extension. virtio-scsi-controller.xml, for example.
      <controller type='scsi' model='virtio-scsi'/>
      
    2. Associate the device controller you just created in virtio-scsi-controller.xml with your guest virtual machine (Guest1, for example):
      # virsh attach-device --config Guest1 ~/virtio-scsi-controller.xml
      
      In this example the --config option behaves the same as it does for disks. Refer to Procedure 15.2, “Adding physical block devices to guests” for more information.
  3. Add a new SCSI disk or CD-ROM. The new disk can be added using the methods in sections Section 15.5.1, “Adding File-based Storage to a Guest” and Section 15.5.2, “Adding Hard Drives and Other Block Devices to a Guest”. In order to create a SCSI disk, specify a target device name that starts with sd. The supported limit for each controller is 1024 virtio-scsi disks, but it is possible that other available resources in the host (such as file descriptors) are exhausted with fewer disks.
    For more information, refer to the following Red Hat Enterprise Linux 6 whitepaper: The next-generation storage interface for the Red Hat Enterprise Linux Kernel Virtual Machine: virtio-scsi.
    # virsh attach-disk Guest1 /var/lib/libvirt/images/FileName.img sdb --cache none
    
    Depending on the version of the driver in the guest virtual machine, the new disk may not be detected immediately by a running guest virtual machine. Follow the steps in the Red Hat Enterprise Linux Storage Administration Guide.

Chapter 16. Using qemu-img

The qemu-img command-line tool is used for formatting, modifying, and verifying various file systems used by KVM. qemu-img options and usages are highlighted in the sections that follow.

Warning

Never use qemu-img to modify images in use by a running virtual machine or any other process. This may destroy the image. Also, be aware that querying an image that is being modified by another process may encounter inconsistent state.

16.1. Checking the Disk Image

To perform a consistency check on a disk image with the file name imgname.
# qemu-img check [-f format] imgname

Note

Only a selected group of formats support consistency checks. These include qcow2, vdi, vhdx, vmdk, and qed.

16.2. Committing Changes to an Image

Commit any changes recorded in the specified image file (imgname) to the file's base image with the qemu-img commit command. Optionally, specify the file's format type (fmt).
 # qemu-img commit [-f fmt] [-t cache] imgname

16.3. Comparing Images

Compare the contents of two specified image files (imgname1 and imgname2) with the qemu-img compare command. Optionally, specify the files' format types (fmt). The images can have different formats and settings.
By default, images with different sizes are considered identical if the larger image contains only unallocated or zeroed sectors in the area after the end of the other image. In addition, if any sector is not allocated in one image and contains only zero bytes in the other one, it is evaluated as equal. If you specify the -s option, the images are not considered identical if the image sizes differ or a sector is allocated in one image and is not allocated in the second one.
 # qemu-img compare [-f fmt] [-F fmt] [-p] [-s] [-q] imgname1 imgname2
The qemu-img compare command exits with one of the following exit codes:
  • 0 - The images are identical
  • 1 - The images are different
  • 2 - There was an error opening one of the images
  • 3 - There was an error checking a sector allocation
  • 4 - There was an error reading the data

16.4. Mapping an Image

Using the qemu-img map command, your can dump the metadata of the specified image file (imgname) and its backing file chain. The dump shows the allocation state of every sector in the (imgname) with the topmost file that allocates it in the backing file chain. Optionally, specify the file's format type (fmt).
 # qemu-img map [-f fmt] [--output=fmt] imgname
There are two output formats, the human format and the json format:

16.4.1. The human Format

The default format (human) only dumps non-zero, allocated parts of the file. The output identifies a file from where data can be read and the offset in the file. Each line includes four fields. The following shows an example of an output:
Offset          Length          Mapped to       File
0               0x20000         0x50000         /tmp/overlay.qcow2
0x100000        0x10000         0x95380000      /tmp/backing.qcow2
The first line means that 0x20000 (131072) bytes starting at offset 0 in the image are available in tmp/overlay.qcow2 (opened in raw format) starting at offset 0x50000 (327680). Data that is compressed, encrypted, or otherwise not available in raw format causes an error if human format is specified.

Note

File names can include newline characters. Therefore, it is not safe to parse output in human format in scripts.

16.4.2. The json Format

If the json option is specified, the output returns an array of dictionaries in JSON format. In addition to the information provided in the human option, the output includes the following information:
  • data - A Boolean field that shows whether or not the sectors contain data
  • zero - A Boolean field that shows whether or not the data is known to read as zero
  • depth - The depth of the backing file of filename

Note

When the json option is specified, the offset field is optional.
For more information about the qemu-img map command and additional options, see the relevant man page.

16.5. Amending an Image

Amend the image format-specific options for the image file. Optionally, specify the file's format type (fmt).
# qemu-img amend [-p] [-f fmt] [-t cache] -o options filename

Note

This operation is only supported for the qcow2 file format.

16.6. Converting an Existing Image to Another Format

The convert option is used to convert one recognized image format to another image format. For a list of accepted formats, refer to Section 16.12, “Supported qemu-img Formats”.
# qemu-img convert [-c] [-p] [-f fmt] [-t cache] [-O output_fmt] [-o options] [-S sparse_size] filename output_filename
The -p parameter shows the progress of the command (optional and not for every command) and -S flag allows for the creation of a sparse file, which is included within the disk image. Sparse files in all purposes function like a standard file, except that the physical blocks that only contain zeros (that is, nothing). When the Operating System sees this file, it treats it as it exists and takes up actual disk space, even though in reality it does not take any. This is particularly helpful when creating a disk for a guest virtual machine as this gives the appearance that the disk has taken much more disk space than it has. For example, if you set -S to 50Gb on a disk image that is 10Gb, then your 10Gb of disk space will appear to be 60Gb in size even though only 10Gb is actually being used.
Convert the disk image filename to disk image output_filename using format output_format. The disk image can be optionally compressed with the -c option, or encrypted with the -o option by setting -o encryption. Note that the options available with the -o parameter differ with the selected format.
Only the qcow2 and qcow2 format supports encryption or compression. qcow2 encryption uses the AES format with secure 128-bit keys. qcow2 compression is read-only, so if a compressed sector is converted from qcow2 format, it is written to the new format as uncompressed data.
Image conversion is also useful to get a smaller image when using a format which can grow, such as qcow or cow. The empty sectors are detected and suppressed from the destination image.

16.7. Creating and Formatting New Images or Devices

Create the new disk image filename of size size and format format.
# qemu-img create [-f format] [-o options] filename [size]
If a base image is specified with -o backing_file=filename, the image will only record differences between itself and the base image. The backing file will not be modified unless you use the commit command. No size needs to be specified in this case.

16.8. Displaying Image Information

The info parameter displays information about a disk image filename. The format for the info option is as follows:
# qemu-img info [-f format] filename
This command is often used to discover the size reserved on disk which can be different from the displayed size. If snapshots are stored in the disk image, they are displayed also. This command will show for example, how much space is being taken by a qcow2 image on a block device. This is done by running the qemu-img. You can check that the image in use is the one that matches the output of the qemu-img info command with the qemu-img check command.
# qemu-img info /dev/vg-90.100-sluo/lv-90-100-sluo
image: /dev/vg-90.100-sluo/lv-90-100-sluo
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 0
cluster_size: 65536

16.9. Rebasing a Backing File of an Image

The qemu-img rebase changes the backing file of an image.
# qemu-img rebase [-f fmt] [-t cache] [-p] [-u] -b backing_file [-F backing_fmt] filename
The backing file is changed to backing_file and (if the format of filename supports the feature), the backing file format is changed to backing_format.

Note

Only the qcow2 format supports changing the backing file (rebase).
There are two different modes in which rebase can operate: safe and unsafe.
safe mode is used by default and performs a real rebase operation. The new backing file may differ from the old one and the qemu-img rebase command will take care of keeping the guest virtual machine-visible content of filename unchanged. In order to achieve this, any clusters that differ between backing_file and old backing file of filename are merged into filename before making any changes to the backing file.
Note that safe mode is an expensive operation, comparable to converting an image. The old backing file is required for it to complete successfully.
unsafe mode is used if the -u option is passed to qemu-img rebase. In this mode, only the backing file name and format of filename is changed, without any checks taking place on the file contents. Make sure the new backing file is specified correctly or the guest-visible content of the image will be corrupted.
This mode is useful for renaming or moving the backing file. It can be used without an accessible old backing file. For instance, it can be used to fix an image whose backing file has already been moved or renamed.

16.10. Re-sizing the Disk Image

Change the disk image filename as if it had been created with size size. Only images in raw format can be resized in both directions, whereas qcow2 version 2 or qcow2 version 3 images can be grown but cannot be shrunk.
Use the following to set the size of the disk image filename to size bytes:
# qemu-img resize filename size
You can also resize relative to the current size of the disk image. To give a size relative to the current size, prefix the number of bytes with + to grow, or - to reduce the size of the disk image by that number of bytes. Adding a unit suffix allows you to set the image size in kilobytes (K), megabytes (M), gigabytes (G) or terabytes (T).
# qemu-img resize filename [+|-]size[K|M|G|T]

Warning

Before using this command to shrink a disk image, you must use file system and partitioning tools inside the VM itself to reduce allocated file systems and partition sizes accordingly. Failure to do so will result in data loss.
After using this command to grow a disk image, you must use file system and partitioning tools inside the VM to actually begin using the new space on the device.

16.11. Listing, Creating, Applying, and Deleting a Snapshot

Using different parameters from the qemu-img snapshot command you can list, apply, create, or delete an existing snapshot (snapshot) of specified image (filename).
# qemu-img snapshot [ -l | -a snapshot | -c snapshot | -d snapshot ] filename
The accepted arguments are as follows:
  • -l lists all snapshots associated with the specified disk image.
  • The apply option, -a, reverts the disk image (filename) to the state of a previously saved snapshot.
  • -c creates a snapshot (snapshot) of an image (filename).
  • -d deletes the specified snapshot.

16.12. Supported qemu-img Formats

When a format is specified in any of the qemu-img commands, the following format types may be used:
  • raw - Raw disk image format (default). This can be the fastest file-based format. If your file system supports holes (for example in ext2 or ext3 ), then only the written sectors will reserve space. Use qemu-img info to obtain the real size used by the image or ls -ls on Unix/Linux. Although Raw images give optimal performance, only very basic features are available with a Raw image. For example, no snapshots are available.
  • qcow2 - QEMU image format, the most versatile format with the best feature set. Use it to have optional AES encryption, zlib-based compression, support of multiple VM snapshots, and smaller images, which are useful on file systems that do not support holes . Note that this expansive feature set comes at the cost of performance.
    Although only the formats above can be used to run on a guest virtual machine or host physical machine, qemu-img also recognizes and supports the following formats in order to convert from them into either raw , or qcow2 format. The format of an image is usually detected automatically. In addition to converting these formats into raw or qcow2 , they can be converted back from raw or qcow2 to the original format. Note that the qcow2 version supplied with Red Hat Enterprise Linux 7 is 1.1. The format that is supplied with previous versions of Red Hat Enterprise Linux will be 0.10. You can revert image files to previous versions of qcow2. To know which version you are using, run qemu-img info qcow2 [imagefilename.img] command. To change the qcow version refer to Section 26.20.2, “Setting Target Elements”.
  • bochs - Bochs disk image format.
  • cloop - Linux Compressed Loop image, useful only to reuse directly compressed CD-ROM images present for example in the Knoppix CD-ROMs.
  • cow - User Mode Linux Copy On Write image format. The cow format is included only for compatibility with previous versions.
  • dmg - Mac disk image format.
  • nbd - Network block device.
  • parallels - Parallels virtualization disk image format.
  • qcow - Old QEMU image format. Only included for compatibility with older versions.
  • qed - Old QEMU image format. Only included for compatibility with older versions.
  • vdi - Oracle VM VirtualBox hard disk image format.
  • vhdx - Microsoft Hyper-V virtual hard disk-X disk image format.
  • vmdk - VMware 3 and 4 compatible image format.
  • vvfat - Virtual VFAT disk image format.

Chapter 17. KVM Migration

This chapter covers the migration guest virtual machines from one host physical machine that runs the KVM hypervisor to another. Migrating guests is possible because virtual machines run in a virtualized environment instead of directly on the hardware.

17.1. Migration Definition and Benefits

Migration works by sending the state of the guest virtual machine's memory and any virtualized devices to a destination host physical machine. It is recommended to use shared, networked storage to store the guest's images to be migrated. It is also recommended to use libvirt-managed storage pools for shared storage when migrating virtual machines.
Migrations can be performed both with live (running) and non-live (shut-down) guests.
In a live migration, the guest virtual machine continues to run on the source host machine, while the guest's memory pages are transferred to the destination host machine. During migration, KVM monitors the source for any changes in pages it has already transferred, and begins to transfer these changes when all of the initial pages have been transferred. KVM also estimates transfer speed during migration, so when the remaining amount of data to transfer will reaches a certain configurable period of time (10ms by default), KVM suspends the original guest virtual machine, transfers the remaining data, and resumes the same guest virtual machine on the destination host physical machine.
In contrast, a non-live migration (offline migration) suspends the guest virtual machine and then copies the guest's memory to the destination host machine. The guest is then resumed on the destination host machine and the memory the guest used on the source host machine is freed. The time it takes to complete such a migration only depends on network bandwidth and latency. If the network is experiencing heavy use or low bandwidth, the migration will take much longer. It should be noted that if the original guest virtual machine modifies pages faster than KVM can transfer them to the destination host physical machine, offline migration must be used, as live migration would never complete.
Migration is useful for:
Load balancing
Guest virtual machines can be moved to host physical machines with lower usage if their host machine becomes overloaded, or if another host machine is under-utilized.
Hardware independence
When you need to upgrade, add, or remove hardware devices on the host physical machine, you can safely relocate guest virtual machines to other host physical machines. This means that guest virtual machines do not experience any downtime for hardware improvements.
Energy saving
Virtual machines can be redistributed to other host physical machines, and the unloaded host systems can thus be powered off to save energy and cut costs in low usage periods.
Geographic migration
Virtual machines can be moved to another location for lower latency or when required by other reasons.

17.2. Migration Requirements and Limitations

Before using KVM migration, make sure that your system fulfills the migration's requirements, and that you are aware of its limitations.

Migration requirements

  • A guest virtual machine installed on shared storage using one of the following protocols:
    • Fibre Channel-based LUNs
    • iSCSI
    • NFS
    • GFS2
    • SCSI RDMA protocols (SCSI RCP): the block export protocol used in Infiniband and 10GbE iWARP adapters
  • Make sure that the libvirtd service is enabled and running.
    # systemctl enable libvirtd.service
    # systemctl restart libvirtd.service
  • The ability to migrate effectively is dependant on the parameter setting in the /etc/libvirt/libvirtd.conf file. To edit this file, use the following procedure:

    Procedure 17.1. Configuring libvirtd.conf

    1. Opening the libvirtd.conf requires running the command as root:
      # vim /etc/libvirt/libvirtd.conf
    2. Change the parameters as needed and save the file.
    3. Restart the libvirtd service:
      # systemctl restart libvirtd
  • The migration platforms and versions should be checked against Table 17.1, “Live Migration Compatibility”
  • Both host systems must have the appropriate TCP/IP ports open. In cases where a firewall is used, refer to the Red Hat Enterprise Linux Virtualization Security Guide for detailed port information.
  • Use a separate system exporting the shared storage medium. Storage should not reside on either of the two host physical machines used for the migration.
  • Shared storage must mount at the same location on source and destination systems. The mounted directory names must be identical. Although it is possible to keep the images using different paths, it is not recommended. Note that, if you intend to use virt-manager to perform the migration, the path names must be identical. If you intend to use virsh to perform the migration, different network configurations and mount directories can be used with the help of --xml option or pre-hooks . For more information on pre-hooks, refer to libvirt.org, and for more information on the XML option, refer to Chapter 26, Manipulating the Domain XML.
  • When migration is attempted on an existing guest virtual machine in a public bridge+tap network, the source and destination host machines must be located on the same network. Otherwise, the guest virtual machine network will not operate after migration.

Migration Limitations

  • Guest virtual machine migration has the following limitations when used on Red Hat Enterprise Linux with virtualization technology based on KVM:
    • Point to point migration – must be done manually to designate destination hypervisor from originating hypervisor
    • No validation or roll-back is available
    • Determination of target may only be done manually
    • Storage migration cannot be performed live on Red Hat Enterprise Linux 7, but you can migrate storage while the guest virtual machine is powered down. Live storage migration is available on Red Hat Enterprise Virtualization. Call your service representative for details.

Note

If you are migrating a guest machine that has virtio devices on it, make sure to set the number of vectors on any virtio device on either platfrom to 32 or fewer. For detailed information, see Section 26.18, “Devices”.

17.3. Live Migration and Red Hat Enterprise Linux Version Compatibility

Live Migration is supported as shown in Table 17.1, “Live Migration Compatibility”:

Table 17.1. Live Migration Compatibility

Migration Method Release Type Example Live Migration Support Notes
Forward Major release 6.5+ → 7.x Fully supported Any issues should be reported
Backward Major release 7.x → 6.y Not supported
ForwardMinor release7.x → 7.y (7.0 → 7.1)Fully supportedAny issues should be reported
BackwardMinor release7.y → 7.x (7.1 → 7.0)Fully supportedAny issues should be reported

Troubleshooting problems with migration

  • Issues with the migration protocol — If backward migration ends with "unknown section error", repeating the migration process can repair the issue as it may be a transient error. If not, report the problem.
  • Issues with audio devices — When migrating from Red Hat Enterprise Linux 6.x to Red Hat Enterprise Linux 7.y, note that the es1370 audio card is no longer supported. Use the ac97 audio card instead.
  • Issues with network cards — When migrating from Red Hat Enterprise Linux 6.x to Red Hat Enterprise Linux 7.y, note that the pcnet and ne2k_pci network cards are no longer supported. Use the virtio-net network device instead.
Configuring Network Storage
Configure shared storage and install a guest virtual machine on the shared storage.

17.4. Shared Storage Example: NFS for a Simple Migration

Important

This example uses NFS to share guest virtual machine images with other KVM host physical machines. Although not practical for large installations, it is presented to demonstrate migration techniques only. Do not use this example for migrating or running more than a few guest virtual machines. In addition, it is required that the synch parameter is enabled. This is required for proper export of the NFS storage.
iSCSI storage is a better choice for large deployments. For configuration details, refer to Section 14.5, “iSCSI-based Storage Pools”.
For detailed information on configuring NFS, opening IP tables, and configuring the firewall, refer to Red Hat Linux Storage Administration Guide.
Make sure that NFS file locking is not used as it is not supported in KVM.
  1. Export your libvirt image directory

    Migration requires storage to reside on a system that is separate to the migration target systems. On this separate system, export the storage by adding the default image directory to the /etc/exports file:
    /var/lib/libvirt/images *.example.com(rw,no_root_squash,sync)
    Change the hostname parameter as required for your environment.
  2. Start NFS

    1. Install the NFS packages if they are not yet installed:
      # yum install nfs-utils
    2. Make sure that the ports for NFS in iptables (2049, for example) are opened and add NFS to the /etc/hosts.allow file.
    3. Start the NFS service:
      # systemctl start nfs-server
  3. Mount the shared storage on the source and the destination

    On the migration source and the destination systems, mount the /var/lib/libvirt/images directory:
    # mount storage_host:/var/lib/libvirt/images /var/lib/libvirt/images
    

    Warning

    Whichever directory is chosen for the source host physical machine must be exactly the same as that on the destination host physical machine. This applies to all types of shared storage. The directory must be the same or the migration with virt-manager will fail.

17.5. Live KVM Migration with virsh

A guest virtual machine can be migrated to another host physical machine with the virsh command. The migrate command accepts parameters in the following format:
# virsh migrate --live GuestName DestinationURL
Note that the --live option may be eliminated when live migration is not desired. Additional options are listed in Section 17.5.2, “Additional Options for the virsh migrate Command”.
The GuestName parameter represents the name of the guest virtual machine which you want to migrate.
The DestinationURL parameter is the connection URL of the destination host physical machine. The destination system must run the same version of Red Hat Enterprise Linux, be using the same hypervisor and have libvirt running.

Note

The DestinationURL parameter for normal migration and peer2peer migration has different semantics:
  • normal migration: the DestinationURL is the URL of the target host physical machine as seen from the source guest virtual machine.
  • peer2peer migration: DestinationURL is the URL of the target host physical machine as seen from the source host physical machine.
Once the command is entered, you will be prompted for the root password of the destination system.

Important

Name resolution must be working on both sides (source and destination) in order for migration to succeed. Each side must be able to find the other. Make sure that you can ping one side to the other to check that the name resolution is working.
Example: live migration with virsh
This example migrates from host1.example.com to host2.example.com. Change the host physical machine names for your environment. This example migrates a virtual machine named guest1-rhel6-64.
This example assumes you have fully configured shared storage and meet all the prerequisites (listed here: Migration requirements).
  1. Verify the guest virtual machine is running

    From the source system, host1.example.com, verify guest1-rhel6-64 is running:
    [root@host1 ~]# virsh list
    Id Name                 State
    ----------------------------------
     10 guest1-rhel6-64     running
    
  2. Migrate the guest virtual machine

    Execute the following command to live migrate the guest virtual machine to the destination, host2.example.com. Append /system to the end of the destination URL to tell libvirt that you need full access.
    # virsh migrate --live guest1-rhel7-64 qemu+ssh://host2.example.com/system
    Once the command is entered you will be prompted for the root password of the destination system.
  3. Wait

    The migration may take some time depending on load and the size of the guest virtual machine. virsh only reports errors. The guest virtual machine continues to run on the source host physical machine until fully migrated.
  4. Verify the guest virtual machine has arrived at the destination host

    From the destination system, host2.example.com, verify guest1-rhel7-64 is running:
    [root@host2 ~]# virsh list
    Id Name                 State
    ----------------------------------
     10 guest1-rhel7-64     running
    
The live migration is now complete.

Note

libvirt supports a variety of networking methods including TLS/SSL, UNIX sockets, SSH, and unencrypted TCP. For more information on using other methods, refer to Chapter 21, Remote Management of Guests.

Note

Non-running guest virtual machines can be migrated using the following command:
# virsh migrate --offline --persistent 

17.5.1. Additional Tips for Migration with virsh

It is possible to perform multiple, concurrent live migrations where each migration runs in a separate command shell. However, this should be done with caution and should involve careful calculations as each migration instance uses one MAX_CLIENT from each side (source and target). As the default setting is 20, there is enough to run 10 instances without changing the settings. Should you need to change the settings, refer to the procedure Procedure 17.1, “Configuring libvirtd.conf”.
  1. Open the libvirtd.conf file as described in Procedure 17.1, “Configuring libvirtd.conf”.
  2. Look for the Processing controls section.
    #################################################################
    #
    # Processing controls
    #
    
    # The maximum number of concurrent client connections to allow
    # over all sockets combined.
    #max_clients = 5000
    
    # The maximum length of queue of connections waiting to be
    # accepted by the daemon. Note, that some protocols supporting
    # retransmission may obey this so that a later reattempt at
    # connection succeeds.
    #max_queued_clients = 1000
    
    # The minimum limit sets the number of workers to start up
    # initially. If the number of active clients exceeds this,
    # then more threads are spawned, upto max_workers limit.
    # Typically you'd want max_workers to equal maximum number
    # of clients allowed
    #min_workers = 5
    #max_workers = 20
    
    
    # The number of priority workers. If all workers from above
    # pool will stuck, some calls marked as high priority
    # (notably domainDestroy) can be executed in this pool.
    #prio_workers = 5
    
    # Total global limit on concurrent RPC calls. Should be
    # at least as large as max_workers. Beyond this, RPC requests
    # will be read into memory and queued. This directly impact
    # memory usage, currently each request requires 256 KB of
    # memory. So by default upto 5 MB of memory is used
    #
    # XXX this isn't actually enforced yet, only the per-client
    # limit is used so far
    #max_requests = 20
    
    # Limit on concurrent requests from a single client
    # connection. To avoid one client monopolizing the server
    # this should be a small fraction of the global max_requests
    # and max_workers parameter
    #max_client_requests = 5
    
    #################################################################
    
  3. Change the max_clients and max_workers parameters settings. It is recommended that the number be the same in both parameters. The max_clients will use 2 clients per migration (one per side) and max_workers will use 1 worker on the source and 0 workers on the destination during the perform phase and 1 worker on the destination during the finish phase.

    Important

    The max_clients and max_workers parameters settings are affected by all guest virtual machine connections to the libvirtd service. This means that any user that is using the same guest virtual machine and is performing a migration at the same time will also beholden to the limits set in the max_clients and max_workers parameters settings. This is why the maximum value needs to be considered carefully before performing a concurrent live migration.

    Important

    The max_clients parameter controls how many clients are allowed to connect to libvirt. When a large number of containers are started at once, this limit can be easily reached and exceeded. The value of the max_clients parameter could be increased to avoid this, but doing so can leave the system more vulnerable to denial of service (DoS) attacks against instances. To alleviate this problem, a new max_anonymous_clients setting has been introduced in Red Hat Enterprise Linux 7.0 that specifies a limit of connections which are accepted but not yet authenticated. You can implement a combination of max_clients and max_anonymous_clients to suit your workload.
  4. Save the file and restart the service.

    Note

    There may be cases where a migration connection drops because there are too many ssh sessions that have been started, but not yet authenticated. By default, sshd allows only 10 sessions to be in a "pre-authenticated state" at any time. This setting is controlled by the MaxStartups parameter in the sshd configuration file (located here: /etc/ssh/sshd_config), which may require some adjustment. Adjusting this parameter should be done with caution as the limitation is put in place to prevent DoS attacks (and over-use of resources in general). Setting this value too high will negate its purpose. To change this parameter, edit the file /etc/ssh/sshd_config, remove the # from the beginning of the MaxStartups line, and change the 10 (default value) to a higher number. Remember to save the file and restart the sshd service. For more information, refer to the sshd_config man page.

17.5.2. Additional Options for the virsh migrate Command

In addition to --live, virsh migrate accepts the following options:
  • --direct - used for direct migration
  • --p2p - used for peer-2-peer migration
  • --tunneled - used for tunneled migration
  • --offline - migrates domain definition without starting the domain on destination and without stopping it on source host. Offline migration may be used with inactive domains and it must be used with the --persistent option.
  • --persistent - leaves the domain persistent on destination host physical machine
  • --undefinesource - undefines the domain on the source host physical machine
  • --suspend - leaves the domain paused on the destination host physical machine
  • --change-protection - enforces that no incompatible configuration changes will be made to the domain while the migration is underway; this flag is implicitly enabled when supported by the hypervisor, but can be explicitly used to reject the migration if the hypervisor lacks change protection support.
  • --unsafe - forces the migration to occur, ignoring all safety procedures.
  • --verbose - displays the progress of migration as it is occurring
  • --compressed - activates compression of memory pages that have to be transferred repeatedly during live migration.
  • --abort-on-error - cancels the migration if a soft error (for example I/O error) happens during the migration.
  • --domain [name] - sets the domain name, id or uuid.
  • --desturi [URI] - connection URI of the destination host as seen from the client (normal migration) or source (p2p migration).
  • --migrateuri [URI] - the migration URI, which can usually be omitted.
  • --graphicsuri [URI] - graphics URI to be used for seamless graphics migration.
  • --listen-address [address] - sets the listen address that the hypervisor on the destination side should bind to for incoming migration.
  • --timeout [seconds] - forces a guest virtual machine to suspend when the live migration counter exceeds N seconds. It can only be used with a live migration. Once the timeout is initiated, the migration continues on the suspended guest virtual machine.
  • --dname [newname] - is used for renaming the domain during migration, which also usually can be omitted
  • --xml [filename] - the filename indicated can be used to supply an alternative XML file for use on the destination to supply a larger set of changes to any host-specific portions of the domain XML, such as accounting for naming differences between source and destination in accessing underlying storage. This option is usually omitted.
  • --migrate-disks [disk_identifiers] - this option can be used to select which disks are copied during the migration. This allows for more efficient live migration when copying certain disks is undesirable, such as when they already exist on the destination, or when they are no longer useful. [disk_identifiers] should be replaced by a comma-separated list of disks to be migrated, identified by their arguments found in the <target dev= /> line of the Domain XML file.
In addtion the following commands may help as well:
  • virsh migrate-setmaxdowntime [domain] [downtime] - will set a maximum tolerable downtime for a domain which is being live-migrated to another host. The specified downtime is in milliseconds. The domain specified must be the same domain that is being migrated.
  • virsh migrate-compcache [domain] --size - will set and or get the size of the cache in bytes which is used for compressing repeatedly transferred memory pages during a live migration. When the --size is not used the command displays the current size of the compression cache. When --size is used, and specified in bytes, the hypervisor is asked to change compression to match the indicated size, following which the current size is displayed. The --size argument is supposed to be used while the domain is being live migrated as a reaction to the migration progress and increasing number of compression cache misses obtained from the domjobinfo.
  • virsh migrate-setspeed [domain] [bandwidth] - sets the migration bandwidth in Mib/sec for the specified domain which is being migrated to another host.
  • virsh migrate-getspeed [domain] - gets the maximum migration bandwidth that is available in Mib/sec for the specified domain.
For more information, refer to Migration Limitations or the virsh man page.

17.6. Migrating with virt-manager

This section covers migrating a KVM guest virtual machine with virt-manager from one host physical machine to another.
  1. Open virt-manager

    Open virt-manager. Choose ApplicationsSystem ToolsVirtual Machine Manager from the main menu bar to launch virt-manager.
    Virt-Manager main menu

    Figure 17.1. Virt-Manager main menu

  2. Connect to the target host physical machine

    Connect to the target host physical machine by clicking on the File menu, then click Add Connection.
  3. Add connection

    The Add Connection window appears.
    Adding a connection to the target host physical machine

    Figure 17.2. Adding a connection to the target host physical machine

    Enter the following details:
    • Hypervisor: Select QEMU/KVM.
    • Method: Select the connection method.
    • Username: Enter the user name for the remote host physical machine.
    • Hostname: Enter the host name for the remote host physical machine.
    Click the Connect button. An SSH connection is used in this example, so the specified user's password must be entered in the next step.
    Enter password

    Figure 17.3. Enter password

  4. Migrate guest virtual machines

    Open the list of guests inside the source host physical machine (click the small triangle on the left of the host name) and right click on the guest that is to be migrated and click Migrate.
    In the New Host field, use the drop-down list to select the host physical machine you wish to migrate the guest virtual machine to and click Migrate.
    Choosing the destination host physical machine and starting the migration process

    Figure 17.4. Choosing the destination host physical machine and starting the migration process

    A progress window will appear.
    Progress window

    Figure 17.5. Progress window

    virt-manager now displays the newly migrated guest virtual machine running in the destination host. The guest virtual machine that was running in the source host physical machine is now listed inthe Shutoff state.
    Migrated guest virtual machine running in the destination host physical machine

    Figure 17.6. Migrated guest virtual machine running in the destination host physical machine

  5. Optional - View the storage details for the host physical machine

    In the Edit menu, click Connection Details, the Connection Details window appears.
    Click the Storage tab. The iSCSI target details for the destination host physical machine is shown. Note that the migrated guest virtual machine is listed as using the storage
    Storage details

    Figure 17.7. Storage details

    This host was defined by the following XML configuration:
    
    <pool type='iscsi'>
        <name>iscsirhel6guest</name>
        <source>
            <host name='virtlab22.example.com.'/>
            <device path='iqn.2001-05.com.iscsivendor:0-8a0906-fbab74a06-a700000017a4cc89-rhevh'/>
        </source>
        <target>
            <path>/dev/disk/by-path</path>
        </target>
    </pool>
      ...
    
    

    Figure 17.8. XML configuration for the destination host physical machine

Chapter 18. Guest Virtual Machine Device Configuration

Red Hat Enterprise Linux 7 supports three classes of devices for guest virtual machines:
  • Emulated devices are purely virtual devices that mimic real hardware, allowing unmodified guest operating systems to work with them using their standard in-box drivers. Red Hat Enterprise Linux 7 supports up to 216 virtio devices.
  • Virtio devices are purely virtual devices designed to work optimally in a virtual machine. Virtio devices are similar to emulated devices, however, non-Linux virtual machines do not include the drivers they require by default. Virtualization management software like the Virtual Machine Manager (virt-manager) and the Red Hat Enterprise Virtualization Hypervisor install these drivers automatically for supported non-Linux guest operating systems. Red Hat Enterprise Linux 7 supports up to 700 scsi disks.
  • Assigned devices are physical devices that are exposed to the virtual machine. This method is also known as 'passthrough'. Device assignment allows virtual machines exclusive access to PCI devices for a range of tasks, and allows PCI devices to appear and behave as if they were physically attached to the guest operating system. Red Hat Enterprise Linux 7 supports up to 32 assigned devices per virtual machine.
    Device assignment is supported on PCIe devices, including select graphics devices. Nvidia K-series Quadro, GRID, and Tesla graphics card GPU functions are now supported with device assignment in Red Hat Enterprise Linux 7. Parallel PCI devices may be supported as assigned devices, but they have severe limitations due to security and system configuration conflicts. Refer to the sections within this chapter for more details regarding specific series and versions that are supported.
Red Hat Enterprise Linux 7 supports PCI hot plug of devices exposed as single function slots to the virtual machine. Single function host devices and individual functions of multi-function host devices may be configured to enable this. Configurations exposing devices as multi-function PCI slots to the virtual machine are recommended only for non-hotplug applications.
For more information on specific devices and for limitations refer to Section 26.18, “Devices”.

Note

Platform support for interrupt remapping is required to fully isolate a guest with assigned devices from the host. Without such support, the host may be vulnerable to interrupt injection attacks from a malicious guest. In an environment where guests are trusted, the admin may opt-in to still allow PCI device assignment using the allow_unsafe_interrupts option to the vfio_iommu_type1 module. This may either be done persistently by adding a .conf file (for example local.conf) to /etc/modprobe.d containing the following:
options vfio_iommu_type1 allow_unsafe_interrupts=1
or dynamically using the sysfs entry to do the same:
# echo 1 > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts

18.1. PCI Devices

PCI device assignment is only available on hardware platforms supporting either Intel VT-d or AMD IOMMU. These Intel VT-d or AMD IOMMU specifications must be enabled in BIOS for PCI device assignment to function.

Procedure 18.1. Preparing an Intel system for PCI device assignment

  1. Enable the Intel VT-d specifications

    The Intel VT-d specifications provide hardware support for directly assigning a physical device to a virtual machine. These specifications are required to use PCI device assignment with Red Hat Enterprise Linux.
    The Intel VT-d specifications must be enabled in the BIOS. Some system manufacturers disable these specifications by default. The terms used to refer to these specifications can differ between manufacturers; consult your system manufacturer's documentation for the appropriate terms.
  2. Activate Intel VT-d in the kernel

    Activate Intel VT-d in the kernel by adding the intel_iommu=pt parameter to the end of the GRUB_CMDLINX_LINUX line, within the quotes, in the /etc/sysconfig/grub file.

    Note

    Instead of using the *_iommu=pt parameter for device assignment, which puts IOMMU into passthrough mode, it is also possible to use *_iommu=on. However, iommu=on should be used with caution, as it enables IOMMU for all devices, including those not used for device assignment by KVM, which may have a negative impact on guest performance.
    The example below is a modified grub file with Intel VT-d activated.
    GRUB_CMDLINE_LINUX="rd.lvm.lv=vg_VolGroup00/LogVol01
    vconsole.font=latarcyrheb-sun16 rd.lvm.lv=vg_VolGroup_1/root
    vconsole.keymap=us $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/
    rhcrashkernel-param || :) rhgb quiet intel_iommu=pt"
  3. Regenerate config file

    Regenerate /etc/grub2.cfg by running:
    grub2-mkconfig -o /etc/grub2.cfg
    Note that if you are using a UEFI-based host, the target file should be /etc/grub2-efi.cfg.
  4. Ready to use

    Reboot the system to enable the changes. Your system is now capable of PCI device assignment.

Procedure 18.2. Preparing an AMD system for PCI device assignment

  1. Enable the AMD IOMMU specifications

    The AMD IOMMU specifications are required to use PCI device assignment in Red Hat Enterprise Linux. These specifications must be enabled in the BIOS. Some system manufacturers disable these specifications by default.
  2. Enable IOMMU kernel support

    Append amd_iommu=pt to the end of the GRUB_CMDLINX_LINUX line, within the quotes, in /etc/sysconfig/grub so that AMD IOMMU specifications are enabled at boot.
  3. Regenerate config file

    Regenerate /etc/grub2.cfg by running:
    grub2-mkconfig -o /etc/grub2.cfg
    Note that if you are using a UEFI-based host, the target file should be /etc/grub2-efi.cfg.
  4. Ready to use

    Reboot the system to enable the changes. Your system is now capable of PCI device assignment.

Note

For further information on IOMMU, see Appendix D, Working with IOMMU Groups.

18.1.1. Assigning a PCI Device with virsh

These steps cover assigning a PCI device to a virtual machine on a KVM hypervisor.
This example uses a PCIe network controller with the PCI identifier code, pci_0000_01_00_0, and a fully virtualized guest machine named guest1-rhel7-64.

Procedure 18.3. Assigning a PCI device to a guest virtual machine with virsh

  1. Identify the device

    First, identify the PCI device designated for device assignment to the virtual machine. Use the lspci command to list the available PCI devices. You can refine the output of lspci with grep.
    This example uses the Ethernet controller highlighted in the following output:
    # lspci | grep Ethernet
    00:19.0 Ethernet controller: Intel Corporation 82567LM-2 Gigabit Network Connection
    01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
    01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
    This Ethernet controller is shown with the short identifier 00:19.0. We need to find out the full identifier used by virsh in order to assign this PCI device to a virtual machine.
    To do so, use the virsh nodedev-list command to list all devices of a particular type (pci) that are attached to the host machine. Then look at the output for the string that maps to the short identifier of the device you wish to use.
    This example shows the string that maps to the Ethernet controller with the short identifier 00:19.0. Note that the : and . characters are replaced with underscores in the full identifier.
    # virsh nodedev-list --cap pci
    pci_0000_00_00_0
    pci_0000_00_01_0
    pci_0000_00_03_0
    pci_0000_00_07_0
    pci_0000_00_10_0
    pci_0000_00_10_1
    pci_0000_00_14_0
    pci_0000_00_14_1
    pci_0000_00_14_2
    pci_0000_00_14_3
    pci_0000_00_19_0
    pci_0000_00_1a_0
    pci_0000_00_1a_1
    pci_0000_00_1a_2
    pci_0000_00_1a_7
    pci_0000_00_1b_0
    pci_0000_00_1c_0
    pci_0000_00_1c_1
    pci_0000_00_1c_4
    pci_0000_00_1d_0
    pci_0000_00_1d_1
    pci_0000_00_1d_2
    pci_0000_00_1d_7
    pci_0000_00_1e_0
    pci_0000_00_1f_0
    pci_0000_00_1f_2
    pci_0000_00_1f_3
    pci_0000_01_00_0
    pci_0000_01_00_1
    pci_0000_02_00_0
    pci_0000_02_00_1
    pci_0000_06_00_0
    pci_0000_07_02_0
    pci_0000_07_03_0
    Record the PCI device number that maps to the device you want to use; this is required in other steps.
  2. Review device information

    Information on the domain, bus, and function are available from output of the virsh nodedev-dumpxml command:
    
    # virsh nodedev-dumpxml pci_0000_00_19_0
    <device>
      <name>pci_0000_00_19_0</name>
      <parent>computer</parent>
      <driver>
        <name>e1000e</name>
      </driver>
      <capability type='pci'>
        <domain>0</domain>
        <bus>0</bus>
        <slot>25</slot>
        <function>0</function>
        <product id='0x1502'>82579LM Gigabit Network Connection</product>
        <vendor id='0x8086'>Intel Corporation</vendor>
        <iommuGroup number='7'>
          <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
        </iommuGroup>
      </capability>
    </device>
    

    Figure 18.1. Dump contents

    Note

    An IOMMU group is determined based on the visibility and isolation of devices from the perspective of the IOMMU. Each IOMMU group may contain one or more devices. When multiple devices are present, all endpoints within the IOMMU group must be claimed for any device within the group to be assigned to a guest. This can be accomplished either by also assigning the extra endpoints to the guest or by detaching them from the host driver using virsh nodedev-detach. Devices contained within a single group may not be split between multiple guests or split between host and guest. Non-endpoint devices such as PCIe root ports, switch ports, and bridges should not be detached from the host drivers and will not interfere with assignment of endpoints.
    Devices within an IOMMU group can be determined using the iommuGroup section of the virsh nodedev-dumpxml output. Each member of the group is provided via a separate "address" field. This information may also be found in sysfs using the following:
    $ ls /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/
    An example of the output from this would be:
    0000:01:00.0  0000:01:00.1
    To assign only 0000.01.00.0 to the guest, the unused endpoint should be detached from the host before starting the guest:
    $ virsh nodedev-detach pci_0000_01_00_1
  3. Determine required configuration details

    Refer to the output from the virsh nodedev-dumpxml pci_0000_00_19_0 command for the values required for the configuration file.
    The example device has the following values: bus = 0, slot = 25 and function = 0. The decimal configuration uses those three values:
    bus='0'
    slot='25'
    function='0'
  4. Add configuration details

    Run virsh edit, specifying the virtual machine name, and add a device entry in the <source> section to assign the PCI device to the guest virtual machine.
    # virsh edit guest1-rhel7-64
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
         <address domain='0' bus='0' slot='25' function='0'/>
      </source>
    </hostdev>
    

    Figure 18.2. Add PCI device

    Alternately, run virsh attach-device, specifying the virtual machine name and the guest's XML file:
    virsh attach-device guest1-rhel7-64 file.xml
  5. Start the virtual machine

    # virsh start guest1-rhel7-64
The PCI device should now be successfully assigned to the virtual machine, and accessible to the guest operating system.

18.1.2. Assigning a PCI Device with virt-manager

PCI devices can be added to guest virtual machines using the graphical virt-manager tool. The following procedure adds a Gigabit Ethernet controller to a guest virtual machine.

Procedure 18.4. Assigning a PCI device to a guest virtual machine using virt-manager

  1. Open the hardware settings

    Open the guest virtual machine and click the Add Hardware button to add a new device to the virtual machine.
    The virtual machine hardware window with the Information button selected on the top taskbar and Overview selected on the left menu pane.

    Figure 18.3. The virtual machine hardware information window

  2. Select a PCI device

    Select PCI Host Device from the Hardware list on the left.
    Select an unused PCI device. Note that selecting PCI devices presently in use by another guest causes errors. In this example, a spare audio controller is used. Click Finish to complete setup.
    The Add new virtual hardware wizard with PCI Host Device selected on the left menu pane, showing a list of host devices for selection in the right menu pane.

    Figure 18.4. The Add new virtual hardware wizard

  3. Add the new device

    The setup is complete and the guest virtual machine now has direct access to the PCI device.
    The virtual machine hardware window with the Information button selected on the top taskbar and Overview selected on the left menu pane, displaying the newly added PCI Device in the list of virtual machine devices in the left menu pane.

    Figure 18.5. The virtual machine hardware information window

Note

If device assignment fails, there may be other endpoints in the same IOMMU group that are still attached to the host. There is no way to retrieve group information using virt-manager, but virsh commands can be used to analyze the bounds of the IOMMU group and if necessary sequester devices.
Refer to the Note in Section 18.1.1, “Assigning a PCI Device with virsh” for more information on IOMMU groups and how to detach endpoint devices using virsh.

18.1.3. PCI Device Assignment with virt-install

To use virt-install to assign a PCI device, use the --host-device parameter.

Procedure 18.5. Assigning a PCI device to a virtual machine with virt-install

  1. Identify the device

    Identify the PCI device designated for device assignment to the guest virtual machine.
    # lspci | grep Ethernet
    00:19.0 Ethernet controller: Intel Corporation 82567LM-2 Gigabit Network Connection
    01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
    01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
    The virsh nodedev-list command lists all devices attached to the system, and identifies each PCI device with a string. To limit output to only PCI devices, enter the following command:
    # virsh nodedev-list --cap pci
    pci_0000_00_00_0
    pci_0000_00_01_0
    pci_0000_00_03_0
    pci_0000_00_07_0
    pci_0000_00_10_0
    pci_0000_00_10_1
    pci_0000_00_14_0
    pci_0000_00_14_1
    pci_0000_00_14_2
    pci_0000_00_14_3
    pci_0000_00_19_0
    pci_0000_00_1a_0
    pci_0000_00_1a_1
    pci_0000_00_1a_2
    pci_0000_00_1a_7
    pci_0000_00_1b_0
    pci_0000_00_1c_0
    pci_0000_00_1c_1
    pci_0000_00_1c_4
    pci_0000_00_1d_0
    pci_0000_00_1d_1
    pci_0000_00_1d_2
    pci_0000_00_1d_7
    pci_0000_00_1e_0
    pci_0000_00_1f_0
    pci_0000_00_1f_2
    pci_0000_00_1f_3
    pci_0000_01_00_0
    pci_0000_01_00_1
    pci_0000_02_00_0
    pci_0000_02_00_1
    pci_0000_06_00_0
    pci_0000_07_02_0
    pci_0000_07_03_0
    Record the PCI device number; the number is needed in other steps.
    Information on the domain, bus and function are available from output of the virsh nodedev-dumpxml command:
    # virsh nodedev-dumpxml pci_0000_01_00_0
    
    
    <device>
      <name>pci_0000_01_00_0</name>
      <parent>pci_0000_00_01_0</parent>
      <driver>
        <name>igb</name>
      </driver>
      <capability type='pci'>
        <domain>0</domain>
        <bus>1</bus>
        <slot>0</slot>
        <function>0</function>
        <product id='0x10c9'>82576 Gigabit Network Connection</product>
        <vendor id='0x8086'>Intel Corporation</vendor>
        <iommuGroup number='7'>
          <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
        </iommuGroup>
      </capability>
    </device>
    

    Figure 18.6. PCI device file contents

    Note

    If there are multiple endpoints in the IOMMU group and not all of them are assigned to the guest, you will need to manually detach the other endpoint(s) from the host by running the following command before you start the guest:
    $ virsh nodedev-detach pci_0000_00_19_1
    Refer to the Note in Section 18.1.1, “Assigning a PCI Device with virsh” for more information on IOMMU groups.
  2. Add the device

    Use the PCI identifier output from the virsh nodedev command as the value for the --host-device parameter.
    virt-install \
    --name=guest1-rhel7-64 \
    --disk path=/var/lib/libvirt/images/guest1-rhel7-64.img,size=8 \
    --vcpus=2 --ram=2048 \
    --location=http://example1.com/installation_tree/RHEL7.0-Server-x86_64/os \
    --nonetworks \
    --os-type=linux \
    --os-variant=rhel7
    --host-device=pci_0000_01_00_0
  3. Complete the installation

    Complete the guest installation. The PCI device should be attached to the guest.

18.1.4. Detaching an Assigned PCI Device

When a host PCI device has been assigned to a guest machine, the host can no longer use the device. Read this section to learn how to detach the device from the guest with virsh or virt-manager so it is available for host use.

Procedure 18.6. Detaching a PCI device from a guest with virsh

  1. Detach the device

    Use the following command to detach the PCI device from the guest by removing it in the guest's XML file:
    # virsh detach-device name_of_guest file.xml
  2. Re-attach the device to the host (optional)

    If the device is in managed mode, skip this step. The device will be returned to the host automatically.
    If the device is not using managed mode, use the following command to re-attach the PCI device to the host machine:
    # virsh nodedev-reattach device
    For example, to re-attach the pci_0000_01_00_0 device to the host:
    virsh nodedev-reattach pci_0000_01_00_0
    The device is now available for host use.

Procedure 18.7. Detaching a PCI Device from a guest with virt-manager

  1. Open the virtual hardware details screen

    In virt-manager, double-click on the virtual machine that contains the device. Select the Show virtual hardware details button to display a list of virtual hardware.
    The Show virtual hardware details button.

    Figure 18.7. The virtual hardware details button

  2. Select and remove the device

    Select the PCI device to be detached from the list of virtual devices in the left panel.
    The PCI device details and the Remove button.

    Figure 18.8. Selecting the PCI device to be detached

    Click the Remove button to confirm. The device is now available for host use.

18.1.5. Creating PCI Bridges

Peripheral Component Interconnects (PCI) bridges are used to attach to devices such as network cards, modems and sound cards. Just like their physical counterparts, virtual devices can also be attached to a PCI Bridge. In the past, only 31 PCI devices could be added to any guest virtual machine. Now, when a 31st PCI device is added, a PCI bridge is automatically placed in the 31st slot moving the additional PCI device to the PCI bridge. Each PCI bridge has 31 slots for 31 additional devices, all of which can be bridges. In this manner, over 900 devices can be available for guest virtual machines. Note that this action cannot be performed when the guest virtual machine is running. You must add the PCI device on a guest virtual machine that is shutdown.

18.1.5.1. PCI Bridge hot plug/hot unplug Support

PCI Bridge hot plug/hot unplug is supported on the following device types:
  • virtio-net-pci
  • virtio-scsi-pci
  • e1000
  • rtl8139
  • virtio-serial-pci
  • virtio-balloon-pci

18.1.6. PCI Device Assignment with SR-IOV Devices

A PCI network device (specified in the domain XML by the <source> element) can be directly connected to the guest using direct device assignment (sometimes referred to as passthrough). Due to limitations in standard single-port PCI ethernet card driver design, only Single Root I/O Virtualization (SR-IOV) virtual function (VF) devices can be assigned in this manner; to assign a standard single-port PCI or PCIe Ethernet card to a guest, use the traditional <hostdev> device definition.

     <devices>
    <interface type='hostdev'>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
      </source>
      <mac address='52:54:00:6d:90:02'>
      <virtualport type='802.1Qbh'>
        <parameters profileid='finance'/>
      </virtualport>
    </interface>
  </devices>

Figure 18.9. XML example for PCI device assignment

18.1.6.1. Configuring PCI Assignment with SR-IOV Devices

SR-IOV network cards provide multiple VFs that can each be individually assigned to a guest virtual machines using PCI device assignment. Once assigned, each behaves as a full physical network device. This permits many guest virtual machines to gain the performance advantage of direct PCI device assignment, while only using a single slot on the host physical machine.
These VFs can be assigned to guest virtual machines in the traditional manner using the <hostdev> element. However, SR-IOV VF network devices do not have permanent unique MAC addresses, which causes problems where the guest virtual machine's network settings need to be re-configured each time the host physical machine is rebooted. To fix this, you need to set the MAC address prior to assigning the VF to the host physical machine after every boot of the guest virtual machine. In order to assign this MAC address, as well as other options, refer to the following procedure:

Procedure 18.8. Configuring MAC addresses, vLAN, and virtual ports for assigning PCI devices on SR-IOV

The <hostdev> element cannot be used for function-specific items like MAC address assignment, vLAN tag ID assignment, or virtual port assignment, because the <mac>, <vlan>, and <virtualport> elements are not valid children for <hostdev>. Instead, these elements can be used with the hostdev interface type: <interface type='hostdev'>. This device type behaves as a hybrid of an <interface> and <hostdev>. Thus, before assigning the PCI device to the guest virtual machine, libvirt initializes the network-specific hardware/switch that is indicated (such as setting the MAC address, setting a vLAN tag, or associating with an 802.1Qbh switch) in the guest virtual machine's XML configuration file. For information on setting the vLAN tag, refer to Section 20.16, “Setting vLAN Tags”.
  1. Gather information

    In order to use <interface type='hostdev'>, you must have an SR-IOV-capable network card, host physical machine hardware that supports either the Intel VT-d or AMD IOMMU extensions, and you must know the PCI address of the VF that you wish to assign.
  2. Shut down the guest virtual machine

    Using virsh shutdown command, shut down the guest virtual machine (here named guestVM).
    # virsh shutdown guestVM
  3. Open the XML file for editing

    Run the # virsh save-image-edit command to open the XML file for editing (refer to Section 23.10.5, “Editing the Guest Virtual Machine Configuration” for more information) with the --running option. The name of the configuration file in this example is guestVM.xml.
     # virsh save-image-edit guestVM.xml --running 
    The guestVM.xml opens in your default editor.
  4. Edit the XML file

    Update the configuration file (guestVM.xml) to have a <devices> entry similar to the following:
    
     <devices>
       ...
       <interface type='hostdev' managed='yes'>
         <source>
           <address type='pci' domain='0x0' bus='0x00' slot='0x07' function='0x0'/> <!--these values can be decimal as well-->
         </source>
         <mac address='52:54:00:6d:90:02'/>                                         <!--sets the mac address-->
         <virtualport type='802.1Qbh'>                                              <!--sets the virtual port for the 802.1Qbh switch-->
           <parameters profileid='finance'/>
         </virtualport>
         <vlan>                                                                     <!--sets the vlan tag-->
          <tag id='42'/>
         </vlan>
       </interface>
       ...
     </devices>
    
    

    Figure 18.10. Sample domain XML for hostdev interface type

    Note that if you do not provide a MAC address, one will be automatically generated, just as with any other type of interface device. In addition, the <virtualport> element is only used if you are connecting to an 802.11Qgh hardware switch. 802.11Qbg (also known as "VEPA") switches are currently not supported.
  5. Restart the guest virtual machine

    Run the virsh start command to restart the guest virtual machine you shut down in step 2. Refer to Section 23.9, “Starting, Resuming, and Restoring a Virtual Machine” for more information.
     # virsh start guestVM 
    When the guest virtual machine starts, it sees the network device provided to it by the physical host machine's adapter, with the configured MAC address. This MAC address remains unchanged across guest virtual machine and host physical machine reboots.

18.1.6.2. Setting PCI device assignment from a pool of SR-IOV virtual functions

Hard coding the PCI addresses of particular Virtual Functions (VFs) into a guest's configuration has two serious limitations:
  • The specified VF must be available any time the guest virtual machine is started. Therefore, the administrator must permanently assign each VF to a single guest virtual machine (or modify the configuration file for every guest virtual machine to specify a currently unused VF's PCI address each time every guest virtual machine is started).
  • If the guest virtual machine is moved to another host physical machine, that host physical machine must have exactly the same hardware in the same location on the PCI bus (or the guest virtual machine configuration must be modified prior to start).
It is possible to avoid both of these problems by creating a libvirt network with a device pool containing all the VFs of an SR-IOV device. Once that is done, configure the guest virtual machine to reference this network. Each time the guest is started, a single VF will be allocated from the pool and assigned to the guest virtual machine. When the guest virtual machine is stopped, the VF will be returned to the pool for use by another guest virtual machine.

Procedure 18.9. Creating a device pool

  1. Shutdown the guest virtual machine

    Using virsh shutdown command, shutdown the guest virtual machine, here named guestVM.
    # virsh shutdown guestVM
  2. Create a configuration file

    Using your editor of choice, create an XML file (named passthrough.xml, for example) in the /tmp directory. Make sure to replace pf dev='eth3' with the netdev name of your own SR-IOV device's Physical Function (PF).
    The following is an example network definition that will make available a pool of all VFs for the SR-IOV adapter with its PF at "eth3' on the host physical machine:
          
    <network>
       <name>passthrough</name> <!-- This is the name of the file you created -->
       <forward mode='hostdev' managed='yes'>
         <pf dev='myNetDevName'/>  <!-- Use the netdev name of your SR-IOV devices PF here -->
       </forward>
    </network>
          
    
    

    Figure 18.11. Sample network definition domain XML

  3. Load the new XML file

    enter the following command, replacing /tmp/passthrough.xml with the name and location of your XML file you created in the previous step:
    # virsh net-define /tmp/passthrough.xml
  4. Restarting the guest

    Run the following, replacing passthrough.xml with the name of your XML file you created in the previous step:
     # virsh net-autostart passthrough # virsh net-start passthrough 
  5. Re-start the guest virtual machine

    Run the virsh start command to restart the guest virtual machine you shutdown in the first step (example uses guestVM as the guest virtual machine's domain name). Refer to Section 23.9, “Starting, Resuming, and Restoring a Virtual Machine” for more information.
     # virsh start guestVM 
  6. Initiating passthrough for devices

    Although only a single device is shown, libvirt will automatically derive the list of all VFs associated with that PF the first time a guest virtual machine is started with an interface definition in its domain XML like the following:
             
    <interface type='network'>
       <source network='passthrough'>
    </interface>
          
    
    

    Figure 18.12. Sample domain XML for interface network definition

  7. Verification

    You can verify this by running virsh net-dumpxml passthrough command after starting the first guest that uses the network; you will get output similar to the following:
          
    <network connections='1'>
       <name>passthrough</name>
       <uuid>a6b49429-d353-d7ad-3185-4451cc786437</uuid>
       <forward mode='hostdev' managed='yes'>
         <pf dev='eth3'/>
         <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x1'/>
         <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x3'/>
         <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x5'/>
         <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x7'/>
         <address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x1'/>
         <address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x3'/>
         <address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x5'/>
       </forward>
    </network>
          
    
    

    Figure 18.13. XML dump file passthrough contents

18.2. USB Devices

This section gives the commands required for handling USB devices.

18.2.1. Assigning USB Devices to Guest Virtual Machines

Most devices such as web cameras, card readers, disk drives, keyboards, mice are connected to a computer using a USB port and cable. There are two ways to pass such devices to a guest virtual machine:
  • Using USB passthrough - this requires the device to be physically connected to the host physical machine that is hosting the guest virtual machine. SPICE is not needed in this case. USB devices on the host can be passed to the guest via the command line or virt-manager. Refer to Section 22.3.2, “Attaching USB Devices to a Guest Virtual Machine” for virt manager directions. Note that the virt-manager directions are not suitable for hot plugging or hot unplugging devices. If you want to hot plug/or hot unplug a USB device, refer to Procedure 23.4, “Hot plugging USB devices for use by the guest virtual machine”.
  • Using USB re-direction - USB re-direction is best used in cases where there is a host physical machine that is running in a data center. The user connects to his/her guest virtual machine from a local machine or thin client. On this local machine there is a SPICE client. The user can attach any USB device to the thin client and the SPICE client will redirect the device to the host physical machine on the data center so it can be used by the guest virtual machine that is running on the thin client. For instructions via the virt-manager refer to Section 22.3.3, “USB Redirection”.

18.2.2. Setting a Limit on USB Device Redirection

To filter out certain devices from redirection, pass the filter property to -device usb-redir. The filter property takes a string consisting of filter rules, the format for a rule is:
<class>:<vendor>:<product>:<version>:<allow>
Use the value -1 to designate it to accept any value for a particular field. You may use multiple rules on the same command line using | as a separator. Note that if a device matches none of the passed in rules, redirecting it will not be allowed!

Example 18.1. An example of limiting redirection with a guest virtual machine

  1. Prepare a guest virtual machine.
  2. Add the following code excerpt to the guest virtual machine's' domain xml file:
        <redirdev bus='usb' type='spicevmc'>
          <alias name='redir0'/>
          <address type='usb' bus='0' port='3'/>
        </redirdev>
        <redirfilter>
          <usbdev class='0x08' vendor='0x1234' product='0xBEEF' version='2.0' allow='yes'/>
          <usbdev class='-1' vendor='-1' product='-1' version='-1' allow='no'/>
        </redirfilter>
    
  3. Start the guest virtual machine and confirm the setting changes by running the following:
    #ps -ef | grep $guest_name
    -device usb-redir,chardev=charredir0,id=redir0,/
    filter=0x08:0x1234:0xBEEF:0x0200:1|-1:-1:-1:-1:0,bus=usb.0,port=3
  4. Plug a USB device into a host physical machine, and use virt-manager to connect to the guest virtual machine.
  5. Click USB device selection in the menu, which will produce the following message: "Some USB devices are blocked by host policy". Click OK to confirm and continue.
    The filter takes effect.
  6. To make sure that the filter captures properly check the USB device vendor and product, then make the following changes in the host physical machine's domain XML to allow for USB redirection.
       <redirfilter>
          <usbdev class='0x08' vendor='0x0951' product='0x1625' version='2.0' allow='yes'/>
          <usbdev allow='no'/>
        </redirfilter>
    
  7. Restart the guest virtual machine, then use virt-viewer to connect to the guest virtual machine. The USB device will now redirect traffic to the guest virtual machine.

18.3. Configuring Device Controllers

Depending on the guest virtual machine architecture, some device buses can appear more than once, with a group of virtual devices tied to a virtual controller. Normally, libvirt can automatically infer such controllers without requiring explicit XML markup, but in some cases it is better to explicitly set a virtual controller element.

  ...
  <devices>
    <controller type='ide' index='0'/>
    <controller type='virtio-serial' index='0' ports='16' vectors='4'/>
    <controller type='virtio-serial' index='1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </controller>
    ...
  </devices>
  ...

Figure 18.14. Domain XML example for virtual controllers

Each controller has a mandatory attribute <controller type>, which must be one of:
  • ide
  • fdc
  • scsi
  • sata
  • usb
  • ccid
  • virtio-serial
  • pci
The <controller> element has a mandatory attribute <controller index> which is the decimal integer describing in which order the bus controller is encountered (for use in controller attributes of <address> elements). When <controller type ='virtio-serial'> there are two additional optional attributes (named ports and vectors), which control how many devices can be connected through the controller.
When <controller type ='scsi'>, there is an optional attribute model model, which can have the following values:
  • auto
  • buslogic
  • ibmvscsi
  • lsilogic
  • lsisas1068
  • lsisas1078
  • virtio-scsi
  • vmpvscsi
When <controller type ='usb'>, there is an optional attribute model model, which can have the following values:
  • piix3-uhci
  • piix4-uhci
  • ehci
  • ich9-ehci1
  • ich9-uhci1
  • ich9-uhci2
  • ich9-uhci3
  • vt82c686b-uhci
  • pci-ohci
  • nec-xhci
Note that if the USB bus needs to be explicitly disabled for the guest virtual machine, <model='none'> may be used. .
For controllers that are themselves devices on a PCI or USB bus, an optional sub-element <address> can specify the exact relationship of the controller to its master bus, with semantics as shown in Section 18.4, “Setting Addresses for Devices”.
An optional sub-element <driver> can specify the driver specific options. Currently it only supports attribute queues, which specifies the number of queues for the controller. For best performance, it's recommended to specify a value matching the number of vCPUs.
USB companion controllers have an optional sub-element <master> to specify the exact relationship of the companion to its master controller. A companion controller is on the same bus as its master, so the companion index value should be equal.
An example XML which can be used is as follows:
   
     ...
  <devices>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0' bus='0' slot='4' function='7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0' bus='0' slot='4' function='0' multifunction='on'/>
    </controller>
    ...
  </devices>
  ...
   

Figure 18.15. Domain XML example for USB controllers

PCI controllers have an optional model attribute with the following possible values:
  • pci-root
  • pcie-root
  • pci-bridge
  • dmi-to-pci-bridge
For machine types which provide an implicit PCI bus, the pci-root controller with index='0' is auto-added and required to use PCI devices. pci-root has no address. PCI bridges are auto-added if there are too many devices to fit on the one bus provided by model='pci-root', or a PCI bus number greater than zero was specified. PCI bridges can also be specified manually, but their addresses should only refer to PCI buses provided by already specified PCI controllers. Leaving gaps in the PCI controller indexes might lead to an invalid configuration. The following XML example can be added to the <devices> section:

  ...
  <devices>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='pci' index='1' model='pci-bridge'>
      <address type='pci' domain='0' bus='0' slot='5' function='0' multifunction='off'/>
    </controller>
  </devices>
  ...

Figure 18.16. Domain XML example for PCI bridge

For machine types which provide an implicit PCI Express (PCIe) bus (for example, the machine types based on the Q35 chipset), the pcie-root controller with index='0' is auto-added to the domain's configuration. pcie-root has also no address, but provides 31 slots (numbered 1-31) and can only be used to attach PCIe devices. In order to connect standard PCI devices on a system which has a pcie-root controller, a pci controller with model='dmi-to-pci-bridge' is automatically added. A dmi-to-pci-bridge controller plugs into a PCIe slot (as provided by pcie-root), and itself provides 31 standard PCI slots (which are not hot-pluggable). In order to have hot-pluggable PCI slots in the guest system, a pci-bridge controller will also be automatically created and connected to one of the slots of the auto-created dmi-to-pci-bridge controller; all guest devices with PCI addresses that are auto-determined by libvirt will be placed on this pci-bridge device.
   
     ...
  <devices>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0' bus='0' slot='0xe' function='0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0' bus='1' slot='1' function='0'/>
    </controller>
  </devices>
  ...
   

Figure 18.17. Domain XML example for PCIe (PCI express)

The following XML configuration is used for USB 3.0 / xHCI emulation:
   
     ...
  <devices>
    <controller type='usb' index='3' model='nec-xhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </controller>
  </devices>
    ...

Figure 18.18. Domain XML example for USB3/xHCI devices

18.4. Setting Addresses for Devices

Many devices have an optional <address> sub-element which is used to describe where the device is placed on the virtual bus presented to the guest virtual machine. If an address (or any optional attribute within an address) is omitted on input, libvirt will generate an appropriate address; but an explicit address is required if more control over layout is required. For domain XML device examples that include an <address> element, see Figure 18.9, “XML example for PCI device assignment”.
Every address has a mandatory attribute type that describes which bus the device is on. The choice of which address to use for a given device is constrained in part by the device and the architecture of the guest virtual machine. For example, a <disk> device uses type='drive', while a <console> device would use type='pci' on i686 or x86_64 guest virtual machine architectures. Each address type has further optional attributes that control where on the bus the device will be placed as described in the table:

Table 18.1. Supported device address types

Address type Description
type='pci' PCI addresses have the following additional attributes:
  • domain (a 2-byte hex integer, not currently used by qemu)
  • bus (a hex value between 0 and 0xff, inclusive)
  • slot (a hex value between 0x0 and 0x1f, inclusive)
  • function (a value between 0 and 7, inclusive)
  • multifunction controls turning on the multifunction bit for a particular slot/function in the PCI control register By default it is set to 'off', but should be set to 'on' for function 0 of a slot that will have multiple functions used.
type='drive' Drive addresses have the following additional attributes:
  • controller (a 2-digit controller number)
  • bus (a 2-digit bus number
  • target (a 2-digit bus number)
  • unit (a 2-digit unit number on the bus)
type='virtio-serial' Each virtio-serial address has the following additional attributes:
  • controller (a 2-digit controller number)
  • bus (a 2-digit bus number)
  • slot (a 2-digit slot within the bus)
type='ccid' A CCID address, for smart-cards, has the following additional attributes:
  • bus (a 2-digit bus number)
  • slot attribute (a 2-digit slot within the bus)
type='usb' USB addresses have the following additional attributes:
  • bus (a hex value between 0 and 0xfff, inclusive)
  • port (a dotted notation of up to four octets, such as 1.2 or 2.1.3.1)
type='isa' ISA addresses have the following additional attributes:
  • iobase
  • irq

18.5. Random Number Generator Device

Random number generators are very important for operating system security. For securing virtual operating systems, Red Hat Enterprise Linux 7 includes virtio-rng, a virtual hardware random number generator device that can provide the guest with fresh entropy on request.
On the host physical machine, the hardware RNG interface creates a chardev at /dev/hwrng, which can be opened and then read to fetch entropy from the host physical machine. In co-operation with the rngd daemon, the entropy from the host physical machine can be routed to the guest virtual machine's /dev/random, which is the primary source of randomness.
Using a random number generator is particularly useful when a device such as a keyboard, mouse, and other inputs are not enough to generate sufficient entropy on the guest virtual machine. The virtual random number generator device allows the host physical machine to pass through entropy to guest virtual machine operating systems. This procedure can be performed using either the command line or the virt-manager interface. For more information about virtio-rng, see Red Hat Enterprise Linux Virtual Machines: Access to Random Numbers Made Easy.

Procedure 18.10. Implementing virtio-rng using the Virtual Machine Manager

  1. Shut down the guest virtual machine.
  2. Select the guest virtual machine and from the Edit menu, select Virtual Machine Details, to open the Details window for the specified guest virtual machine.
  3. Click the Add Hardware button.
  4. In the Add New Virtual Hardware window, select RNG to open the Random Number Generator window.
    Random Number Generator window

    Figure 18.19. Random Number Generator window

    Enter the desired parameters and click Finish when done. The parameters are explained in virtio-rng elements.

Procedure 18.11. Implementing virtio-rng using command-line tools

  1. Shutdown the guest virtual machine.
  2. Using the virsh edit domain-name command, open the XML file for the desired guest virtual machine.
  3. Edit the <devices> element to include the following:
    
      ...
      <devices>
        <rng model='virtio'>
          <rate period='2000' bytes='1234'/>
          <backend model='random'>/dev/random</backend>
          <!-- OR -->
          <backend model='egd' type='udp'>
            <source mode='bind' service='1234'/>
            <source mode='connect' host='1.2.3.4' service='1234'/>
          </backend>
        </rng>
      </devices>
      ...

    Figure 18.20. Random number generator device

    The random number generator device allows the following XML attributes and elements:

    virtio-rng elements

    • <model> - The required model attribute specifies what type of RNG device is provided.
    • <backend model> - The <backend> element specifies the source of entropy to be used for the guest. The source model is configured using the model attribute. Supported source models include 'random' and 'egd' .
      • <backend model='random'> - This <backend> type expects a non-blocking character device as input. Examples of such devices are /dev/random and /dev/urandom. The file name is specified as contents of the <backend> element. When no file name is specified the hypervisor default is used.
      • <backend model='egd'> - This backend connects to a source using the EGD protocol. The source is specified as a character device. Refer to character device host physical machine interface for more information.

18.6. Assigning GPU Devices

Red Hat Enterprise Linux 7 supports PCI device assignment of NVIDIA K-Series Quadro (model 2000 series or higher), GRID, and Tesla as non-VGA graphics devices. Currently up to two GPUs may be attached to the virtual machine in addition to one of the standard, emulated VGA interfaces. The emulated VGA is used for pre-boot and installation and the NVIDIA GPU takes over when the NVIDIA graphics drivers are loaded. Note that the NVIDIA Quadro 2000 is not supported, nor is the Quadro K420 card.
This procedure will, in short, identify the device from lspci, detach it from host physical machine and then attach it to the guest virtual machine.
  1. Enable IOMMU support in the host physical machine kernel

    For an Intel VT-d system this is done by adding the intel_iommu=pt parameter to the kernel command line. For an AMD-Vi system, the option is amd_iommu=pt. To enable this option you will need to edit or add the GRUB_CMDLINX_LINUX line to the /etc/sysconfig/grub configuration file as follows:
    GRUB_CMDLINE_LINUX="rd.lvm.lv=vg_VolGroup00/LogVol01
    vconsole.font=latarcyrheb-sun16 rd.lvm.lv=vg_VolGroup_1/root
    vconsole.keymap=us $([ -x /usr/sbin/rhcrashkernel-param ]  &&
    /usr/sbin/rhcrashkernel-param || :) rhgb quiet intel_iommu=pt"
    

    Note

    For further information on IOMMU, see Appendix D, Working with IOMMU Groups.
  2. Regenerate the boot loader configuration

    Regenerate the boot loader configuration using the grub2-mkconfig to include this option, by running the following command:
    # grub2-mkconfig -o /etc/grub2.cfg
    
    Note that if you are using a UEFI-based host, the target file should be /etc/grub2-efi.cfg.
  3. Reboot the host physical machine

    In order for this option to take effect, reboot the host physical machine with the following command:
    # reboot

Procedure 18.12. Excluding the GPU device from binding to the host physical machine driver

For GPU assignment it is recommended to exclude the device from binding to host drivers as these drivers often do not support dynamic unbinding of the device.
  1. Identify the PCI bus address

    To identify the PCI bus address and IDs of the device, run the following lspci command. In this example, a VGA controller such as a Quadro or GRID card is used as follows:
    # lspci -Dnn | grep VGA
    0000:02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK106GL [Quadro K4000] [10de:11fa] (rev a1)
    
    The resulting search reveals that the PCI bus address of this device is 0000:02:00.0 and the PCI IDs for the device are 10de:11fa.
  2. Prevent the native host physical machine driver from using the GPU device

    To prevent the native host physical machine driver from using the GPU device you can use a PCI ID with the pci-stub driver. To do this, append the following additional option to the GRUB_CMDLINX_LINUX configuration file located in /etc/sysconfig/grub as follows:
    pci-stub.ids=10de:11fa
    To add additional PCI IDs for pci-stub, separate them with a comma.
  3. Regenerate the boot loader configuration

    Regenerate the boot loader configuration using the grub2-mkconfig to include this option, by running the following command:
    # grub2-mkconfig -o /etc/grub2.cfg
    
    Note that if you are using a UEFI-based host, the target file should be /etc/grub2-efi.cfg.
  4. Reboot the host physical machine

    In order for this option to take effect, reboot the host physical machine with the following command:
    # reboot
The virsh commands can be used to further evaluate the device, however in order to use virsh with the devices you need to convert the PCI bus address to libvirt compatible format by appending pci_ and converting delimiters to underscores. In this example the libvirt address of PCI device 0000:02:00.0 becomes pci_0000_02_00_0. The nodedev-dumpxml option provides additional information for the device as shown:
# virsh nodedev-dumpxml pci_0000_02_00_0
	
<device>
  <name>pci_0000_02_00_0</name>
  <path>/sys/devices/pci0000:00/0000:00:03.0/0000:02:00.0</path>
  <parent>pci_0000_00_03_0</parent>
  <driver>
    <name>pci-stub</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>2</bus>
    <slot>0</slot>
    <function>0</function>
    <product id='0x11fa'>GK106GL [Quadro K4000]</product>
    <vendor id='0x10de'>NVIDIA Corporation</vendor>
	<!-- pay attention to this part -->
    <iommuGroup number='13'>
      <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
    </iommuGroup>
    <pci-express>
      <link validity='cap' port='0' speed='8' width='16'/>
      <link validity='sta' speed='2.5' width='16'/>
    </pci-express>
  </capability>
</device>
	

Figure 18.21. XML file adaptation for GPU - Example

Particularly important in this output is the <iommuGroup> element. The iommuGroup indicates the set of devices which are considered isolated from other devices due to IOMMU capabilities and PCI bus topologies. All of the endpoint devices within the iommuGroup (that is devices that are not PCIe root ports, bridges, or switch ports) need to be unbound from the native host drivers in order to be assigned to a guest. In the example above, the group is composed of the GPU device (0000:02:00.0) as well as the companion audio device (0000:02:00.1). For more information, refer to Appendix D, Working with IOMMU Groups.

Note

Assignment of Nvidia audio functions is not supported due to hardware issues with legacy interrupt support. In order to assign the GPU to a guest, the audio function must first be detached from native host drivers. This can either be done by using lspci to find the PCI IDs for the device and appending it to the pci-stub.ids option or dynamically using the nodedev-detach option of virsh. For example:
# virsh nodedev-detach pci_0000_02_00_1
Device pci_0000_02_00_1 detached
The GPU audio function is generally not useful without the GPU itself, so it’s generally recommended to use the pci-stub.ids option instead.
The GPU can be attached to the VM using virt-manager or using virsh, either by directly editing the VM XML ( virsh edit [domain]) or attaching the GPU to the domain with virsh attach-device. If you are using the virsh attach-device command, an XML fragment first needs to be created for the device, such as the following:
		
<hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
  </source>
</hostdev>

Figure 18.22. XML file for attaching GPU - Example

Save this to a file and run virsh attach-device [domain] [file] --persistent to include the XML in the VM configuration. Note that the assigned GPU is added in addition to the existing emulated graphics device in the guest virtual machine. The assigned GPU is handled as a secondary graphics device in the VM. Assignment as a primary graphics device is not supported and emulated graphics devices in the VM's XML should not be removed.

Note

When using an assigned Nvidia GPU in the guest, only the Nvidia drivers are supported. Other drivers may not work and may generate errors. For a Red Hat Enterprise Linux 7 guest, the nouveau driver can be blacklisted using the option modprobe.blacklist=nouveau on the kernel command line during install. For information on other guest virtual machines refer to the operating system's specific documentation.
When configuring Xorg for use with an assigned GPU in a KVM guest, the BusID option must be added to xorg.conf to specify the guest address of the GPU. For example, within the guest determine the PCI bus address of the GPU (this will be different than the host address):
# lspci | grep VGA
00:02.0 VGA compatible controller: Device 1234:1111
00:09.0 VGA compatible controller: NVIDIA Corporation GK106GL [Quadro K4000] (rev a1)
In this example the address is 00:09.0. The file /etc/X11/xorg.conf is then modified to add the highlighted entry below.
Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID          "PCI:0:9:0"
EndSection
Depending on the guest operating system, with the Nvidia drivers loaded, the guest may support using both the emulated graphics and assigned graphics simultaneously or may disable the emulated graphics. Note that access to the assigned graphics framebuffer is not provided by tools such as virt-manager. If the assigned GPU is not connected to a physical display, guest-based remoting solutions may be necessary to access the GPU desktop. As with all PCI device assignment, migration of a guest with an assigned GPU is not supported and each GPU is owned exclusively by a single guest. Depending on the guest operating system, hotplug support of GPUs may be available.

Chapter 19. SR-IOV

Developed by the PCI-SIG (PCI Special Interest Group), the Single Root I/O Virtualization (SR-IOV) specification is a standard for a type of PCI device assignment that can share a single device to multiple virtual machines. SR-IOV improves device performance for virtual machines.

Note

Virtual machines that use the Xeon E3-1200 series chip set, do not support SR-IOV. More information can be found on Intel's website or in this article.
How SR-IOV works

Figure 19.1. How SR-IOV works

SR-IOV enables a Single Root Function (for example, a single Ethernet port), to appear as multiple, separate, physical devices. A physical device with SR-IOV capabilities can be configured to appear in the PCI configuration space as multiple functions. Each device has its own configuration space complete with Base Address Registers (BARs).
SR-IOV uses two PCI functions:
  • Physical Functions (PFs) are full PCIe devices that include the SR-IOV capabilities. Physical Functions are discovered, managed, and configured as normal PCI devices. Physical Functions configure and manage the SR-IOV functionality by assigning Virtual Functions.
  • Virtual Functions (VFs) are simple PCIe functions that only process I/O. Each Virtual Function is derived from a Physical Function. The number of Virtual Functions a device may have is limited by the device hardware. A single Ethernet port, the Physical Device, may map to many Virtual Functions that can be shared to virtual machines.
The hypervisor can map one or more Virtual Functions to a virtual machine. The Virtual Function's configuration space is then mapped to the configuration space presented to the guest.
Each Virtual Function can only be mapped to a single guest at a time, as Virtual Functions require real hardware resources. A virtual machine can have multiple Virtual Functions. A Virtual Function appears as a network card in the same way as a normal network card would appear to an operating system.
The SR-IOV drivers are implemented in the kernel. The core implementation is contained in the PCI subsystem, but there must also be driver support for both the Physical Function (PF) and Virtual Function (VF) devices. An SR-IOV capable device can allocate VFs from a PF. The VFs appear as PCI devices which are backed on the physical PCI device by resources such as queues and register sets.

19.1. Advantages of SR-IOV

SR-IOV devices can share a single physical port with multiple virtual machines.
Virtual Functions have near-native performance and provide better performance than paravirtualized drivers and emulated access. Virtual Functions provide data protection between virtual machines on the same physical server as the data is managed and controlled by the hardware.
These features allow for increased virtual machine density on hosts within a data center.
SR-IOV is better able to utilize the bandwidth of devices with multiple guests.

19.2. Using SR-IOV

This section covers the use of PCI passthrough to assign a Virtual Function of an SR-IOV capable multiport network card to a virtual machine as a network device.
SR-IOV Virtual Functions (VFs) can be assigned to virtual machines by adding a device entry in <hostdev> with the virsh edit or virsh attach-device command. However, this can be problematic because unlike a regular network device, an SR-IOV VF network device does not have a permanent unique MAC address, and is assigned a new MAC address each time the host is rebooted. Because of this, even if the guest is assigned the same VF after a reboot, when the host is rebooted the guest determines its network adapter to have a new MAC address. As a result, the guest believes there is new hardware connected each time, and will usually require re-configuration of the guest's network settings.
libvirt-0.9.10 and newer contains the <interface type='hostdev'> interface device. Using this interface device, libvirt will first perform any network-specific hardware/switch initialization indicated (such as setting the MAC address, VLAN tag, or 802.1Qbh virtualport parameters), then perform the PCI device assignment to the guest.
Using the <interface type='hostdev'> interface device requires:
  • an SR-IOV-capable network card,
  • host hardware that supports either the Intel VT-d or the AMD IOMMU extensions
  • the PCI address of the VF to be assigned.

Important

Assignment of an SR-IOV device to a virtual machine requires that the host hardware supports the Intel VT-d or the AMD IOMMU specification.
To attach an SR-IOV network device on an Intel or an AMD system, follow this procedure:

Procedure 19.1. Attach an SR-IOV network device on an Intel or AMD system

  1. Enable Intel VT-d or the AMD IOMMU specifications in the BIOS and kernel

    On an Intel system, enable Intel VT-d in the BIOS if it is not enabled already. Refer to Procedure 18.1, “Preparing an Intel system for PCI device assignment” for procedural help on enabling Intel VT-d in the BIOS and kernel.
    Skip this step if Intel VT-d is already enabled and working.
    On an AMD system, enable the AMD IOMMU specifications in the BIOS if they are not enabled already. Refer to Procedure 18.2, “Preparing an AMD system for PCI device assignment” for procedural help on enabling IOMMU in the BIOS.
  2. Verify support

    Verify if the PCI device with SR-IOV capabilities is detected. This example lists an Intel 82576 network interface card which supports SR-IOV. Use the lspci command to verify whether the device was detected.
    # lspci
    03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
    03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
    Note that the output has been modified to remove all other devices.
  3. Start the SR-IOV kernel modules

    If the device is supported the driver kernel module should be loaded automatically by the kernel. Optional parameters can be passed to the module using the modprobe command. The Intel 82576 network interface card uses the igb driver kernel module.
    # modprobe igb [<option>=<VAL1>,<VAL2>,]
    # lsmod |grep igb
    igb    87592  0
    dca    6708    1 igb
  4. Activate Virtual Functions

    The max_vfs parameter of the igb module allocates the maximum number of Virtual Functions. The max_vfs parameter causes the driver to spawn, up to the value of the parameter in, Virtual Functions. For this particular card the valid range is 0 to 7.
    Remove the module to change the variable.
    # modprobe -r igb
    Restart the module with the max_vfs set to 7 or any number of Virtual Functions up to the maximum supported by your device.
    # modprobe igb max_vfs=7
  5. Make the Virtual Functions persistent

    To make the Virtual Functions persistent across reboots, add the following to the /etc/rc.d/rc.local file. Specify the number of VFs desired (in this example, 2), up to the limit supported by the network interface card, and replace enp14s0f0 with the PF network device name(s):
    # echo 2 > /sys/class/net/enp14s0f0/device/sriov_numvfs
    This will ensure the feature is enabled at boot-time.

    Note

    If the /etc/rc.d/rc.local file does not already exist on your system, first create the file, then make it executable with this command:
    # chmod +x /etc/rc.d/rc.local
  6. Inspect the new Virtual Functions

    Using the lspci command, list the newly added Virtual Functions attached to the Intel 82576 network device. (Alternatively, use grep to search for Virtual Function, to search for devices that support Virtual Functions.)
    # lspci | grep 82576
    0b:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
    0b:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
    0b:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    0b:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    0b:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    0b:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    0b:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    0b:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    0b:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    0b:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    0b:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    0b:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    0b:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    0b:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    0b:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    0b:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    The identifier for the PCI device is found with the -n parameter of the lspci command. The Physical Functions correspond to 0b:00.0 and 0b:00.1. All Virtual Functions have Virtual Function in the description.
  7. Verify devices exist with virsh

    The libvirt service must recognize the device before adding a device to a virtual machine. libvirt uses a similar notation to the lspci output. All punctuation characters, : and ., in lspci output are changed to underscores (_).
    Use the virsh nodedev-list command and the grep command to filter the Intel 82576 network device from the list of available host devices. 0b is the filter for the Intel 82576 network devices in this example. This may vary for your system and may result in additional devices.
    # virsh nodedev-list | grep 0b
    pci_0000_0b_00_0
    pci_0000_0b_00_1
    pci_0000_0b_10_0
    pci_0000_0b_10_1
    pci_0000_0b_10_2
    pci_0000_0b_10_3
    pci_0000_0b_10_4
    pci_0000_0b_10_5
    pci_0000_0b_10_6
    pci_0000_0b_11_7
    pci_0000_0b_11_1
    pci_0000_0b_11_2
    pci_0000_0b_11_3
    pci_0000_0b_11_4
    pci_0000_0b_11_5
    The serial numbers for the Virtual Functions and Physical Functions should be in the list.
  8. Get device details with virsh

    The pci_0000_0b_00_0 is one of the Physical Functions and pci_0000_0b_10_0 is the first corresponding Virtual Function for that Physical Function. Use the virsh nodedev-dumpxml command to get advanced output for both devices.
    # virsh nodedev-dumpxml pci_0000_03_00_0
    <device>
      <name>pci_0000_03_00_0</name>
      <path>/sys/devices/pci0000:00/0000:00:01.0/0000:03:00.0</path>
      <parent>pci_0000_00_01_0</parent>
      <driver>
        <name>igb</name>
      </driver>
      <capability type='pci'>
        <domain>0</domain>
        <bus>3</bus>
        <slot>0</slot>
        <function>0</function>
        <product id='0x10c9'>82576 Gigabit Network Connection</product>
        <vendor id='0x8086'>Intel Corporation</vendor>
        <capability type='virt_functions'>
          <address domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
          <address domain='0x0000' bus='0x03' slot='0x10' function='0x2'/>
          <address domain='0x0000' bus='0x03' slot='0x10' function='0x4'/>
          <address domain='0x0000' bus='0x03' slot='0x10' function='0x6'/>
          <address domain='0x0000' bus='0x03' slot='0x11' function='0x0'/>
          <address domain='0x0000' bus='0x03' slot='0x11' function='0x2'/>
          <address domain='0x0000' bus='0x03' slot='0x11' function='0x4'/>
        </capability>
        <iommuGroup number='14'>
          <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
          <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
        </iommuGroup>
      </capability>
    </device>
    # virsh nodedev-dumpxml pci_0000_03_11_5
    <device>
      <name>pci_0000_03_11_5</name>
      <path>/sys/devices/pci0000:00/0000:00:01.0/0000:03:11.5</path>
      <parent>pci_0000_00_01_0</parent>
      <driver>
        <name>igbvf</name>
      </driver>
      <capability type='pci'>
        <domain>0</domain>
        <bus>3</bus>
        <slot>17</slot>
        <function>5</function>
        <product id='0x10ca'>82576 Virtual Function</product>
        <vendor id='0x8086'>Intel Corporation</vendor>
        <capability type='phys_function'>
          <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
        </capability>
        <iommuGroup number='35'>
          <address domain='0x0000' bus='0x03' slot='0x11' function='0x5'/>
        </iommuGroup>
      </capability>
    </device>
    This example adds the Virtual Function pci_0000_0b_10_0 to the virtual machine in Step 9. Note the bus, slot and function parameters of the Virtual Function: these are required for adding the device.
    Copy these parameters into a temporary XML file, such as /tmp/new-interface.xml for example.
       <interface type='hostdev' managed='yes'>
         <source>
           <address type='pci' domain='0' bus='11' slot='16' function='0'/>
         </source>
       </interface>

    Note

    If you do not specify a MAC address, one will be automatically generated. The <virtualport> element is only used when connecting to an 802.11Qbh hardware switch. The <vlan> element will transparently put the guest's device on the VLAN tagged 42.
    When the virtual machine starts, it should see a network device of the type provided by the physical adapter, with the configured MAC address. This MAC address will remain unchanged across host and guest reboots.
    The following <interface> example shows the syntax for the optional <mac address>, <virtualport>, and <vlan> elements. In practice, use either the <vlan> or <virtualport> element, not both simultaneously as shown in the example:
    ...
     <devices>
       ...
       <interface type='hostdev' managed='yes'>
         <source>
           <address type='pci' domain='0' bus='11' slot='16' function='0'/>
         </source>
         <mac address='52:54:00:6d:90:02'>
         <vlan>
            <tag id='42'/>
         </vlan>
         <virtualport type='802.1Qbh'>
           <parameters profileid='finance'/>
         </virtualport>
       </interface>
       ...
     </devices>
  9. Add the Virtual Function to the virtual machine

    Add the Virtual Function to the virtual machine using the following command with the temporary file created in the previous step. This attaches the new device immediately and saves it for subsequent guest restarts.
    virsh attach-device MyGuest /tmp/new-interface.xml --live --config
    
    Specifying the --live option with virsh attach-device attaches the new device to the running guest. Using the --config option ensures the new device is available after future guest restarts.

    Note

    The --live option is only accepted when the guest is running. virsh will return an error if the --live option is used on a non-running guest.
The virtual machine detects a new network interface card. This new card is the Virtual Function of the SR-IOV device.

19.3. Troubleshooting SR-IOV

This section contains solutions for problems which may affect SR-IOV. If you need additional help, refer to Appendix A, Troubleshooting as well as Section 18.1.6.2, “Setting PCI device assignment from a pool of SR-IOV virtual functions”.
Error starting the guest
When starting a configured virtual machine, an error occurs as follows:
# virsh start test
error: Failed to start domain test
error: Requested operation is not valid: PCI device 0000:03:10.1 is in use by domain rhel7
This error is often caused by a device that is already assigned to another guest or to the host itself.
Error migrating, saving, or dumping the guest
Attempts to migrate and dump the virtual machine cause an error similar to the following:
# virsh dump rhel7/tmp/rhel7.dump

error: Failed to core dump domain rhel7 to /tmp/rhel7.dump
error: internal error: unable to execute QEMU command 'migrate': State blocked by non-migratable device '0000:00:03.0/vfio-pci'
Because device assignment uses hardware on the specific host where the virtual machine was started, guest migration and save are not supported when device assignment is in use. Currently, the same limitation also applies to core-dumping a guest; this may change in the future. It is important to note that QEMU does not currently support migrate, save, and dump operations on guest virtual machines with PCI devices attached. Currently it only can support these actions with USB devices. Work is currently being done to improve this in the future.

Chapter 20. Virtual Networking

This chapter introduces the concepts needed to create, start, stop, remove, and modify virtual networks with libvirt.
Additional information can be found in the libvirt reference chapter

20.1. Virtual Network Switches

Libvirt virtual networking uses the concept of a virtual network switch. A virtual network switch is a software construct that operates on a host physical machine server, to which virtual machines (guests) connect. The network traffic for a guest is directed through this switch:
Virtual network switch with two guests

Figure 20.1. Virtual network switch with two guests

Linux host physical machine servers represent a virtual network switch as a network interface. When the libvirtd daemon (libvirtd) is first installed and started, the default network interface representing the virtual network switch is virbr0.
Linux host physical machine with an interface to a virtual network switch

Figure 20.2. Linux host physical machine with an interface to a virtual network switch

This virbr0 interface can be viewed with the ip command like any other interface:
 $ ip addr show virbr0
 3: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
     link/ether 1b:c4:94:cf:fd:17 brd ff:ff:ff:ff:ff:ff
     inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

20.2. Bridged Mode

When using Bridged mode, all of the guest virtual machines appear within the same subnet as the host physical machine. All other physical machines on the same physical network are aware of the virtual machines, and can access the virtual machines. Bridging operates on Layer 2 of the OSI networking model.
It is possible to use multiple physical interfaces on the hypervisor by joining them together with a bond. The bond is then added to a bridge and then guest virtual machines are added onto the bridge as well. However, the bonding driver has several modes of operation, and only a few of these modes work with a bridge where virtual guest machines are in use.
Virtual network switch in bridged mode

Figure 20.3. Virtual network switch in bridged mode

Warning

The only bonding modes that should be used with a guest virtual machine are Mode 1, Mode 2, and Mode 4. Under no circumstances should Modes 0, 3, 5, or 6 be used. It should also be noted that mii-monitoring should be used to monitor bonding modes as arp-monitoring does not work.
For more information on bonding modes, refer to the knowledgebase article on bonding modes, or The Red Hat Enterprise Linux 7 Networking Guide.
For a detailed explanation of bridge_opts parameters, see the Red Hat Enterprise Virtualization Administration Guide.

20.3. Network Address Translation

By default, virtual network switches operate in NAT mode. They use IP masquerading rather than SNAT (Source-NAT) or DNAT (Destination-NAT). IP masquerading enables connected guests to use the host physical machine IP address for communication to any external network. By default, computers that are placed externally to the host physical machine cannot communicate to the guests inside when the virtual network switch is operating in NAT mode, as shown in the following diagram:
Virtual network switch using NAT with two guests

Figure 20.4. Virtual network switch using NAT with two guests

Warning

Virtual network switches use NAT configured by iptables rules. Editing these rules while the switch is running is not recommended, as incorrect rules may result in the switch being unable to communicate.
If the switch is not running, you can set th public IP range for forward mode NAT in order to create a port masquerading range by running:
# iptables -j SNAT --to-source [start]-[end]

20.4. DNS and DHCP

IP information can be assigned to guests via DHCP. A pool of addresses can be assigned to a virtual network switch for this purpose. Libvirt uses the dnsmasq program for this. An instance of dnsmasq is automatically configured and started by libvirt for each virtual network switch that needs it.
Virtual network switch running dnsmasq

Figure 20.5. Virtual network switch running dnsmasq

20.5. Routed Mode

When using Routed mode, the virtual switch connects to the physical LAN connected to the host physical machine, passing traffic back and forth without the use of NAT. The virtual switch can examine all traffic and use the information contained within the network packets to make routing decisions. When using this mode, all of the virtual machines are in their own subnet, routed through a virtual switch. This situation is not always ideal as no other host physical machines on the physical network are aware of the virtual machines without manual physical router configuration, and cannot access the virtual machines. Routed mode operates at Layer 3 of the OSI networking model.
Virtual network switch in routed mode

Figure 20.6. Virtual network switch in routed mode

20.6. Isolated Mode

When using Isolated mode, guests connected to the virtual switch can communicate with each other, and with the host physical machine, but their traffic will not pass outside of the host physical machine, nor can they receive traffic from outside the host physical machine. Using dnsmasq in this mode is required for basic functionality such as DHCP. However, even if this network is isolated from any physical network, DNS names are still resolved. Therefore a situation can arise when DNS names resolve but ICMP echo request (ping) commands fail.
Virtual network switch in isolated mode

Figure 20.7. Virtual network switch in isolated mode

20.7. The Default Configuration

When the libvirtd daemon (libvirtd) is first installed, it contains an initial virtual network switch configuration in NAT mode. This configuration is used so that installed guests can communicate to the external network, through the host physical machine. The following image demonstrates this default configuration for libvirtd:
Default libvirt network configuration

Figure 20.8. Default libvirt network configuration

Note

A virtual network can be restricted to a specific physical interface. This may be useful on a physical system that has several interfaces (for example, eth0, eth1 and eth2). This is only useful in routed and NAT modes, and can be defined in the dev=<interface> option, or in virt-manager when creating a new virtual network.

20.8. Examples of Common Scenarios

This section demonstrates different virtual networking modes and provides some example scenarios.

20.8.1. Bridged Mode

Bridged mode operates on Layer 2 of the OSI model. When used, all of the guest virtual machines will appear on the same subnet as the host physical machine. The most common use cases for bridged mode include:
  • Deploying guest virtual machines in an existing network alongside host physical machines making the difference between virtual and physical machines transparent to the end user.
  • Deploying guest virtual machines without making any changes to existing physical network configuration settings.
  • Deploying guest virtual machines which must be easily accessible to an existing physical network. Placing guest virtual machines on a physical network where they must access services within an existing broadcast domain, such as DHCP.
  • Connecting guest virtual machines to an exsting network where VLANs are used.

20.8.2. Routed Mode

DMZ
Consider a network where one or more nodes are placed in a controlled sub-network for security reasons. The deployment of a special sub-network such as this is a common practice, and the sub-network is known as a DMZ. Refer to the following diagram for more details on this layout:
Sample DMZ configuration

Figure 20.9. Sample DMZ configuration

Host physical machines in a DMZ typically provide services to WAN (external) host physical machines as well as LAN (internal) host physical machines. As this requires them to be accessible from multiple locations, and considering that these locations are controlled and operated in different ways based on their security and trust level, routed mode is the best configuration for this environment.
Virtual Server Hosting
Consider a virtual server hosting company that has several host physical machines, each with two physical network connections. One interface is used for management and accounting, the other is for the virtual machines to connect through. Each guest has its own public IP address, but the host physical machines use private IP address as management of the guests can only be performed by internal administrators. Refer to the following diagram to understand this scenario:
Virtual server hosting sample configuration

Figure 20.10. Virtual server hosting sample configuration

When the host physical machine has a public IP address and the virtual machines have static public IP addresses, bridged networking cannot be used, as the provider only accepts packets from the MAC address of the public host physical machine. The following diagram demonstrates this:
Virtual server using static IP addresses

Figure 20.11. Virtual server using static IP addresses

20.8.3. NAT Mode

NAT (Network Address Translation) mode is the default mode. It can be used for testing when there is no need for direct network visibility.

20.8.4. Isolated Mode

Isolated mode allows virtual machines to communicate with each other only. They are unable to interact with the physical network.

20.9. Managing a Virtual Network

To configure a virtual network on your system:
  1. From the Edit menu, select Connection Details.
  2. This will open the Connection Details menu. Click the Virtual Networks tab.
    Virtual network configuration

    Figure 20.12. Virtual network configuration

  3. All available virtual networks are listed on the left-hand box of the menu. You can edit the configuration of a virtual network by selecting it from this box and editing as you see fit.

20.10. Creating a Virtual Network

To create a virtual network on your system using the Virtual Manager (virt-manager):
  1. Open the Virtual Networks tab from within the Connection Details menu. Click the Add Network button, identified by a plus sign (+) icon. For more information, refer to Section 20.9, “Managing a Virtual Network”.
    Virtual network configuration

    Figure 20.13. Virtual network configuration

    This will open the Create a new virtual network window. Click Forward to continue.
    Naming your new virtual network

    Figure 20.14. Naming your new virtual network

  2. Enter an appropriate name for your virtual network and click Forward.
    Choosing an IPv4 address space

    Figure 20.15. Choosing an IPv4 address space

  3. Check the Enable IPv4 network address space definition check box.
    Enter an IPv4 address space for your virtual network in the Network field.
    Check the Enable DHCPv4 check box.
    Define the DHCP range for your virtual network by specifying a Start and End range of IP addresses.
    Choosing an IPv4 address space

    Figure 20.16. Choosing an IPv4 address space

    Click Forward to continue.
  4. If you want to enable IPv6, check the Enable `IPv6` network address space definition.
    Enabling IPv6

    Figure 20.17. Enabling IPv6

    Additional fields appear in the Create a new virtual network window.
    Configuring IPv6

    Figure 20.18. Configuring IPv6

    Enter an IPv6 address in the Network field.
  5. If you want to enable DHCPv6, check the Enable DHCPv6 check box.
    Additional fields appear in the Create a new virtual network window.
    Configuring DHCPv6

    Figure 20.19. Configuring DHCPv6

    (Optional) Edit the start and end of the DHCPv6 range.
  6. If you want to enable static route definitions, check the Enable Static Route Definition check box.
    Additional fields appear in the Create a new virtual network window.
    Defining static routes

    Figure 20.20. Defining static routes

    Enter a network address and the gateway that will be used for the route to the network in the appropriate fields.
    Click Forward.
  7. Select how the virtual network should connect to the physical network.
    Connecting to the physical network

    Figure 20.21. Connecting to the physical network

    If you want the virtual network to be isolated, ensure that the Isolated virtual network radio button is selected.
    If you want the virtual network to connect to a physical network, select Forwarding to physical network, and choose whether the Destination should be Any physical device or a specific physical device. Also select whether the Mode should be NAT or Routed.
    If you want to enable IPv6 routing within the virtual network, check the Enable IPv6 internal routing/networking check box.
    Enter a DNS domain name for the virtual network.
    Click Finish to create the virtual network.
  8. The new virtual network is now available in the Virtual Networks tab of the Connection Details window.

20.11. Attaching a Virtual Network to a Guest

To attach a virtual network to a guest:
  1. In the Virtual Machine Manager window, highlight the guest that will have the network assigned.
    Selecting a virtual machine to display

    Figure 20.22. Selecting a virtual machine to display

  2. From the Virtual Machine Manager Edit menu, select Virtual Machine Details.
  3. Click the Add Hardware button on the Virtual Machine Details window.
  4. In the Add new virtual hardware window, select Network from the left pane, and select your network name (network1 in this example) from the Host device menu and click Finish.
    Select your network from the Add new virtual hardware window

    Figure 20.23. Select your network from the Add new virtual hardware window

  5. The new network is now displayed as a virtual network interface that will be presented to the guest upon launch.
    New network shown in guest hardware list

    Figure 20.24. New network shown in guest hardware list

20.12. Directly Attaching to Physical Interface

The instructions provided in this chapter will assist in the direct attachment of the virtual machine's NIC to the given physical interface of the host physical machine. If you want to use an assignment method (known as passthrough), refer to Chapter 18, Guest Virtual Machine Device Configuration and Section 23.44, “Interface Commands”. This setup requires the Linux macvtap driver to be available. There are four modes that you can choose for the operation mode of the macvtap device, with 'vepa' being the default mode. Their behavior is as follows:

Physical interface delivery modes

vepa
All VMs' packets are sent to the external bridge. Packets whose destination is a VM on the same host physical machine as where the packet originates from are sent back to the host physical machine by the VEPA capable bridge (today's bridges are typically not VEPA capable).
bridge
Packets whose destination is on the same host physical machine as where they originate from are directly delivered to the target macvtap device. Both origin and destination devices need to be in bridge mode for direct delivery. If either one of them is in vepa mode, a VEPA capable bridge is required.
private
All packets are sent to the external bridge and will only be delivered to a target VM on the same host physical machine if they are sent through an external router or gateway and that device sends them back to the host physical machine. This procedure is followed if either the source or destination device is in private mode.
passthrough
This feature attaches a virtual function of a SRIOV capable NIC directly to a VM without losing the migration capability. All packets are sent to the VF/IF of the configured network device. Depending on the capabilities of the device additional prerequisites or limitations may apply; for example, on Linux this requires kernel 2.6.38 or newer.
Each of the four modes is configured by changing the domain xml file. Once this file is opened, change the mode setting as shown:
  <devices>
    ...
    <interface type='direct'>
      <source dev='eth0' mode='vepa'/>
    </interface>
  </devices>
The network access of direct attached guest virtual machines can be managed by the hardware switch to which the physical interface of the host physical machine is connected to.
The interface can have additional parameters as shown below, if the switch is conforming to the IEEE 802.1Qbg standard. The parameters of the virtualport element are documented in more detail in the IEEE 802.1Qbg standard. The values are network specific and should be provided by the network administrator. In 802.1Qbg terms, the Virtual Station Interface (VSI) represents the virtual interface of a virtual machine.
Note that IEEE 802.1Qbg requires a non-zero value for the VLAN ID. Also if the switch is conforming to the IEEE 802.1Qbh standard, the values are network specific and should be provided by the network administrator.

Virtual Station Interface types

managerid
The VSI Manager ID identifies the database containing the VSI type and instance definitions. This is an integer value and the value 0 is reserved.
typeid
The VSI Type ID identifies a VSI type characterizing the network access. VSI types are typically managed by network administrator. This is an integer value.
typeidversion
The VSI Type Version allows multiple versions of a VSI Type. This is an integer value.
instanceid
The VSI Instance ID Identifier is generated when a VSI instance (a virtual interface of a virtual machine) is created. This is a globally unique identifier.
profileid
The profile ID contains the name of the port profile that is to be applied onto this interface. This name is resolved by the port profile database into the network parameters from the port profile, and those network parameters will be applied to this interface.
Each of the four types is configured by changing the domain xml file. Once this file is opened, change the mode setting as shown:
  <devices>
    ...
    <interface type='direct'>
      <source dev='eth0.2' mode='vepa'/>
      <virtualport type="802.1Qbg">
        <parameters managerid="11" typeid="1193047" typeidversion="2" instanceid="09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f"/>
      </virtualport>
    </interface>
  </devices>
The profile ID is shown here:
  <devices>
    ...
    <interface type='direct'>
      <source dev='eth0' mode='private'/>
      <virtualport type='802.1Qbh'>
        <parameters profileid='finance'/>
      </virtualport>
    </interface>
  </devices>
  ...

20.13. Dynamically Changing a Host Physical Machine or a Network Bridge that is Attached to a Virtual NIC

This section demonstrates how to move the vNIC of a guest virtual machine from one bridge to another while the guest virtual machine is running without compromising the guest virtual machine
  1. Prepare guest virtual machine with a configuration similar to the following:
    <interface type='bridge'>
          <mac address='52:54:00:4a:c9:5e'/>
          <source bridge='virbr0'/>
          <model type='virtio'/>
    </interface>
    
  2. Prepare an XML file for interface update:
    # cat br1.xml
    <interface type='bridge'>
          <mac address='52:54:00:4a:c9:5e'/>
          <source bridge='virbr1'/>
          <model type='virtio'/>
    </interface>
    
  3. Start the guest virtual machine, confirm the guest virtual machine's network functionality, and check that the guest virtual machine's vnetX is connected to the bridge you indicated.
    # brctl show
    bridge name     bridge id               STP enabled     interfaces
    virbr0          8000.5254007da9f2       yes                  virbr0-nic
    
    vnet0
    virbr1          8000.525400682996       yes                  virbr1-nic
    
  4. Update the guest virtual machine's network with the new interface parameters with the following command:
    # virsh update-device test1 br1.xml 
    
    Device updated successfully
    
    
  5. On the guest virtual machine, run service network restart. The guest virtual machine gets a new IP address for virbr1. Check the guest virtual machine's vnet0 is connected to the new bridge(virbr1)
    # brctl show
    bridge name     bridge id               STP enabled     interfaces
    virbr0          8000.5254007da9f2       yes             virbr0-nic
    virbr1          8000.525400682996       yes             virbr1-nic     vnet0
    

20.14. Applying Network Filtering

This section provides an introduction to libvirt's network filters, their goals, concepts and XML format.

20.14.1. Introduction

The goal of the network filtering, is to enable administrators of a virtualized system to configure and enforce network traffic filtering rules on virtual machines and manage the parameters of network traffic that virtual machines are allowed to send or receive. The network traffic filtering rules are applied on the host physical machine when a virtual machine is started. Since the filtering rules cannot be circumvented from within the virtual machine, it makes them mandatory from the point of view of a virtual machine user.
From the point of view of the guest virtual machine, the network filtering system allows each virtual machine's network traffic filtering rules to be configured individually on a per interface basis. These rules are applied on the host physical machine when the virtual machine is started and can be modified while the virtual machine is running. The latter can be achieved by modifying the XML description of a network filter.
Multiple virtual machines can make use of the same generic network filter. When such a filter is modified, the network traffic filtering rules of all running virtual machines that reference this filter are updated. The machines that are not running will update on start.
As previously mentioned, applying network traffic filtering rules can be done on individual network interfaces that are configured for certain types of network configurations. Supported network types include:
  • network
  • ethernet -- must be used in bridging mode
  • bridge

Example 20.1. An example of network filtering

The interface XML is used to reference a top-level filter. In the following example, the interface description references the filter clean-traffic.
   <devices>
    <interface type='bridge'>
      <mac address='00:16:3e:5d:c7:9e'/>
      <filterref filter='clean-traffic'/>
    </interface>
  </devices>
Network filters are written in XML and may either contain: references to other filters, rules for traffic filtering, or hold a combination of both. The above referenced filter clean-traffic is a filter that only contains references to other filters and no actual filtering rules. Since references to other filters can be used, a tree of filters can be built. The clean-traffic filter can be viewed using the command: # virsh nwfilter-dumpxml clean-traffic.
As previously mentioned, a single network filter can be referenced by multiple virtual machines. Since interfaces will typically have individual parameters associated with their respective traffic filtering rules, the rules described in a filter's XML can be generalized using variables. In this case, the variable name is used in the filter XML and the name and value are provided at the place where the filter is referenced.

Example 20.2. Description extended

In the following example, the interface description has been extended with the parameter IP and a dotted IP address as a value.
  <devices>
    <interface type='bridge'>
      <mac address='00:16:3e:5d:c7:9e'/>
      <filterref filter='clean-traffic'>
        <parameter name='IP' value='10.0.0.1'/>
      </filterref>
    </interface>
  </devices>
In this particular example, the clean-traffic network traffic filter will be represented with the IP address parameter 10.0.0.1 and as per the rule dictates that all traffic from this interface will always be using 10.0.0.1 as the source IP address, which is one of the purpose of this particular filter.

20.14.2. Filtering Chains

Filtering rules are organized in filter chains. These chains can be thought of as having a tree structure with packet filtering rules as entries in individual chains (branches).
Packets start their filter evaluation in the root chain and can then continue their evaluation in other chains, return from those chains back into the root chain or be dropped or accepted by a filtering rule in one of the traversed chains.
Libvirt's network filtering system automatically creates individual root chains for every virtual machine's network interface on which the user chooses to activate traffic filtering. The user may write filtering rules that are either directly instantiated in the root chain or may create protocol-specific filtering chains for efficient evaluation of protocol-specific rules.
The following chains exist:
  • root
  • mac
  • stp (spanning tree protocol)
  • vlan
  • arp and rarp
  • ipv4
  • ipv6
Multiple chains evaluating the mac, stp, vlan, arp, rarp, ipv4, or ipv6 protocol can be created using the protocol name only as a prefix in the chain's name.

Example 20.3. ARP traffic filtering

This example allows chains with names arp-xyz or arp-test to be specified and have their ARP protocol packets evaluated in those chains.
The following filter XML shows an example of filtering ARP traffic in the arp chain.
<filter name='no-arp-spoofing' chain='arp' priority='-500'>
  <uuid>f88f1932-debf-4aa1-9fbe-f10d3aa4bc95</uuid>
  <rule action='drop' direction='out' priority='300'>
    <mac match='no' srcmacaddr='$MAC'/>
  </rule>
  <rule action='drop' direction='out' priority='350'>
    <arp match='no' arpsrcmacaddr='$MAC'/>
  </rule>
  <rule action='drop' direction='out' priority='400'>
    <arp match='no' arpsrcipaddr='$IP'/>
  </rule>
  <rule action='drop' direction='in' priority='450'>
    <arp opcode='Reply'/>
    <arp match='no' arpdstmacaddr='$MAC'/>
  </rule>
  <rule action='drop' direction='in' priority='500'>
    <arp match='no' arpdstipaddr='$IP'/>
  </rule>
  <rule action='accept' direction='inout' priority='600'>
    <arp opcode='Request'/>
  </rule>
  <rule action='accept' direction='inout' priority='650'>
    <arp opcode='Reply'/>
  </rule>
  <rule action='drop' direction='inout' priority='1000'/>
</filter>
The consequence of putting ARP-specific rules in the arp chain, rather than for example in the root chain, is that packets protocols other than ARP do not need to be evaluated by ARP protocol-specific rules. This improves the efficiency of the traffic filtering. However, one must then pay attention to only putting filtering rules for the given protocol into the chain since other rules will not be evaluated. For example, an IPv4 rule will not be evaluated in the ARP chain since IPv4 protocol packets will not traverse the ARP chain.

20.14.3. Filtering Chain Priorities

As previously mentioned, when creating a filtering rule, all chains are connected to the root chain. The order in which those chains are accessed is influenced by the priority of the chain. The following table shows the chains that can be assigned a priority and their default priorities.

Table 20.1. Filtering chain default priorities values

Chain (prefix) Default priority
stp -810
mac -800
vlan -750
ipv4 -700
ipv6 -600
arp -500
rarp -400

Note

A chain with a lower priority value is accessed before one with a higher value.
The chains listed in Table 20.1, “Filtering chain default priorities values” can be also be assigned custom priorities by writing a value in the range [-1000 to 1000] into the priority (XML) attribute in the filter node. Section 20.14.2, “Filtering Chains”filter shows the default priority of -500 for arp chains, for example.

20.14.4. Usage of Variables in Filters

There are two variables that have been reserved for usage by the network traffic filtering subsystem: MAC and IP.
MAC is designated for the MAC address of the network interface. A filtering rule that references this variable will automatically be replaced with the MAC address of the interface. This works without the user having to explicitly provide the MAC parameter. Even though it is possible to specify the MAC parameter similar to the IP parameter above, it is discouraged since libvirt knows what MAC address an interface will be using.
The parameter IP represents the IP address that the operating system inside the virtual machine is expected to use on the given interface. The IP parameter is special in so far as the libvirt daemon will try to determine the IP address (and thus the IP parameter's value) that is being used on an interface if the parameter is not explicitly provided but referenced. For current limitations on IP address detection, consult the section on limitations Section 20.14.12, “Limitations” on how to use this feature and what to expect when using it. The XML file shown in Section 20.14.2, “Filtering Chains” contains the filter no-arp-spoofing, which is an example of using a network filter XML to reference the MAC and IP variables.
Note that referenced variables are always prefixed with the character $. The format of the value of a variable must be of the type expected by the filter attribute identified in the XML. In the above example, the IP parameter must hold a legal IP address in standard format. Failure to provide the correct structure will result in the filter variable not being replaced with a value and will prevent a virtual machine from starting or will prevent an interface from attaching when hot-plugging is being used. Some of the types that are expected for each XML attribute are shown in the example Example 20.4, “Sample variable types”.

Example 20.4. Sample variable types

As variables can contain lists of elements, (the variable IP can contain multiple IP addresses that are valid on a particular interface, for example), the notation for providing multiple elements for the IP variable is:
  <devices>
    <interface type='bridge'>
      <mac address='00:16:3e:5d:c7:9e'/>
      <filterref filter='clean-traffic'>
        <parameter name='IP' value='10.0.0.1'/>
        <parameter name='IP' value='10.0.0.2'/>
        <parameter name='IP' value='10.0.0.3'/>
      </filterref>
    </interface>
  </devices>
This XML file creates filters to enable multiple IP addresses per interface. Each of the IP addresses will result in a separate filtering rule. Therefore using the XML above and the following rule, three individual filtering rules (one for each IP address) will be created:
  <rule action='accept' direction='in' priority='500'>
    <tcp srpipaddr='$IP'/>
  </rule>
As it is possible to access individual elements of a variable holding a list of elements, a filtering rule like the following accesses the 2nd element of the variable DSTPORTS.
  <rule action='accept' direction='in' priority='500'>
    <udp dstportstart='$DSTPORTS[1]'/>
  </rule>

Example 20.5. Using a variety of variables

As it is possible to create filtering rules that represent all of the permissible rules from different lists using the notation $VARIABLE[@<iterator id="x">]. The following rule allows a virtual machine to receive traffic on a set of ports, which are specified in DSTPORTS, from the set of source IP address specified in SRCIPADDRESSES. The rule generates all combinations of elements of the variable DSTPORTS with those of SRCIPADDRESSES by using two independent iterators to access their elements.
  <rule action='accept' direction='in' priority='500'>
    <ip srcipaddr='$SRCIPADDRESSES[@1]' dstportstart='$DSTPORTS[@2]'/>
  </rule>
Assign concrete values to SRCIPADDRESSES and DSTPORTS as shown:
  SRCIPADDRESSES = [ 10.0.0.1, 11.1.2.3 ]
  DSTPORTS = [ 80, 8080 ]
Assigning values to the variables using $SRCIPADDRESSES[@1] and $DSTPORTS[@2] would then result in all variants of addresses and ports being created as shown:
  • 10.0.0.1, 80
  • 10.0.0.1, 8080
  • 11.1.2.3, 80
  • 11.1.2.3, 8080
Accessing the same variables using a single iterator, for example by using the notation $SRCIPADDRESSES[@1] and $DSTPORTS[@1], would result in parallel access to both lists and result in the following combination:
  • 10.0.0.1, 80
  • 11.1.2.3, 8080

Note

$VARIABLE is short-hand for $VARIABLE[@0]. The former notation always assumes the role of iterator with iterator id="0" added as shown in the opening paragraph at the top of this section.

20.14.5. Automatic IP Address Detection and DHCP Snooping

This section provides information about automatic IP address detection and DHCP snooping.

20.14.5.1. Introduction

The detection of IP addresses used on a virtual machine's interface is automatically activated if the variable IP is referenced but no value has been assigned to it. The variable CTRL_IP_LEARNING can be used to specify the IP address learning method to use. Valid values include: any, dhcp, or none.
The value any instructs libvirt to use any packet to determine the address in use by a virtual machine, which is the default setting if the variable CTRL_IP_LEARNING is not set. This method will only detect a single IP address per interface. Once a guest virtual machine's IP address has been detected, its IP network traffic will be locked to that address, if for example, IP address spoofing is prevented by one of its filters. In that case, the user of the VM will not be able to change the IP address on the interface inside the guest virtual machine, which would be considered IP address spoofing. When a guest virtual machine is migrated to another host physical machine or resumed after a suspend operation, the first packet sent by the guest virtual machine will again determine the IP address that the guest virtual machine can use on a particular interface.
The value of dhcp instructs libvirt to only honor DHCP server-assigned addresses with valid leases. This method supports the detection and usage of multiple IP address per interface. When a guest virtual machine resumes after a suspend operation, any valid IP address leases are applied to its filters. Otherwise the guest virtual machine is expected to use DHCP to obtain a new IP addresses. When a guest virtual machine migrates to another physical host physical machine, the guest virtual machine is required to re-run the DHCP protocol.
If CTRL_IP_LEARNING is set to none, libvirt does not do IP address learning and referencing IP without assigning it an explicit value is an error.

20.14.5.2. DHCP Snooping

CTRL_IP_LEARNING=dhcp (DHCP snooping) provides additional anti-spoofing security, especially when combined with a filter allowing only trusted DHCP servers to assign IP addresses. To enable this, set the variable DHCPSERVER to the IP address of a valid DHCP server and provide filters that use this variable to filter incoming DHCP responses.
When DHCP snooping is enabled and the DHCP lease expires, the guest virtual machine will no longer be able to use the IP address until it acquires a new, valid lease from a DHCP server. If the guest virtual machine is migrated, it must get a new valid DHCP lease to use an IP address (for example by bringing the VM interface down and up again).

Note

Automatic DHCP detection listens to the DHCP traffic the guest virtual machine exchanges with the DHCP server of the infrastructure. To avoid denial-of-service attacks on libvirt, the evaluation of those packets is rate-limited, meaning that a guest virtual machine sending an excessive number of DHCP packets per second on an interface will not have all of those packets evaluated and thus filters may not get adapted. Normal DHCP client behavior is assumed to send a low number of DHCP packets per second. Further, it is important to setup appropriate filters on all guest virtual machines in the infrastructure to avoid them being able to send DHCP packets. Therefore guest virtual machines must either be prevented from sending UDP and TCP traffic from port 67 to port 68 or the DHCPSERVER variable should be used on all guest virtual machines to restrict DHCP server messages to only be allowed to originate from trusted DHCP servers. At the same time anti-spoofing prevention must be enabled on all guest virtual machines in the subnet.

Example 20.6. Activating IPs for DHCP snooping

The following XML provides an example for the activation of IP address learning using the DHCP snooping method:
    <interface type='bridge'>
      <source bridge='virbr0'/>
      <filterref filter='clean-traffic'>
        <parameter name='CTRL_IP_LEARNING' value='dhcp'/>
      </filterref>
    </interface>

20.14.6. Reserved Variables

Table 20.2, “Reserved variables” shows the variables that are considered reserved and are used by libvirt:

Table 20.2. Reserved variables

Variable Name Definition
MAC The MAC address of the interface
IP The list of IP addresses in use by an interface
IPV6 Not currently implemented: the list of IPV6 addresses in use by an interface
DHCPSERVER The list of IP addresses of trusted DHCP servers
DHCPSERVERV6 Not currently implemented: The list of IPv6 addresses of trusted DHCP servers
CTRL_IP_LEARNING The choice of the IP address detection mode

20.14.7. Element and Attribute Overview

The root element required for all network filters is named <filter> with two possible attributes. The name attribute provides a unique name of the given filter. The chain attribute is optional but allows certain filters to be better organized for more efficient processing by the firewall subsystem of the underlying host physical machine. Currently the system only supports the following chains: root, ipv4, ipv6, arp and rarp.

20.14.8. References to Other Filters

Any filter may hold references to other filters. Individual filters may be referenced multiple times in a filter tree but references between filters must not introduce loops.

Example 20.7. An Example of a clean traffic filter

The following shows the XML of the clean-traffic network filter referencing several other filters.
<filter name='clean-traffic'>
  <uuid>6ef53069-ba34-94a0-d33d-17751b9b8cb1</uuid>
  <filterref filter='no-mac-spoofing'/>
  <filterref filter='no-ip-spoofing'/>
  <filterref filter='allow-incoming-ipv4'/>
  <filterref filter='no-arp-spoofing'/>
  <filterref filter='no-other-l2-traffic'/>
  <filterref filter='qemu-announce-self'/>
</filter>
To reference another filter, the XML node <filterref> needs to be provided inside a filter node. This node must have the attribute filter whose value contains the name of the filter to be referenced.
New network filters can be defined at any time and may contain references to network filters that are not known to libvirt, yet. However, once a virtual machine is started or a network interface referencing a filter is to be hot-plugged, all network filters in the filter tree must be available. Otherwise the virtual machine will not start or the network interface cannot be attached.

20.14.9. Filter Rules

The following XML shows a simple example of a network traffic filter implementing a rule to drop traffic if the IP address (provided through the value of the variable IP) in an outgoing IP packet is not the expected one, thus preventing IP address spoofing by the VM.

Example 20.8. Example of network traffic filtering

<filter name='no-ip-spoofing' chain='ipv4'>
  <uuid>fce8ae33-e69e-83bf-262e-30786c1f8072</uuid>
  <rule action='drop' direction='out' priority='500'>
    <ip match='no' srcipaddr='$IP'/>
  </rule>
</filter>
The traffic filtering rule starts with the rule node. This node may contain up to three of the following attributes:
  • action is mandatory can have the following values:
    • drop (matching the rule silently discards the packet with no further analysis)
    • reject (matching the rule generates an ICMP reject message with no further analysis)
    • accept (matching the rule accepts the packet with no further analysis)
    • return (matching the rule passes this filter, but returns control to the calling filter for further analysis)
    • continue (matching the rule goes on to the next rule for further analysis)
  • direction is mandatory can have the following values:
    • in for incoming traffic
    • out for outgoing traffic
    • inout for incoming and outgoing traffic
  • priority is optional. The priority of the rule controls the order in which the rule will be instantiated relative to other rules. Rules with lower values will be instantiated before rules with higher values. Valid values are in the range of -1000 to 1000. If this attribute is not provided, priority 500 will be assigned by default. Note that filtering rules in the root chain are sorted with filters connected to the root chain following their priorities. This allows to interleave filtering rules with access to filter chains. Refer to Section 20.14.3, “Filtering Chain Priorities” for more information.
  • statematch is optional. Possible values are '0' or 'false' to turn the underlying connection state matching off. The default setting is 'true' or 1
The above example Example 20.7, “An Example of a clean traffic filter” indicates that the traffic of type ip will be associated with the chain ipv4 and the rule will have priority=500. If for example another filter is referenced whose traffic of type ip is also associated with the chain ipv4 then that filter's rules will be ordered relative to the priority=500 of the shown rule.
A rule may contain a single rule for filtering of traffic. The above example shows that traffic of type ip is to be filtered.

20.14.10. Supported Protocols

The following sections list and give some details about the protocols that are supported by the network filtering subsystem. This type of traffic rule is provided in the rule node as a nested node. Depending on the traffic type a rule is filtering, the attributes are different. The above example showed the single attribute srcipaddr that is valid inside the ip traffic filtering node. The following sections show what attributes are valid and what type of data they are expecting. The following datatypes are available:
  • UINT8 : 8 bit integer; range 0-255
  • UINT16: 16 bit integer; range 0-65535
  • MAC_ADDR: MAC address in dotted decimal format, for example 00:11:22:33:44:55
  • MAC_MASK: MAC address mask in MAC address format, for instance, FF:FF:FF:FC:00:00
  • IP_ADDR: IP address in dotted decimal format, for example 10.1.2.3
  • IP_MASK: IP address mask in either dotted decimal format (255.255.248.0) or CIDR mask (0-32)
  • IPV6_ADDR: IPv6 address in numbers format, for example FFFF::1
  • IPV6_MASK: IPv6 mask in numbers format (FFFF:FFFF:FC00::) or CIDR mask (0-128)
  • STRING: A string
  • BOOLEAN: 'true', 'yes', '1' or 'false', 'no', '0'
  • IPSETFLAGS: The source and destination flags of the ipset described by up to 6 'src' or 'dst' elements selecting features from either the source or destination part of the packet header; example: src,src,dst. The number of 'selectors' to provide here depends on the type of ipset that is referenced
Every attribute except for those of type IP_MASK or IPV6_MASK can be negated using the match attribute with value no. Multiple negated attributes may be grouped together. The following XML fragment shows such an example using abstract attributes.
[...]
  <rule action='drop' direction='in'>
    <protocol match='no' attribute1='value1' attribute2='value2'/>
    <protocol attribute3='value3'/>
  </rule>
[...]
Rules behave evaluate the rule as well as look at it logically within the boundaries of the given protocol attributes. Thus, if a single attribute's value does not match the one given in the rule, the whole rule will be skipped during the evaluation process. Therefore, in the above example incoming traffic will only be dropped if: the protocol property attribute1 does not match both value1 and the protocol property attribute2 does not match value2 and the protocol property attribute3 matches value3.

20.14.10.1. MAC (Ethernet)

Protocol ID: mac
Rules of this type should go into the root chain.

Table 20.3. MAC protocol types

Attribute Name Datatype Definition
srcmacaddr MAC_ADDR MAC address of sender
srcmacmask MAC_MASK Mask applied to MAC address of sender
dstmacaddr MAC_ADDR MAC address of destination
dstmacmask MAC_MASK Mask applied to MAC address of destination
protocolid UINT16 (0x600-0xffff), STRING Layer 3 protocol ID. Valid strings include [arp, rarp, ipv4, ipv6]
comment STRING text string up to 256 characters
The filter can be written as such:
[...]
<mac match='no' srcmacaddr='$MAC'/>
[...]

20.14.10.2. VLAN (802.1Q)

Protocol ID: vlan
Rules of this type should go either into the root or vlan chain.

Table 20.4. VLAN protocol types

Attribute Name Datatype Definition
srcmacaddr MAC_ADDR MAC address of sender
srcmacmask MAC_MASK Mask applied to MAC address of sender
dstmacaddr MAC_ADDR MAC address of destination
dstmacmask MAC_MASK Mask applied to MAC address of destination
vlan-id UINT16 (0x0-0xfff, 0 - 4095) VLAN ID
encap-protocol UINT16 (0x03c-0xfff), String Encapsulated layer 3 protocol ID, valid strings are arp, ipv4, ipv6
comment STRING text string up to 256 characters

20.14.10.3. STP (Spanning Tree Protocol)

Protocol ID: stp
Rules of this type should go either into the root or stp chain.

Table 20.5. STP protocol types

Attribute Name Datatype Definition
srcmacaddr MAC_ADDR MAC address of sender
srcmacmask MAC_MASK Mask applied to MAC address of sender
type UINT8 Bridge Protocol Data Unit (BPDU) type
flags UINT8 BPDU flagdstmacmask
root-priority UINT16 Root priority range start
root-priority-hi UINT16 (0x0-0xfff, 0 - 4095) Root priority range end
root-address MAC _ADDRESS root MAC Address
root-address-mask MAC _MASK root MAC Address mask
roor-cost UINT32 Root path cost (range start)
root-cost-hi UINT32 Root path cost range end
sender-priority-hi UINT16 Sender prioriry range end
sender-address MAC_ADDRESS BPDU sender MAC address
sender-address-mask MAC_MASK BPDU sender MAC address mask
port UINT16 Port identifier (range start)
port_hi UINT16 Port identifier range end
msg-age UINT16 Message age timer (range start)
msg-age-hi UINT16 Message age timer range end
max-age-hi UINT16 Maximum age time range end
hello-time UINT16 Hello time timer (range start)
hello-time-hi UINT16 Hello time timer range end
forward-delay UINT16 Forward delay (range start)
forward-delay-hi UINT16 Forward delay range end
comment STRING text string up to 256 characters

20.14.10.4. ARP/RARP

Protocol ID: arp or rarp
Rules of this type should either go into the root or arp/rarp chain.

Table 20.6. ARP and RARP protocol types

Attribute Name Datatype Definition
srcmacaddr MAC_ADDR MAC address of sender
srcmacmask MAC_MASK Mask applied to MAC address of sender
dstmacaddr MAC_ADDR MAC address of destination
dstmacmask MAC_MASK Mask applied to MAC address of destination
hwtype UINT16 Hardware type
protocoltype UINT16 Protocol type
opcode UINT16, STRING Opcode valid strings are: Request, Reply, Request_Reverse, Reply_Reverse, DRARP_Request, DRARP_Reply, DRARP_Error, InARP_Request, ARP_NAK
arpsrcmacaddr MAC_ADDR Source MAC address in ARP/RARP packet
arpdstmacaddr MAC _ADDR Destination MAC address in ARP/RARP packet
arpsrcipaddr IP_ADDR Source IP address in ARP/RARP packet
arpdstipaddr IP_ADDR Destination IP address in ARP/RARP packet
gratututous BOOLEAN Boolean indiating whether to check for a gratuitous ARP packet
comment STRING text string up to 256 characters

20.14.10.5. IPv4

Protocol ID: ip
Rules of this type should either go into the root or ipv4 chain.

Table 20.7. IPv4 protocol types

Attribute Name Datatype Definition
srcmacaddr MAC_ADDR MAC address of sender
srcmacmask MAC_MASK Mask applied to MAC address of sender
dstmacaddr MAC_ADDR MAC address of destination
dstmacmask MAC_MASK Mask applied to MAC address of destination
srcipaddr IP_ADDR Source IP address
srcipmask IP_MASK Mask applied to source IP address
dstipaddr IP_ADDR Destination IP address
dstipmask IP_MASK Mask applied to destination IP address
protocol UINT8, STRING Layer 4 protocol identifier. Valid strings for protocol are: tcp, udp, udplite, esp, ah, icmp, igmp, sctp
srcportstart UINT16 Start of range of valid source ports; requires protocol
srcportend UINT16 End of range of valid source ports; requires protocol
dstportstart UNIT16 Start of range of valid destination ports; requires protocol
dstportend UNIT16 End of range of valid destination ports; requires protocol
comment STRING text string up to 256 characters

20.14.10.6. IPv6

Protocol ID: ipv6
Rules of this type should either go into the root or ipv6 chain.

Table 20.8. IPv6 protocol types

Attribute Name Datatype Definition
srcmacaddr MAC_ADDR MAC address of sender
srcmacmask MAC_MASK Mask applied to MAC address of sender
dstmacaddr MAC_ADDR MAC address of destination
dstmacmask MAC_MASK Mask applied to MAC address of destination
srcipaddr IP_ADDR Source IP address
srcipmask IP_MASK Mask applied to source IP address
dstipaddr IP_ADDR Destination IP address
dstipmask IP_MASK Mask applied to destination IP address
protocol UINT8, STRING Layer 4 protocol identifier. Valid strings for protocol are: tcp, udp, udplite, esp, ah, icmpv6, sctp
scrportstart UNIT16 Start of range of valid source ports; requires protocol
srcportend UINT16 End of range of valid source ports; requires protocol
dstportstart UNIT16 Start of range of valid destination ports; requires protocol
dstportend UNIT16 End of range of valid destination ports; requires protocol
comment STRING text string up to 256 characters

20.14.10.7. TCP/UDP/SCTP

Protocol ID: tcp, udp, sctp
The chain parameter is ignored for this type of traffic and should either be omitted or set to root. .

Table 20.9. TCP/UDP/SCTP protocol types

Attribute Name Datatype Definition
srcmacaddr MAC_ADDR MAC address of sender
srcipaddr IP_ADDR Source IP address
srcipmask IP_MASK Mask applied to source IP address
dstipaddr IP_ADDR Destination IP address
dstipmask IP_MASK Mask applied to destination IP address
scripto IP_ADDR Start of range of source IP address
srcipfrom IP_ADDR End of range of source IP address
dstipfrom IP_ADDR Start of range of destination IP address
dstipto IP_ADDR End of range of destination IP address
scrportstart UNIT16 Start of range of valid source ports; requires protocol
srcportend UINT16 End of range of valid source ports; requires protocol
dstportstart UNIT16 Start of range of valid destination ports; requires protocol
dstportend UNIT16 End of range of valid destination ports; requires protocol
comment STRING text string up to 256 characters
state STRING comma separated list of NEW,ESTABLISHED,RELATED,INVALID or NONE
flags STRING TCP-only: format of mask/flags with mask and flags each being a comma separated list of SYN,ACK,URG,PSH,FIN,RST or NONE or ALL
ipset STRING The name of an IPSet managed outside of libvirt
ipsetflags IPSETFLAGS flags for the IPSet; requires ipset attribute

20.14.10.8. ICMP

Protocol ID: icmp
Note: The chain parameter is ignored for this type of traffic and should either be omitted or set to root.

Table 20.10. ICMP protocol types

Attribute Name Datatype Definition
srcmacaddr MAC_ADDR MAC address of sender
srcmacmask MAC_MASK Mask applied to the MAC address of the sender
dstmacaddr MAD_ADDR MAC address of the destination
dstmacmask MAC_MASK Mask applied to the MAC address of the destination
srcipaddr IP_ADDR Source IP address
srcipmask IP_MASK Mask applied to source IP address
dstipaddr IP_ADDR Destination IP address
dstipmask IP_MASK Mask applied to destination IP address
srcipfrom IP_ADDR start of range of source IP address
scripto IP_ADDR end of range of source IP address
dstipfrom IP_ADDR Start of range of destination IP address
dstipto IP_ADDR End of range of destination IP address
type UNIT16 ICMP type
code UNIT16 ICMP code
comment STRING text string up to 256 characters
state STRING comma separated list of NEW,ESTABLISHED,RELATED,INVALID or NONE
ipset STRING The name of an IPSet managed outside of libvirt
ipsetflags IPSETFLAGS flags for the IPSet; requires ipset attribute

20.14.10.9. IGMP, ESP, AH, UDPLITE, 'ALL'

Protocol ID: igmp, esp, ah, udplite, all
The chain parameter is ignored for this type of traffic and should either be omitted or set to root.

Table 20.11. IGMP, ESP, AH, UDPLITE, 'ALL'

Attribute Name Datatype Definition
srcmacaddr MAC_ADDR MAC address of sender
srcmacmask MAC_MASK Mask applied to the MAC address of the sender
dstmacaddr MAD_ADDR MAC address of the destination
dstmacmask MAC_MASK Mask applied to the MAC address of the destination
srcipaddr IP_ADDR Source IP address
srcipmask IP_MASK Mask applied to source IP address
dstipaddr IP_ADDR Destination IP address
dstipmask IP_MASK Mask applied to destination IP address
srcipfrom IP_ADDR start of range of source IP address
scripto IP_ADDR end of range of source IP address
dstipfrom IP_ADDR Start of range of destination IP address
dstipto IP_ADDR End of range of destination IP address
comment STRING text string up to 256 characters
state STRING comma separated list of NEW,ESTABLISHED,RELATED,INVALID or NONE
ipset STRING The name of an IPSet managed outside of libvirt
ipsetflags IPSETFLAGS flags for the IPSet; requires ipset attribute

20.14.10.10. TCP/UDP/SCTP over IPV6

Protocol ID: tcp-ipv6, udp-ipv6, sctp-ipv6
The chain parameter is ignored for this type of traffic and should either be omitted or set to root.

Table 20.12. TCP, UDP, SCTP over IPv6 protocol types

Attribute Name Datatype Definition
srcmacaddr MAC_ADDR MAC address of sender
srcipaddr IP_ADDR Source IP address
srcipmask IP_MASK Mask applied to source IP address
dstipaddr IP_ADDR Destination IP address
dstipmask IP_MASK Mask applied to destination IP address
srcipfrom IP_ADDR start of range of source IP address
scripto IP_ADDR end of range of source IP address
dstipfrom IP_ADDR Start of range of destination IP address
dstipto IP_ADDR End of range of destination IP address
srcportstart UINT16 Start of range of valid source ports
srcportend UINT16 End of range of valid source ports
dstportstart UINT16 Start of range of valid destination ports
dstportend UINT16 End of range of valid destination ports
comment STRING text string up to 256 characters
state STRING comma separated list of NEW,ESTABLISHED,RELATED,INVALID or NONE
ipset STRING The name of an IPSet managed outside of libvirt
ipsetflags IPSETFLAGS flags for the IPSet; requires ipset attribute

20.14.10.11. ICMPv6

Protocol ID: icmpv6
The chain parameter is ignored for this type of traffic and should either be omitted or set to root.

Table 20.13. ICMPv6 protocol types

Attribute Name Datatype Definition
srcmacaddr MAC_ADDR MAC address of sender
srcipaddr IP_ADDR Source IP address
srcipmask IP_MASK Mask applied to source IP address
dstipaddr IP_ADDR Destination IP address
dstipmask IP_MASK Mask applied to destination IP address
srcipfrom IP_ADDR start of range of source IP address
scripto IP_ADDR end of range of source IP address
dstipfrom IP_ADDR Start of range of destination IP address
dstipto IP_ADDR End of range of destination IP address
type UINT16 ICMPv6 type
code UINT16 ICMPv6 code
comment STRING text string up to 256 characters
state STRING comma separated list of NEW,ESTABLISHED,RELATED,INVALID or NONE
ipset STRING The name of an IPSet managed outside of libvirt
ipsetflags IPSETFLAGS flags for the IPSet; requires ipset attribute

20.14.10.12. IGMP, ESP, AH, UDPLITE, 'ALL' over IPv6

Protocol ID: igmp-ipv6, esp-ipv6, ah-ipv6, udplite-ipv6, all-ipv6
The chain parameter is ignored for this type of traffic and should either be omitted or set to root.

Table 20.14. IGMP, ESP, AH, UDPLITE, 'ALL' over IPv protocol types

Attribute Name Datatype Definition
srcmacaddr MAC_ADDR MAC address of sender
srcipaddr IP_ADDR Source IP address
srcipmask IP_MASK Mask applied to source IP address
dstipaddr IP_ADDR Destination IP address
dstipmask IP_MASK Mask applied to destination IP address
srcipfrom IP_ADDR start of range of source IP address
scripto IP_ADDR end of range of source IP address
dstipfrom IP_ADDR Start of range of destination IP address
dstipto IP_ADDR End of range of destination IP address
comment STRING text string up to 256 characters
state STRING comma separated list of NEW,ESTABLISHED,RELATED,INVALID or NONE
ipset STRING The name of an IPSet managed outside of libvirt
ipsetflags IPSETFLAGS flags for the IPSet; requires ipset attribute

20.14.11. Advanced Filter Configuration Topics

The following sections discuss advanced filter configuration topics.

20.14.11.1. Connection tracking

The network filtering subsystem (on Linux) makes use of the connection tracking support of IP tables. This helps in enforcing the direction of the network traffic (state match) as well as counting and limiting the number of simultaneous connections towards a guest virtual machine. As an example, if a guest virtual machine has TCP port 8080 open as a server, clients may connect to the guest virtual machine on port 8080. Connection tracking and enforcement of the direction and then prevents the guest virtual machine from initiating a connection from (TCP client) port 8080 to the host physical machine back to a remote host physical machine. More importantly, tracking helps to prevent remote attackers from establishing a connection back to a guest virtual machine. For example, if the user inside the guest virtual machine established a connection to port 80 on an attacker site, then the attacker will not be able to initiate a connection from TCP port 80 back towards the guest virtual machine. By default the connection state match that enables connection tracking and then enforcement of the direction of traffic is turned on.

Example 20.9. XML example for turning off connections to the TCP port

The following shows an example XML fragment where this feature has been turned off for incoming connections to TCP port 12345.
   [...]
    <rule direction='in' action='accept' statematch='false'>
      <cp dstportstart='12345'/>
    </rule>
   [...]
This now allows incoming traffic to TCP port 12345, but would also enable the initiation from (client) TCP port 12345 within the VM, which may or may not be desirable.

20.14.11.2. Limiting number of connections

To limit the number of connections a guest virtual machine may establish, a rule must be provided that sets a limit of connections for a given type of traffic. If for example a VM is supposed to be allowed to only ping one other IP address at a time and is supposed to have only one active incoming ssh connection at a time.

Example 20.10. XML sample file that sets limits to connections

The following XML fragment can be used to limit connections
  [...]
  <rule action='drop' direction='in' priority='400'>
    <tcp connlimit-above='1'/>
  </rule>
  <rule action='accept' direction='in' priority='500'>
    <tcp dstportstart='22'/>
  </rule>
  <rule action='drop' direction='out' priority='400'>
    <icmp connlimit-above='1'/>
  </rule>
  <rule action='accept' direction='out' priority='500'>
    <icmp/>
  </rule>
  <rule action='accept' direction='out' priority='500'>
    <udp dstportstart='53'/>
  </rule>
  <rule action='drop' direction='inout' priority='1000'>
    <all/>
  </rule>
  [...]

Note

Limitation rules must be listed in the XML prior to the rules for accepting traffic. According to the XML file in Example 20.10, “XML sample file that sets limits to connections”, an additional rule for allowing DNS traffic sent to port 22 go out the guest virtual machine, has been added to avoid ssh sessions not getting established for reasons related to DNS lookup failures by the ssh daemon. Leaving this rule out may result in the ssh client hanging unexpectedly as it tries to connect. Additional caution should be used in regards to handling timeouts related to tracking of traffic. An ICMP ping that the user may have terminated inside the guest virtual machine may have a long timeout in the host physical machine's connection tracking system and will therefore not allow another ICMP ping to go through.
The best solution is to tune the timeout in the host physical machine's sysfs with the following command:# echo 3 > /proc/sys/net/netfilter/nf_conntrack_icmp_timeout. This command sets the ICMP connection tracking timeout to 3 seconds. The effect of this is that once one ping is terminated, another one can start after 3 seconds.
If for any reason the guest virtual machine has not properly closed its TCP connection, the connection to be held open for a longer period of time, especially if the TCP timeout value was set for a large amount of time on the host physical machine. In addition, any idle connection may result in a time out in the connection tracking system which can be re-activated once packets are exchanged.
However, if the limit is set too low, newly initiated connections may force an idle connection into TCP backoff. Therefore, the limit of connections should be set rather high so that fluctuations in new TCP connections do not cause odd traffic behavior in relation to idle connections.

20.14.11.3. Command-line tools

virsh has been extended with life-cycle support for network filters. All commands related to the network filtering subsystem start with the prefix nwfilter. The following commands are available:
  • nwfilter-list : lists UUIDs and names of all network filters
  • nwfilter-define : defines a new network filter or updates an existing one (must supply a name)
  • nwfilter-undefine : deletes a specified network filter (must supply a name). Do not delete a network filter currently in use.
  • nwfilter-dumpxml : displays a specified network filter (must supply a name)
  • nwfilter-edit : edits a specified network filter (must supply a name)

20.14.11.4. Pre-existing network filters

The following is a list of example network filters that are automatically installed with libvirt:

Table 20.15. ICMPv6 protocol types

Command Name Description
no-arp-spoofing Prevents a guest virtual machine from spoofing ARP traffic; this filter only allows ARP request and reply messages and enforces that those packets contain the MAC and IP addresses of the guest virtual machine.
allow-dhcp Allows a guest virtual machine to request an IP address via DHCP (from any DHCP server)
allow-dhcp-server Allows a guest virtual machine to request an IP address from a specified DHCP server. The dotted decimal IP address of the DHCP server must be provided in a reference to this filter. The name of the variable must be DHCPSERVER.
no-ip-spoofing Prevents a guest virtual machine from sending IP packets with a source IP address different from the one inside the packet.
no-ip-multicast Prevents a guest virtual machine from sending IP multicast packets.
clean-traffic Prevents MAC, IP and ARP spoofing. This filter references several other filters as building blocks.
These filters are only building blocks and require a combination with other filters to provide useful network traffic filtering. The most used one in the above list is the clean-traffic filter. This filter itself can for example be combined with the no-ip-multicast filter to prevent virtual machines from sending IP multicast traffic on top of the prevention of packet spoofing.

20.14.11.5. Writing your own filters

Since libvirt only provides a couple of example networking filters, you may consider writing your own. When planning on doing so there are a couple of things you may need to know regarding the network filtering subsystem and how it works internally. Certainly you also have to know and understand the protocols very well that you want to be filtering on so that no further traffic than what you want can pass and that in fact the traffic you want to allow does pass.
The network filtering subsystem is currently only available on Linux host physical machines and only works for QEMU and KVM type of virtual machines. On Linux, it builds upon the support for ebtables, iptables and ip6tables and makes use of their features. Considering the list found in Section 20.14.10, “Supported Protocols” the following protocols can be implemented using ebtables:
  • mac
  • stp (spanning tree protocol)
  • vlan (802.1Q)
  • arp, rarp
  • ipv4
  • ipv6
Any protocol that runs over IPv4 is supported using iptables, those over IPv6 are implemented using ip6tables.
Using a Linux host physical machine, all traffic filtering rules created by libvirt's network filtering subsystem first passes through the filtering support implemented by ebtables and only afterwards through iptables or ip6tables filters. If a filter tree has rules with the protocols including: mac, stp, vlan arp, rarp, ipv4, or ipv6; the ebtable rules and values listed will automatically be used first.
Multiple chains for the same protocol can be created. The name of the chain must have a prefix of one of the previously enumerated protocols. To create an additional chain for handling of ARP traffic, a chain with name arp-test, can for example be specified.
As an example, it is possible to filter on UDP traffic by source and destination ports using the ip protocol filter and specifying attributes for the protocol, source and destination IP addresses and ports of UDP packets that are to be accepted. This allows early filtering of UDP traffic with ebtables. However, once an IP or IPv6 packet, such as a UDP packet, has passed the ebtables layer and there is at least one rule in a filter tree that instantiates iptables or ip6tables rules, a rule to let the UDP packet pass will also be necessary to be provided for those filtering layers. This can be achieved with a rule containing an appropriate udp or udp-ipv6 traffic filtering node.

Example 20.11. Creating a custom filter

Suppose a filter is needed to fulfill the following list of requirements:
  • prevents a VM's interface from MAC, IP and ARP spoofing
  • opens only TCP ports 22 and 80 of a VM's interface
  • allows the VM to send ping traffic from an interface but not let the VM be pinged on the interface
  • allows the VM to do DNS lookups (UDP towards port 53)
The requirement to prevent spoofing is fulfilled by the existing clean-traffic network filter, thus the way to do this is to reference it from a custom filter.
To enable traffic for TCP ports 22 and 80, two rules are added to enable this type of traffic. To allow the guest virtual machine to send ping traffic a rule is added for ICMP traffic. For simplicity reasons, general ICMP traffic will be allowed to be initiated from the guest virtual machine, and will not be specified to ICMP echo request and response messages. All other traffic will be prevented to reach or be initiated by the guest virtual machine. To do this a rule will be added that drops all other traffic. Assuming the guest virtual machine is called test and the interface to associate our filter with is called eth0, a filter is created named test-eth0.
The result of these considerations is the following network filter XML:
<filter name='test-eth0'>
  <!- - This rule references the clean traffic filter to prevent MAC, IP and ARP spoofing. By not providing an IP address parameter, libvirt will detect the IP address the guest virtual machine is using. - ->
  <filterref filter='clean-traffic'/>

  <!- - This rule enables TCP ports 22 (ssh) and 80 (http) to be reachable - ->
  <rule action='accept' direction='in'>
    <tcp dstportstart='22'/>
  </rule>

  <rule action='accept' direction='in'>
    <tcp dstportstart='80'/>
  </rule>

  <!- - This rule enables general ICMP traffic to be initiated by the guest virtual machine including ping traffic - ->
  <rule action='accept' direction='out'>
    <icmp/>
  </rule>>

  <!- - This rule enables outgoing DNS lookups using UDP - ->
  <rule action='accept' direction='out'>
    <udp dstportstart='53'/>
  </rule>

  <!- - This rule drops all other traffic - ->
  <rule action='drop' direction='inout'>
    <all/>
  </rule>

</filter>

20.14.11.6. Sample custom filter

Although one of the rules in the above XML contains the IP address of the guest virtual machine as either a source or a destination address, the filtering of the traffic works correctly. The reason is that whereas the rule's evaluation occurs internally on a per-interface basis, the rules are additionally evaluated based on which (tap) interface has sent or will receive the packet, rather than what their source or destination IP address may be.

Example 20.12. Sample XML for network interface descriptions

An XML fragment for a possible network interface description inside the domain XML of the test guest virtual machine could then look like this:
   [...]
    <interface type='bridge'>
      <source bridge='mybridge'/>
      <filterref filter='test-eth0'/>
    </interface>
   [...]
To more strictly control the ICMP traffic and enforce that only ICMP echo requests can be sent from the guest virtual machine and only ICMP echo responses be received by the guest virtual machine, the above ICMP rule can be replaced with the following two rules:
  <!- - enable outgoing ICMP echo requests- ->
  <rule action='accept' direction='out'>
    <icmp type='8'/>
  </rule>
  <!- - enable incoming ICMP echo replies- ->
  <rule action='accept' direction='in'>
    <icmp type='0'/>
  </rule>

Example 20.13. Second example custom filter

This example demonstrates how to build a similar filter as in the example above, but extends the list of requirements with an ftp server located inside the guest virtual machine. The requirements for this filter are:
  • prevents a guest virtual machine's interface from MAC, IP, and ARP spoofing
  • opens only TCP ports 22 and 80 in a guest virtual machine's interface
  • allows the guest virtual machine to send ping traffic from an interface but does not allow the guest virtual machine to be pinged on the interface
  • allows the guest virtual machine to do DNS lookups (UDP towards port 53)
  • enables the ftp server (in active mode) so it can run inside the guest virtual machine
The additional requirement of allowing an FTP server to be run inside the guest virtual machine maps into the requirement of allowing port 21 to be reachable for FTP control traffic as well as enabling the guest virtual machine to establish an outgoing TCP connection originating from the guest virtual machine's TCP port 20 back to the FTP client (FTP active mode). There are several ways of how this filter can be written and two possible solutions are included in this example.
The first solution makes use of the state attribute of the TCP protocol that provides a hook into the connection tracking framework of the Linux host physical machine. For the guest virtual machine-initiated FTP data connection (FTP active mode) the RELATED state is used to enable detection that the guest virtual machine-initiated FTP data connection is a consequence of ( or 'has a relationship with' ) an existing FTP control connection, thereby allowing it to pass packets through the firewall. The RELATED state, however, is only valid for the very first packet of the outgoing TCP connection for the FTP data path. Afterwards, the state is ESTABLISHED, which then applies equally to the incoming and outgoing direction. All this is related to the FTP data traffic originating from TCP port 20 of the guest virtual machine. This then leads to the following solution:
<filter name='test-eth0'>
  <!- - This filter (eth0) references the clean traffic filter to prevent MAC, IP, and ARP spoofing. By not providing an IP address parameter, libvirt will detect the IP address the guest virtual machine is using. - ->
  <filterref filter='clean-traffic'/>

  <!- - This rule enables TCP port 21 (FTP-control) to be reachable - ->
  <rule action='accept' direction='in'>
    <tcp dstportstart='21'/>
  </rule>

  <!- - This rule enables TCP port 20 for guest virtual machine-initiated FTP data connection related to an existing FTP control connection - ->
  <rule action='accept' direction='out'>
    <tcp srcportstart='20' state='RELATED,ESTABLISHED'/>
  </rule>

  <!- - This rule accepts all packets from a client on the FTP data connection - ->
  <rule action='accept' direction='in'>
    <tcp dstportstart='20' state='ESTABLISHED'/>
  </rule>

  <!- - This rule enables TCP port 22 (SSH) to be reachable - ->
  <rule action='accept' direction='in'>
    <tcp dstportstart='22'/>
  </rule>

  <!- -This rule enables TCP port 80 (HTTP) to be reachable - ->
  <rule action='accept' direction='in'>
    <tcp dstportstart='80'/>
  </rule>

  <!- - This rule enables general ICMP traffic to be initiated by the guest virtual machine, including ping traffic - ->
  <rule action='accept' direction='out'>
    <icmp/>
  </rule>

  <!- - This rule enables outgoing DNS lookups using UDP - ->
  <rule action='accept' direction='out'>
    <udp dstportstart='53'/>
  </rule>

  <!- - This rule drops all other traffic - ->
  <rule action='drop' direction='inout'>
    <all/>
  </rule>

</filter>
Before trying out a filter using the RELATED state, you have to make sure that the appropriate connection tracking module has been loaded into the host physical machine's kernel. Depending on the version of the kernel, you must run either one of the following two commands before the FTP connection with the guest virtual machine is established:
  • #modprobe nf_conntrack_ftp - where available OR
  • #modprobe ip_conntrack_ftp if above is not available
If protocols other than FTP are used in conjunction with the RELATED state, their corresponding module must be loaded. Modules are available for the protocols: ftp, tftp, irc, sip, sctp, and amanda.
The second solution makes use of the state flags of connections more than the previous solution did. This solution takes advantage of the fact that the NEW state of a connection is valid when the very first packet of a traffic flow is detected. Subsequently, if the very first packet of a flow is accepted, the flow becomes a connection and thus enters into the ESTABLISHED state. Therefore a general rule can be written for allowing packets of ESTABLISHED connections to reach the guest virtual machine or be sent by the guest virtual machine. This is done writing specific rules for the very first packets identified by the NEW state and dictates the ports that the data is acceptable. All packets meant for ports that are not explicitly accepted are dropped, thus not reaching an ESTABLISHED state. Any subsequent packets sent from that port are dropped as well.
<filter name='test-eth0'>
  <!- - This filter references the clean traffic filter to prevent MAC, IP and ARP spoofing. By not providing and IP address parameter, libvirt will detect the IP address the VM is using. - ->
  <filterref filter='clean-traffic'/>

  <!- - This rule allows the packets of all previously accepted connections to reach the guest virtual machine - ->
  <rule action='accept' direction='in'>
    <all state='ESTABLISHED'/>
  </rule>

  <!- - This rule allows the packets of all previously accepted and related connections be sent from the guest virtual machine - ->
  <rule action='accept' direction='out'>
    <all state='ESTABLISHED,RELATED'/>
  </rule>

  <!- - This rule enables traffic towards port 21 (FTP) and port 22 (SSH)- ->
  <rule action='accept' direction='in'>
    <tcp dstportstart='21' dstportend='22' state='NEW'/>
  </rule>

  <!- - This rule enables traffic towards port 80 (HTTP) - ->
  <rule action='accept' direction='in'>
    <tcp dstportstart='80' state='NEW'/>
  </rule>

  <!- - This rule enables general ICMP traffic to be initiated by the guest virtual machine, including ping traffic - ->
  <rule action='accept' direction='out'>
    <icmp state='NEW'/>
  </rule>

  <!- - This rule enables outgoing DNS lookups using UDP - ->
  <rule action='accept' direction='out'>
    <udp dstportstart='53' state='NEW'/>
  </rule>

  <!- - This rule drops all other traffic - ->
  <rule action='drop' direction='inout'>
    <all/>
  </rule>

</filter>

20.14.12. Limitations

The following is a list of the currently known limitations of the network filtering subsystem.
  • VM migration is only supported if the whole filter tree that is referenced by a guest virtual machine's top level filter is also available on the target host physical machine. The network filter clean-traffic for example should be available on all libvirt installations and thus enable migration of guest virtual machines that reference this filter. To assure version compatibility is not a problem make sure you are using the most current version of libvirt by updating the package regularly.
  • Migration must occur between libvirt insallations of version 0.8.1 or later in order not to lose the network traffic filters associated with an interface.
  • VLAN (802.1Q) packets, if sent by a guest virtual machine, cannot be filtered with rules for protocol IDs arp, rarp, ipv4 and ipv6. They can only be filtered with protocol IDs, MAC and VLAN. Therefore, the example filter clean-traffic Example 20.1, “An example of network filtering” will not work as expected.

20.15. Creating Tunnels

This section will demonstrate how to implement different tunneling scenarios.

20.15.1. Creating Multicast Tunnels

A multicast group is setup to represent a virtual network. Any guest virtual machines whose network devices are in the same multicast group can talk to each other even across host physical machines. This mode is also available to unprivileged users. There is no default DNS or DHCP support and no outgoing network access. To provide outgoing network access, one of the guest virtual machines should have a second NIC which is connected to one of the first four network types thus providing appropriate routing. The multicast protocol is compatible the guest virtual machine user mode. Note that the source address that you provide must be from the address used fot the multicast address block.
To create a multicast tunnel place the following XML details into the <devices> element:
      
  ...
  <devices>
    <interface type='mcast'>
      <mac address='52:54:00:6d:90:01'>
      <source address='230.0.0.1' port='5558'/>
    </interface>
  </devices>
  ...


Figure 20.25. Multicast tunnel domain XMl example

20.15.2. Creating TCP Tunnels

A TCP client-server architecture provides a virtual network. In this configuration, one guest virtual machine provides the server end of the network while all other guest virtual machines are configured as clients. All network traffic is routed between the guest virtual machine clients via the guest virtual machine server. This mode is also available for unprivileged users. Note that this mode does not provide default DNS or DHCP support nor does it provide outgoing network access. To provide outgoing network access, one of the guest virtual machines should have a second NIC which is connected to one of the first four network types thus providing appropriate routing.
To create a TCP tunnel place the following XML details into the <devices> element:
      
       ...
  <devices>
    <interface type='server'>
      <mac address='52:54:00:22:c9:42'>
      <source address='192.168.0.1' port='5558'/>
    </interface>
    ...
    <interface type='client'>
      <mac address='52:54:00:8b:c9:51'>
      <source address='192.168.0.1' port='5558'/>
    </interface>
  </devices>
  ...
      

Figure 20.26. TCP tunnel domain XMl example

20.16. Setting vLAN Tags

virtual local area network (vLAN) tags are added using the virsh net-edit command. This tag can also be used with PCI device assignment with SR-IOV devices. For more information, refer to Section 18.1.6.1, “Configuring PCI Assignment with SR-IOV Devices”.

<network>
  <name>ovs-net</name>
  <forward mode='bridge'/>
  <bridge name='ovsbr0'/>
  <virtualport type='openvswitch'>
    <parameters interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
  </virtualport>
  <vlan trunk='yes'>
    <tag id='42' nativeMode='untagged'/>
    <tag id='47'/>
  </vlan>
  <portgroup name='dontpanic'>
    <vlan>
      <tag id='42'/>
    </vlan>
  </portgroup>
</network>

Figure 20.27. vSetting VLAN tag (on supported network types only)

If (and only if) the network type supports vlan tagging transparent to the guest, an optional <vlan> element can specify one or more vlan tags to apply to the traffic of all guests using this network. (openvswitch and type='hostdev' SR-IOV networks do support transparent vlan tagging of guest traffic; everything else, including standard linux bridges and libvirt's own virtual networks, do not support it. 802.1Qbh (vn-link) and 802.1Qbg (VEPA) switches provide their own way (outside of libvirt) to tag guest traffic onto specific vlans.) As expected, the tag attribute specifies which vlan tag to use. If a network has more than one <vlan> element defined, it is assumed that the user wants to do VLAN trunking using all the specified tags. In the case that vlan trunking with a single tag is desired, the optional attribute trunk='yes' can be added to the vlan element.
For network connections using openvswitch it is possible to configure the 'native-tagged' and 'native-untagged' vlan modes. This uses the optional nativeMode attribute on the <tag> element: nativeMode may be set to 'tagged' or 'untagged'. The id attribute of the element sets the native vlan.
<vlan> elements can also be specified in a <portgroup> element, as well as directly in a domain's <interface> element. In the case that a vlan tag is specified in multiple locations, the setting in <interface> takes precedence, followed by the setting in the <portgroup> selected by the interface config. The <vlan> in <network> will be selected only if none is given in <portgroup> or <interface>.

20.17. Applying QoS to Your Virtual Network

Quality of Service (QoS) refers to the resource control systems that guarantees an optimal experience for all users on a network, making sure that there is no delay, jitter, or packet loss. QoS can be application specific or user / group specific. Refer to Section 26.18.9.14, “Quality of service (QoS)” for more information.

Chapter 21. Remote Management of Guests

This section explains how to remotely manage your guests.

21.1. Transport Modes

For remote management, libvirt supports the following transport modes:
Transport Layer Security (TLS)
Transport Layer Security TLS 1.0 (SSL 3.1) authenticated and encrypted TCP/IP socket, usually listening on a public port number. To use this, you will need to generate client and server certificates. The standard port is 16514. For detailed instructions, see Section 21.3, “Remote Management over TLS and SSL”.
SSH
Transported over a Secure Shell protocol (SSH) connection. The libvirt daemon (libvirtd) must be running on the remote machine. Port 22 must be open for SSH access. You should use some sort of SSH key management (for example, the ssh-agent utility) or you will be prompted for a password. For detailed instructions, see Section 21.2, “Remote Management with SSH”.
UNIX Sockets
UNIX domain sockets are only accessible on the local machine. Sockets are not encrypted, and use UNIX permissions or SELinux for authentication. The standard socket names are /var/run/libvirt/libvirt-sock and /var/run/libvirt/libvirt-sock-ro (for read-only connections).
ext
The ext parameter is used for any external program which can make a connection to the remote machine by means outside the scope of libvirt. This parameter is unsupported.
TCP
Unencrypted TCP/IP socket. Not recommended for production use, this is normally disabled, but an administrator can enable it for testing or use over a trusted network. The default port is 16509.
The default transport, if no other is specified, is TLS.
Remote URIs
A Uniform Resource Identifier (URI) is used by virsh and libvirt to connect to a remote host. URIs can also be used with the --connect parameter for the virsh command to execute single commands or migrations on remote hosts. Remote URIs are formed by taking ordinary local URIs and adding a host name or a transport name, or both. As a special case, using a URI scheme of 'remote', will tell the remote libvirtd server to probe for the optimal hypervisor driver. This is equivalent to passing a NULL URI for a local connection
libvirt URIs take the general form (content in square brackets, "[]", represents optional functions):
driver[+transport]://[username@][hostname][:port]/path[?extraparameters]
Note that if the hypervisor (driver) is QEMU, the path is mandatory.
The following are examples of valid remote URIs:
  • qemu://hostname/
The transport method or the host name must be provided to target an external location. For more information, refer to http://libvirt.org/guide/html/Application_Development_Guide-Architecture-Remote_URIs.html.

Examples of remote management parameters

  • Connect to a remote KVM host named host2, using SSH transport and the SSH user name virtuser. The connect command for each is connect [<name>] [--readonly], where <name> is a valid URI. For more information about the virsh connect command, refer to Section 23.7, “Connecting to the Hypervisor with virsh Connect”
    qemu+ssh://virtuser@host2/
  • Connect to a remote KVM hypervisor on the host named host2 using TLS.
    qemu://host2/

Testing examples

  • Connect to the local KVM hypervisor with a non-standard UNIX socket. The full path to the UNIX socket is supplied explicitly in this case.
    qemu+unix:///system?socket=/opt/libvirt/run/libvirt/libvirt-sock
  • Connect to the libvirt daemon with an non-encrypted TCP/IP connection to the server with the IP address 10.1.1.10 on port 5000. This uses the test driver with default settings.
    test+tcp://10.1.1.10:5000/default
Extra URI Parameters
Extra parameters can be appended to remote URIs. The table below covers the recognized parameters. All other parameters are ignored. Note that parameter values must be URI-escaped (that is, a question mark (?) is appended before the parameter and special characters are converted into the URI format).

Table 21.1. Extra URI parameters

Name Transport mode Description Example usage
name all modes The name passed to the remote virConnectOpen function. The name is normally formed by removing transport, hostname, port number, username, and extra parameters from the remote URI, but in certain very complex cases it may be better to supply the name explicitly. name=qemu:///system
command ssh and ext The external command. For ext transport this is required. For ssh the default is ssh. The PATH is searched for the command. command=/opt/openssh/bin/ssh
socket unix and ssh The path to the UNIX domain socket, which overrides the default. For ssh transport, this is passed to the remote netcat command (see netcat). socket=/opt/libvirt/run/libvirt/libvirt-sock
no_verify tls If set to a non-zero value, this disables client checks of the server's certificate. Note that to disable server checks of the client's certificate or IP address you must change the libvirtd configuration. no_verify=1
no_tty ssh If set to a non-zero value, this stops ssh from asking for a password if it cannot log in to the remote machine automatically . Use this when you do not have access to a terminal . no_tty=1

21.2. Remote Management with SSH

The ssh package provides an encrypted network protocol that can securely send management functions to remote virtualization servers. The method described below uses the libvirt management connection, securely tunneled over an SSH connection, to manage the remote machines. All the authentication is done using SSH public key cryptography and passwords or passphrases gathered by your local SSH agent. In addition, the VNC console for each guest is tunneled over SSH.
When using using SSH for remotely managing your virtual machines, be aware of the following problems:
  • You require root log in access to the remote machine for managing virtual machines.
  • The initial connection setup process may be slow.
  • There is no standard or trivial way to revoke a user's key on all hosts or guests.
  • SSH does not scale well with larger numbers of remote machines.

Note

Red Hat Enterprise Virtualization enables remote management of large numbers of virtual machines. For further details, refer to the Red Hat Enterprise Virtualization documentation.
The following packages are required for SSH access:
  • openssh
  • openssh-askpass
  • openssh-clients
  • openssh-server
Configuring Password-less or Password-managed SSH Access for virt-manager
The following instructions assume you are starting from scratch and do not already have SSH keys set up. If you have SSH keys set up and copied to the other systems, you can skip this procedure.

Important

SSH keys are user-dependent and may only be used by their owners. A key's owner is the user who generated it. Keys may not be sharedacross different users.
virt-manager must be run by the user who owns the keys to connect to the remote host. That means, if the remote systems are managed by a non-root user, virt-manager must be run in unprivileged mode. If the remote systems are managed by the local root user, then the SSH keys must be owned and created by root.
You cannot manage the local host as an unprivileged user with virt-manager.
  1. Optional: Changing user

    Change user, if required. This example uses the local root user for remotely managing the other hosts and the local host.
    $ su -
  2. Generating the SSH key pair

    Generate a public key pair on the machine where virt-manager is used. This example uses the default key location, in the ~/.ssh/ directory.
    # ssh-keygen -t rsa
  3. Copying the keys to the remote hosts

    Remote login without a password, or with a pass-phrase, requires an SSH key to be distributed to the systems being managed. Use the ssh-copy-id command to copy the key to root user at the system address provided (in the example, root@host2.example.com).
    # ssh-copy-id -i ~/.ssh/id_rsa.pub root@host2.example.com
    root@host2.example.com's password:
    
    Afterwards, try logging into the machine and check the .ssh/authorized_keys file to make sure unexpected keys have not been added:
    ssh root@host2.example.com
    Repeat for other systems, as required.
  4. Optional: Add the passphrase to the ssh-agent

    Add the pass-phrase for the SSH key to the ssh-agent, if required. On the local host, use the following command to add the pass-phrase (if there was one) to enable password-less login.
    # ssh-add ~/.ssh/id_rsa
    This command will fail to run if the ssh-agent is not running. To avoid errors or conflicts, make sure that your SSH parameters are set correctly. Refer to the Red Hat Enterprise System Administration Guide for more information.
The libvirt daemon (libvirtd)
The libvirt daemon provides an interface for managing virtual machines. You must have the libvirtd daemon installed and running on every remote host that you intend to manage this way.
$ ssh root@somehost
# systemctl enable libvirtd.service
# systemctl start libvirtd.service
After libvirtd and SSH are configured, you should be able to remotely access and manage your virtual machines. You should also be able to access your guests with VNC at this point.
Accessing Remote Hosts with virt-manager
Remote hosts can be managed with the virt-manager GUI tool. SSH keys must belong to the user executing virt-manager for password-less login to work.
  1. Start virt-manager.
  2. Open the File->Add Connection menu.
    Add connection menu

    Figure 21.1. Add connection menu

  3. Use the drop down menu to select hypervisor type, and click the Connect to remote host check box to open the Connection Method (in this case Remote tunnel over SSH), and enter the desired User name and Hostname, then click Connect.

21.3. Remote Management over TLS and SSL

You can manage virtual machines using the TLS and SSL protocols. TLS and SSL provides greater scalability but is more complicated than SSH (refer to Section 21.2, “Remote Management with SSH”). TLS and SSL is the same technology used by web browsers for secure connections. The libvirt management connection opens a TCP port for incoming connections, which is securely encrypted and authenticated based on x509 certificates. The following procedures provide instructions on creating and deploying authentication certificates for TLS and SSL management.

Procedure 21.1. Creating a certificate authority (CA) key for TLS management

  1. Before you begin, confirm that gnutls-utils is installed. If not, install it:
    # yum install gnutls-utils 
  2. Generate a private key, using the following command:
    # certtool --generate-privkey > cakey.pem
  3. After the key is generated, create a signature file so the key can be self-signed. To do this, create a file with signature details and name it ca.info. This file should contain the following:
    cn = Name of your organization
    ca
    cert_signing_key
    
  4. Generate the self-signed key with the following command:
    # certtool --generate-self-signed --load-privkey cakey.pem --template ca.info --outfile cacert.pem
    After the file is generated, the ca.info file can be deleted using the rm command. The file that results from the generation process is named cacert.pem. This file is the public key (certificate). The loaded file cakey.pem is the private key. For security purposes, this file should be kept private, and not reside in a shared space.
  5. Install the cacert.pem CA certificate file on all clients and servers in the /etc/pki/CA/cacert.pem directory to let them know that the certificate issued by your CA can be trusted. To view the contents of this file, run:
    # certtool -i --infile cacert.pem
    This is all that is required to set up your CA. Keep the CA's private key safe as you will need it in order to issue certificates for your clients and servers.

Procedure 21.2. Issuing a server certificate

This procedure demonstrates how to issue a certificate with the X.509 Common Name (CN) field set to the host name of the server. The CN must match the host name which clients will be using to connect to the server. In this example, clients will be connecting to the server using the URI: qemu://mycommonname/system, so the CN field should be identical, for this example "mycommoname".
  1. Create a private key for the server.
    # certtool --generate-privkey > serverkey.pem
  2. Generate a signature for the CA's private key by first creating a template file called server.info. Make sure that the CN is set to be the same as the server's host name:
    organization = Name of your organization
    cn = mycommonname
    tls_www_server
    encryption_key
    signing_key
    
  3. Create the certificate:
    # certtool --generate-certificate --load-privkey serverkey.pem --load-ca-certificate cacert.pem --load-ca-privkey cakey.pem \ --template server.info --outfile servercert.pem
    This results in two files being generated:
    • serverkey.pem - The server's private key
    • servercert.pem - The server's public key
  4. Make sure to keep the location of the private key secret. To view the contents of the file, use the following command:
    # certtool -i --infile servercert.pem
    When opening this file, the CN= parameter should be the same as the CN that you set earlier. For example, mycommonname.
  5. Install the two files in the following locations:
    • serverkey.pem - the server's private key. Place this file in the following location: /etc/pki/libvirt/private/serverkey.pem
    • servercert.pem - the server's certificate. Install it in the following location on the server: /etc/pki/libvirt/servercert.pem

Procedure 21.3. Issuing a client certificate

  1. For every client (that is to say any program linked with libvirt, such as virt-manager), you need to issue a certificate with the X.509 Distinguished Name (DN) field set to a suitable name. This needs to be decided on a corporate level.
    For example purposes, the following information will be used:
    C=USA,ST=North Carolina,L=Raleigh,O=Red Hat,CN=name_of_client
  2. Create a private key:
    # certtool --generate-privkey > clientkey.pem
  3. Generate a signature for the CA's private key by first creating a template file called client.info. The file should contain the following (fields should be customized to reflect your region/location):
    country = USA
    state = North Carolina
    locality = Raleigh
    organization = Red Hat
    cn = client1
    tls_www_client
    encryption_key
    signing_key
    
  4. Sign the certificate with the following command:
    # certtool --generate-certificate --load-privkey clientkey.pem --load-ca-certificate cacert.pem \ --load-ca-privkey cakey.pem --template client.info --outfile clientcert.pem
  5. Install the certificates on the client machine:
    # cp clientkey.pem /etc/pki/libvirt/private/clientkey.pem
    # cp clientcert.pem /etc/pki/libvirt/clientcert.pem

21.4. Configuring a VNC Server

To configure a VNC server, use the Remote Desktop Viewer in Application > Utilities. Alternatively, you can run the vinagre command.
Use the following step set up a dedicated VNC server session:
If needed, create and then edit the ~/.vnc/xstartup file to start a GNOME session whenever vncserver is started. The first time you run the vncserver script, it will ask you for a password you want to use for your VNC session. For more information on VNC server files, refer to the Red Hat Enterprise Linux Installation Guide.

21.5. Enhancing Remote Management of Virtual Machines with NSS

In Red Hat Enterprise Linux 7.3 and later, you can use the libvirt Network Security Services (NSS) module to make it easier to connect to guests with SSH, TLS, SSL, as well as other remote login services. In addition, the module also benefits utilities that use host name translation, such as ping.
To be able to use this functionality, install the libvirt-nss package:
# yum install libvirt-nss
Finally, enable the module by adding libvirt to the hosts line of the /etc/nsswitch.conf file, for example as follows:
passwd:      compat
shadow:      compat
group:       compat
hosts:       files libvirt dns
...
As a result, libvirt's NSS module will be added to modules that translate host domain names to IP addresses, which for example enables connecting to a remote guest in NAT mode without setting a static IP address. The order in which modules are listed on the hosts line determines the order in which these modules are consulted to find the specified remote guest.

Chapter 22. Managing Guests with the Virtual Machine Manager (virt-manager)

This chapter describes the Virtual Machine Manager (virt-manager) windows, dialog boxes, and various GUI controls.
virt-manager provides a graphical view of hypervisors and guests on your host system and on remote host systems. virt-manager can perform virtualization management tasks, including:
  • defining and creating guests,
  • assigning memory,
  • assigning virtual CPUs,
  • monitoring operational performance,
  • saving and restoring, pausing and resuming, and shutting down and starting guests,
  • links to the textual and graphical consoles, and
  • live and offline migrations.

Important

It is important to note which user you are using. If you create a guest virtual machine with one user you will not be able to retrieve information about it using another user. This is especially important when you create a virtual machine in virt-manager. The default user is root in that case unless otherwise specified. Should you have a case where you cannot list the virtual machine using the virsh list --all command, it is most likely due to you running the command using a different user than you used to create the virtual machine.

22.1. Starting virt-manager

To start virt-manager session open the Applications menu, then the System Tools menu and select Virtual Machine Manager (virt-manager).
The virt-manager main window appears.
Starting virt-manager

Figure 22.1. Starting virt-manager

Alternatively, virt-manager can be started remotely using ssh as demonstrated in the following command:
ssh -X host's address
[remotehost]# virt-manager
Using ssh to manage virtual machines and hosts is discussed further in Section 21.2, “Remote Management with SSH”.

22.2. The Virtual Machine Manager Main Window

This main window displays all the running guests and resources used by guests. Select a guest by double clicking the guest's name.
Virtual Machine Manager main window

Figure 22.2. Virtual Machine Manager main window

22.3. The Virtual Hardware Details Window

The virtual hardware details window displays information about the virtual hardware configured for the guest. Virtual hardware resources can be added, removed and modified in this window. To access the virtual hardware details window, click on the icon in the toolbar.
The virtual hardware details icon

Figure 22.3. The virtual hardware details icon

Clicking the icon displays the virtual hardware details window.
The virtual hardware details window

Figure 22.4. The virtual hardware details window

22.3.1. Applying Boot Options to Guest Virtual Machines

Using virt-manager you can select how the guest virtual machine will act on boot. The boot options will not take effect until the guest virtual machine reboots. You can either power down the virtual machine before making any changes, or you can reboot the machine afterwards. If you do not do either of these options, the changes will happen the next time the guest reboots.

Procedure 22.1. Configuring boot options

  1. From the Virtual Machine Manager Edit menu, select Virtual Machine Details.
  2. From the side panel, select Boot Options and then complete any or all of the following optional steps:
    1. To indicate that this guest virtual machine should start each time the host physical machine boots, select the Autostart check box.
    2. To indicate the order in which guest virtual machine should boot, click the Enable boot menu check box. After this is checked, you can then check the devices you want to boot from and using the arrow keys change the order that the guest virtual machine will use when booting.
    3. If you want to boot directly from the Linux kernel, expand the Direct kernel boot menu. Fill in the Kernel path, Initrd path, and the Kernel arguments that you want to use.
  3. Click Apply.
    Configuring boot options

    Figure 22.5. Configuring boot options

22.3.2. Attaching USB Devices to a Guest Virtual Machine

Note

In order to attach the USB device to the guest virtual machine, you first must attach it to the host physical machine and confirm that the device is working. If the guest is running, you need to shut it down before proceeding.

Procedure 22.2. Attaching USB devices using Virt-Manager

  1. Open the guest virtual machine's Virtual Machine Details screen.
  2. Click Add Hardware
  3. In the Add New Virtual Hardware popup, select USB Host Device, select the device you want to attach from the list and Click Finish.
    Add USB Device

    Figure 22.6. Add USB Device

  4. To use the USB device in the guest virtual machine, start the guest virtual machine.

22.3.3. USB Redirection

USB re-direction is best used in cases where there is a host physical machine that is running in a data center. The user connects to his/her guest virtual machine from a local machine or thin client. On this local machine there is a SPICE client. The user can attach any USB device to the thin client and the SPICE client will redirect the device to the host physical machine on the data center so it can be used by the VM that is running on the thin client.

Procedure 22.3. Redirecting USB devices

  1. Open the guest virtual machine's Virtual Machine Details screen.
  2. Click Add Hardware
  3. In the Add New Virtual Hardware popup, select USB Redirection. Make sure to select Spice channel from the Type drop-down menu and click Finish.
    Add New Virtual Hardware window

    Figure 22.7. Add New Virtual Hardware window

  4. Open the Virtual Machine menu and select Redirect USB device. A pop-up window opens with a list of USB devices.
    Select a USB device

    Figure 22.8. Select a USB device

  5. Select a USB device for redirection by checking its check box and click OK.

22.4. Virtual Machine Graphical Console

This window displays a guest's graphical console. Guests can use several different protocols to export their graphical frame buffers: virt-manager supports VNC and SPICE. If your virtual machine is set to require authentication, the Virtual Machine graphical console prompts you for a password before the display appears.
Graphical console window

Figure 22.9. Graphical console window

Note

VNC is considered insecure by many security experts, however, several changes have been made to enable the secure usage of VNC for virtualization on Red Hat enterprise Linux. The guest machines only listen to the local host's loopback address (127.0.0.1). This ensures only those with shell privileges on the host can access virt-manager and the virtual machine through VNC. Although virt-manager is configured to listen to other public network interfaces and alternative methods can be configured, it is not recommended.
Remote administration can be performed by tunneling over SSH which encrypts the traffic. Although VNC can be configured to access remotely without tunneling over SSH, for security reasons, it is not recommended. To remotely administer the guest follow the instructions in: Chapter 21, Remote Management of Guests. TLS can provide enterprise level security for managing guest and host systems.
Your local desktop can intercept key combinations (for example, Ctrl+Alt+F1) to prevent them from being sent to the guest machine. You can use the Send key menu option to send these sequences. From the guest machine window, click the Send key menu and select the key sequence to send. In addition, from this menu you can also capture the screen output.
SPICE is an alternative to VNC available for Red Hat Enterprise Linux.

22.5. Adding a Remote Connection

This procedure covers how to set up a connection to a remote system using virt-manager.
  1. To create a new connection open the File menu and select the Add Connection... menu item.
  2. The Add Connection wizard appears. Select the hypervisor. For Red Hat Enterprise Linux 7 systems select QEMU/KVM. Select Local for the local system or one of the remote connection options and click Connect. This example uses Remote tunnel over SSH which works on default installations. For more information on configuring remote connections refer to Chapter 21, Remote Management of Guests
    Add Connection

    Figure 22.10. Add Connection

  3. Enter the root password for the selected host when prompted.
A remote host is now connected and appears in the main virt-manager window.
Remote host in the main virt-manager window

Figure 22.11. Remote host in the main virt-manager window

22.6. Displaying Guest Details

You can use the Virtual Machine Monitor to view activity information for any virtual machines on your system.
To view a virtual system's details:
  1. In the Virtual Machine Manager main window, highlight the virtual machine that you want to view.
    Selecting a virtual machine to display

    Figure 22.12. Selecting a virtual machine to display

  2. From the Virtual Machine Manager Edit menu, select Virtual Machine Details.
    When the Virtual Machine details window opens, there may be a console displayed. Should this happen, click View and then select Details. The Overview window opens first by default. To go back to this window, select Overview from the navigation pane on the left hand side.
    The Overview view shows a summary of configuration details for the guest.
    Displaying guest details overview

    Figure 22.13. Displaying guest details overview

  3. Select Performance from the navigation pane on the left hand side.
    The Performance view shows a summary of guest performance, including CPU and Memory usage.
    Displaying guest performance details

    Figure 22.14. Displaying guest performance details

  4. Select CPUs from the navigation pane on the left hand side. The CPUs view allows you to view or change the current processor allocation.
    It is also possible to increase the number of virtual CPUs (vCPUs) while the virtual machine is running, which is referred to as hot plugging.

    Important

    Hot unplugging vCPUs is not currently supported in Red Hat Enterprise Linux 7.
    Processor allocation panel

    Figure 22.15. Processor allocation panel

  5. Select Memory from the navigation pane on the left hand side. The Memory view allows you to view or change the current memory allocation.
    Displaying memory allocation

    Figure 22.16. Displaying memory allocation

  6. Select Boot Options from the navigation pane on the left hand side. The Boot Options view allows you to view or change the boot options including whether or not the virtual machine starts when the host boots and the boot device order for the virtual machine.
    Displaying boot options

    Figure 22.17. Displaying boot options

  7. Each virtual disk attached to the virtual machine is displayed in the navigation pane. Click on a virtual disk to modify or remove it.
    Displaying disk configuration

    Figure 22.18. Displaying disk configuration

  8. Each virtual network interface attached to the virtual machine is displayed in the navigation pane. Click on a virtual network interface to modify or remove it.
    Displaying network configuration

    Figure 22.19. Displaying network configuration

22.7. Managing Snapshots

Using virt-manager, it is possible to create, run, and delete guest snapshots. A snapshot is a saved image of the guest's hard disk, memory, and device state at a single point in time. After a snapshot is created, the guest can be returned to the snapshot's configuration at any time.
  • To manage snapshots in virt-manager, click on the guest console.
    The snapshot management interface opens:
  • To create a new snapshot, click the button under the snapshot list. In the snapshot creation interface, input the name of the snapshot and, optionally, a comment, and click Finish.
  • To revert the guest to a snapshot's configuration, select the snapshot and click .
  • To remove the selected snapshot, click .

Warning

Creating and running live snapshots, that is while the virtual machine is running, is only supported with qcow2 disk images.

Note

For more in-depth snapshot management, use the virsh snapshot-create command. See Section 23.45, “Managing Snapshots” for details about managing snapshots with virsh.

22.8. Performance Monitoring

Performance monitoring preferences can be modified with virt-manager's preferences window.
To configure performance monitoring:
  1. From the Edit menu, select Preferences.
    The Preferences window appears.
  2. From the Polling tab specify the time in seconds or stats polling options.
    Configuring performance monitoring

    Figure 22.20. Configuring performance monitoring

22.9. Displaying CPU Usage for Guests

To view the CPU usage for all guests on your system:
  1. From the View menu, select Graph, then the Guest CPU Usage check box.
  2. The Virtual Machine Manager shows a graph of CPU usage for all virtual machines on your system.
    Guest CPU usage graph

    Figure 22.21. Guest CPU usage graph

22.10. Displaying CPU Usage for Hosts

To view the CPU usage for all hosts on your system:
  1. From the View menu, select Graph, then the Host CPU Usage check box.
  2. The Virtual Machine Manager shows a graph of host CPU usage on your system.
    Host CPU usage graph

    Figure 22.22. Host CPU usage graph

22.11. Displaying Disk I/O

To view the disk I/O for all virtual machines on your system:
  1. Make sure that the Disk I/O statistics collection is enabled. To do this, from the Edit menu, select Preferences and click the Polling tab.
  2. Select the Disk I/O check box.
    Enabling Disk I/O

    Figure 22.23. Enabling Disk I/O

  3. To enable the Disk I/O display, from the View menu, select Graph, then the Disk I/O check box.
  4. The Virtual Machine Manager shows a graph of Disk I/O for all virtual machines on your system.
    Displaying Disk I/O

    Figure 22.24. Displaying Disk I/O

22.12. Displaying Network I/O

To view the network I/O for all virtual machines on your system:
  1. Make sure that the Network I/O statistics collection is enabled. To do this, from the Edit menu, select Preferences and click the Pollingtab.
  2. Select the Network I/O check box.
    Enabling Network I/O

    Figure 22.25. Enabling Network I/O

  3. To display the Network I/O statistics, from the View menu, select Graph, then the Network I/O check box.
  4. The Virtual Machine Manager shows a graph of Network I/O for all virtual machines on your system.
    Displaying Network I/O

    Figure 22.26. Displaying Network I/O

22.13. Displaying Memory Usage

To view the memory usage for all virtual machines on your system:
  1. Make sure that the memory usage statistics collection is enabled. To do this, from the Edit menu, select Preferences and click the Pollingtab.
  2. Select the Poll Memory stats check box.
    Enabling memory usage

    Figure 22.27. Enabling memory usage

  3. To display the memory usage, from the View menu, select Graph, then the Memory Usage check box.
  4. The Virtual Machine Manager lists the percentage of memory in use (in megabytes) for all virtual machines on your system.
    Displaying memory usage

    Figure 22.28. Displaying memory usage

Chapter 23. Managing Guest Virtual Machines with virsh

23.1. Guest Virtual Machine States and Types
23.2. Running the virsh Program
23.3. Interactive Mode Commands
23.3.1. Quitting and Exiting from the virsh Interactive Terminal
23.3.2. Changing and Displaying Directories in a Virtual Machine
23.4. Displaying the virsh Version
23.5. Getting Help
23.6. Sending Commands with echo
23.7. Connecting to the Hypervisor with virsh Connect
23.8. Displaying Information about a Guest Virtual Machine
23.9. Starting, Resuming, and Restoring a Virtual Machine
23.9.1. Starting a Guest Virtual Machine
23.9.2. Configuring a Virtual Machine to be Started Automatically at Boot
23.9.3. Rebooting a Guest Virtual Machine
23.9.4. Restoring a Guest Virtual Machine
23.9.5. Resuming a Guest Virtual Machine
23.9.6. Restoring a Guest Virtual Machine from a Suspended State
23.10. Managing a Virtual Machine Configuration
23.10.1. Saving a Guest Virtual Machine's Configuration
23.10.2. Defining a Guest Virtual Machine with an XML File
23.10.3. Updating the XML File That will be Used for Restoring a Guest Virtual Machine
23.10.4. Extracting the Guest Virtual Machine XML File
23.10.5. Editing the Guest Virtual Machine Configuration
23.11. Shutting off, Shutting down, Rebooting, and Forcing a Shutdown of a Guest Virtual Machine
23.11.1. Suspending a Running Guest Virtual Machine Using Power Management
23.11.2. Shutting down a Guest Virtual Machine
23.11.3. Suspending a Guest Virtual Machine
23.11.4. Resetting a Virtual Machine
23.11.5. Stopping a Running Guest Virtual Machine in Order to Restart It Later
23.12. Removing and Deleting a Virtual Machine
23.12.1. Undefining a Virtual Machine
23.12.2. Forcing a Guest Virtual Machine to Stop
23.13. Connecting the Serial Console for the Guest Virtual Machine
23.14. Injecting NMI
23.15. Retrieving Information about Your Virtual Machine
23.15.1. Displaying Device Block Statistics
23.15.2. Retrieving Network Statistics
23.15.3. Modifying the Link State of a Guest Virtual Machine's Virtual Interface
23.15.4. Listing the Link State of a Guest Virtual Machine's Virtual Interface
23.15.5. Setting Network Interface Bandwidth Parameters
23.15.6. Retrieving Memory Statistics
23.15.7. Displaying Errors on Block Devices
23.15.8. Displaying the Block Device Size
23.15.9. Displaying the Block Devices Associated with a Guest Virtual Machine
23.15.10. Displaying Virtual Interfaces Associated with a Guest Virtual Machine
23.16. Working with Snapshots
23.16.1. Shortening a Backing Chain by Coping the Data
23.16.2. Shortening a Backing Chain by Flattening the Image
23.16.3. Changing the Size of a Guest Virtual Machine's Block Device
23.17. Displaying a URI for Connection to a Graphical Display
23.18. Discarding Blocks Not in Use
23.19. Guest Virtual Machine Retrieval Commands
23.19.1. Displaying the Host Physical Machine Name
23.19.2. Displaying General Information about a Virtual Machine
23.19.3. Displaying a Virtual Machine's ID Number
23.19.4. Aborting Running Jobs on a Guest Virtual Machine
23.19.5. Displaying Information about Jobs Running on the Guest Virtual Machine
23.19.6. Displaying the Guest Virtual Machine's Name
23.19.7. Displaying the Virtual Machine's State
23.19.8. Displaying the Connection State to the Virtual Machine
23.20. Converting QEMU Arguments to Domain XML
23.21. Creating a Dump File of a Guest Virtual Machine's Core
23.22. Creating a Virtual Machine XML Dump (Configuration File)
23.23. Creating a Guest Virtual Machine from a Configuration File
23.24. Editing a Guest Virtual Machine's Configuration Settings
23.25. Adding Multifunction PCI Devices to KVM Guest Virtual Machines
23.26. Displaying CPU Statistics for a Specified Guest Virtual Machine
23.27. Saving a Screenshot
23.28. Sending a Keystroke Combination to a Specified Guest Virtual machine
23.29. Sending Process Signal Names to Virtual Processes
23.30. Displaying the IP Address and Port Number for the VNC Display
23.31. NUMA Node Management
23.31.1. Displaying Node Information
23.31.2. Setting NUMA Parameters
23.31.3. Displaying the Amount of Free Memory in a NUMA Cell
23.31.4. Displaying a CPU List
23.31.5. Displaying CPU Statistics
23.31.6. Managing Devices
23.31.7. Suspending the Host Physical Machine
23.31.8. Setting and Displaying the Node Memory Parameters
23.31.9. Listing Devices on a Node
23.31.10. Creating Devices on Host Nodes
23.31.11. Removing a Device
23.31.12. Collect a Device Configuration Settings
23.31.13. Triggering a Reset for a Device
23.32. Retrieving Guest Virtual Machine Information
23.32.1. Getting the Domain ID of a Guest Virtual Machine
23.32.2. Getting the Domain Name of a Guest Virtual Machine
23.32.3. Getting the UUID of a Guest Virtual Machine
23.32.4. Displaying Guest Virtual Machine Information
23.33. Storage Pool Commands
23.33.1. Searching for a Storage Pool XML
23.33.2. Finding a storage Pool
23.33.3. Listing Storage Pool Information
23.33.4. Listing the Available Storage Pools
23.33.5. Refreshing a Storage Pool List
23.33.6. Creating, Defining, and Starting Storage Pools
23.33.7. Stopping and Deleting Storage Pools
23.33.8. Creating an XML Dump File for a Pool
23.33.9. Editing the Storage Pool's Configuration File
23.34. Storage Volume Commands
23.34.1. Creating Storage Volumes
23.34.2. Creating a Storage Volume from Parameters
23.34.3. Creating a Storage Volume from an XML File
23.34.4. Cloning a Storage Volume
23.35. Deleting Storage Volumes
23.36. Deleting a Storage Volume's Contents
23.37. Dumping Storage Volume Information to an XML File
23.38. Listing Volume Information
23.39. Retrieving Storage Volume Information
23.40. Uploading and Downloading Storage Volumes
23.41. Resizing Storage Volumes
23.42. Displaying Per-guest Virtual Machine Information
23.42.1. Displaying the Guest Virtual Machines
23.42.2. Displaying Virtual CPU Information
23.42.3. Pinning vCPU to a Host Physical Machine's CPU
23.42.4. Displaying Information about the Virtual CPU Counts of a Given Domain
23.42.5. Configuring Virtual CPU Affinity
23.42.6. Configuring Virtual CPU Count
23.42.7. Configuring Memory Allocation
23.42.8. Changing the Memory Allocation for the Domain
23.42.9. Displaying Guest Virtual Machine Block Device Information
23.42.10. Displaying Guest Virtual Machine Network Device Information
23.43. Managing Virtual Networks
23.43.1. Autostarting a Virtual Network
23.43.2. Creating a Virtual Network from an XML File
23.43.3. Defining a Virtual Network from an XML File
23.43.4. Stopping a Virtual Network
23.43.5. Creating a Dump File
23.43.6. Editing a Virtual Network's XML Configuration File
23.43.7. Getting Information about a Virtual Network
23.43.8. Listing Information about a Virtual Network
23.43.9. Converting a Network UUID to Network Name
23.43.10. Starting a (previously defined) Inactive Network
23.43.11. Undefining the Configuration for an Inactive Network
23.43.12. Converting a Network Name to Network UUID
23.43.13. Updating an Existing Network Definition File
23.43.14. Migrating Guest Virtual Machines with virsh
23.43.15. Setting a Static IP Address for the Guest Virtual Machine
23.44. Interface Commands
23.44.1. Defining and Starting a Host Physical Machine Interface via an XML File
23.44.2. Editing the XML Configuration File for the Host Interface
23.44.3. Listing Active Host Interfaces
23.44.4. Converting a MAC Address into an Interface Name
23.44.5. Stopping a Specific Host Physical Machine Interface
23.44.6. Displaying the Host Configuration File
23.44.7. Creating Bridge Devices
23.44.8. Tearing Down a Bridge Device
23.44.9. Manipulating Interface Snapshots
23.45. Managing Snapshots
23.45.1. Creating Snapshots
23.45.2. Creating a Snapshot for the Current Guest Virtual Machine
23.45.3. Displaying the Current Snapshot of a Guest Virtual Machine
23.45.4. snapshot-edit
23.45.5. snapshot-info
23.45.6. snapshot-list
23.45.7. snapshot-dumpxml
23.45.8. snapshot-parent
23.45.9. snapshot-revert
23.45.10. snapshot-delete
23.46. Guest Virtual Machine CPU Model Configuration
23.46.1. Introduction
23.46.2. Learning about the Host Physical Machine CPU Model
23.46.3. Determining Support for VFIO IOMMU Devices
23.46.4. Determining a Compatible CPU Model to Suit a Pool of Host Physical Machines
23.47. Configuring the Guest Virtual Machine CPU Model
23.48. Managing Resources for Guest Virtual Machines
23.49. Setting Schedule Parameters
23.50. Disk I/O Throttling
23.51. Display or Set Block I/O Parameters
23.52. Configuring Memory Tuning
virsh is a command-line interface tool for managing guest virtual machines, and works as the primary means of controlling virtualization on Red Hat Enterprise Linux 7. The virsh command-line tool is built on the libvirt management API, and can be used to create, deploy, and manage guest virtual machines. The virsh utility is ideal for creating virtualization administration scripts, and users without root privileges can use it in read-only mode. The virsh package is installed with yum as part of the libvirt-client package. For installation instructions refer to libvirt-client. The remaining sections of this chapter cover the virsh command set in a logical order based on usage.

Note

Note that when using the help or when reading the man pages, the term 'domain' will be used instead of the term guest virtual machine. This is the term used by libvirt. In cases where the screen output is displayed and the word 'domain' is used, it will not be switched to guest or guest virtual machine. In all examples, the guest virtual machine 'guest1' will be used. You should replace this with the name of your guest virtual machine in all cases. When creating a name for a guest virtual machine you should use a short easy to remember integer (0,1,2...), a text string name, or in all cases you can also use the virtual machine's full UUID.

Important

It is important to note which user you are using. If you create a guest virtual machine using one user you will not be able to retrieve information about it using another user. This is especially important when you create a virtual machine in virt-manager. The default user is root in that case unless otherwise specified. Should you have a case where you cannot list the virtual machine using the virsh list --all command, it is most likely due to you running the command using a different user than you used to create the virtual machine. Refer to Important for more information.

23.1. Guest Virtual Machine States and Types

There are two virtual machine classifications and five states. The two classifications are transient and persistent. A transient guest virtual machine is one that does not survive reboot. It will only exist until the guest is shutdown or when the host is restarted. A persistent guest virtual machine survives reboot and will last until it is deleted.
During the life cycle of a virtual machine, libvirt will classify the guest as any of the following states:
  • Undefined - This is a guest virtual machine that has not been defined or created. As such libvirt is unaware of any guest in this state and will not report about guest virtual machines in this state.
  • Defined / Stopped - This is a guest virtual machine which is defined, but is not running. Only persistent guests can be considered defined or stopped. As such, when a transient guest virtual machine is put into this state it ceases to exist.
  • Running - The guest virtual machine in this state has been defined and is currently working. This state can be used with both persistent and transient guest virtual machines.
  • Paused - The guest virtual machine's execution on the hypervisor has been suspended, or its state has been temporarily stored until it is resumed. Guest virtual machines in this state are not aware they have been suspended and do not notice that time has passed when they are resumed.
  • Saved - This state is similar to the paused state, however the guest virtual machine's configuration is saved to persistent storage. Any guest virtual machine in this state is not aware it is paused and does not notice that time has passed once it has been restored.

23.2. Running the virsh Program

There are two ways to run virsh. One way is line by line and the other is inside an interactive terminal. The commands presented in this guide are shown in a line by line format. The same commands may be used in the interactive terminal, but the prefix 'virsh' is not needed, and the interactive terminal automatically registers the user as root. To enter the interactive terminal run $ virsh and click enter. The following prompt appears: virsh #.

Example 23.1. How to launch the virsh interactive terminal

In the following output, you can see the virsh welcome screen for the interactive terminal.
[~]$ virsh
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit


virsh #

23.3. Interactive Mode Commands

The following commands are best used in the virsh interactive mode, or interactive terminal as explained in Example 23.1, “How to launch the virsh interactive terminal”. They can be used in batch mode, but may behave differently as indicated.

23.3.1. Quitting and Exiting from the virsh Interactive Terminal

The virsh quit command and the virsh exit command will close the interactive terminal. For example:
virsh # exit
virsh # quit
The quit and exit commands are mainly useful in the interactive terminal. Note that these commands may also be used in bash mode from the command line as well. In that case, the command virsh exit exits automatically.

23.3.2. Changing and Displaying Directories in a Virtual Machine

</