Red Hat Training
A Red Hat training course is available for RHEL 8
Chapter 18. Creating nested virtual machines
On RHEL 8 systems and later, it is possible to create nested virtual machines (VMs). This means that a RHEL 8 VM (also referred to as a level 1, or L1) that runs on a RHEL 8 physical host (level 0, or L0) can act as a hypervisor and create its own VMs (level 2 or L2).
In other words, an L0 host can run L1 virtual machines (VMs), and each of these L1 VMs can host their own L2 VMs. Note that, in such cases, both L0 and L1 hosts must be RHEL 8 systems, whereas the L2 guest can be any supported RHEL or Windows system.
Red Hat currently provides nested virtualization only as a Technology Preview, and is therefore unsupported.
In addition, Red Hat does not recommend using nested virtualization in production user environments, due to various limitations in functionality. Instead, nested virtualization is primarily intended for development and testing scenarios, such as:
- Debugging hypervisors in a constrained environment
- Testing larger virtual deployments on a limited amount of physical resources
It is also possible to create nested VMs on multiple architectures, such as Intel, AMD, IBM POWER9, and IBM Z. Note that on all these architectures, nesting VMs is a Technology Preview, and therefore not supported by Red Hat.
18.1. Creating a nested virtual machine on Intel
Follow the steps below to enable and configure nested virtualization on an Intel host.
Nested virtualization is currently provided only as a Technology Preview on the Intel architecture, and is therefore unsupported.
Prerequisites
- An L0 RHEL 8 host running an L1 virtual machine (VM).
-
The hypervisor CPU must support nested virtualization. To verify, use the
cat /proc/cpuinfo
command on the L0 hypervisor. If the output of the command includes thevmx
andept
flags, creating L2 VMs is possible. This is generally the case on Intel Xeon v3 cores and later. Ensure that nested virtualization is enabled on the L0 host:
# cat /sys/module/kvm_intel/parameters/nested
- If the command returns 1 or Y, the feature is enabled. Skip the remaining prerequisite steps, and continue with the Procedure section.
If the command returns 0 or N but your system supports nested virtualization, use the following steps to enable the feature.
Unload the
kvm_intel
module:# modprobe -r kvm_intel
Activate the nesting feature:
# modprobe kvm_intel nested=1
The nesting feature is now enabled, but only until the next reboot of the L0 host. To enable it permanently, add the following line to the
/etc/modprobe.d/kvm.conf
file:options kvm_intel nested=1
Procedure
Configure your L1 VM for nested virtualization.
Open the XML configuration of the VM. The following example opens the configuration of the Intel-L1 VM:
# virsh edit Intel-L1
Configure the VM to use
host-passthrough
CPU mode by editing the<cpu>
element:<cpu mode='host-passthrough'/>
If you require the VM to use a specific CPU model, configure the VM to use
custom
CPU mode. Inside the<cpu>
element, add a<feature policy='require' name='vmx'/>
element and a<model>
element with the CPU model specified inside. For example:<cpu mode ='custom' match ='exact' check='partial'> <model fallback='allow'>Haswell-noTSX</model> <feature policy='require' name='vmx'/> ... </cpu>
- Create an L2 VM within the L1 VM. To do this, follow the same procedure as when creating the L1 VM.
18.2. Creating a nested virtual machine on AMD
Follow the steps below to enable and configure nested virtualization on an AMD host.
Nested virtualization is currently provided only as a Technology Preview on the AMD64 architecture, and is therefore unsupported.
Prerequisites
- An L0 RHEL 8 host running an L1 virtual machine (VM).
-
The hypervisor CPU must support nested virtualization. To verify, use the
cat /proc/cpuinfo
command on the L0 hypervisor. If the output of the command includes thesvm
andnpt
flags, creating L2 VMs is possible. This is generally the case on AMD EPYC cores and later. Ensure that nested virtualization is enabled on the L0 host:
# cat /sys/module/kvm_amd/parameters/nested
- If the command returns 1 or Y, the feature is enabled. Skip the remaining prerequisite steps, and continue with the Procedure section.
If the command returns 0 or N, use the following steps to enable the feature.
- Stop all running VMs on the L0 host.
Unload the
kvm_amd
module:# modprobe -r kvm_amd
Activate the nesting feature:
# modprobe kvm_amd nested=1
The nesting feature is now enabled, but only until the next reboot of the L0 host. To enable it permanently, add the following to the
/etc/modprobe.d/kvm.conf
file:options kvm_amd nested=1
Procedure
Configure your L1 VM for nested virtualization.
Open the XML configuration of the VM. The following example opens the configuration of the AMD-L1 VM:
# virsh edit AMD-L1
Configure the VM to use
host-passthrough
CPU mode by editing the<cpu>
element:<cpu mode='host-passthrough'/>
If you require the VM to use a specific CPU model, configure the VM to use
custom
CPU mode. Inside the<cpu>
element, add a<feature policy='require' name='svm'/>
element and a<model>
element with the CPU model specified inside. For example:<cpu mode="custom" match="exact" check="none"> <model fallback="allow">EPYC-IBPB</model> <feature policy="require" name="svm"/> ... </cpu>
- Create an L2 VM within the L1 VM. To do this, follow the same procedure as when creating the L1 VM.
18.3. Creating a nested virtual machine on IBM Z
Follow the steps below to enable and configure nested virtualization on an IBM Z host.
Nested virtualization is currently provided only as a Technology Preview on the IBM Z architecture, and is therefore unsupported.
Prerequisites
- An L0 RHEL 8 host running an L1 virtual machine (VM).
-
The hypervisor CPU must support nested virtualization. To verify this is the case, use the
cat /proc/cpuinfo
command on the L0 hypervisor. If the output of the command includes thesie
flag, creating L2 VMs is possible. Ensure that nested virtualization is enabled on the L0 host:
# cat /sys/module/kvm/parameters/nested
- If the command returns 1 or Y, the feature is enabled. Skip the remaining prerequisite steps, and continue with the Procedure section.
If the command returns 0 or N, use the following steps to enable the feature.
- Stop all running VMs on the L0 host.
Unload the
kvm
module:# modprobe -r kvm
Activate the nesting feature:
# modprobe kvm nested=1
The nesting feature is now enabled, but only until the next reboot of the L0 host. To enable it permanently, add the following line to the
/etc/modprobe.d/kvm.conf
file:options kvm nested=1
Procedure
- Create an L2 VM within the L1 VM. To do this, follow the same procedure as when creating the L1 VM.
18.4. Creating a nested virtual machine on IBM POWER9
Follow the steps below to enable and configure nested virtualization on an IBM POWER9 host.
Nested virtualization is currently provided only as a Technology Preview on the IBM POWER9 architecture, and is therefore unsupported. In addition, creating nested virtual machines (VMs) is not possible on previous versions of IBM POWER systems, such as IBM POWER8.
Prerequisites
- An L0 RHEL 8 host is running an L1 VM. The L1 VM is using RHEL 8 as the guest operating system.
Nested virtualization is enabled on the L0 host:
# cat /sys/module/kvm_hv/parameters/nested
- If the command returns 1 or Y, the feature is enabled. Skip the remaining prerequisite steps, and continue with the Procedure section.
If the command returns 0 or N, use the following steps to enable the feature:
- Stop all running VMs on the L0 host.
Unload the
kvm
module:# modprobe -r kvm_hv
Activate the nesting feature:
# modprobe kvm_hv nested=1
The nesting feature is now enabled, but only until the next reboot of the L0 host. To enable it permanently, add the following line to the
/etc/modprobe.d/kvm.conf
file:options kvm_hv nested=1
Procedure
To ensure that the L1 VM can create L2 VMs, add the
cap-nested-hv
parameter to the machine type of the L1 VM. To do so, use thevirsh edit
command to modify the L1 VM’s XML configuration, and the following line to the<features>
section:<nested-hv state='on'/>
Create an L2 VM within the L1 VM. To do this, follow the same procedure as when creating the L1 VM.
To significantly improve the performance of L2 VMs, Red Hat recommends adding the`cap-nested-hv` parameter to the XML configurations of L2 VMs as well. For instructions, see the previous step.
Additional information
-
Note that
IBM POWER8
as the architecture for the L2 VM currently does not supported.
18.5. Restrictions and limitations for nested virtualization
Keep the following restrictions in mind when using nested virtualization.
Red Hat currently does not support nested virtualization, and only provides nesting as a Technology Preview.
Supported architectures
- The L0 host must be an Intel, AMD, IBM POWER9, or IBM Z system. Nested virtualization currently does not work on other architectures.
Supported guest operating systems
To create nested VMs, you must use the following guest operating systems (OSs):
- On the L0 host - RHEL 8.2 and later
On the L1 VMs - RHEL 7.8 and later, or RHEL 8.2 and later
NoteThis support does not apply to using virtualization offerings based on RHEL 7 and RHEL 8 in L1 VMs. These include:
- Red Hat Virtualization
- Red Hat OpenStack Platform
- OpenShift Virtualization
On the L2 VMs - you must use one of the following OSs:
- RHEL 7.8 and later
- RHEL 8.2 and later
- Microsoft Windows Server 2016
- Microsoft Windows Server 2019
In addition, on IBM POWER9, nested virtualization currently only works under the following circumstances:
- Both the L0 host and the L1 VM use RHEL 8
-
The L2 VM uses RHEL 8, or RHEL 7 with a
rhel-alt
kernel. - The L1 VM and L2 VM are not running in POWER8 compatibility mode.
Hypervisor limitations
- Currently, Red Hat supports nesting only on RHEL-KVM. When RHEL is used as the L0 hypervisor, you can use RHEL 8 or Windows for WSL 2 as the L1 hypervisor.
- When using an L1 RHEL 8 VM on a non-KVM L0 hypervisor, such as VMware ESXi or Amazon Web Services (AWS), creating L2 VMs in the RHEL 8 guest OS may work, but is not supported.
Feature limitations
- Use of L2 VMs as hypervisors and creating L3 guests has not been properly tested and is not expected to work.
- Migrating VMs currently does not work on AMD systems if nested virtualization has been enabled on the L0 host.
On an IBM Z system, huge-page backing storage and nested virtualization cannot be used at the same time.
# modprobe kvm hpage=1 nested=1 modprobe: ERROR: could not insert 'kvm': Invalid argument # dmesg |tail -1 [90226.508366] kvm-s390: A KVM host that supports nesting cannot back its KVM guests with huge pages
Some features available on the L0 host may be unavailable for the L1 hypervisor.
For example, on IBM POWER 9 hardware, the External Interrupt Virtualization Engine (XIVE) does not work. However, L1 VMs can use the emulated XIVE interrupt controller to launch L2 VMs.