Chapter 18. Creating nested virtual machines

RHEL 8.2 and later provide full support for the KVM nested virtualization feature on Intel hosts. This makes it possible for a virtual machine (also referred to as a level 1, or L1) that runs on a RHEL 8 physical host (level 0, or L0) to act as a hypervisor and create its own virtual machines (level 2 or L2).

In other words, a RHEL 8 host can run L1 virtual machines (VMs), and each of these VMs can host nested L2 VMs.

Nested virtualization can be useful in a variety of scenarios, such as debugging hypervisors in a constrained environment or testing larger virtual deployments on a limited amount of physical resources. However, note that nested virtualization is not recommended in production user environments, is subject to various limitations in functionality, and is primarily intended for development and testing.

It is possible to create nested VMs on multiple architectures, but Red Hat currently supports nested VMs only on Intel systems. In contrast, nested virtualization on AMD, IBM POWER9, and IBM Z systems is only provided as a Technology Preview and is therefore unsupported.

18.1. Creating a nested virtual machine on Intel

Follow the steps below to enable and configure nested virtualization on an Intel host.

Prerequisites

  • An L0 RHEL8 host running an L1 virtual machine (VM).
  • The hypervisor CPU must support nested virtualization. To verify, use the cat /proc/cpuinfo command on the L0 hypervisor. If the output of the command includes the vmx and ept flags, creating L2 VMs is possible. This is generally the case on Intel Xeon v3 cores and later.
  • Ensure that nested virtualization is enabled on the L0 host:

    # cat /sys/module/kvm_intel/parameters/nested
    • If the command returns 1, the feature is enabled, and you can start the Procedure below..
    • If the command returns 0 or N but your system supports nested virtualization, use the following steps to enable the feature.

      1. Unload the kvm_intel module:

        # modprobe -r kvm_intel
      2. Activate the nesting feature:

        # modprobe kvm_intel nested=1
      3. The nesting feature is now enabled, but only until the next reboot of the L0 host. To enable it permanently, add the following line to the /etc/modprobe.d/kvm.conf file:

        options kvm_intel nested=1

Procedure

  1. Configure your L1 VM for nested virtualization.

    1. Open the XML configuration of the VM. The following example opens the configuration of the Intel-L1 VM:

      # virsh edit Intel-L1
    2. Add the following line to the configuration:

      <cpu mode='host-passthrough'/>

      If the VM’s XML configuration file already contains a <cpu> element, rewrite it.

  2. Create an L2 VM within the L1 VM. To do this, follow the same procedure as when creating the L1 VM.

18.2. Creating a nested virtual machine on AMD

Follow the steps below to enable and configure nested virtualization on an AMD host.

Warning

Nested virtualization is currently provided only as a Technology Preview on the AMD64 architecture, and is therefore unsupported.

Prerequisites

  • An L0 RHEL8 host running an L1 virtual machine (VM).
  • The hypervisor CPU must support nested virtualization. To verify, use the cat /proc/cpuinfo command on the L0 hypervisor. If the output of the command includes the svm and npt flags, creating L2 VMs is possible. This is generally the case on AMD EPYC cores and later.
  • Ensure that nested virtualization is enabled on the L0 host:

    # cat /sys/module/kvm_amd/parameters/nested
    • If the command returns Y or 1, the feature is enabled, and you can start the Procedure below..
    • If the command returns 0 or N, use the following steps to enable the feature.

      1. Stop all running VMs on the L0 host.
      2. Unload the kvm_amd module:

        # modprobe -r kvm_amd
      3. Activate the nesting feature:

        # modprobe kvm_amd nested=1
      4. The nesting feature is now enabled, but only until the next reboot of the L0 host. To enable it permanently, add the following to the /etc/modprobe.d/kvm.conf file:

        options kvm_amd nested=1

Procedure

  1. Configure your L1 VM for nested virtualization.

    1. Open the XML configuration of the VM. The following example opens the configuration of the AMD-L1 VM:

      # virsh edit AMD-L1
    2. Add the following line to the configuration:

      <cpu mode='host-passthrough'/>

      If the guest’s XML configuration file already contains a <cpu> element, rewrite it.

  2. Create an L2 VM within the L1 VM. To do this, follow the same procedure as when creating the L1 VM.

18.3. Creating a nested virtual machine on IBM Z

Follow the steps below to enable and configure nested virtualization on an IBM Z host.

Warning

Nested virtualization is currently provided only as a Technology Preview on the IBM Z architecture, and is therefore unsupported.

Prerequisites

  • An L0 RHEL8 host running an L1 virtual machine (VM).
  • The hypervisor CPU must support nested virtualization. To verify this is the case, use the cat /proc/cpuinfo command on the L0 hypervisor. If the output of the command includes the sie flag, creating L2 VMs is possible.
  • Ensure that nested virtualization is enabled on the L0 host:

    # cat /sys/module/kvm/parameters/nested
    • If the command returns Y or 1, the feature is enabled, and you can start the Procedure below..
    • If the command returns 0 or N, use the following steps to enable the feature.

      1. Stop all running VMs on the L0 host.
      2. Unload the kvm module:

        # modprobe -r kvm
      3. Activate the nesting feature:

        # modprobe kvm nested=1
      4. The nesting feature is now enabled, but only until the next reboot of the L0 host. To enable it permanently, add the following line to the /etc/modprobe.d/kvm.conf file:

        options kvm nested=1

Procedure

  • Create an L2 VM within the L1 VM. To do this, follow the same procedure as when creating the L1 VM.

18.4. Creating a nested virtual machine on IBM POWER9

Follow the steps below to enable and configure nested virtualization on an IBM POWER9 host.

Warning

Nested virtualization is currently provided only as a Technology Preview on the IBM POWER9 architecture, and is therefore unsupported. In addition, creating nested virtual machines (VMs) is not possible on previous versions of IBM POWER systems, such as IBM POWER8.

Prerequisites

  • An L0 RHEL8 host running an L1 VM. The L1 VM must be using RHEL 8 as the guest operating system.
  • Ensure that nested virtualization is enabled on the L0 host:

    # cat /sys/module/kvm_hv/parameters/nested
    • If the command returns Y or 1, the feature is enabled, and you can start the Procedure below..
    • If the command returns 0 or N, use the following steps to enable the feature:

      1. Stop all running VMs on the L0 host.
      2. Unload the kvm module:

        # modprobe -r kvm_hv
      3. Activate the nesting feature:

        # modprobe kvm_hv nested=1
      4. The nesting feature is now enabled, but only until the next reboot of the L0 host. To enable it permanently, add the following line to the /etc/modprobe.d/kvm.conf file:

        options kvm_hv nested=1

Procedure

  1. To ensure that the L1 VM can create L2 VMs, add the cap-nested-hv parameter to the machine type of the L1 VM. To do so, use the virsh edit command to modify the L1 VM’s XML configuration, and the following line to the <features> section:

    <nested-hv state='on'/>
  2. Create an L2 VM within the L1 VM. To do this, follow the same procedure as when creating the L1 VM.

    To significantly improve the performance of L2 VMs, Red Hat recommends adding the`cap-nested-hv` parameter to the XML configurations of L2 VMs as well. For instructions, see the previous step.

Additional information

  • Note that using IBM POWER8 as the architecture for the L2 VM currently does not work.

18.5. Restrictions and limitations for nested virtualization

Keep the following restrictions in mind when using nested virtualization.

  • The L0 host must be an Intel, AMD, IBM POWER9, or IBM Z system. Nested virtualization currently does not work on other architectures. In addition, Red Hat currently only supports Intel as a host for nested virtual machines (VMs), and all other architectures are provided only as Technology Previews.
  • For nested virtualization to be supported, you must use the following guest operating systems (OSs):

    • On the L0 host - RHEL 8.2 and later
    • On the L1 VMs - RHEL 7.8 and later, or RHEL 8.2 and later
    • On the L2 VMs - one of the following OSs:

      • RHEL 7.8 and later
      • RHEL 8.2 and later
      • Microsoft Windows Server 2016
      • Microsoft Windows Server 2019

    In addition, on IBM POWER9, nested virtualization currently only works under the following circumstances:

    • Both the L0 host and the L1 VM use RHEL 8
    • The L2 VM uses RHEL 8, or RHEL 7 with a rhel-alt kernel.
    • The L1 VM and L2 VM are not running in POWER8 compatibility mode.
  • When using an L1 RHEL 8 VM on a non-KVM L0 hypervisor, such as VMware ESXi or Amazon Web Services (AWS), creating L2 VMs in the RHEL 8 guest OS may work, but is not supported.

    Note that in RHEL 8.2 and later, creating nested RHEL 8 VMs on the Microsoft Hyper-V and Microsoft Azure hypervisors is provided as a Technology Preview.

  • Use of L2 VMs as hypervisors and creating L3 guests has not been properly tested and is not expected to work.
  • Migrating L1 VMs does not work if they are hosting L2 VMs.

    On AMD and Intel systems, migrating L2 VMs does not work properly, and is not recommended by Red Hat. However, on IBM Z systems, L2 VM migration is expected to work.

  • On an IBM Z system, huge-page backing storage and nested virtualization cannot be used at the same time.

    # modprobe kvm hpage=1 nested=1
    modprobe: ERROR: could not insert 'kvm': Invalid argument
    # dmesg |tail -1
    [90226.508366] kvm-s390: A KVM host that supports nesting cannot back its KVM guests with huge pages
  • Not all features available on the L0 host are available for the L1 hypervisor. For example, IOMMU/VT-d or APICv cannot be used by the L1 hypervisor.