Chapter 17. Optimizing virtual machine performance

Procedure

1. Stop libvirtd and its sockets.

   $ systemctl stop libvirtd.service
   $ systemctl stop libvirtd{,-ro,-admin,-tcp,-tls}.socket

2. Disable libvirtd to prevent it from starting on boot.

   $ systemctl disable libvirtd.service
   $ systemctl disable libvirtd{,-ro,-admin,-tcp,-tls}.socket

3. Enable the modular libvirt daemons.

   # for drv in qemu interface network nodedev nwfilter secret storage; do systemctl unmask virt${drv}d.service; systemctl unmask virt${drv}d{,-ro,-admin}.socket; systemctl enable virt${drv}d.service; systemctl enable virt${drv}d{,-ro,-admin}.socket; done

4. Start the sockets for the modular daemons.

   # for drv in qemu network nodedev nwfilter secret storage; do systemctl start virt${drv}d{,-ro,-admin}.socket; done

5. Optional: If you require connecting to your host from remote hosts, enable and start the virtualization proxy daemon.

   1. Check whether the libvirtd-tls.socket service is enabled on your system.

      # cat /etc/libvirt/libvirt.conf | grep listen_tls
      
      listen_tls = 0

   2. If libvirtd-tls.socket is not enabled (listen_tls = 0), activate virtproxyd as follows:

      # systemctl unmask virtproxyd.service
      # systemctl unmask virtproxyd{,-ro,-admin}.socket
      # systemctl enable virtproxyd.service
      # systemctl enable virtproxyd{,-ro,-admin}.socket
      # systemctl start virtproxyd{,-ro,-admin}.socket

   3. If libvirtd-tls.socket is enabled (listen_tls = 1), activate virtproxyd as follows:

      # systemctl unmask virtproxyd.service
      # systemctl unmask virtproxyd{,-ro,-admin,-tls}.socket
      # systemctl enable virtproxyd.service
      # systemctl enable virtproxyd{,-ro,-admin,-tls}.socket
      # systemctl start virtproxyd{,-ro,-admin,-tls}.socket

      To enable the TLS socket of virtproxyd, your host must have TLS certificates configured to work with libvirt. For more information, see the Upstream libvirt documentation.

Verification

1. Activate the enabled virtualization daemons.

   # virsh uri
   qemu:///system

2. Verify that your host is using the virtqemud modular daemon.

   # systemctl is-active virtqemud.service
   active

   If the status is active, you have successfully enabled modular libvirt daemons.

Procedure

1. Optional: Obtain the information about the maximum memory and currently used memory for a VM. This will serve as a baseline for your changes, and also for verification.

   # virsh dominfo testguest
   Max memory:     2097152 KiB
   Used memory:    2097152 KiB

2. In the Virtual Machines interface, click the VM whose information you want to see.

   A new page opens with an Overview section with basic information about the selected VM and a Console section to access the VM’s graphical interface.

3. Click edit next to the Memory line in the Overview pane.

   The Memory Adjustment dialog appears.

   

4. Configure the virtual CPUs for the selected VM.

   • Maximum allocation - Sets the maximum amount of host memory that the VM can use for its processes. You can specify the maximum memory when creating the VM or increase it later. You can specify memory as multiples of MiB or GiB.

     Adjusting maximum memory allocation is only possible on a shut-off VM.

   • Current allocation - Sets the actual amount of memory allocated to the VM. This value can be less than the Maximum allocation but cannot exceed it. You can adjust the value to regulate the memory available to the VM for its processes. You can specify memory as multiples of MiB or GiB.

     If you do not specify this value, the default allocation is the Maximum allocation value.

5. Click Save.

   The memory allocation of the VM is adjusted.

Procedure

1. Optional: Obtain the information about the maximum memory and currently used memory for a VM. This will serve as a baseline for your changes, and also for verification.

   # virsh dominfo testguest
   Max memory:     2097152 KiB
   Used memory:    2097152 KiB

2. Adjust the maximum memory allocated to a VM. Increasing this value improves the performance potential of the VM, and reducing the value lowers the performance footprint the VM has on your host. Note that this change can only be performed on a shut-off VM, so adjusting a running VM requires a reboot to take effect.

   For example, to change the maximum memory that the testguest VM can use to 4096 MiB:

   # virt-xml testguest --edit --memory memory=4096,currentMemory=4096
   Domain 'testguest' defined successfully.
   Changes will take effect after the domain is fully powered off.

   To increase the maximum memory of a running VM, you can attach a memory device to the VM. This is also referred to as memory hot plug. For details, see Attaching memory devices to virtual machines.

   Warning

   Removing memory devices from a running VM (also referred as a memory hot unplug) is not supported, and highly discouraged by Red Hat.

3. Optional: You can also adjust the memory currently used by the VM, up to the maximum allocation. This regulates the memory load that the VM has on the host until the next reboot, without changing the maximum VM allocation.

   # virsh setmem testguest --current 2048

Verification

1. Confirm that the memory used by the VM has been updated:

   # virsh dominfo testguest
   Max memory:     4194304 KiB
   Used memory:    2097152 KiB

2. Optional: If you adjusted the current VM memory, you can obtain the memory balloon statistics of the VM to evaluate how effectively it regulates its memory use.

    # virsh domstats --balloon testguest
   Domain: 'testguest'
     balloon.current=365624
     balloon.maximum=4194304
     balloon.swap_in=0
     balloon.swap_out=0
     balloon.major_fault=306
     balloon.minor_fault=156117
     balloon.unused=3834448
     balloon.available=4035008
     balloon.usable=3746340
     balloon.last-update=1587971682
     balloon.disk_caches=75444
     balloon.hugetlb_pgalloc=0
     balloon.hugetlb_pgfail=0
     balloon.rss=1005456

1. Display the current <blkio> parameters for a VM:

   # virsh dumpxml VM-name

   <domain>
     [...]
     <blkiotune>
       <weight>800</weight>
       <device>
         <path>/dev/sda</path>
         <weight>1000</weight>
       </device>
       <device>
         <path>/dev/sdb</path>
         <weight>500</weight>
       </device>
     </blkiotune>
     [...]
   </domain>

2. Edit the I/O weight of a specified device:

   # virsh blkiotune VM-name --device-weights device, I/O-weight

   For example, the following changes the weight of the /dev/sda device in the liftrul VM to 500.

   # virsh blkiotune liftbrul --device-weights /dev/sda, 500

Procedure

1. Use the virsh domblklist command to list the names of all the disk devices on a specified VM.

   # virsh domblklist rollin-coal
   Target     Source
   ------------------------------------------------
   vda        /var/lib/libvirt/images/rollin-coal.qcow2
   sda        -
   sdb        /home/horridly-demanding-processes.iso

2. Find the host block device where the virtual disk that you want to throttle is mounted.

   For example, if you want to throttle the sdb virtual disk from the previous step, the following output shows that the disk is mounted on the /dev/nvme0n1p3 partition.

   $ lsblk
   NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
   zram0                                         252:0    0     4G  0 disk  [SWAP]
   nvme0n1                                       259:0    0 238.5G  0 disk
   ├─nvme0n1p1                                   259:1    0   600M  0 part  /boot/efi
   ├─nvme0n1p2                                   259:2    0     1G  0 part  /boot
   └─nvme0n1p3                                   259:3    0 236.9G  0 part
     └─luks-a1123911-6f37-463c-b4eb-fxzy1ac12fea 253:0    0 236.9G  0 crypt /home

3. Set I/O limits for the block device using the virsh blkiotune command.

   # virsh blkiotune VM-name --parameter device,limit

   The following example throttles the sdb disk on the rollin-coal VM to 1000 read and write I/O operations per second and to 50 MB per second read and write throughput.

   # virsh blkiotune rollin-coal --device-read-iops-sec /dev/nvme0n1p3,1000 --device-write-iops-sec /dev/nvme0n1p3,1000 --device-write-bytes-sec /dev/nvme0n1p3,52428800 --device-read-bytes-sec /dev/nvme0n1p3,52428800

Procedure

1. Adjust the maximum number of vCPUs that can be attached to a VM, which takes effect on the VM’s next boot.

   For example, to increase the maximum vCPU count for the testguest VM to 8:

   # virsh setvcpus testguest 8 --maximum --config

   Note that the maximum may be limited by the CPU topology, host hardware, the hypervisor, and other factors.

2. Adjust the current number of vCPUs attached to a VM, up to the maximum configured in the previous step. For example:

   • To increase the number of vCPUs attached to the running testguest VM to 4:

     # virsh setvcpus testguest 4 --live

     This increases the VM’s performance and host load footprint of testguest until the VM’s next boot.

   • To permanently decrease the number of vCPUs attached to the testguest VM to 1:

     # virsh setvcpus testguest 1 --config

     This decreases the VM’s performance and host load footprint of testguest after the VM’s next boot. However, if needed, additional vCPUs can be hot plugged to the VM to temporarily increase its performance.

Procedure

1. In the Virtual Machines interface, click the VM whose information you want to see.

   A new page opens with an Overview section with basic information about the selected VM and a Console section to access the VM’s graphical interface.

2. Click edit next to the number of vCPUs in the Overview pane.

   The vCPU details dialog appears.

   

1. Configure the virtual CPUs for the selected VM.

   • vCPU Count - The number of vCPUs currently in use.

     Note

     The vCPU count cannot be greater than the vCPU Maximum.

   • vCPU Maximum - The maximum number of virtual CPUs that can be configured for the VM. If this value is higher than the vCPU Count, additional vCPUs can be attached to the VM.
   • Sockets - The number of sockets to expose to the VM.
   • Cores per socket - The number of cores for each socket to expose to the VM.

   • Threads per core - The number of threads for each core to expose to the VM.

     Note that the Sockets, Cores per socket, and Threads per core options adjust the CPU topology of the VM. This may be beneficial for vCPU performance and may impact the functionality of certain software in the guest OS. If a different setting is not required by your deployment, keep the default values.

2. Click Apply.

   The virtual CPUs for the VM are configured.

   Note

   Changes to virtual CPU settings only take effect after the VM is restarted.

1. Pin specific vCPU threads to a specific host CPU or range of CPUs. This is also possible on non-NUMA hosts and VMs, and is recommended as a safe method of vCPU performance improvement.

   For example, the following commands pin vCPU threads 0 to 5 of the testguest6 VM to host CPUs 1, 3, 5, 7, 9, and 11, respectively:

   # virsh vcpupin testguest6 0 1
   # virsh vcpupin testguest6 1 3
   # virsh vcpupin testguest6 2 5
   # virsh vcpupin testguest6 3 7
   # virsh vcpupin testguest6 4 9
   # virsh vcpupin testguest6 5 11

   Afterwards, you can verify whether this was successful:

   # virsh vcpupin testguest6
   VCPU   CPU Affinity
   ----------------------
   0      1
   1      3
   2      5
   3      7
   4      9
   5      11

2. After pinning vCPU threads, you can also pin QEMU process threads associated with a specified VM to a specific host CPU or range of CPUs. For example, the following commands pin the QEMU process thread of testguest6 to CPUs 13 and 15, and verify this was successful:

   # virsh emulatorpin testguest6 13,15
   # virsh emulatorpin testguest6
   emulator: CPU Affinity
   ----------------------------------
          *: 13,15

3. Finally, you can also specify which host NUMA nodes will be assigned specifically to a certain VM. This can improve the host memory usage by the VM’s vCPU. For example, the following commands set testguest6 to use host NUMA nodes 3 to 5, and verify this was successful:

   # virsh numatune testguest6 --nodeset 3-5
   # virsh numatune testguest6

Solution

1. Obtain the information about the host topology:

   # virsh capabilities

   The output should include a section that looks similar to the following:

   <topology>
     <cells num="2">
       <cell id="0">
         <memory unit="KiB">15624346</memory>
         <pages unit="KiB" size="4">3906086</pages>
         <pages unit="KiB" size="2048">0</pages>
         <pages unit="KiB" size="1048576">0</pages>
         <distances>
           <sibling id="0" value="10" />
           <sibling id="1" value="21" />
         </distances>
         <cpus num="6">
           <cpu id="0" socket_id="0" core_id="0" siblings="0,3" />
           <cpu id="1" socket_id="0" core_id="1" siblings="1,4" />
           <cpu id="2" socket_id="0" core_id="2" siblings="2,5" />
           <cpu id="3" socket_id="0" core_id="0" siblings="0,3" />
           <cpu id="4" socket_id="0" core_id="1" siblings="1,4" />
           <cpu id="5" socket_id="0" core_id="2" siblings="2,5" />
         </cpus>
       </cell>
       <cell id="1">
         <memory unit="KiB">15624346</memory>
         <pages unit="KiB" size="4">3906086</pages>
         <pages unit="KiB" size="2048">0</pages>
         <pages unit="KiB" size="1048576">0</pages>
         <distances>
           <sibling id="0" value="21" />
           <sibling id="1" value="10" />
         </distances>
         <cpus num="6">
           <cpu id="6" socket_id="1" core_id="3" siblings="6,9" />
           <cpu id="7" socket_id="1" core_id="4" siblings="7,10" />
           <cpu id="8" socket_id="1" core_id="5" siblings="8,11" />
           <cpu id="9" socket_id="1" core_id="3" siblings="6,9" />
           <cpu id="10" socket_id="1" core_id="4" siblings="7,10" />
           <cpu id="11" socket_id="1" core_id="5" siblings="8,11" />
         </cpus>
       </cell>
     </cells>
   </topology>

2. Optional: Test the performance of the VM using the applicable tools and utilities.

3. Set up and mount 1 GiB huge pages on the host:

   1. Add the following line to the host’s kernel command line:

      default_hugepagesz=1G hugepagesz=1G

   2. Create the /etc/systemd/system/hugetlb-gigantic-pages.service file with the following content:

      [Unit]
      Description=HugeTLB Gigantic Pages Reservation
      DefaultDependencies=no
      Before=dev-hugepages.mount
      ConditionPathExists=/sys/devices/system/node
      ConditionKernelCommandLine=hugepagesz=1G
      
      [Service]
      Type=oneshot
      RemainAfterExit=yes
      ExecStart=/etc/systemd/hugetlb-reserve-pages.sh
      
      [Install]
      WantedBy=sysinit.target

   3. Create the /etc/systemd/hugetlb-reserve-pages.sh file with the following content:

      #!/bin/sh
      
      nodes_path=/sys/devices/system/node/
      if [ ! -d $nodes_path ]; then
      	echo "ERROR: $nodes_path does not exist"
      	exit 1
      fi
      
      reserve_pages()
      {
      	echo $1 > $nodes_path/$2/hugepages/hugepages-1048576kB/nr_hugepages
      }
      
      reserve_pages 4 node1
      reserve_pages 4 node2

      This reserves four 1GiB huge pages from node1 and four 1GiB huge pages from node2.

   4. Make the script created in the previous step executable:

      # chmod +x /etc/systemd/hugetlb-reserve-pages.sh

   5. Enable huge page reservation on boot:

      # systemctl enable hugetlb-gigantic-pages

4. Use the virsh edit command to edit the XML configuration of the VM you wish to optimize, in this example super-VM:

   # virsh edit super-vm

5. Adjust the XML configuration of the VM in the following way:

   1. Set the VM to use 8 static vCPUs. Use the <vcpu/> element to do this.

   2. Pin each of the vCPU threads to the corresponding host CPU threads that it mirrors in the topology. To do so, use the <vcpupin/> elements in the <cputune> section.

      Note that, as shown by the virsh capabilities utility above, host CPU threads are not ordered sequentially in their respective cores. In addition, the vCPU threads should be pinned to the highest available set of host cores on the same NUMA node. For a table illustration, see the Sample topology section below.

      The XML configuration for steps a. and b. can look similar to:

      <cputune>
        <vcpupin vcpu='0' cpuset='1'/>
        <vcpupin vcpu='1' cpuset='4'/>
        <vcpupin vcpu='2' cpuset='2'/>
        <vcpupin vcpu='3' cpuset='5'/>
        <vcpupin vcpu='4' cpuset='7'/>
        <vcpupin vcpu='5' cpuset='10'/>
        <vcpupin vcpu='6' cpuset='8'/>
        <vcpupin vcpu='7' cpuset='11'/>
        <emulatorpin cpuset='6,9'/>
      </cputune>

   3. Set the VM to use 1 GiB huge pages:

      <memoryBacking>
        <hugepages>
          <page size='1' unit='GiB'/>
        </hugepages>
      </memoryBacking>

   4. Configure the VM’s NUMA nodes to use memory from the corresponding NUMA nodes on the host. To do so, use the <memnode/> elements in the <numatune/> section:

      <numatune>
        <memory mode="preferred" nodeset="1"/>
        <memnode cellid="0" mode="strict" nodeset="0"/>
        <memnode cellid="1" mode="strict" nodeset="1"/>
      </numatune>

   5. Ensure the CPU mode is set to host-passthrough, and that the CPU uses cache in passthrough mode:

      <cpu mode="host-passthrough">
        <topology sockets="2" cores="2" threads="2"/>
        <cache mode="passthrough"/>

Verification

1. Confirm that the resulting XML configuration of the VM includes a section similar to the following:

   [...]
     <memoryBacking>
       <hugepages>
         <page size='1' unit='GiB'/>
       </hugepages>
     </memoryBacking>
     <vcpu placement='static'>8</vcpu>
     <cputune>
       <vcpupin vcpu='0' cpuset='1'/>
       <vcpupin vcpu='1' cpuset='4'/>
       <vcpupin vcpu='2' cpuset='2'/>
       <vcpupin vcpu='3' cpuset='5'/>
       <vcpupin vcpu='4' cpuset='7'/>
       <vcpupin vcpu='5' cpuset='10'/>
       <vcpupin vcpu='6' cpuset='8'/>
       <vcpupin vcpu='7' cpuset='11'/>
       <emulatorpin cpuset='6,9'/>
     </cputune>
     <numatune>
       <memory mode="preferred" nodeset="1"/>
       <memnode cellid="0" mode="strict" nodeset="0"/>
       <memnode cellid="1" mode="strict" nodeset="1"/>
     </numatune>
     <cpu mode="host-passthrough">
       <topology sockets="2" cores="2" threads="2"/>
       <cache mode="passthrough"/>
       <numa>
         <cell id="0" cpus="0-3" memory="2" unit="GiB">
           <distances>
             <sibling id="0" value="10"/>
             <sibling id="1" value="21"/>
           </distances>
         </cell>
         <cell id="1" cpus="4-7" memory="2" unit="GiB">
           <distances>
             <sibling id="0" value="21"/>
             <sibling id="1" value="10"/>
           </distances>
         </cell>
       </numa>
     </cpu>
   </domain>

2. Optional: Test the performance of the VM using the applicable tools and utilities to evaluate the impact of the VM’s optimization.

1. Install the ksmtuned service:

   # dnf install ksmtuned

2. Start the service:

   • To enable KSM for a single session, use the systemctl utility to start the ksm and ksmtuned services.

     # systemctl start ksm
     # systemctl start ksmtuned

   • To enable KSM persistently, use the systemctl utility to enable the ksm and ksmtuned services.

     # systemctl enable ksm
     Created symlink /etc/systemd/system/multi-user.target.wants/ksm.service → /usr/lib/systemd/system/ksm.service
     
     # systemctl enable ksmtuned
     Created symlink /etc/systemd/system/multi-user.target.wants/ksmtuned.service → /usr/lib/systemd/system/ksmtuned.service