Chapter 13. Using virsh emulatorpin in virtual environments with NFV
Use this procedure to determine the impact of using virsh emulatorpin in Red Hat OpenStack Platform with NFV.
13.1. Symptom
You experience packet loss in Red Hat OpenStack Platform {vernum} NFV environment, and have not configured emulator thread pinning.
13.2. Solution
Use this section to investigate and configure emulator thread pinning.
13.2.1. qemu-kvm Emulator Threads
Emulator threads handle interrupt requests and non-blocking processes for virtual machine hardware emulation. Threads not running vCPUs are qemu-kvm emulator threads. See the following example.
[root@overcloud-compute-0 ~]# ps -Tp `pgrep -f instance-00000009`
PID SPID TTY TIME CMD
364936 364936 ? 00:00:02 qemu-kvm
364936 364946 ? 00:00:00 qemu-kvm
364936 364952 ? 00:00:52 CPU 0/KVM
364936 364953 ? 00:00:26 CPU 1/KVM
364936 364954 ? 00:00:30 CPU 2/KVM
364936 364956 ? 00:00:00 vnc_workerDue to the Linux CFS (completely fair scheduler), emulator threads normally move periodically from one pCPU to another, within the defined in libvirt’s emulator pin set.
In NFV contexts, you might experience issues if you configure emulator threads when using the isolcpus parameter, because this kernel configuration disables the CFS scheduling on those CPUs. If you are not using the isolcpus parameter, you can experience packet loss when the emulator threads interrupt CPUs that are processing packets.
Examples of emulator threads include:
- qemu-kvm threads
- vnc_worker threads
- vhost-<qemu-kvm PID> kernel threads (When virtio-net is used (kernel networking on the hypervisor)
13.2.2. Default Behavior for Emulator Thread Pinning
By default, nova will configure an emulator thread pin set which spans the pCPUs assigned to all vCPUs. If you are not using the isolcpus parameter, then emulator threads can be scheduled on any pCPU, and will periodically move from one pCPU to another.
virsh dumpxml instance-0000001d
(...)
<vcpu placement='static'>4</vcpu>
<cputune>
<shares>4096</shares>
<vcpupin vcpu='0' cpuset='34'/>
<vcpupin vcpu='1' cpuset='14'/>
<vcpupin vcpu='2' cpuset='10'/>
<vcpupin vcpu='3' cpuset='30'/>
<emulatorpin cpuset='10,14,30,34'/>
</cputune>
(...)[root@overcloud-compute-0 ~]# virsh dumpxml instance-00000009
(...)
<nova:vcpus>3</nova:vcpus>
<vcpu placement='static'>3</vcpu>
<vcpupin vcpu='0' cpuset='1'/>
<vcpupin vcpu='1' cpuset='2'/>
<vcpupin vcpu='2' cpuset='3'/>
(...)
<emulatorpin cpuset='1-3'/>
(...)Therefore any of these CPUs can be preempted by qemu’s emulator threads, risking packet drops.
For details on the current progress of new features for emulator thread pinning, see Bug 1468004 and OpenStack Change 510897
At the time of this writing, the draft specified the following thread policies:
Valid THREAD-POLICY values are:
- ``share``: (default) The emulator threads float across the pCPUs
associated to the guest. To place a workload's emulator threads on
a set of isolated physical CPUs, set ``share``` and
``[compute]/cpu_shared_set`` configuration option to the set of
host CPUs that should be used for best-effort CPU resources.
- ``isolate``: The emulator threads are isolated on a single pCPU.13.2.3. About the Impact of isolcpus on Emulator Thread Scheduling
When isolcpus is used, CFS scheduler is disabled and all emulator threads will run on the first available, lowest indexed pCPU. As a consequence, without intervention or further configuration, one vCPU of the instance runs a high risk for resource contention with the emulator threads.
Further details can be found at Kernel.org Bugzilla – Bug 116701.
Use the following algorithm to determine which vCPU the emulator threads are using:
PCPU=MIN([EMULATORPINSET]) VCPU=REVERSE_CPUSET(PCPU) REVERSE_CPUSET := SELECT pcpu from `virsh dumpxml <instance name> | grep "cpuset=$PCPU"`
For example, in this instance, all emulator threads and children inherit affinity 1-3 from the default emulator pin set:
[root@overcloud-compute-0 ~]# taskset -a -c -p `pgrep -f instance-00000009`
pid 364936's current affinity list: 1-3
pid 364946's current affinity list: 1-3
pid 364952's current affinity list: 1
pid 364953's current affinity list: 2
pid 364954's current affinity list: 3
pid 364956's current affinity list: 1-3
[root@overcloud-compute-0 ~]# ps -Tp `pgrep -f instance-00000009`
PID SPID TTY TIME CMD
364936 364936 ? 00:00:02 qemu-kvm
364936 364946 ? 00:00:00 qemu-kvm
364936 364952 ? 00:00:51 CPU 0/KVM
364936 364953 ? 00:00:26 CPU 1/KVM
364936 364954 ? 00:00:30 CPU 2/KVM
364936 364956 ? 00:00:00 vnc_worker
[root@overcloud-compute-0 ~]# pgrep -f vhost- | xargs -I {} taskset -a -c -p {}
pid 364948's current affinity list: 1-3
pid 364949's current affinity list: 1-3
pid 364950's current affinity list: 1-3
[root@overcloud-compute-0 ~]#In combination with isolcpus, all emulator threads and the vhost-* threads execute on pCPU1 and are never rescheduled:
cat /proc/sched_debug | sed '/^cpu#/,/^runnable/{//!d}' | grep vhost -C3
(...)
cpu#1, 2099.998 MHz
runnable tasks:
task PID tree-key switches prio wait-time sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
watchdog/1 11 -2.995579 410285 0 0.000000 5025.887998 0.000000 0 /
migration/1 12 0.000000 79 0 0.000000 3.375060 0.000000 0 /
ksoftirqd/1 13 5172444.259776 54 120 0.000000 0.570500 0.000000 0 /
kworker/1:0 14 5188475.472257 370 120 0.000000 14.707114 0.000000 0 /
kworker/1:0H 15 8360.049510 10 100 0.000000 0.150151 0.000000 0 /
kworker/1:1 2707 5045807.055876 16370 120 0.000000 793.611916 0.000000 0 /
kworker/1:1H 2763 5187682.987749 11755 100 0.000000 191.949725 0.000000 0 /
qemu-kvm 364936 3419.522791 50276 120 0.000000 2476.880384 0.000000 0 /machine.slice/machine-qemu\x2d6\x2dinstance\x2d00000009.scope/emulator
qemu-kvm 364946 1270.815296 102 120 0.000000 23.204111 0.000000 0 /machine.slice/machine-qemu\x2d6\x2dinstance\x2d00000009.scope/emulator
CPU 0/KVM 364952 52703.660314 53709 120 0.000000 52715.105472 0.000000 0 /machine.slice/machine-qemu\x2d6\x2dinstance\x2d00000009.scope/vcpu0
vnc_worker 364956 123.609634 1 120 0.000000 0.016849 0.000000 0 /machine.slice/machine-qemu\x2d6\x2dinstance\x2d00000009.scope/emulator
vhost-364936 364948 3410.527677 1039 120 0.000000 84.254772 0.000000 0 /machine.slice/machine-qemu\x2d6\x2dinstance\x2d00000009.scope/emulator
vhost-364936 364949 3407.341502 55 120 0.000000 2.894394 0.000000 0 /machine.slice/machine-qemu\x2d6\x2dinstance\x2d00000009.scope/emulator
vhost-364936 364950 3410.395220 174 120 0.000000 10.969077 0.000000 0 /machine.slice/machine-qemu\x2d6\x2dinstance\x2d00000009.scope/emulator
cpu#2, 2099.998 MHz
runnable tasks:
task PID tree-key switches prio wait-time sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
watchdog/2 16 -5.995418 410285 0 0.000000 5197.571153 0.000000 0 /
migration/2 17 0.000000 79 0 0.000000 3.384688 0.000000 0 /
ksoftirqd/2 18 -7.031102 3 120 0.000000 0.019079 0.000000 0 /
kworker/2:0 19 0.119413 39 120 0.000000 0.588589 0.000000 0 /
kworker/2:0H 20 -1.047613 8 100 0.000000 0.086272 0.000000 0 /
kworker/2:1 2734 1475469.236026 11322 120 0.000000 241.388582 0.000000 0 /
CPU 1/KVM 364953 27258.370583 33294 120 0.000000 27269.017017 0.000000 0 /machine.slice/machine-qemu\x2d6\x2dinstance\x2d00000009.scope/vcpu1
cpu#3, 2099.998 MHz
runnable tasks:
task PID tree-key switches prio wait-time sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
watchdog/3 21 -5.996592 410285 0 0.000000 4970.777439 0.000000 0 /
migration/3 22 0.000000 79 0 0.000000 3.886799 0.000000 0 /
ksoftirqd/3 23 -7.035295 3 120 0.000000 0.014677 0.000000 0 /
kworker/3:0 24 17.758583 38 120 0.000000 0.637152 0.000000 0 /
kworker/3:0H 25 -1.047727 8 100 0.000000 0.077141 0.000000 0 /
kworker/3:1 362530 154177.523420 83 120 0.000000 6.544285 0.000000 0 /
CPU 2/KVM 364954 32456.061889 25966 120 0.000000 32466.719084 0.000000 0 /machine.slice/machine-qemu\x2d6\x2dinstance\x2d00000009.scope/vcpu213.2.4. Optimal Location of Emulator Threads
This section provides descriptions for placing emulator threads on the following networks:
- DPDK networking within the instance and netdev datapath in Open vSwitch
- DPDK networking within the instance, system datapath in Open vSwitch and kernel space networking on the hypervisor
- Kernel networking within the instance and netdev datapath in Open vSwitch
13.2.4.1. Optimal Placement of Emulator Threads with DPDK Networking Within the Instance and netdev datapath in Open vSwitch
If DPDK runs within the instance, packet processing is done entirely in the user space. Do not schedule PMDs to run on vCPU0, as this should remain for the OS and interrupt handling. Because PMD CPUs within the instance run an active loop and need 100% of the CPU, they should not be preempted. Packet loss can occur if one of these vCPUs is preempted. Thus, the emulatorpin cpuset needs to be configured in such a way that it does not overlap with the physical CPUs that handle the virtual CPUs numbered 1 and above.
With DPDK networking within the instance, the optimal location for emulator threads is either the pCPU that is handling vCPU 0 or a dedicated physical CPU that is not handling any virtual CPUs at all.
If OVS-DPDK is used on the hypervisor and DPDK within the instance, place the emulator thread on vCPU 0.
13.2.4.2. Optimal Placement of Emulator Threads with DPDK Networking Within the Instance and System datapath in Open vSwitch
If kernel space networking is used on the hypervisor, then packet processing on the hypervisor is executed within the kernel.
With DPDK networking within the instance, the optimal location for emulator threads is either the pCPU that is handling vCPU 0, or a dedicated physical CPU that is not handling any virtual CPUs.
Note that in this scenario, packet processing for the vNIC queues is executed within vhost-<qemu-kvm PID> kernel threads of the hypervisor. Under high traffic, these kernel threads can generate a significant CPU load. The optimal location of the emulator threads needs to be determined on a case-by-case basis.
[root@overcloud-compute-0 ~]# ps aux | grep vhost- root 364948 0.0 0.0 0 0 ? S 20:32 0:00 [vhost-364936] root 364949 0.0 0.0 0 0 ? S 20:32 0:00 [vhost-364936] root 364950 0.0 0.0 0 0 ? S 20:32 0:00 [vhost-364936]
13.2.4.3. Optimal Placement of Emulator Threads with Kernel Networking within the Instance and netdev datapath in Open vSwitch
With kernel networking within the instance, there are two options:
- Optimize the interrupt distribution, for example, softirqs within the instance. In such a case, you do not have to allocate an additional pCPU for emulator threads and can assign the emulator threads to a pCPU that is not handling any network interrupts.
- Use a dedicated pCPU on the same NUMA node for emulator threads.
Due to the complexity of the first option, the second option is recommended.
13.3. Diagnosis
13.3.1. The Demonstration Environment
The demonstration environment runs one instance: instance-0000001d. Its associated qemu-kvm thread has the following PID:
[root@overcloud-compute-0 ~]# pidof qemu-kvm 73517
13.3.2. How Emulatorpin works
By default, a Red Hat OpenStack Platform deployment uses the following settings:
virsh dumpxml instance-0000001d
(...)
<vcpu placement='static'>4</vcpu>
<cputune>
<shares>4096</shares>
<vcpupin vcpu='0' cpuset='34'/>
<vcpupin vcpu='1' cpuset='14'/>
<vcpupin vcpu='2' cpuset='10'/>
<vcpupin vcpu='3' cpuset='30'/>
<emulatorpin cpuset='10,14,30,34'/>
</cputune>
(...)This leads to an unpredictable allocation of the emulator threads, such as qemu-kvm, vnc_worker, and so on:
[root@overcloud-compute-0 ~]# ps -T -p 73517
PID SPID TTY TIME CMD
73517 73517 ? 00:00:00 qemu-kvm
73517 73527 ? 00:00:00 qemu-kvm
73517 73535 ? 00:00:06 CPU 0/KVM
73517 73536 ? 00:00:02 CPU 1/KVM
73517 73537 ? 00:00:03 CPU 2/KVM
73517 73538 ? 00:00:02 CPU 3/KVM
73517 73540 ? 00:00:00 vnc_worker
[root@overcloud-compute-0 ~]# taskset -apc 73517
pid 73517's current affinity list: 10,14,30,34
pid 73527's current affinity list: 10,14,30,34
pid 73535's current affinity list: 34
pid 73536's current affinity list: 14
pid 73537's current affinity list: 10
pid 73538's current affinity list: 30
pid 73540's current affinity list: 10,14,30,34[root@overcloud-compute-0 ~]# virsh vcpupin instance-0000001d | awk '$NF~/[0-9]+/ {print $NF}' | sort -n | while read CPU; do sed '/cpu#/,/runnable task/{//!d}' /proc/sched_debug | sed -n "/^cpu#${CPU},/,/^$/p" ; done
cpu#10, 2197.477 MHz
runnable tasks:
task PID tree-key switches prio wait-time sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
migration/10 64 0.000000 107 0 0.000000 90.232791 0.000000 0 /
ksoftirqd/10 65 -13.045337 3 120 0.000000 0.004679 0.000000 0 /
kworker/10:0 66 -12.892617 40 120 0.000000 0.157359 0.000000 0 /
kworker/10:0H 67 -9.320550 8 100 0.000000 0.015065 0.000000 0 /
kworker/10:1 17996 9695.675528 23 120 0.000000 0.222805 0.000000 0 /
qemu-kvm 73517 1994.534332 27105 120 0.000000 886.203254 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
qemu-kvm 73527 722.347466 84 120 0.000000 18.236155 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
CPU 2/KVM 73537 3356.749162 18051 120 0.000000 3370.045619 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/vcpu2
vnc_worker 73540 354.007735 1 120 0.000000 0.047002 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
worker 74584 1970.499537 5 120 0.000000 0.130143 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
worker 74585 1970.492700 4 120 0.000000 0.071887 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
worker 74586 1982.467246 3 120 0.000000 0.033604 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
worker 74587 1994.520768 1 120 0.000000 0.076039 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
worker 74588 2006.500153 1 120 0.000000 0.004878 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
cpu#14, 2197.477 MHz
runnable tasks:
task PID tree-key switches prio wait-time sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
migration/14 88 0.000000 107 0 0.000000 90.107596 0.000000 0 /
ksoftirqd/14 89 -13.045376 3 120 0.000000 0.004782 0.000000 0 /
kworker/14:0 90 -12.921990 40 120 0.000000 0.128166 0.000000 0 /
kworker/14:0H 91 -9.321186 8 100 0.000000 0.016870 0.000000 0 /
kworker/14:1 17999 6247.571171 5 120 0.000000 0.028576 0.000000 0 /
CPU 1/KVM 73536 2274.381281 6679 120 0.000000 2287.691654 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/vcpu1
cpu#30, 2197.477 MHz
runnable tasks:
task PID tree-key switches prio wait-time sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
migration/30 180 0.000000 107 0 0.000000 89.206960 0.000000 0 /
ksoftirqd/30 181 -13.045892 3 120 0.000000 0.003828 0.000000 0 /
kworker/30:0 182 -12.929272 40 120 0.000000 0.120754 0.000000 0 /
kworker/30:0H 183 -9.321056 8 100 0.000000 0.018042 0.000000 0 /
kworker/30:1 18012 6234.935501 5 120 0.000000 0.026505 0.000000 0 /
CPU 3/KVM 73538 2474.183301 12595 120 0.000000 2487.479666 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/vcpu3
cpu#34, 2197.477 MHz
runnable tasks:
task PID tree-key switches prio wait-time sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
migration/34 204 0.000000 107 0 0.000000 89.067908 0.000000 0 /
ksoftirqd/34 205 -13.046824 3 120 0.000000 0.002884 0.000000 0 /
kworker/34:0 206 -12.922407 40 120 0.000000 0.127423 0.000000 0 /
kworker/34:0H 207 -9.320822 8 100 0.000000 0.017381 0.000000 0 /
kworker/34:1 18016 10788.797590 7 120 0.000000 0.042631 0.000000 0 /
CPU 0/KVM 73535 5969.227225 14233 120 0.000000 5983.425363 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/vcpu0The emulator threads can be moved by using virsh emulatorpin:
virsh emulatorpin instance-0000001d 34
Note that the affinity for all non-CPU threads changes.
[root@overcloud-compute-0 ~]# ps -T -p 73517
PID SPID TTY TIME CMD
73517 73517 ? 00:00:00 qemu-kvm
73517 73527 ? 00:00:00 qemu-kvm
73517 73535 ? 00:00:06 CPU 0/KVM
73517 73536 ? 00:00:02 CPU 1/KVM
73517 73537 ? 00:00:03 CPU 2/KVM
73517 73538 ? 00:00:02 CPU 3/KVM
73517 73540 ? 00:00:00 vnc_worker
[root@overcloud-compute-0 ~]# taskset -apc 73517
pid 73517's current affinity list: 34
pid 73527's current affinity list: 34
pid 73535's current affinity list: 34
pid 73536's current affinity list: 14
pid 73537's current affinity list: 10
pid 73538's current affinity list: 30
pid 73540's current affinity list: 34
Note the number of switches in the historic data in /proc/sched_debug. In the following example, PID 73517 already moved to cpu#34. The other emulator workers did not run since the last output, and therefore still show on cpu#10:
[root@overcloud-compute-0 ~]# virsh vcpupin instance-0000001d | awk '$NF~/[0-9]+/ {print $NF}' | sort -n | while read CPU; do sed '/cpu#/,/runnable task/{//!d}' /proc/sched_debug | sed -n "/^cpu#${CPU},/,/^$/p" ; done
cpu#10, 2197.477 MHz
runnable tasks:
task PID tree-key switches prio wait-time sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
migration/10 64 0.000000 107 0 0.000000 90.232791 0.000000 0 /
ksoftirqd/10 65 -13.045337 3 120 0.000000 0.004679 0.000000 0 /
kworker/10:0 66 -12.892617 40 120 0.000000 0.157359 0.000000 0 /
kworker/10:0H 67 -9.320550 8 100 0.000000 0.015065 0.000000 0 /
kworker/10:1 17996 9747.429082 26 120 0.000000 0.255547 0.000000 0 /
qemu-kvm 73527 722.347466 84 120 0.000000 18.236155 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
CPU 2/KVM 73537 3424.520709 21610 120 0.000000 3437.817166 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/vcpu2
vnc_worker 73540 354.007735 1 120 0.000000 0.047002 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
cpu#14, 2197.477 MHz
runnable tasks:
task PID tree-key switches prio wait-time sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
migration/14 88 0.000000 107 0 0.000000 90.107596 0.000000 0 /
ksoftirqd/14 89 -13.045376 3 120 0.000000 0.004782 0.000000 0 /
kworker/14:0 90 -12.921990 40 120 0.000000 0.128166 0.000000 0 /
kworker/14:0H 91 -9.321186 8 100 0.000000 0.016870 0.000000 0 /
kworker/14:1 17999 6247.571171 5 120 0.000000 0.028576 0.000000 0 /
CPU 1/KVM 73536 2283.094453 7028 120 0.000000 2296.404826 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/vcpu1
cpu#30, 2197.477 MHz
runnable tasks:
task PID tree-key switches prio wait-time sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
migration/30 180 0.000000 107 0 0.000000 89.206960 0.000000 0 /
ksoftirqd/30 181 -13.045892 3 120 0.000000 0.003828 0.000000 0 /
kworker/30:0 182 -12.929272 40 120 0.000000 0.120754 0.000000 0 /
kworker/30:0H 183 -9.321056 8 100 0.000000 0.018042 0.000000 0 /
kworker/30:1 18012 6234.935501 5 120 0.000000 0.026505 0.000000 0 /
CPU 3/KVM 73538 2521.828931 14047 120 0.000000 2535.125296 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/vcpu3
cpu#34, 2197.477 MHz
runnable tasks:
task PID tree-key switches prio wait-time sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
migration/34 204 0.000000 107 0 0.000000 89.067908 0.000000 0 /
ksoftirqd/34 205 -13.046824 3 120 0.000000 0.002884 0.000000 0 /
kworker/34:0 206 -12.922407 40 120 0.000000 0.127423 0.000000 0 /
kworker/34:0H 207 -9.320822 8 100 0.000000 0.017381 0.000000 0 /
kworker/34:1 18016 10788.797590 7 120 0.000000 0.042631 0.000000 0 /
qemu-kvm 73517 2.613794 27706 120 0.000000 941.839262 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
CPU 0/KVM 73535 5994.533905 15169 120 0.000000 6008.732043 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/vcpu0Note how thread 73517 moves to cpu#34. If you now interact with a VNC session, you can see that /proc/sched_debug shows the vnc_worker threads on cpu#34 as well.
[root@overcloud-compute-0 ~]# virsh vcpupin instance-0000001d | awk '$NF~/[0-9]+/ {print $NF}' | sort -n | while read CPU; do sed '/cpu#/,/runnable task/{//!d}' /proc/sched_debug | sed -n "/^cpu#${CPU},/,/^$/p" ; done
cpu#10, 2197.477 MHz
runnable tasks:
task PID tree-key switches prio wait-time sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
migration/10 64 0.000000 107 0 0.000000 90.232791 0.000000 0 /
ksoftirqd/10 65 -13.045337 3 120 0.000000 0.004679 0.000000 0 /
kworker/10:0 66 -12.892617 40 120 0.000000 0.157359 0.000000 0 /
kworker/10:0H 67 -9.320550 8 100 0.000000 0.015065 0.000000 0 /
kworker/10:1 17996 9963.300958 27 120 0.000000 0.273007 0.000000 0 /
qemu-kvm 73527 722.347466 84 120 0.000000 18.236155 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
CPU 2/KVM 73537 3563.793234 26162 120 0.000000 3577.089691 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/vcpu2
cpu#14, 2197.477 MHz
runnable tasks:
task PID tree-key switches prio wait-time sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
migration/14 88 0.000000 107 0 0.000000 90.107596 0.000000 0 /
ksoftirqd/14 89 -13.045376 3 120 0.000000 0.004782 0.000000 0 /
kworker/14:0 90 -12.921990 40 120 0.000000 0.128166 0.000000 0 /
kworker/14:0H 91 -9.321186 8 100 0.000000 0.016870 0.000000 0 /
kworker/14:1 17999 6247.571171 5 120 0.000000 0.028576 0.000000 0 /
CPU 1/KVM 73536 2367.789075 9648 120 0.000000 2381.099448 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/vcpu1
cpu#30, 2197.477 MHz
runnable tasks:
task PID tree-key switches prio wait-time sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
migration/30 180 0.000000 107 0 0.000000 89.206960 0.000000 0 /
ksoftirqd/30 181 -13.045892 3 120 0.000000 0.003828 0.000000 0 /
kworker/30:0 182 -12.929272 40 120 0.000000 0.120754 0.000000 0 /
kworker/30:0H 183 -9.321056 8 100 0.000000 0.018042 0.000000 0 /
kworker/30:1 18012 6234.935501 5 120 0.000000 0.026505 0.000000 0 /
CPU 3/KVM 73538 2789.628278 24788 120 0.000000 2802.924643 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/vcpu3
cpu#34, 2197.477 MHz
runnable tasks:
task PID tree-key switches prio wait-time sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
migration/34 204 0.000000 107 0 0.000000 89.067908 0.000000 0 /
ksoftirqd/34 205 -13.046824 3 120 0.000000 0.002884 0.000000 0 /
kworker/34:0 206 -12.922407 40 120 0.000000 0.127423 0.000000 0 /
kworker/34:0H 207 -9.320822 8 100 0.000000 0.017381 0.000000 0 /
kworker/34:1 18016 11315.391422 25 120 0.000000 0.196078 0.000000 0 /
qemu-kvm 73517 471.930276 30975 120 0.000000 1295.543576 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
CPU 0/KVM 73535 6160.062172 19201 120 0.000000 6174.260310 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/vcpu0
vnc_worker 73540 459.653524 38 120 0.000000 7.535037 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
worker 78703 449.098251 2 120 0.000000 0.120313 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
worker 78704 449.131175 3 120 0.000000 0.066961 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator
worker 78705 461.100994 4 120 0.000000 0.022897 0.000000 0 /machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000001d.scope/emulator