netperf shows UDP stream sender throughput decreased by 10%~73% on RHEL8 guests. Severe performance degradation caused by some potential bug.

Issue

• Possible severe performance degradation is observed on UDP stream sender throuput on RHEL8 KVM guests.

• netperf shows UDP stream sender throughput is decreased by 10%~73% on RHEL8 KVM guests running on RHEL8 KVM host.

• From the data just right below we can at least make sure that Spectre v2 mitigation has nothing to do with the performance difference.

  • RHEL7.7 guest is on RHEL7.7 KVM host and RHEL8.2 guest is on RHEL8.2 KVM host.
  • (A) means Spectre v2 mitigation is enabled (default)
  • (B) means Spectre v2 mitigation is disabled.

Tool    Machine           Category     Test_item        8.2(A)  8.2(B)  ratio   7.7(B)  8.2(B)  ratio
-------------------------------------------------------------------------------------------------------------------------------
netperf hpg10-4.MultiCore.TEST_V2V  Sender Throughput UDP_STREAM_1472   6455    13514   109.36% 15733   13514   -14.10%
netperf hpg10-4.MultiCore.TEST_V2V  Sender Throughput UDP_STREAM_466    2027    3318    63.69%  5697    3318    -41.76%
netperf hpg10-4.MultiCore.TEST_V2V  Sender Throughput UDP_STREAM_18 79  122 54.43%  228 122 -46.55%
netperf hpg10-4.MultiCore.TEST_V2V  Sender Throughput UDP_STREAM_210    914 1396    52.74%  2453    1396    -43.08%
netperf hpg10-4.MultiCore.TEST_V2V  Sender Throughput UDP_STREAM_978    4273    6667    56.03%  10850   6667    -38.55%
netperf hpg10-4.MultiCore.TEST_V2V  Sender Throughput UDP_STREAM_82 356 550 54.49%  1071    550 -48.65%
netperf hpg10-4.MultiCore.TEST_V2P  Sender Throughput UDP_STREAM_1472   6219    13872   123.06% 15406   13872   -9.95%
netperf hpg10-4.MultiCore.TEST_V2P  Sender Throughput UDP_STREAM_466    1951    3372    72.83%  5220    3372    -35.41%
netperf hpg10-4.MultiCore.TEST_V2P  Sender Throughput UDP_STREAM_18 76  124 63.16%  225 124 -44.78%
netperf hpg10-4.MultiCore.TEST_V2P  Sender Throughput UDP_STREAM_210    883 1483    67.95%  2490    1483    -40.45%
netperf hpg10-4.MultiCore.TEST_V2P  Sender Throughput UDP_STREAM_978    4106    7000    70.48%  10759   7000    -34.94%
netperf hpg10-4.MultiCore.TEST_V2P  Sender Throughput UDP_STREAM_82 341 550 61.29%  1031    550 -46.65%
netperf hpg10-4.SingleCore.TEST_V2V Sender Throughput UDP_STREAM_1472   1695    2345    38.35%  6756    2345    -65.29%
netperf hpg10-4.SingleCore.TEST_V2V Sender Throughput UDP_STREAM_466    555 711 28.11%  2262    711 -68.56%
netperf hpg10-4.SingleCore.TEST_V2V Sender Throughput UDP_STREAM_18 22  24  9.09%   93  24  -74.10%
netperf hpg10-4.SingleCore.TEST_V2V Sender Throughput UDP_STREAM_210    251 306 21.91%  1021    306 -70.03%
netperf hpg10-4.SingleCore.TEST_V2V Sender Throughput UDP_STREAM_978    1148    1427    24.30%  4552    1427    -68.65%
netperf hpg10-4.SingleCore.TEST_V2V Sender Throughput UDP_STREAM_82 98  116 18.37%  419 116 -72.35%
netperf hpg10-4.SingleCore.TEST_V2P Sender Throughput UDP_STREAM_1472   1687    2286    35.51%  6715    2286    -65.96%
netperf hpg10-4.SingleCore.TEST_V2P Sender Throughput UDP_STREAM_466    558 732 31.18%  2245    732 -67.39%
netperf hpg10-4.SingleCore.TEST_V2P Sender Throughput UDP_STREAM_18 22  24  9.09%   92  24  -73.84%
netperf hpg10-4.SingleCore.TEST_V2P Sender Throughput UDP_STREAM_210    254 309 21.65%  1014    309 -69.51%
netperf hpg10-4.SingleCore.TEST_V2P Sender Throughput UDP_STREAM_978    1143    1415    23.80%  4533    1415    -68.79%
netperf hpg10-4.SingleCore.TEST_V2P Sender Throughput UDP_STREAM_82 100 113 13.00%  417 113 -72.92%

• Perf data captured on RHEL7/8 guests during the time frame when netperf run the UDP stream sender throughput tests:

[RHEL7.7 guest]

# Children      Self       Samples  Command  Shared Object     Symbol                                 
# ........  ........  ............  .......  ................  .......................................
#
    98.28%    13.69%         32825  netperf  libc-2.17.so      [.] __GI___libc_sendto
13.69% __libc_start_main;main;send_udp_stream;send_omni_inner;__GI___libc_sendto
9.77% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;udp_send_skb;ip_send_skb;ip_local_out_sk;ip_output;ip_finish_output;dev_queue_xmit;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc023e584;0xffffffffc018b740;iowrite16
6.67% __GI___libc_sendto;system_call
3.97% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;udp_send_skb;ip_send_skb;ip_local_out_sk;ip_output;ip_finish_output;neigh_resolve_output;dev_queue_xmit;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc023e584;0xffffffffc018b740;iowrite16
3.18% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;ip_route_output_flow;__ip_route_output_key_hash;fib_table_lookup
2.04% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg
1.89% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;udp_send_skb;ip_send_skb;ip_local_out_sk;ip_output;ip_finish_output
1.80% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.50
1.53% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;udp_send_skb;ip_send_skb;ip_local_out_sk;ip_output;ip_finish_output;dev_queue_xmit;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc023e37f;0xffffffffc023da40;0xffffffffc018bda1
1.50% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;ip_make_skb;__ip_make_skb
1.46% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.50;sock_alloc_send_skb;sock_alloc_send_pskb;alloc_skb_with_frags;__alloc_skb
1.44% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;ip_route_output_flow;__ip_route_output_key_hash
1.38% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;udp_send_skb;ip_send_skb;ip_local_out_sk;ip_output;ip_finish_output;dev_queue_xmit;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc023e4f5;sock_wfree;sock_def_write_space
1.34% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.50;ip_generic_getfrag;copy_user_enhanced_fast_string
1.26% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;copy_user_enhanced_fast_string
1.02% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg
1.01% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.50;ip_generic_getfrag;memcpy_fromiovecend;__check_object_size;__check_heap_object
1.01% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;udp_send_skb;ip_send_skb;ip_local_out_sk;ip_output;ip_finish_output;dev_queue_xmit;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc023e440;memset
0.92% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sockfd_lookup_light;fget_light
0.92% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.50;sock_alloc_send_skb;sock_alloc_send_pskb;alloc_skb_with_frags;__alloc_skb;kmem_cache_alloc_node
0.75% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;inet_sendmsg
0.74% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.50;sock_alloc_send_skb;sock_alloc_send_pskb;alloc_skb_with_frags;__alloc_skb;__kmalloc_reserve.isra.32;__kmalloc_node_track_caller
0.70% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg
0.67% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;ip_make_skb;__ip_make_skb;__ip_select_ident;ip_idents_reserve
0.64% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;udp_send_skb;ip_send_skb;ip_local_out_sk;ip_output;ip_finish_output;dev_queue_xmit;__dev_queue_xmit;sch_direct_xmit;validate_xmit_skb_list;validate_xmit_skb.isra.109.part.110;netif_skb_features
0.64% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;cap_sk_getsecid
0.63% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;udp_send_skb;ip_send_skb;ip_local_out_sk;__ip_local_out_sk
0.61% __GI___libc_sendto;auditsys;__audit_syscall_entry
0.60% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto
0.60% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;ip_make_skb;__ip_make_skb;__ip_select_ident
0.58% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;udp_send_skb;ip_send_skb;ip_local_out_sk;ip_output
0.58% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;udp_sendmsg
0.57% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;cap_socket_sendmsg
0.57% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;udp_send_skb;ip_send_skb;ip_local_out_sk;ip_output;ip_finish_output;dev_queue_xmit;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit
0.56% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.50;ip_generic_getfrag;memcpy_fromiovecend
0.55% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;ip_make_skb;ipv4_mtu
0.53% __GI___libc_sendto;sysret_audit;__audit_syscall_exit
0.51% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;udp_send_skb
0.51% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;udp_send_skb;ip_send_skb;ip_local_out_sk;ip_output;ip_finish_output;dev_queue_xmit;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc023e440;sg_init_table
0.50% __GI___libc_sendto;system_call;sys_sendto;SYSC_sendto;sock_sendmsg;inet_sendmsg;udp_sendmsg;udp_send_skb;ip_send_skb;ip_local_out_sk;ip_output;ip_finish_output;dev_queue_xmit;__dev_queue_xmit;sch_direct_xmit;validate_xmit_skb_list;validate_xmit_skb.isra.109.part.110;passthru_features_check
    ...

[RHEL8.0 guest]

# Children      Self       Samples  Command  Shared Object     Symbol                                        
# ........  ........  ............  .......  ................  ..............................................
#
   100.00%     0.00%             0  netperf  [unknown]         [k] 0x5541f689495641d7
23.67% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.43;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc01dc088;virtqueue_kick;vp_notify;iowrite16
6.56% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64
6.04% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto
4.43% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.43;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc01dbdfe;0xffffffffc01da239;virtqueue_get_buf_ctx
3.61% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_route_output_flow;ip_route_output_key_hash;ip_route_output_key_hash_rcu;fib_table_lookup
2.73% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg
2.64% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.43;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc01dc163;sock_wfree;sock_def_write_space
1.89% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.49
1.79% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.43;ip_send_skb;ip_output;ip_finish_output2
1.48% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.49;sock_alloc_send_pskb;alloc_skb_with_frags;__alloc_skb;kmem_cache_alloc_node
1.23% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.43;ip_send_skb;ip_output
1.22% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.49;sock_alloc_send_pskb;alloc_skb_with_frags;__alloc_skb
1.19% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_make_skb;__ip_make_skb
1.05% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.43;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit
1.04% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;inet_sendmsg
1.03% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.49;sock_alloc_send_pskb;alloc_skb_with_frags;__alloc_skb;__kmalloc_reserve.isra.49;__kmalloc_node_track_caller
0.96% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.49;ip_generic_getfrag
0.95% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner
0.93% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.49;ip_generic_getfrag;_copy_from_iter_full;copyin;copy_user_generic_unrolled
0.91% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.43;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc01dc031;virtqueue_add_outbuf
0.89% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.49;ip_generic_getfrag;__check_object_size;__check_heap_object
0.88% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_route_output_flow;ip_route_output_key_hash;ip_route_output_key_hash_rcu
0.86% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto
0.85% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;send_data
0.83% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.43;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;validate_xmit_skb_list;validate_xmit_skb;netif_skb_features
0.76% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sockfd_lookup_light;__fget_light
0.76% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.43;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc01dc163;sock_wfree
0.72% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;move_addr_to_kernel.part.22;_copy_from_user;copy_user_generic_unrolled
0.69% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;syscall_slow_exit_work;__audit_syscall_exit
0.68% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.43;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc01dc088;virtqueue_kick;virtqueue_kick_prepare
0.64% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_make_skb;__ip_make_skb;ipv4_mtu
0.61% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.49;ip_generic_getfrag;__check_object_size;__virt_addr_valid
0.59% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_make_skb;__ip_make_skb;__ip_select_ident;ip_idents_reserve
0.56% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;syscall_trace_enter;__audit_syscall_entry
0.55% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.43;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc01dbd90
0.53% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.43;ip_send_skb;ip_local_out;__ip_local_out;ip_send_check
0.53% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.43
0.53% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.43;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit
0.52% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.49;sock_alloc_send_pskb
0.52% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.49;ip_generic_getfrag;__check_object_size
0.51% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.43;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc01dbdfe;0xffffffffc01da22b;consume_skb;kfree
    ...

[RHEL8.2 guest]

# Children      Self       Samples  Command  Shared Object     Symbol                                        
# ........  ........  ............  .......  ................  ..............................................
#
    99.99%     0.00%             0  netperf  [unknown]         [k] 0x5541f689495641d7
63.46% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.45;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc02760df;virtqueue_kick;vp_notify;iowrite16
5.47% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto
2.33% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64
1.43% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_route_output_flow;ip_route_output_key_hash;ip_route_output_key_hash_rcu;fib_table_lookup
1.40% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.45;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc0275e5c;0xffffffffc0274239;virtqueue_get_buf_ctx
1.14% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg
0.91% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.45;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc02760df;virtqueue_kick;virtqueue_kick_prepare
0.90% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.45;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc02761ba;sock_wfree;sock_def_write_space
0.84% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.45;ip_send_skb;ip_output;ip_finish_output2
0.79% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.52;sock_alloc_send_pskb;alloc_skb_with_frags;__alloc_skb;kmem_cache_alloc_node
0.69% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.45;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;__qdisc_run;fq_codel_dequeue
0.61% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.45;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit;0xffffffffc02760df;virtqueue_kick;vp_notify
0.59% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_route_output_flow;ip_route_output_key_hash;ip_route_output_key_hash_rcu
0.56% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;ip_make_skb;__ip_append_data.isra.52;sock_alloc_send_pskb;alloc_skb_with_frags;__alloc_skb
0.55% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.45;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit
0.54% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_sendto;__sys_sendto;sockfd_lookup_light;__fget_light
0.52% 0x5541f689495641d7;__libc_start_main;main;send_udp_stream;send_omni_inner;__libc_sendto;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_sendto;__sys_sendto;sock_sendmsg;udp_sendmsg;udp_send_skb.isra.45;ip_send_skb;ip_output;ip_finish_output2;__dev_queue_xmit;sch_direct_xmit;dev_hard_start_xmit
    ...