The system is experiencing high %sys CPU usage, primarily due to IBM Tivoli Monitoring processes (klzagent and k08agent) handling a large volume of incoming TCP packet processing

Issue

• The system is showing relatively high %sys CPU usage.
• Perf data captured during the issue window implies that IBM Tivoli Monitoring processes (klzagent and k08agent) together, are consuming ~44.20+% of total sampled CPU cycles:
  • klzagent: ~24.37% Total
  • k08agent: ~19.82% Total

Samples: 160K of event 'cpu-clock:pppH', Event count (approx.)
  Overhead  Command          Shared Object
-   46.47%  swapper          [kernel.kallsyms]               ◆
   + 42.91% native_safe_halt                                 ▒
   + 2.38% finish_task_switch                                ▒
-   17.83%  klzagent         [kernel.kallsyms]               ▒
           - 4.50% _raw_spin_unlock_irqrestore                       ▒
              - 3.88% __wake_up_common_lock                          ▒
                 - 3.86% sock_def_readable                           ▒
                    - 3.86% tcp_rcv_established                      ▒
                         tcp_v4_do_rcv                               ▒
                         tcp_v4_rcv                                  ▒
                         ip_protocol_deliver_rcu                     ▒
                         ip_local_deliver_finish                     ▒
                         ip_local_deliver                            ▒
                         ip_rcv                                      ▒
                         __netif_receive_skb_core                    ▒
                         process_backlog                             ▒
                         __napi_poll                                 ▒
                         net_rx_action                               ▒
                         __softirqentry_text_start                   ▒
                         do_softirq_own_stack                        ▒
                         do_softirq.part.16                          ▒
                         __local_bh_enable_ip                        ▒
                         ip_finish_output2                           ▒
                         ip_output                                   ▒
                         __ip_queue_xmit                             ▒
                         __tcp_transmit_skb                          ▒
                         tcp_write_xmit                              ▒
                         __tcp_push_pending_frames                   ▒
                         tcp_sendmsg_locked                          ▒
                         tcp_sendmsg                                 ▒
                         __sock_sendmsg                              ▒
                         __sys_sendto                                ▒
                         __x64_sys_sendto                            ▒
                         do_syscall_64                               ▒
                         entry_SYSCALL_64_after_hwframe              ▒
                         0x153ef7f1bb66                              ▒
           + 1.35% do_syscall_64                                     ▒
           + 0.75% finish_task_switch                                ▒
-   14.39%  k08agent         [kernel.kallsyms]               ▒
           - 3.45% _raw_spin_unlock_irqrestore                       ▒
              - 2.79% __wake_up_common_lock                          ▒
                 + 2.78% sock_def_readable                           ▒
   + 1.20% do_syscall_64                                     ▒
   + 0.64% finish_task_switch                                ▒
+    3.24%  klzagent         libpthread-2.28.so              ▒
+    2.65%  k08agent         libpthread-2.28.so              ▒
        ...