High CPU utilization (%sys) and load average due to simultaneous sshd processes

Issue

• High CPU utilization (%sys) and load average due to simultaneous sshd processes causing slowness.

• The top command shows the high CPU utilization is due to the sshd processes.

top - 17:07:31 up 40 min,  3 users,  load average: 50.00, 22.20, 31.46
Tasks: 337 total,  73 running, 263 sleeping,   0 stopped,   1 zombie
%Cpu(s):  8.2 us, 91.6 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem : 32761428 total, 28172572 free,  1128708 used,  3460148 buff/cache
KiB Swap:  4063228 total,  4063228 free,        0 used. 31193440 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 88231 root      20   0  148364   5172   3900 R  20.5  0.0   0:02.53 sshd
 88130 root      20   0  148364   5172   3900 R  20.1  0.0   0:02.83 sshd
 88582 root      20   0  148364   5172   3900 R  20.1  0.0   0:01.09 sshd
 88075 root      20   0  148364   5172   3900 R  19.8  0.0   0:03.28 sshd
 88292 root      20   0  148364   5172   3900 R  19.8  0.0   0:02.14 sshd
 88567 root      20   0  148364   5172   3900 R  19.8  0.0   0:01.21 sshd
 87791 root      20   0  148364   5168   3896 R  19.5  0.0   0:03.31 sshd
 87796 root      20   0  148364   5172   3900 R  19.5  0.0   0:03.48 sshd
 88015 root      20   0  148364   5164   3896 R  19.5  0.0   0:02.95 sshd
 88049 root      20   0  148364   5172   3900 R  19.5  0.0   0:03.11 sshd
 88451 root      20   0  148364   5168   3900 R  19.5  0.0   0:01.55 sshd

• The sshd sessions are getting hung or timed out, and the CPU usage (%sys) and load average are skyrocketing.

04:30:02 PM     CPU      %usr     %nice      %sys   %iowait    %steal      %irq     %soft    %guest    %gnice     %idle
05:40:01 PM     all      9.07      0.00     90.73      0.00      0.00      0.00      0.18      0.00      0.00      0.01
05:40:01 PM       0      8.28      0.00     91.37      0.00      0.00      0.00      0.33      0.00      0.00      0.01
05:40:01 PM       1      8.85      0.00     90.97      0.00      0.00      0.00      0.16      0.00      0.00      0.01
05:40:01 PM       2      9.54      0.00     90.28      0.00      0.00      0.00      0.17      0.00      0.00      0.01
05:40:01 PM       3     10.07      0.00     89.71      0.00      0.00      0.00      0.20      0.00      0.00      0.01
05:40:01 PM       4      8.55      0.00     91.43      0.00      0.00      0.00      0.00      0.00      0.00      0.02
05:40:01 PM       5      9.24      0.00     90.74      0.00      0.00      0.00      0.00      0.00      0.00      0.02
05:40:01 PM       6      8.24      0.00     91.74      0.00      0.00      0.00      0.00      0.00      0.00      0.01
05:40:01 PM       7      8.18      0.00     91.80      0.00      0.00      0.00      0.00      0.00      0.00      0.01
05:40:01 PM       8     10.25      0.00     89.45      0.00      0.00      0.00      0.28      0.00      0.00      0.02
05:40:01 PM       9      9.01      0.00     90.34      0.00      0.00      0.00      0.64      0.00      0.00      0.01
05:40:01 PM      10      9.25      0.00     90.51      0.00      0.00      0.00      0.22      0.00      0.00      0.01
05:40:01 PM      11      9.41      0.01     90.37      0.00      0.00      0.00      0.21      0.00      0.00      0.01

04:30:02 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
04:40:01 PM        85       466     71.93     63.77     37.06         0
04:50:01 PM        72       483     69.17     68.32     52.33         0
05:00:01 PM         2       366      0.55     25.65     41.42         0
05:10:01 PM        59       470     66.66     41.16     37.32         0
05:20:01 PM        80       533     68.94     64.53     52.03         0
05:30:01 PM        89       528     69.12     68.23     59.93         0
05:40:01 PM        86       548     66.95     67.53     63.72         0

• SSH logins exhibit a delay of 10 seconds or more when attempting to connect to a server.