Why does a system high-load occur? Load average is high

Issue

• Why does a system high-load occur?
• System's load averages go up.

$ sar -q 
00:00:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
<snip>
14:50:01            0       322      0.07      0.05      0.04
15:20:01            0       331      2.46     53.30     68.72
15:42:19           95       343    109.15    107.25     91.37
16:00:01            0       311     38.94     87.63     98.53
16:10:01            0       297      0.00     11.75     51.62
16:20:01            0       295      0.00      1.56     27.03

For 8 CPU server, load average is more than 8 and hence high:

$ uptime
 15:11:10 up 29 days, 20:57,  1 user,  load average: 43.41, 27.10, 21.40