High Load on Compute nodes and VM creation failing

Issue

• High Load Average on compute node and VM creation failing on this node

• We can see the load is high using sar -q ALL:

12:00:00 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
12:10:05 AM        21      3890    128.23    136.66    141.83         0
12:20:05 AM        28      3686    119.47    126.71    134.28         0
12:30:00 AM        20      3658    114.30    121.70    128.62         0
12:40:01 AM        29      3710    125.96    129.01    129.70         0
12:50:00 AM        16      3821    137.67    130.63    129.05         0
01:00:00 AM        29      3556    146.85    132.35    129.71         1
01:10:05 AM        24      3616    159.36    140.78    134.55         1
01:20:00 AM        25      4084    125.34    140.30    137.95         1
01:30:01 AM        24      4052    168.70    151.68    143.21         0
01:40:01 AM        43      3980    150.89    146.35    143.93         1
01:50:00 AM        38      3691    132.94    131.21    137.66         0
02:00:01 AM        31      3888    171.58    153.28    143.48         1
02:10:05 AM        17      4034    143.62    152.79    149.33         0
02:20:05 AM        16      3898    147.37    147.48    148.68         0
02:30:01 AM        39      4231    147.04    144.43    146.14         0
02:40:00 AM        22      3704    147.74    139.46    141.47         0
02:50:05 AM        19      4087    139.70    131.96    134.96         0
03:00:01 AM        30      3826    148.53    149.53    144.57         1
03:10:05 AM        22      3785    152.98    152.73    150.21         1
03:20:05 AM        24      4027    178.73    164.60    156.03         1

• CPUs are idling with sar:

12:00:00 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:05 AM     all     26.79      0.00      5.81      0.29      0.00     67.12
12:20:05 AM     all     26.79      0.00      5.93      0.33      0.00     66.96
12:30:00 AM     all     28.50      0.00      6.11      0.31      0.00     65.08
12:40:01 AM     all     26.99      0.00      6.32      0.35      0.00     66.34
12:50:00 AM     all     24.87      0.00      5.84      0.31      0.00     68.97
01:00:00 AM     all     25.67      0.00      5.48      0.33      0.00     68.52
01:10:05 AM     all     26.22      0.00      6.33      0.35      0.00     67.10
01:20:00 AM     all     26.56      0.00      6.25      0.35      0.00     66.84
01:30:01 AM     all     26.19      0.00      5.72      0.36      0.00     67.73
01:40:01 AM     all     25.25      0.00      5.34      0.35      0.00     69.06
01:50:00 AM     all     24.83      0.00      5.97      0.29      0.00     68.91
02:00:01 AM     all     26.31      0.00      5.90      0.36      0.00     67.43
02:10:05 AM     all     26.67      0.00      5.68      0.39      0.00     67.26
02:20:05 AM     all     25.98      0.00      5.57      0.36      0.00     68.10
02:30:01 AM     all     25.54      0.00      5.53      0.37      0.00     68.57
02:40:00 AM     all     26.45      0.00      5.66      0.34      0.00     67.55
02:50:05 AM     all     24.83      0.00      5.25      0.31      0.00     69.61
03:00:01 AM     all     25.83      0.00      5.30      0.38      0.00     68.48
03:10:05 AM     all     26.87      0.00      5.44      0.37      0.00     67.32
03:20:05 AM     all     28.38      0.00      5.38      0.35      0.00     65.90