High Load on Compute nodes and VM creation failing
Issue
-
High Load Average on compute node and VM creation failing on this node
-
We can see the load is high using
sar -q ALL
:
12:00:00 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
12:10:05 AM 21 3890 128.23 136.66 141.83 0
12:20:05 AM 28 3686 119.47 126.71 134.28 0
12:30:00 AM 20 3658 114.30 121.70 128.62 0
12:40:01 AM 29 3710 125.96 129.01 129.70 0
12:50:00 AM 16 3821 137.67 130.63 129.05 0
01:00:00 AM 29 3556 146.85 132.35 129.71 1
01:10:05 AM 24 3616 159.36 140.78 134.55 1
01:20:00 AM 25 4084 125.34 140.30 137.95 1
01:30:01 AM 24 4052 168.70 151.68 143.21 0
01:40:01 AM 43 3980 150.89 146.35 143.93 1
01:50:00 AM 38 3691 132.94 131.21 137.66 0
02:00:01 AM 31 3888 171.58 153.28 143.48 1
02:10:05 AM 17 4034 143.62 152.79 149.33 0
02:20:05 AM 16 3898 147.37 147.48 148.68 0
02:30:01 AM 39 4231 147.04 144.43 146.14 0
02:40:00 AM 22 3704 147.74 139.46 141.47 0
02:50:05 AM 19 4087 139.70 131.96 134.96 0
03:00:01 AM 30 3826 148.53 149.53 144.57 1
03:10:05 AM 22 3785 152.98 152.73 150.21 1
03:20:05 AM 24 4027 178.73 164.60 156.03 1
- CPUs are idling with
sar
:
12:00:00 AM CPU %user %nice %system %iowait %steal %idle
12:10:05 AM all 26.79 0.00 5.81 0.29 0.00 67.12
12:20:05 AM all 26.79 0.00 5.93 0.33 0.00 66.96
12:30:00 AM all 28.50 0.00 6.11 0.31 0.00 65.08
12:40:01 AM all 26.99 0.00 6.32 0.35 0.00 66.34
12:50:00 AM all 24.87 0.00 5.84 0.31 0.00 68.97
01:00:00 AM all 25.67 0.00 5.48 0.33 0.00 68.52
01:10:05 AM all 26.22 0.00 6.33 0.35 0.00 67.10
01:20:00 AM all 26.56 0.00 6.25 0.35 0.00 66.84
01:30:01 AM all 26.19 0.00 5.72 0.36 0.00 67.73
01:40:01 AM all 25.25 0.00 5.34 0.35 0.00 69.06
01:50:00 AM all 24.83 0.00 5.97 0.29 0.00 68.91
02:00:01 AM all 26.31 0.00 5.90 0.36 0.00 67.43
02:10:05 AM all 26.67 0.00 5.68 0.39 0.00 67.26
02:20:05 AM all 25.98 0.00 5.57 0.36 0.00 68.10
02:30:01 AM all 25.54 0.00 5.53 0.37 0.00 68.57
02:40:00 AM all 26.45 0.00 5.66 0.34 0.00 67.55
02:50:05 AM all 24.83 0.00 5.25 0.31 0.00 69.61
03:00:01 AM all 25.83 0.00 5.30 0.38 0.00 68.48
03:10:05 AM all 26.87 0.00 5.44 0.37 0.00 67.32
03:20:05 AM all 28.38 0.00 5.38 0.35 0.00 65.90
Environment
- Red Hat OpenStack Platform 16.2 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.