High CPU Consumption due the process kblockd
Hi Guys,
We are having issues with one of our database server. The vm is working stremly slow and theses processes are consumming the most of resources. We apply some updates in the database but the issue persist.
This is the information we have
# ps -eo pcpu,pid,user,args | sort -k1 -r | head -10
%CPU PID USER COMMAND
9.8 48 root [kblockd/2]
9.5 46 root [kblockd/0]
8.7 14601 oracle oracleCDB (LOCAL=NO)
7.3 14417 oracle ora_j006_CDB
6.6 12177 oracle ora_j007_CDB
6.5 51 root [kblockd/5]
6.5 49 root [kblockd/3]
6.4 47 root [kblockd/1]
6.3 15182 oracle ora_smon_CDB
1 17:28:22 2017 from 10.2.222.198
[root@cebdbprod01 ~]# sar
Linux 2.6.32-504.1.3.el6.x86_64 (cebdbprod01.cebroker.com) 09/21/2017 _x86_64_ (6 CPU)
12:00:01 AM CPU %user %nice %system %iowait %steal %idle
12:10:01 AM all 3.63 0.00 11.36 29.31 0.00 55.71
12:20:01 AM all 3.06 0.00 5.59 24.87 0.00 66.48
12:30:01 AM all 6.41 0.00 8.41 32.60 0.00 52.59
12:40:01 AM all 10.94 0.00 11.19 38.67 0.00 39.20
12:50:01 AM all 8.91 0.00 11.79 34.07 0.00 45.23
01:00:01 AM all 10.22 0.00 5.42 15.92 0.00 68.44
01:10:01 AM all 35.41 0.00 14.01 30.25 0.00 20.34
01:20:01 AM all 28.42 0.00 9.51 30.85 0.00 31.22
01:30:01 AM all 39.56 0.00 12.17 25.90 0.00 22.37
01:40:01 AM all 47.50 0.00 19.24 25.53 0.00 7.74
01:50:01 AM all 37.65 0.00 15.22 26.60 0.00 20.53
02:00:01 AM all 26.99 0.00 10.25 28.61 0.00 34.15
02:10:01 AM all 29.95 0.00 16.34 35.41 0.00 18.30
02:20:01 AM all 30.37 0.00 14.95 35.29 0.00 19.40
02:30:02 AM all 37.89 0.00 21.95 32.17 0.00 7.99
02:40:01 AM all 58.70 0.00 22.53 15.47 0.00 3.29
02:50:01 AM all 47.85 0.00 18.74 23.91 0.00 9.50
03:00:01 AM all 42.69 0.00 14.20 27.53 0.00 15.59
03:10:01 AM all 51.03 0.00 23.60 19.91 0.00 5.46
03:20:01 AM all 49.73 0.00 18.00 23.91 0.00 8.36
03:30:01 AM all 46.78 0.00 29.57 19.39 0.00 4.26
03:40:01 AM all 68.18 0.00 21.77 8.15 0.00 1.90
03:50:01 AM all 73.76 0.14 22.48 3.43 0.00 0.19
04:00:01 AM all 71.07 0.00 22.90 5.40 0.00 0.62
04:10:02 AM all 59.38 0.00 36.65 3.64 0.00 0.33
04:20:01 AM all 61.65 0.00 28.05 9.06 0.00 1.23
04:30:01 AM all 51.38 0.00 27.67 17.33 0.00 3.62
04:40:01 AM all 47.51 0.00 29.23 20.01 0.00 3.25
04:50:01 AM all 36.59 0.00 30.28 25.68 0.00 7.44
05:00:01 AM all 29.89 0.00 20.25 33.53 0.00 16.33
05:10:01 AM all 37.57 0.00 29.82 27.26 0.00 5.34
05:20:01 AM all 34.77 0.00 27.37 29.05 0.00 8.82
05:30:01 AM all 35.83 0.00 22.95 27.44 0.00 13.78
05:40:01 AM all 35.47 0.00 33.59 24.54 0.00 6.41
05:50:01 AM all 29.64 0.00 16.59 33.34 0.00 20.44
06:00:01 AM all 28.46 0.00 16.47 32.12 0.00 22.95
06:10:01 AM all 31.38 0.00 20.10 33.80 0.00 14.72
06:20:01 AM all 29.19 0.00 13.33 29.30 0.00 28.17
06:30:01 AM all 27.09 0.00 12.41 26.92 0.00 33.58
06:40:01 AM all 34.54 0.00 17.64 30.60 0.00 17.21
06:50:02 AM all 18.68 0.00 16.08 34.68 0.00 30.57
07:00:01 AM all 7.73 0.00 8.33 18.33 0.00 65.60
07:10:01 AM all 4.79 0.00 4.90 18.05 0.00 72.26
07:20:02 AM all 2.77 0.00 4.20 16.49 0.00 76.54
07:30:01 AM all 1.77 0.00 3.14 12.51 0.00 82.58
07:40:01 AM all 2.15 0.00 3.27 12.77 0.00 81.81
07:50:01 AM all 1.50 0.00 3.39 11.60 0.00 83.51
08:00:01 AM all 3.36 0.00 5.31 18.88 0.00 72.45
08:10:01 AM all 7.86 0.00 14.78 36.24 0.00 41.12
08:20:01 AM all 6.83 0.00 8.07 29.93 0.00 55.17
08:30:01 AM all 11.01 0.00 10.28 33.32 0.00 45.39
08:40:01 AM all 23.15 0.00 41.55 31.22 0.00 4.09
08:50:01 AM all 22.18 0.00 29.65 33.86 0.00 14.30
09:00:01 AM all 11.53 0.00 11.98 29.73 0.00 46.76
09:10:02 AM all 15.62 0.00 28.27 37.97 0.00 18.14
We need you recommendation for this case.
Thanks
Responses
Red Hat
Guru
7257 points
kblockd is the kernel's block I/O task.
Usually high usage by that indicates a storage performance problem.
Red Hat
Guru
7257 points
I see you've got support entitlements, a support case would be a good step.
Supply a sosreport from the system on the case, and start collecting the watcher-cron script every minute, then advise within that collected data when the high usage by kblockd is seen. That'll get investigation off to a good start.
Close
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.