RHEL5: CPU %sys high load issue, many processes in sys_nanosleep system call, stuck in enqueue_hrtimer

Issue

• There is an Oracle instance(oracle 11g 11.2.0.3) operating on the Dell R910 server, the database is very slow on 20:00~23:00 every night, it affected their key application.

• Use the command 'top' to check the CPU usage and found the sys% is up to 79%, and there is a lot of oracle process that used the CPU

# top
top - 01:29:10 up 89 days, 14:26,  4 users,  load average: 511.60, 288.10, 154.00
Tasks: 6111 total, 440 running, 5670 sleeping,   0 stopped,   1 zombie
Cpu(s): 31.8%us, 66.3%sy,  0.0%ni,  0.4%id,  0.2%wa,  0.1%hi,  1.2%si,  0.0%st
Mem:  528493904k total, 465468904k used, 63025000k free,  1489424k buffers
Swap: 393215992k total,  3369436k used, 389846556k free, 279441308k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                        
31740 oracle    18   0 28.3g 6.0g 6.0g R 100.0  1.2   3:13.45 oracle                                                                                                                                                                        
26634 oracle    16   0 28.3g 1.4g 1.4g R 100.0  0.3   0:58.96 oracle                                                                                                                                                                        
30391 oracle    16   0 28.3g 3.4g 3.4g S 100.0  0.7   1:07.10 oracle                                                                                                                                                                        
26124 oracle    20   0 28.3g 2.0g 2.0g R 100.0  0.4   0:34.15 oracle                                                                                                                                                                        
26644 oracle    16   0 28.3g 3.0g 3.0g S 100.0  0.6   0:48.34 oracle                                                                                                                                                                        
30341 oracle    19   0 28.3g 2.7g 2.6g S 100.0  0.5   0:42.12 oracle                                                                                                                                                                        
26333 oracle    17   0 28.3g 1.7g 1.6g S 100.0  0.3   0:27.77 oracle                                                                                                                                                                        
29162 oracle    15   0 28.3g 2.4g 2.4g R 100.0  0.5   0:32.08 oracle                                                                                                                                                                        
26013 oracle    16   0 28.3g 3.4g 3.4g S 100.0  0.7   1:06.18 oracle                                                                                                                                                                        
27866 oracle    19   0 28.3g 2.8g 2.8g R 100.0  0.6   0:36.99 oracle                                                                                                                                                                        
25912 oracle    19   0 28.3g 2.8g 2.7g S 100.0  0.5   0:46.80 

• Many processes blocked in the following backtrace during the high cpu load time:

Call Trace:
 [<ffffffff800a4cb5>] enqueue_hrtimer+0x55/0x70
 [<ffffffff8003c491>] hrtimer_start+0xc6/0xd8
 [<ffffffff8003ad50>] hrtimer_try_to_cancel+0x4a/0x53  
 [<ffffffff80063ce1>] do_nanosleep+0x3f/0x70
 [<ffffffff8005a01c>] hrtimer_nanosleep+0x58/0x118
 [<ffffffff800a4e22>] hrtimer_wakeup+0x0/0x22
 [<ffffffff800549ca>] sys_nanosleep+0x4c/0x62
 [<ffffffff8005d28d>] tracesys+0xd5/0xe0