Server running a Sybase database hangs from time to time

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 5.8
  • Sybase Database Application

Issue

  • We execute the following command on a NFS mounted (Netapp filer) directory:
dd if=/dev/zero of=zerofile bs=1M count=1000000

This turns the machine unresponsive.

no more logins possible (timeout)
power comsumption spikes from 120W to 350W
only "Force power off” works, “Graceful power off” and “Force system Reset” do nothing

Last top collected:

top - 15:59:07 up  1:02,  4 users,  load average: 65.00, 55.35, 194.76
Tasks: 613 total,  22 running, 588 sleeping,   0 stopped,   3 zombie
Cpu(s):  0.0%us, 89.7%sy,  0.0%ni,  2.9%id,  7.3%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  264115840k total, 263535616k used,   580224k free,     1352k buffers
Swap: 32768056k total,   452772k used, 32315284k free, 23833536k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                      
29203 root      19   0  648m 2908 1284 S 502.6  0.0  36:45.49 java                                                                        
28955 root      15   0  140m 1580  932 S 419.3  0.0  40:23.88 splunkd                                                                     
23650 root      15  -5     0    0    0 S 146.2  0.0  11:00.03 vxfs_thread                                                                 
20965 <user>   19   0 32.4g 2392 1164 S 138.5  0.0  15:20.95 java                                                                        
25345 root      18   0 11636  216  216 R 100.0  0.0   7:25.62 irqbalance                                                                  
 2900 root      20  -5     0    0    0 R 95.9  0.0  10:34.70 kswapd1                                                                      
22890 root      18   0 13552   44   44 D 93.4  0.0   7:17.87 cmahostd                                                                     
23210 root      18   0 10780  184   40 D 92.3  0.0   8:02.22 cmaperfd                                                                     
 2187 root      12  -5     0    0    0 R 90.9  0.0   8:29.61 kthread                                                                      
  130 root      13  -5     0    0    0 R 90.6  0.0   9:18.62 khelper                                                                      
 2899 root      20  -5     0    0    0 R 85.1  0.0   7:52.26 kswapd0                                                                      
32337 root      18   0  119m  640  640 R 84.5  0.0   7:30.83 snmpd                                                                        
32326 root      18   0 13348  556  556 R 84.0  0.0   6:14.48 perfstat                                                                     
23039 root      16   0 12640  400  184 R 76.2  0.0   6:22.47 cmasm2d                                                                      
22215 root      23   0  226m 1116  972 S 75.4  0.0   4:15.87 ssh-servant-g3                                                               
22258 root      18   0 19252  408  408 R 75.1  0.0   4:24.40 crond                                                                        
10461 root      18   0  9560  596  596 D 74.7  0.0   5:16.32 bash                                                                         
 9846 root      18   0  3828  580  580 D 74.4  0.0   6:26.27 mpstat                                                                       
12005 root      18   0  9420  584  584 D 71.9  0.0   7:04.89 sh                                                                           
32307 nscd      18   0  146m  712  668 S 71.3  0.0   4:01.93 nscd                                                                         
21693 root      18   0 72656  568  568 D 70.8  0.0   6:47.02 bpcd                                                                         
 9832 root      18   0  7304  596  596 D 70.8  0.0   7:11.11 iostat                                                                       
 9840 root      18   0  7304  596  596 R 70.7  0.0   7:13.85 iostat

The file created by the dd looks normal, an strace on dd only shows:

read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576) = 1048576
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576) = 1048576

Resolution

  • Update the Sybase Database application to address a known issue related to the application no longer using DirectIO and overrunning PageCache as a result.

  • Alternatively, contact Sybase support in order to verify if the revision installed to the system is susceptible to this 3rd party application bug.

Root Cause

  • The used version of Sybase has a bug, which is only triggered during resizing of an existing disk. It will ignore the directIO option and use up all available memory if the backing disk is larger than the amount of RAM installed to the system.

Diagnostic Steps

  • Verify that the system has the following Sybase application load installed which is susceptible:
$ rpm -qa | grep Sybase
CS_SybaseIQ_VCSAgent-1.0.0-2.x86_64
CS_SybaseIQ_VCSTypes-1.0.0-2.x86_64 
CS_SybasePatrol_VCSAgent-1.0-1.0.2-1.x86_64
CS_SybasePatrol_VCSTypes-1.0-1.0.2-2.x86_64
CS_Sybase_ASE155-1.0.0-5.x86_64            
CS_Sybase_ASE157-1.0.0-6.x86_64            
CS_Sybase_Client-2.1.0-2.x86_64            
CS_Sybase_DBATools-4.2.5-6.x86_64          
CS_Sybase_VCSAgent-1.0-1.0.1-1.x86_64      
CS_Sybase_VCSTypes-1.0-1.0.2-2.x86_64

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.