Oracle RAC eviction on s390

Issue

• Since upgrading to kernel 2.6.18-194.3.1.el5 and glibc 2.5-49.el5_5.2, we have had 4 Oracle RAC node evictions/reboots.

• We see the following messages in/var/log/messages:

  Aug  9 20:40:01 rac1 auditd[2552]: Audit daemon rotating log files
  Aug 10 21:10:12 rac1 logger: Cluster Ready Services completed waiting on dependencies.
  Aug 10 21:10:13 rac1 logger: Oracle CSS Family monitor restarting.
  Aug 10 21:10:14 rac1 logger: Oracle CSS restart. 0, 1
  Aug 10 21:11:45 rac1 syslogd 1.4.1: restart.
  Aug 10 21:11:45 rac1 kernel: klogd 1.4.1, log source = /proc/kmsg started.
  Aug 10 21:11:45 rac1 kernel: Linux version 2.6.18-194.3.1.el5 (mockbuild@s390-001.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Sun May 2 04:26:53 EDT 2010
  

• init.cssd reboots the node without any indication as to why.