The system crashes/reboot(run_timer_softirq) on RHEL5

Solution Unverified - Updated -

Issue

icgemsmw02p is one of 2 node VCS (VCS 5.1RP2) cluster crashing/rebooting few times a week though we have removed the node out of cluster and nothing running on it. HP ran diag and did not find any.

# uname -a

Linux icgemsmw02p 2.6.18-194.32.1.el5 #1 SMP Mon Dec 20 10:52:42 EST 2010 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
# last reboot
reboot   system boot  2.6.18-194.32.1. Sun May 29 10:28         (1+12:01)  
reboot   system boot  2.6.18-194.32.1. Sat May 28 13:57          (20:25)   
reboot   system boot  2.6.18-194.32.1. Wed May 25 21:27         (3+12:56)  
reboot   system boot  2.6.18-194.32.1. Wed May 25 13:44          (07:37

$ cat sys      KERNEL: /cores/20110601221938/work/vmlinux
    DUMPFILE: /cores/20110601221938/work/vmcore-icgemsmw02p-00480908-1306933685
        CPUS: 24
        DATE: Wed Jun  1 01:06:20 2011
      UPTIME: 04:10:17
LOAD AVERAGE: 0.20, 0.14, 0.10
       TASKS: 773
    NODENAME: icgemsmw02p
     RELEASE: 2.6.18-194.32.1.el5
     VERSION: #1 SMP Mon Dec 20 10:52:42 EST 2010
     MACHINE: x86_64  (2666 Mhz)
      MEMORY: 71 GB
       PANIC: ""

LOG
last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
CPU 11
Modules linked in: nfs fscache nfs_acl vxodm(PFU) vxglm(PU) autofs4 dmpaa(PU) vxspec(PFU) vxio(PFU) vxdmp(PU) lockd sunrpc bonding ipv6 xfrm_nalgo crypto_api vxportal(PFU) fdd(PFU) vxfs(PU) emcpdm(PU) emcpgpx(PU) emcpmpx(PU) emcp(PU) dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom shpchp bnx2 serio_raw hpilo sg pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod lpfc scsi_transport_fc usb_storage ata_piix libata cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 0, comm: swapper Tainted: PF     2.6.18-194.32.1.el5 #1
RIP: 0010:[<ffffffff80097b5d>]  [<ffffffff80097b5d>] run_timer_softirq+0xf0/0x241
RSP: 0018:ffff81091fda3f10  EFLAGS: 00010006
RAX: ffff81121fc60e00 RBX: 00000000000000e0 RCX: ffff81121fc60e18
RDX: 0000a1ff00000020 RSI: 12230ef92f061ca2 RDI: ffff81121fc20000
RBP: 0000000000000021 R08: 0000000001ca4c3a R09: ffff81091fd9fe08
R10: 0000000000000001 R11: 0000000000000282 R12: ffff81121fc60000
R13: 000000000000000a R14: 000000000000000b R15: ffffffff803f43c0
FS:  0000000000000000(0000) GS:ffff81121fc28cc0(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00002aaaaaed7000 CR3: 00000009066c5000 CR4: 00000000000006e0
Process swapper (pid: 0, threadinfo ffff81091fd9e000, task ffff81121fc2e7a0)
Stack:  ffff81121febfb40 0000a1ff00000020 ffff81121fc2e7a0 0000000000000046
0000000000000021 ffffffff803e1f90 000000000000000a 000000000000000b
ffffffff803f43c0 ffffffff8001241d 0000000000000046 ffff81091fda3f98
Call Trace:
<IRQ>  [<ffffffff8001241d>] __do_softirq+0x89/0x133
[<ffffffff8005e2fc>] call_softirq+0x1c/0x28
[<ffffffff8006cb8a>] do_softirq+0x2c/0x85
[<ffffffff8005dc8e>] apic_timer_interrupt+0x66/0x6c
<EOI>  [<ffffffff8019d72a>] acpi_processor_idle_simple+0x17d/0x30e
[<ffffffff8019d619>] acpi_processor_idle_simple+0x6c/0x30e
[<ffffffff8019d5ad>] acpi_processor_idle_simple+0x0/0x30e
[<ffffffff8019d5ad>] acpi_processor_idle_simple+0x0/0x30e
[<ffffffff800492c4>] cpu_idle+0x95/0xb8
[<ffffffff80077991>] start_secondary+0x498/0x4a7

Code: 48 89 22 48 89 49 08 48 89 48 18 e9 fe 00 00 00 4c 8b 7d 18
RIP  [<ffffffff80097b5d>] run_timer_softirq+0xf0/0x241
RSP <ffff81091fda3f10>

Environment

  • Red Hat Enterprise Linux 5
    • 2.6.18-194.32.1
  • Veritas HA cluster

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content