The system crashes/reboot(run_timer_softirq) on RHEL5
Issue
icgemsmw02p is one of 2 node VCS (VCS 5.1RP2) cluster crashing/rebooting few times a week though we have removed the node out of cluster and nothing running on it. HP ran diag and did not find any.
# uname -a
Linux icgemsmw02p 2.6.18-194.32.1.el5 #1 SMP Mon Dec 20 10:52:42 EST 2010 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
# last reboot
reboot system boot 2.6.18-194.32.1. Sun May 29 10:28 (1+12:01)
reboot system boot 2.6.18-194.32.1. Sat May 28 13:57 (20:25)
reboot system boot 2.6.18-194.32.1. Wed May 25 21:27 (3+12:56)
reboot system boot 2.6.18-194.32.1. Wed May 25 13:44 (07:37
$ cat sys KERNEL: /cores/20110601221938/work/vmlinux
DUMPFILE: /cores/20110601221938/work/vmcore-icgemsmw02p-00480908-1306933685
CPUS: 24
DATE: Wed Jun 1 01:06:20 2011
UPTIME: 04:10:17
LOAD AVERAGE: 0.20, 0.14, 0.10
TASKS: 773
NODENAME: icgemsmw02p
RELEASE: 2.6.18-194.32.1.el5
VERSION: #1 SMP Mon Dec 20 10:52:42 EST 2010
MACHINE: x86_64 (2666 Mhz)
MEMORY: 71 GB
PANIC: ""LOG
last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
CPU 11
Modules linked in: nfs fscache nfs_acl vxodm(PFU) vxglm(PU) autofs4 dmpaa(PU) vxspec(PFU) vxio(PFU) vxdmp(PU) lockd sunrpc bonding ipv6 xfrm_nalgo crypto_api vxportal(PFU) fdd(PFU) vxfs(PU) emcpdm(PU) emcpgpx(PU) emcpmpx(PU) emcp(PU) dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom shpchp bnx2 serio_raw hpilo sg pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod lpfc scsi_transport_fc usb_storage ata_piix libata cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 0, comm: swapper Tainted: PF 2.6.18-194.32.1.el5 #1
RIP: 0010:[<ffffffff80097b5d>] [<ffffffff80097b5d>] run_timer_softirq+0xf0/0x241
RSP: 0018:ffff81091fda3f10 EFLAGS: 00010006
RAX: ffff81121fc60e00 RBX: 00000000000000e0 RCX: ffff81121fc60e18
RDX: 0000a1ff00000020 RSI: 12230ef92f061ca2 RDI: ffff81121fc20000
RBP: 0000000000000021 R08: 0000000001ca4c3a R09: ffff81091fd9fe08
R10: 0000000000000001 R11: 0000000000000282 R12: ffff81121fc60000
R13: 000000000000000a R14: 000000000000000b R15: ffffffff803f43c0
FS: 0000000000000000(0000) GS:ffff81121fc28cc0(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00002aaaaaed7000 CR3: 00000009066c5000 CR4: 00000000000006e0
Process swapper (pid: 0, threadinfo ffff81091fd9e000, task ffff81121fc2e7a0)
Stack: ffff81121febfb40 0000a1ff00000020 ffff81121fc2e7a0 0000000000000046
0000000000000021 ffffffff803e1f90 000000000000000a 000000000000000b
ffffffff803f43c0 ffffffff8001241d 0000000000000046 ffff81091fda3f98
Call Trace:
<IRQ> [<ffffffff8001241d>] __do_softirq+0x89/0x133
[<ffffffff8005e2fc>] call_softirq+0x1c/0x28
[<ffffffff8006cb8a>] do_softirq+0x2c/0x85
[<ffffffff8005dc8e>] apic_timer_interrupt+0x66/0x6c
<EOI> [<ffffffff8019d72a>] acpi_processor_idle_simple+0x17d/0x30e
[<ffffffff8019d619>] acpi_processor_idle_simple+0x6c/0x30e
[<ffffffff8019d5ad>] acpi_processor_idle_simple+0x0/0x30e
[<ffffffff8019d5ad>] acpi_processor_idle_simple+0x0/0x30e
[<ffffffff800492c4>] cpu_idle+0x95/0xb8
[<ffffffff80077991>] start_secondary+0x498/0x4a7Code: 48 89 22 48 89 49 08 48 89 48 18 e9 fe 00 00 00 4c 8b 7d 18
RIP [<ffffffff80097b5d>] run_timer_softirq+0xf0/0x241
RSP <ffff81091fda3f10>
Environment
- Red Hat Enterprise Linux 5
- 2.6.18-194.32.1
- Veritas HA cluster
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.