System panics with very short scsi timeout and/or high level of console logging
Issue
- System panics with very short scsi timeout and/or high level of console logging
BUG: unable to handle kernel NULL pointer dereference at 00000000000000b0
IP: [<ffffffff8127834f>] blk_done_softirq+0x7f/0xa0
PGD dfd338b067 PUD cd80ed6067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:07.0/0000:0b:00.1/host6/rport-6:0-4/target6:0:2/6:0:2:83/state
CPU 50
Modules linked in: hangcheck_timer oracleacfs(P)(U) oracleadvm(P)(U) oracleoks(P)(U) krg_11_0_0_1130_impRHEL6K1smp-x86_64(P)(U) mptctl mptbase nfsd exportfs oracleasm(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc bonding 8021q garp stp llc ipv6 ext3 jbd dm_round_robin iTCO_wdt iTCO_vendor_support be2net ixgbe dca mdio e1000e ptp pps_core microcode ipmi_devintf serio_raw lpc_ich mfd_core hpilo hpwdt i7core_edac edac_core sg power_meter acpi_ipmi ipmi_si ipmi_msghandler bnx2 shpchp ext4 jbd2 mbcache sr_mod cdrom sd_mod lpfc scsi_transport_fc scsi_tgt crc_t10dif pata_acpi ata_generic ata_piix hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Pid: 205, comm: ksoftirqd/50 Tainted: P --------------- 2.6.32-504.1.3.el6.x86_64 #1 HP ProLiant DL980 G7
RIP: 0010:[<ffffffff8127834f>] [<ffffffff8127834f>] blk_done_softirq+0x7f/0xa0
RSP: 0018:ffff88a070c03f18 EFLAGS: 00010216
RAX: 0000000000000000 RBX: ffff88a070c03f18 RCX: ffff88adba447490
RDX: ffff88a070c03f18 RSI: ffff881fd299bbc0 RDI: ffff88a070c137c0
RBP: ffff88a070c03f38 R08: 0000000000000000 R09: 0000000000000000
R10: ffff881fd1a74400 R11: 00000000055b9b1f R12: ffffffff81a830a0
R13: 0000000000000020 R14: 0000000000000100 R15: 0000000000000004
FS: 0000000000000000(0000) GS:ffff88a070c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000000000000b0 CR3: 000000cd920a0000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ksoftirqd/50 (pid: 205, threadinfo ffff881fd3a42000, task ffff881fd3a3eaa0)
Stack:
ffff88adba447490 ffff88ae8b7d14d0 0000000000000100 0000000000000029
<d> ffff88a070c03fa8 ffffffff8107d8b1 0000000000000032 ffff88a070c03f68
<d> 0000003200000005 ffff881fd3a43fd8 ffff881fd3a43fd8 0000000000011440
Call Trace:
<IRQ>
[<ffffffff8107d8b1>] __do_softirq+0xc1/0x1e0
[<ffffffff8100c30c>] call_softirq+0x1c/0x30
<EOI>
[<ffffffff8100fc15>] ? do_softirq+0x65/0xa0
[<ffffffff8107d470>] ksoftirqd+0x80/0x110
[<ffffffff8107d3f0>] ? ksoftirqd+0x0/0x110
[<ffffffff8109e66e>] kthread+0x9e/0xc0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8109e5d0>] ? kthread+0x0/0xc0
[<ffffffff8100c200>] ? child_rip+0x0/0x20
Code: e0 48 39 d8 74 34 66 0f 1f 44 00 00 48 8d 78 f0 48 8b 4f 10 48 8b 57 18 48 89 51 08 48 89 0a 48 89 47 10 48 89 47 18 48 8b 47 38 <ff> 90 b0 00 00 00 48 8b 45 e0 48 39 d8 75 d2 48 83 c4 18 5b c9
RIP [<ffffffff8127834f>] blk_done_softirq+0x7f/0xa0
RSP <ffff88a070c03f18>
CR2: 00000000000000b0
Environment
- Red Hat Enterprise Linux 6
- Short scsi timeout set (4 seconds in this case)
- Great deal of logging to the console (not required but exacerbates issue)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.