System crashed with NULL pointer dereference, RIP: swiotlb_unmap_sg+0x2b/0x123 error

Solution Unverified - Updated -

Issue

  • Server crashed and rebooted with following lpfc, and kernel panic errors on console:
lpfc 0000:04:00.2: 0:0310 Mailbox command x9b timeout Data: x20 x300 xffff81063af79c00
lpfc 0000:04:00.2: 0:0345 Resetting board due to mailbox timeout
lpfc 0000:04:00.2: 0:2564 POST_SGL_BLOCK mailbox command failed status x0 add_status x0 mbx status xfffffe
lpfc 0000:04:00.2: 0:2813 Mgmt IO is Blocked 900 - mbox cmd 9b still active
Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP: 
 [<ffffffff8015c991>] swiotlb_unmap_sg+0x2b/0x123
PGD c19612067 PUD c1e5df067 PMD 0 
Oops: 0000 [1] SMP 
last sysfs file: /devices/pci0000:00/0000:00:03.0/0000:05:00.7/device
CPU 6 
Modules linked in: nls_utf8 raid456 xor raid1 raid0 linear ide_cd cdrom vfat msdos fat nfs nfs_acl hpwdt(U) mptctl mptbase loop seos(PU) hidp rfcomm l2cap bluetooth lockd sunrpc bonding ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 xfrm_nalgo crypto_api dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev hpilo shpchp sg serio_raw be2net 8021q pcspkr tpm_tis tpm tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage lpfc(U) scsi_transport_fc cciss(U) ext3 jbd uhci_hcd ohci_hcd ehci_hcd hpvsa(PU) sd_mod scsi_mod
Pid: 1138, comm: lpfc_worker_0 Tainted: P     ---- 2.6.18-274.18.1.el5 #1
RIP: 0010:[<ffffffff8015c991>]  [<ffffffff8015c991>] swiotlb_unmap_sg+0x2b/0x123
RSP: 0018:ffff81063bd9bc00  EFLAGS: 00010297
RAX: ffffffff8031eb20 RBX: 0000000000000000 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff810c3f558070
RBP: 0000000000000001 R08: ffff81063bcce3a0 R09: ffff8106400c5cc0
R10: 00000000000000c8 R11: ffffffff8015c966 R12: 0000000000000000
R13: 0000000000000001 R14: ffff810c3f558070 R15: 00000000000e0008
FS:  0000000000000000(0000) GS:ffff810c3fe7c8c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000c1f081000 CR4: 00000000000006a0
Process lpfc_worker_0 (pid: 1138, threadinfo ffff81063bd9a000, task ffff81065528b860)
Stack:  0000000000000001 00000000000e0008 ffff81063f768800 0000000000000003
 ffff81063bcce3a0 ffffffff88361891 ffff81063bd9bc30 0000000000000000
 ffff81061e9b0410 ffffffff8009a5d0 ffff810c3f78c068 ffff81063bcce000
Call Trace:
 [<ffffffff88361891>] :lpfc:lpfc_scsi_cmd_iocb_cmpl+0x11c0/0x1554
 [<ffffffff8009a5d0>] msleep+0x21/0x2c
 [<ffffffff800df1a2>] __drain_alien_cache+0x51/0x66
 [<ffffffff801cfaca>] dma_pool_free+0xf1/0x14b
 [<ffffffff88356568>] :lpfc:lpfc_hba_down_post_s4+0x0/0x177
 [<ffffffff88325f0a>] :lpfc:lpfc_sli_cancel_iocbs+0x68/0x78
 [<ffffffff883534da>] :lpfc:lpfc_hba_down_post_s3+0xf2/0x12b
 [<ffffffff88356583>] :lpfc:lpfc_hba_down_post_s4+0x1b/0x177
 [<ffffffff883311fc>] :lpfc:lpfc_sli_brdrestart_s4+0xfe/0x10d
 [<ffffffff88354572>] :lpfc:lpfc_reset_hba+0x2e/0x3f
 [<ffffffff8834b006>] :lpfc:lpfc_do_work+0x8df/0xb72
 [<ffffffff800a2fa2>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8834a727>] :lpfc:lpfc_do_work+0x0/0xb72
 [<ffffffff800a2d8a>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032731>] kthread+0xfe/0x132
 [<ffffffff8005dfb1>] child_rip+0xa/0x11
 [<ffffffff800a2d8a>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032633>] kthread+0x0/0x132
 [<ffffffff8005dfa7>] child_rip+0x0/0x11

Code: 48 8b 0b 48 8b 73 10 48 8b 39 48 c1 ef 33 48 89 f8 48 c1 e8 
RIP  [<ffffffff8015c991>] swiotlb_unmap_sg+0x2b/0x123
 RSP <ffff81063bd9bc00>
crash>                           

Environment

  • Red Hat Enterprise Linux 5.7
  • Vendor provided lpfc modules

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content