RHEL6/7: kernel crashes during EMC SAN upgrade in sd_remove due to use-after-free of struct scsi_disk memory in size-1024 slab accessed from device.device_private.driver_data field
Issue
- server crashed during EMC SAN upgrade
sd 1:0:6:2: [sdf] Mode Sense: 9f 00 00 00
sd 1:0:6:2: [sdf] Write cache: disabled, read cache: disabled, doesn't support DPO or FUA
sdew:
sdf: sdf1
sdew1
sdfb: sdfb1
sd 1:0:6:2: [sdf] Attached SCSI disk
Info:emcp:scsi disk structure has no genhd ptr
general protection fault: 0000 [#1] SMP
sd 1:0:6:2: Attached scsi generic sg5 type 0
last sysfs file: /sys/devices/pci0000:00/0000:00:03.0/0000:08:00.0/host1/rport-1:0-8/target1:0:6/1:0:6:2/type
CPU 2
Modules linked in: oracleacfs(P)(U) oracleadvm(P)(U) oracleoks(P)(U) nfs fscache mptctl mptbase nfsd nfs_acl auth_rpcgss exportfs oracleasm(U) autofs4 lockd
scsi 1:0:6:3: Direct-Access EMC Invista 2110 PQ: 0 ANSI: 4
sunrpc pcc_cpufreq bonding ipv6 ext3 jbd emcpdm(P)(U) emcpgpx(P)(U) emcpmpx(P)(U) emcp(P)(U)
sd 1:0:6:3: [sdh] 31457280 512-byte logical blocks: (16.1 GB/15.0 GiB)
iTCO_wdt iTCO_vendor_support microcode sb_edac edac_core lpc_ich mfd_core i2c_i801 tg3 hpilo hpwdt power_meter acpi_ipmi ipmi_si ipmi_msghandler igb i2c_algo_bit i2c_core ixgbe dca ptp pps_core mdio
sd 1:0:6:3: [sdh] Write Protect is off
sd 1:0:6:3: [sdh] Mode Sense: 9f 00 00 00
sg ext4 jbd2 mbcache sd_mod xhci_hcd lpfc(U) scsi_transport_fc scsi_tgt crc_t10dif hpsa wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: freq_table]
Pid: 632, comm: fc_wq_2 Tainted: P W --------------- 2.6.32-504.23.4.el6.x86_64 #1 HP ProLiant DL380 Gen9
RIP: 0010:[<ffffffffa0164c83>] [<ffffffffa0164c83>] sd_remove+0x43/0xc0 [sd_mod]
RSP: 0018:ffff88204d60bcc0 EFLAGS: 00010286
RAX: 006e647300000010 RBX: ffff8804d485e938 RCX: ffffffff81b0b790
RDX: ffffffff81ebc5dc RSI: ffffffff81388a90 RDI: ffff8804d485e938
RBP: ffff88204d60bce0 R08: 0000000000000000 R09: ffffffff816490c0
R10: 0000000000000000 R11: 0000000000000000 R12: ffff881a02cb8000
R13: ffff881a02cb8010 R14: ffff88204f282060 R15: ffff88204da53010
FS: 0000000000000000(0000) GS:ffff880028240000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007fa01228dd50 CR3: 000000031a926000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process fc_wq_2 (pid: 632, threadinfo ffff88204d60a000, task ffff88204f0d8040)
Stack:
ffff88204f282060 ffff8804d485e938 ffffffffa016a9c8 ffffffff81b0bca0
<d> ffff88204d60bd00 ffffffff8136c72f ffff8804d485e998 ffff8804d485e938
<d> ffff88204d60bd20 ffffffff8136c89d ffff8804d485e938 ffff880c20c39c28
Call Trace:
[<ffffffff8136c72f>] __device_release_driver+0x6f/0xe0
[<ffffffff8136c89d>] device_release_driver+0x2d/0x40
[<ffffffff8136b7b3>] bus_remove_device+0xa3/0x100
[<ffffffff813692ad>] device_del+0x12d/0x1e0
[<ffffffff8138e245>] __scsi_remove_device+0xc5/0xd0
[<ffffffff8138e280>] scsi_remove_device+0x30/0x50
[<ffffffff8138e422>] scsi_remove_target+0x162/0x200
[<ffffffffa005e570>] ? fc_starget_delete+0x0/0x30 [scsi_transport_fc]
sd 1:0:6:3: [sdh] Write cache: disabled, read cache: disabled, doesn't support DPO or FUA
[<ffffffffa005e596>] fc_starget_delete+0x26/0x30 [scsi_transport_fc]
[<ffffffff81098100>] worker_thread+0x170/0x2a0
[<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
[<ffffffff81097f90>] ? worker_thread+0x0/0x2a0
[<ffffffff8109e78e>] kthread+0x9e/0xc0
[<ffffffff8100c28a>] child_rip+0xa/0x20
[<ffffffff8109e6f0>] ? kthread+0x0/0xc0
[<ffffffff8100c280>] ? child_rip+0x0/0x20
Code: 00 48 89 fb 48 c7 c7 80 b7 b0 81 e8 a8 28 f4 e0 48 89 df e8 b0 76 20 e1 49 89 c4 48 8b 40 08 48 c7 c6 90 8a 38 81 4d 8d 6c 24 10 <48> 8b 78 08 e8 34 13 11 e1 49 8b 44 24 08 31 f6 48 8b 78 08 e8
RIP [<ffffffffa0164c83>] sd_remove+0x43/0xc0 [sd_mod]
RSP <ffff88204d60bcc0>
Environment
- Red Hat Enterprise Linux 6
- 3rd party modules: lpfc (U) emcpgpx P(U) emcp P(U) emcpmpx P(U) emcpdm P(U) oracleasm (U) oracleoks P(U) oracleadvm P(U) oracleacfs P(U)
- EMC powerpath
- EMCpower.LINUX-6.0.0.00.00-158.el6 on RHEL 6.x
- EMCpower.LINUX-6.1.0.00.00-091.el7 on RHEL 7.x
- EMC SAN upgrade (host equivalent of port up / down type test)
- Emulex FC HBA Detail
08:00.0 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)
08:00.1 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)
- Emulex FC Driver version (non-Red Hat)
Emulex LightPulse Fibre Channel SCSI driver 10.2.477.17
Copyright(c) 2004-2013 Emulex. All rights reserved.
scsi1 : Emulex LPe12000 PCIe Fibre Channel Adapter on PCI bus 08 device 00 irq 16
scsi2 : Emulex LPe12000 PCIe Fibre Channel Adapter on PCI bus 08 device 01 irq 17
- Oracle ASM
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.