RHEL6/7: kernel crashes during EMC SAN upgrade in sd_remove due to use-after-free of struct scsi_disk memory in size-1024 slab accessed from device.device_private.driver_data field

Solution Verified - Updated -

Issue

  • server crashed during EMC SAN upgrade
sd 1:0:6:2: [sdf] Mode Sense: 9f 00 00 00
sd 1:0:6:2: [sdf] Write cache: disabled, read cache: disabled, doesn't support DPO or FUA
 sdew:
 sdf: sdf1
 sdew1
 sdfb: sdfb1
sd 1:0:6:2: [sdf] Attached SCSI disk
Info:emcp:scsi disk structure has no genhd ptr
general protection fault: 0000 [#1] SMP 
sd 1:0:6:2: Attached scsi generic sg5 type 0
last sysfs file: /sys/devices/pci0000:00/0000:00:03.0/0000:08:00.0/host1/rport-1:0-8/target1:0:6/1:0:6:2/type
CPU 2 
Modules linked in: oracleacfs(P)(U) oracleadvm(P)(U) oracleoks(P)(U) nfs fscache mptctl mptbase nfsd nfs_acl auth_rpcgss exportfs oracleasm(U) autofs4 lockd
scsi 1:0:6:3: Direct-Access     EMC      Invista          2110 PQ: 0 ANSI: 4
 sunrpc pcc_cpufreq bonding ipv6 ext3 jbd emcpdm(P)(U) emcpgpx(P)(U) emcpmpx(P)(U) emcp(P)(U)
sd 1:0:6:3: [sdh] 31457280 512-byte logical blocks: (16.1 GB/15.0 GiB)
 iTCO_wdt iTCO_vendor_support microcode sb_edac edac_core lpc_ich mfd_core i2c_i801 tg3 hpilo hpwdt power_meter acpi_ipmi ipmi_si ipmi_msghandler igb i2c_algo_bit i2c_core ixgbe dca ptp pps_core mdio
sd 1:0:6:3: [sdh] Write Protect is off
sd 1:0:6:3: [sdh] Mode Sense: 9f 00 00 00
 sg ext4 jbd2 mbcache sd_mod xhci_hcd lpfc(U) scsi_transport_fc scsi_tgt crc_t10dif hpsa wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: freq_table]

Pid: 632, comm: fc_wq_2 Tainted: P        W  ---------------    2.6.32-504.23.4.el6.x86_64 #1 HP ProLiant DL380 Gen9
RIP: 0010:[<ffffffffa0164c83>]  [<ffffffffa0164c83>] sd_remove+0x43/0xc0 [sd_mod]
RSP: 0018:ffff88204d60bcc0  EFLAGS: 00010286
RAX: 006e647300000010 RBX: ffff8804d485e938 RCX: ffffffff81b0b790
RDX: ffffffff81ebc5dc RSI: ffffffff81388a90 RDI: ffff8804d485e938
RBP: ffff88204d60bce0 R08: 0000000000000000 R09: ffffffff816490c0
R10: 0000000000000000 R11: 0000000000000000 R12: ffff881a02cb8000
R13: ffff881a02cb8010 R14: ffff88204f282060 R15: ffff88204da53010
FS:  0000000000000000(0000) GS:ffff880028240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007fa01228dd50 CR3: 000000031a926000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process fc_wq_2 (pid: 632, threadinfo ffff88204d60a000, task ffff88204f0d8040)
Stack:
 ffff88204f282060 ffff8804d485e938 ffffffffa016a9c8 ffffffff81b0bca0
<d> ffff88204d60bd00 ffffffff8136c72f ffff8804d485e998 ffff8804d485e938
<d> ffff88204d60bd20 ffffffff8136c89d ffff8804d485e938 ffff880c20c39c28
Call Trace:
 [<ffffffff8136c72f>] __device_release_driver+0x6f/0xe0
 [<ffffffff8136c89d>] device_release_driver+0x2d/0x40
 [<ffffffff8136b7b3>] bus_remove_device+0xa3/0x100
 [<ffffffff813692ad>] device_del+0x12d/0x1e0
 [<ffffffff8138e245>] __scsi_remove_device+0xc5/0xd0
 [<ffffffff8138e280>] scsi_remove_device+0x30/0x50
 [<ffffffff8138e422>] scsi_remove_target+0x162/0x200
 [<ffffffffa005e570>] ? fc_starget_delete+0x0/0x30 [scsi_transport_fc]
sd 1:0:6:3: [sdh] Write cache: disabled, read cache: disabled, doesn't support DPO or FUA
 [<ffffffffa005e596>] fc_starget_delete+0x26/0x30 [scsi_transport_fc]
 [<ffffffff81098100>] worker_thread+0x170/0x2a0
 [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81097f90>] ? worker_thread+0x0/0x2a0
 [<ffffffff8109e78e>] kthread+0x9e/0xc0
 [<ffffffff8100c28a>] child_rip+0xa/0x20
 [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
Code: 00 48 89 fb 48 c7 c7 80 b7 b0 81 e8 a8 28 f4 e0 48 89 df e8 b0 76 20 e1 49 89 c4 48 8b 40 08 48 c7 c6 90 8a 38 81 4d 8d 6c 24 10 <48> 8b 78 08 e8 34 13 11 e1 49 8b 44 24 08 31 f6 48 8b 78 08 e8 
RIP  [<ffffffffa0164c83>] sd_remove+0x43/0xc0 [sd_mod]
 RSP <ffff88204d60bcc0>

Environment

  • Red Hat Enterprise Linux 6
    • 3rd party modules: lpfc (U) emcpgpx P(U) emcp P(U) emcpmpx P(U) emcpdm P(U) oracleasm (U) oracleoks P(U) oracleadvm P(U) oracleacfs P(U)
  • EMC powerpath
    • EMCpower.LINUX-6.0.0.00.00-158.el6 on RHEL 6.x
    • EMCpower.LINUX-6.1.0.00.00-091.el7 on RHEL 7.x
  • EMC SAN upgrade (host equivalent of port up / down type test)
  • Emulex FC HBA Detail
08:00.0 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)
08:00.1 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)
  • Emulex FC Driver version (non-Red Hat)
Emulex LightPulse Fibre Channel SCSI driver 10.2.477.17
Copyright(c) 2004-2013 Emulex.  All rights reserved.
scsi1 : Emulex LPe12000 PCIe Fibre Channel Adapter  on PCI bus 08 device 00 irq 16
scsi2 : Emulex LPe12000 PCIe Fibre Channel Adapter  on PCI bus 08 device 01 irq 17
  • Oracle ASM

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.