RHEL6/7: crash in sysfs_find_dirent with EMC PowerPath after cable rport goes down and comes back
Issue
kernel crash in sysfs_find_dirent called from sd_remove
[ 1716.471636] Hardware name: ProLiant DL580 G7
[ 1716.478431] Modules linked in: emcpdm(P)(U) emcpmpx(P)(U) emcpgpx(P)(U) emcp(P)(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge autofs4 8021q garp stp llc target_core_iblock target_core_file target_core_pscsi target_core_mod configfs cpufreq_ondemand freq_table pcc_cpufreq ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables vhost_net macvtap macvlan tun kvm_intel kvm uinput ipmi_devintf power_meter acpi_ipmi ipmi_si ipmi_msghandler microcode iTCO_wdt iTCO_vendor_support netxen_nic sg serio_raw lpc_ich mfd_core hpilo hpwdt i7core_edac edac_core shpchp ext4 jbd2 mbcache mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core sr_mod cdrom sd_mod lpfc scsi_transport_fc scsi_tgt crc_t10dif pata_acpi ata_generic ata_piix hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: emcpioc]
[ 1716.534442] Pid: 7481, comm: fc_wq_8 Tainted: P --------------- 2.6.32-504.23.4.el6.x86_64 #1
[ 1716.541418] Call Trace:
[ 1716.548257] [<ffffffff81074e47>] ? warn_slowpath_common+0x87/0xc0
[ 1716.555084] [<ffffffff81074e9a>] ? warn_slowpath_null+0x1a/0x20
[ 1716.561818] [<ffffffff8128f90d>] ? kref_get+0x2d/0x30
[ 1716.568472] [<ffffffff8128e58a>] ? kobject_get+0x1a/0x30
[ 1716.575001] [<ffffffff81367f39>] ? get_device+0x19/0x20
[ 1716.581433] [<ffffffffa032d456>] ? __scsi_disk_get+0x46/0x50 [sd_mod]
[ 1716.587802] [<ffffffffa032e7a8>] ? sd_open+0x48/0x1d0 [sd_mod]
[ 1716.594097] [<ffffffffa06d2a96>] ? emcp_sd_bdops_open+0x26/0x60 [emcp]
[ 1716.600300] [<ffffffffa06ee9cc>] ? emcp_remove+0x3ec/0x8a0 [emcp]
[ 1716.606478] [<ffffffffa06eeef5>] ? emcp_chg_device_notify+0x75/0x90 [emcp]
[ 1716.612537] [<ffffffff815300d5>] ? notifier_call_chain+0x55/0x80
[ 1716.618501] [<ffffffff810a529a>] ? __blocking_notifier_call_chain+0x5a/0x80
[ 1716.624432] [<ffffffff810a52d6>] ? blocking_notifier_call_chain+0x16/0x20
[ 1716.630266] [<ffffffff8136c718>] ? __device_release_driver+0x58/0xe0
[ 1716.636004] [<ffffffff8136c89d>] ? device_release_driver+0x2d/0x40
[ 1716.641684] [<ffffffff8136b7b3>] ? bus_remove_device+0xa3/0x100
[ 1716.647272] [<ffffffff813692ad>] ? device_del+0x12d/0x1e0
[ 1716.652743] [<ffffffff8138e245>] ? __scsi_remove_device+0xc5/0xd0
[ 1716.658144] [<ffffffff8138e280>] ? scsi_remove_device+0x30/0x50
[ 1716.663471] [<ffffffff8138e422>] ? scsi_remove_target+0x162/0x200
[ 1716.668682] [<ffffffffa026b570>] ? fc_starget_delete+0x0/0x30 [scsi_transport_fc]
[ 1716.673859] [<ffffffffa026b596>] ? fc_starget_delete+0x26/0x30 [scsi_transport_fc]
[ 1716.678945] [<ffffffff81098100>] ? worker_thread+0x170/0x2a0
[ 1716.683934] [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
[ 1716.688843] [<ffffffff81097f90>] ? worker_thread+0x0/0x2a0
[ 1716.693661] [<ffffffff8109e78e>] ? kthread+0x9e/0xc0
[ 1716.698367] [<ffffffff8100c28a>] ? child_rip+0xa/0x20
[ 1716.702950] [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
[ 1716.707406] [<ffffffff8100c280>] ? child_rip+0x0/0x20
[ 1716.711818] ---[ end trace 7cf8438742997625 ]---
[ 1716.716650] BUG: unable to handle kernel NULL pointer dereference at 0000000000000068
[ 1716.721018] IP: [<ffffffff8120b2ec>] sysfs_find_dirent+0xc/0x50
[ 1716.725528] PGD 0
[ 1716.729717] Oops: 0000 [#1] SMP
[ 1716.733813] last sysfs file: /sys/devices/pci0000:00/0000:00:0a.0/0000:05:00.0/host4/rport-4:0-3/target4:0:1/4:0:1:0/block/sdv/uevent
[ 1716.742041] CPU 48
[ 1716.742079] Modules linked in: emcpdm(P)(U) emcpmpx(P)(U) emcpgpx(P)(U) emcp(P)(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge autofs4 8021q garp stp llc target_core_iblock target_core_file target_core_pscsi target_core_mod configfs cpufreq_ondemand freq_table pcc_cpufreq ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables vhost_net macvtap macvlan tun kvm_intel kvm uinput ipmi_devintf power_meter acpi_ipmi ipmi_si ipmi_msghandler microcode iTCO_wdt iTCO_vendor_support netxen_nic sg serio_raw lpc_ich mfd_core hpilo hpwdt i7core_edac edac_core shpchp ext4 jbd2 mbcache mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core sr_mod cdrom sd_mod lpfc scsi_transport_fc scsi_tgt crc_t10dif pata_acpi ata_generic ata_piix hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: emcpioc]
[ 1716.780281]
[ 1716.784280] Pid: 7481, comm: fc_wq_8 Tainted: P W --------------- 2.6.32-504.23.4.el6.x86_64 #1 HP ProLiant DL580 G7
[ 1716.792524] RIP: 0010:[<ffffffff8120b2ec>] [<ffffffff8120b2ec>] sysfs_find_dirent+0xc/0x50
[ 1716.797037] RSP: 0018:ffff88341e397c10 EFLAGS: 00010286
[ 1716.801333] RAX: ffff88341e396000 RBX: 0000000000000000 RCX: ffff88341d962500
[ 1716.805703] RDX: 0000000000000000 RSI: ffffffff817e9361 RDI: 0000000000000000
[ 1716.810111] RBP: ffff88341e397c20 R08: ffffffb0ffffffb0 R09: ffffff98ffffff98
[ 1716.814509] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff817e9361
[ 1716.818916] R13: 0000000000000000 R14: ffff88142414b460 R15: ffff8834298ce010
[ 1716.823326] FS: 0000000000000000(0000) GS:ffff88286a600000(0000) knlGS:0000000000000000
[ 1716.827774] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[ 1716.832219] CR2: 0000000000000068 CR3: 0000002824583000 CR4: 00000000000007e0
[ 1716.836704] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1716.841162] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1716.845596] Process fc_wq_8 (pid: 7481, threadinfo ffff88341e396000, task ffff883429d1c040)
[ 1716.850094] Stack:
[ 1716.854538] 0000000000000000 ffffffff817e9361 ffff88341e397c40 ffffffff8120c3d3
[ 1716.854678] <d> ffffffff81b09ce0 ffff883414d38420 ffff88341e397c70 ffffffff8120dd00
[ 1716.859294] <d> ffff883414d38410 ffff883414d38410 ffff88342bafb138 ffff883414d38410
[ 1716.868423] Call Trace:
[ 1716.873013] [<ffffffff8120c3d3>] sysfs_get_dirent+0x33/0x80
[ 1716.877664] [<ffffffff8120dd00>] sysfs_remove_group+0x30/0x100
[ 1716.882296] [<ffffffff81371559>] dpm_sysfs_remove+0x19/0x20
[ 1716.886931] [<ffffffff813691c9>] device_del+0x49/0x1e0
[ 1716.891534] [<ffffffffa032dca4>] sd_remove+0x64/0xc0 [sd_mod]
[ 1716.896160] [<ffffffff8136c72f>] __device_release_driver+0x6f/0xe0
[ 1716.900777] [<ffffffff8136c89d>] device_release_driver+0x2d/0x40
[ 1716.905414] [<ffffffff8136b7b3>] bus_remove_device+0xa3/0x100
[ 1716.910055] [<ffffffff813692ad>] device_del+0x12d/0x1e0
[ 1716.914688] [<ffffffff8138e245>] __scsi_remove_device+0xc5/0xd0
[ 1716.919345] [<ffffffff8138e280>] scsi_remove_device+0x30/0x50
[ 1716.923993] [<ffffffff8138e422>] scsi_remove_target+0x162/0x200
[ 1716.928583] [<ffffffffa026b570>] ? fc_starget_delete+0x0/0x30 [scsi_transport_fc]
[ 1716.933173] [<ffffffffa026b596>] fc_starget_delete+0x26/0x30 [scsi_transport_fc]
[ 1716.937764] [<ffffffff81098100>] worker_thread+0x170/0x2a0
[ 1716.942371] [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
[ 1716.946880] [<ffffffff81097f90>] ? worker_thread+0x0/0x2a0
[ 1716.951275] [<ffffffff8109e78e>] kthread+0x9e/0xc0
[ 1716.955681] [<ffffffff8100c28a>] child_rip+0xa/0x20
[ 1716.959979] [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
[ 1716.964143] [<ffffffff8100c280>] ? child_rip+0x0/0x20
[ 1716.968202] Code: c6 48 c7 c7 c0 08 7d 81 31 c0 e8 14 e1 31 00 0f 0b eb fe 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 54 53 0f 1f 44 00 00 <48> 8b 47 68 49 89 f4 eb 1e 0f 1f 00 48 8b 70 e0 4c 89 e7 48 8d
[ 1716.977030] RIP [<ffffffff8120b2ec>] sysfs_find_dirent+0xc/0x50
[ 1716.981101] RSP <ffff88341e397c10>
[ 1716.985006] CR2: 0000000000000068
Environment
- Red Hat Enterprise Linux 6
- 3rd party modules: emcpgpx P(U) emcp P(U) emcpmpx P(U) emcpdm P(U)
- EMC powerpath
- EMCpower.LINUX-6.0.0.00.00-158.el6 on RHEL 6.x
- EMCpower.LINUX-6.1.0.00.00-091.el7 on RHEL 7.x
- cable pull test (rport removal and add)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.