RHEL6/7: crash in sysfs_find_dirent with EMC PowerPath after cable rport goes down and comes back
Issue
kernel crash in sysfs_find_dirent called from sd_remove
[ 1716.471636] Hardware name: ProLiant DL580 G7
[ 1716.478431] Modules linked in: emcpdm(P)(U) emcpmpx(P)(U) emcpgpx(P)(U) emcp(P)(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge autofs4 8021q garp stp llc target_core_iblock target_core_file target_core_pscsi target_core_mod configfs cpufreq_ondemand freq_table pcc_cpufreq ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables vhost_net macvtap macvlan tun kvm_intel kvm uinput ipmi_devintf power_meter acpi_ipmi ipmi_si ipmi_msghandler microcode iTCO_wdt iTCO_vendor_support netxen_nic sg serio_raw lpc_ich mfd_core hpilo hpwdt i7core_edac edac_core shpchp ext4 jbd2 mbcache mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core sr_mod cdrom sd_mod lpfc scsi_transport_fc scsi_tgt crc_t10dif pata_acpi ata_generic ata_piix hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: emcpioc]
[ 1716.534442] Pid: 7481, comm: fc_wq_8 Tainted: P --------------- 2.6.32-504.23.4.el6.x86_64 #1
[ 1716.541418] Call Trace:
[ 1716.548257] [<ffffffff81074e47>] ? warn_slowpath_common+0x87/0xc0
[ 1716.555084] [<ffffffff81074e9a>] ? warn_slowpath_null+0x1a/0x20
[ 1716.561818] [<ffffffff8128f90d>] ? kref_get+0x2d/0x30
[ 1716.568472] [<ffffffff8128e58a>] ? kobject_get+0x1a/0x30
[ 1716.575001] [<ffffffff81367f39>] ? get_device+0x19/0x20
[ 1716.581433] [<ffffffffa032d456>] ? __scsi_disk_get+0x46/0x50 [sd_mod]
[ 1716.587802] [<ffffffffa032e7a8>] ? sd_open+0x48/0x1d0 [sd_mod]
[ 1716.594097] [<ffffffffa06d2a96>] ? emcp_sd_bdops_open+0x26/0x60 [emcp]
[ 1716.600300] [<ffffffffa06ee9cc>] ? emcp_remove+0x3ec/0x8a0 [emcp]
[ 1716.606478] [<ffffffffa06eeef5>] ? emcp_chg_device_notify+0x75/0x90 [emcp]
[ 1716.612537] [<ffffffff815300d5>] ? notifier_call_chain+0x55/0x80
[ 1716.618501] [<ffffffff810a529a>] ? __blocking_notifier_call_chain+0x5a/0x80
[ 1716.624432] [<ffffffff810a52d6>] ? blocking_notifier_call_chain+0x16/0x20
[ 1716.630266] [<ffffffff8136c718>] ? __device_release_driver+0x58/0xe0
[ 1716.636004] [<ffffffff8136c89d>] ? device_release_driver+0x2d/0x40
[ 1716.641684] [<ffffffff8136b7b3>] ? bus_remove_device+0xa3/0x100
[ 1716.647272] [<ffffffff813692ad>] ? device_del+0x12d/0x1e0
[ 1716.652743] [<ffffffff8138e245>] ? __scsi_remove_device+0xc5/0xd0
[ 1716.658144] [<ffffffff8138e280>] ? scsi_remove_device+0x30/0x50
[ 1716.663471] [<ffffffff8138e422>] ? scsi_remove_target+0x162/0x200
[ 1716.668682] [<ffffffffa026b570>] ? fc_starget_delete+0x0/0x30 [scsi_transport_fc]
[ 1716.673859] [<ffffffffa026b596>] ? fc_starget_delete+0x26/0x30 [scsi_transport_fc]
[ 1716.678945] [<ffffffff81098100>] ? worker_thread+0x170/0x2a0
[ 1716.683934] [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
[ 1716.688843] [<ffffffff81097f90>] ? worker_thread+0x0/0x2a0
[ 1716.693661] [<ffffffff8109e78e>] ? kthread+0x9e/0xc0
[ 1716.698367] [<ffffffff8100c28a>] ? child_rip+0xa/0x20
[ 1716.702950] [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
[ 1716.707406] [<ffffffff8100c280>] ? child_rip+0x0/0x20
[ 1716.711818] ---[ end trace 7cf8438742997625 ]---
[ 1716.716650] BUG: unable to handle kernel NULL pointer dereference at 0000000000000068
[ 1716.721018] IP: [<ffffffff8120b2ec>] sysfs_find_dirent+0xc/0x50
[ 1716.725528] PGD 0
[ 1716.729717] Oops: 0000 [#1] SMP
[ 1716.733813] last sysfs file: /sys/devices/pci0000:00/0000:00:0a.0/0000:05:00.0/host4/rport-4:0-3/target4:0:1/4:0:1:0/block/sdv/uevent
[ 1716.742041] CPU 48
[ 1716.742079] Modules linked in: emcpdm(P)(U) emcpmpx(P)(U) emcpgpx(P)(U) emcp(P)(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge autofs4 8021q garp stp llc target_core_iblock target_core_file target_core_pscsi target_core_mod configfs cpufreq_ondemand freq_table pcc_cpufreq ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables vhost_net macvtap macvlan tun kvm_intel kvm uinput ipmi_devintf power_meter acpi_ipmi ipmi_si ipmi_msghandler microcode iTCO_wdt iTCO_vendor_support netxen_nic sg serio_raw lpc_ich mfd_core hpilo hpwdt i7core_edac edac_core shpchp ext4 jbd2 mbcache mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core sr_mod cdrom sd_mod lpfc scsi_transport_fc scsi_tgt crc_t10dif pata_acpi ata_generic ata_piix hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: emcpioc]
[ 1716.780281]
[ 1716.784280] Pid: 7481, comm: fc_wq_8 Tainted: P W --------------- 2.6.32-504.23.4.el6.x86_64 #1 HP ProLiant DL580 G7
[ 1716.792524] RIP: 0010:[<ffffffff8120b2ec>] [<ffffffff8120b2ec>] sysfs_find_dirent+0xc/0x50
[ 1716.797037] RSP: 0018:ffff88341e397c10 EFLAGS: 00010286
[ 1716.801333] RAX: ffff88341e396000 RBX: 0000000000000000 RCX: ffff88341d962500
[ 1716.805703] RDX: 0000000000000000 RSI: ffffffff817e9361 RDI: 0000000000000000
[ 1716.810111] RBP: ffff88341e397c20 R08: ffffffb0ffffffb0 R09: ffffff98ffffff98
[ 1716.814509] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff817e9361
[ 1716.818916] R13: 0000000000000000 R14: ffff88142414b460 R15: ffff8834298ce010
[ 1716.823326] FS: 0000000000000000(0000) GS:ffff88286a600000(0000) knlGS:0000000000000000
[ 1716.827774] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[ 1716.832219] CR2: 0000000000000068 CR3: 0000002824583000 CR4: 00000000000007e0
[ 1716.836704] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1716.841162] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1716.845596] Process fc_wq_8 (pid: 7481, threadinfo ffff88341e396000, task ffff883429d1c040)
[ 1716.850094] Stack:
[ 1716.854538] 0000000000000000 ffffffff817e9361 ffff88341e397c40 ffffffff8120c3d3
[ 1716.854678] <d> ffffffff81b09ce0 ffff883414d38420 ffff88341e397c70 ffffffff8120dd00
[ 1716.859294] <d> ffff883414d38410 ffff883414d38410 ffff88342bafb138 ffff883414d38410
[ 1716.868423] Call Trace:
[ 1716.873013] [<ffffffff8120c3d3>] sysfs_get_dirent+0x33/0x80
[ 1716.877664] [<ffffffff8120dd00>] sysfs_remove_group+0x30/0x100
[ 1716.882296] [<ffffffff81371559>] dpm_sysfs_remove+0x19/0x20
[ 1716.886931] [<ffffffff813691c9>] device_del+0x49/0x1e0
[ 1716.891534] [<ffffffffa032dca4>] sd_remove+0x64/0xc0 [sd_mod]
[ 1716.896160] [<ffffffff8136c72f>] __device_release_driver+0x6f/0xe0
[ 1716.900777] [<ffffffff8136c89d>] device_release_driver+0x2d/0x40
[ 1716.905414] [<ffffffff8136b7b3>] bus_remove_device+0xa3/0x100
[ 1716.910055] [<ffffffff813692ad>] device_del+0x12d/0x1e0
[ 1716.914688] [<ffffffff8138e245>] __scsi_remove_device+0xc5/0xd0
[ 1716.919345] [<ffffffff8138e280>] scsi_remove_device+0x30/0x50
[ 1716.923993] [<ffffffff8138e422>] scsi_remove_target+0x162/0x200
[ 1716.928583] [<ffffffffa026b570>] ? fc_starget_delete+0x0/0x30 [scsi_transport_fc]
[ 1716.933173] [<ffffffffa026b596>] fc_starget_delete+0x26/0x30 [scsi_transport_fc]
[ 1716.937764] [<ffffffff81098100>] worker_thread+0x170/0x2a0
[ 1716.942371] [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
[ 1716.946880] [<ffffffff81097f90>] ? worker_thread+0x0/0x2a0
[ 1716.951275] [<ffffffff8109e78e>] kthread+0x9e/0xc0
[ 1716.955681] [<ffffffff8100c28a>] child_rip+0xa/0x20
[ 1716.959979] [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
[ 1716.964143] [<ffffffff8100c280>] ? child_rip+0x0/0x20
[ 1716.968202] Code: c6 48 c7 c7 c0 08 7d 81 31 c0 e8 14 e1 31 00 0f 0b eb fe 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 54 53 0f 1f 44 00 00 <48> 8b 47 68 49 89 f4 eb 1e 0f 1f 00 48 8b 70 e0 4c 89 e7 48 8d
[ 1716.977030] RIP [<ffffffff8120b2ec>] sysfs_find_dirent+0xc/0x50
[ 1716.981101] RSP <ffff88341e397c10>
[ 1716.985006] CR2: 0000000000000068
Environment
- Red Hat Enterprise Linux 6
- 3rd party modules: emcpgpx P(U) emcp P(U) emcpmpx P(U) emcpdm P(U)
- EMC powerpath
- EMCpower.LINUX-6.0.0.00.00-158.el6 on RHEL 6.x
- EMCpower.LINUX-6.1.0.00.00-091.el7 on RHEL 7.x
- cable pull test (rport removal and add)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
