RHEL 7 for Power (ppc64le) SR-IOV interface goes down with error: kernel BUG at drivers/net/ethernet/emulex/benet/be_main.c:2802!

Issue

• An Emulex be2net-based SR-IOV interface on a PPC64 host stops responding causes the following backtrace:

  Mar 16 16:21:24 localhost kernel: ------------[ cut here ]------------
  Mar 16 16:21:24 localhost kernel: kernel BUG at drivers/net/ethernet/emulex/benet/be_main.c:2802!
  Mar 16 16:21:24 localhost kernel: Oops: Exception in kernel mode, sig: 5 [#1]
  Mar 16 16:21:24 localhost kernel: SMP NR_CPUS=2048 NUMA pSeries
  Mar 16 16:21:24 localhost kernel: Modules linked in: tracedev(OE) sctp_diag sctp dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter pseries_rng sg be2net nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c dm_service_time sd_mod crc_t10dif crct10dif_generic crct10dif_common ibmvscsi scsi_transport_srp ibmveth scsi_tgt dm_multipath dm_mirror
  Mar 16 16:21:24 localhost kernel: dm_region_hash dm_log dm_mod [last unloaded: mmfslinux]
  Mar 16 16:21:24 localhost kernel: CPU: 32 PID: 14820 Comm: ip Tainted: G        W  OE  ------------   3.10.0-693.17.1.el7.ppc64le #1
  Mar 16 16:21:24 localhost kernel: task: c00000151ca19620 ti: c0000014d4250000 task.ti: c0000014d4250000
  Mar 16 16:21:24 localhost kernel: NIP: d000000013a38d60 LR: d000000013a38d44 CTR: c000000000826a70
  Mar 16 16:21:24 localhost kernel: REGS: c0000014d4252e70 TRAP: 0700   Tainted: G        W  OE  ------------    (3.10.0-693.17.1.el7.ppc64le)
  Mar 16 16:21:24 localhost kernel: MSR: 8000000100029033 <SF,EE,ME,IR,DR,RI,LE>  CR: 28222824  XER: 00000000
  Mar 16 16:21:24 localhost kernel: CFAR: d000000013a3643c SOFTE: 1 #012GPR00: d000000013a38d44 c0000014d42530f0 d000000013a5ee28 0000000000000004 #012GPR04: 0000000000000000 0000000000000021 00000000000005c2 00000000000005c2 #012GPR08: 0000000000000000 ffffffffffffe000 0000000000000001 d000000013a4fb28 #012GPR12: c000000000826a70 c000000007b32000 00003fffeca708b8 0000000000000000 #012GPR16: 0000000000000000 c0000014fa7ae410 c000001dd0e05c78 c000001dfe0a8098 #012GPR20: 0000000000000000 c000001dd0f50000 0000000000000000 c000001dd0e05c00 #012GPR24: c000001dd0e05c08 c000001dd0e00000 00000000000005be c000001dd0e05c08 #012GPR28: c000001dd0e00900 c000001dd0e05c18 0000000000000000 c000001dd0e05c00 
  Mar 16 16:21:24 localhost kernel: NIP [d000000013a38d60] be_close+0x8a0/0x990 [be2net]
  Mar 16 16:21:24 localhost kernel: LR [d000000013a38d44] be_close+0x884/0x990 [be2net]
  Mar 16 16:21:24 localhost kernel: Call Trace:
  Mar 16 16:21:24 localhost kernel: [c0000014d42530f0] [d000000013a38d44] be_close+0x884/0x990 [be2net] (unreliable)
  Mar 16 16:21:24 localhost kernel: [c0000014d42531d0] [c00000000082bdc4] __dev_close+0x1a4/0x260
  Mar 16 16:21:24 localhost kernel: [c0000014d4253290] [c00000000083238c] __dev_change_flags+0x3fc/0x520
  Mar 16 16:21:24 localhost kernel: [c0000014d4253330] [c0000000008324e4] dev_change_flags+0x34/0x150
  Mar 16 16:21:24 localhost kernel: [c0000014d4253380] [c00000000084a600] do_setlink+0x400/0xfe0
  Mar 16 16:21:24 localhost kernel: [c0000014d42534e0] [c00000000084e43c] rtnl_newlink+0x8ec/0xa80
  Mar 16 16:21:24 localhost kernel: [c0000014d42537d0] [c000000000848cf4] rtnetlink_rcv_msg+0xd4/0x300
  Mar 16 16:21:24 localhost kernel: [c0000014d4253850] [c00000000087c8a8] netlink_rcv_skb+0x128/0x140
  Mar 16 16:21:24 localhost kernel: [c0000014d4253890] [c000000000848bfc] rtnetlink_rcv+0x3c/0x60
  Mar 16 16:21:24 localhost kernel: [c0000014d42538c0] [c00000000087c468] netlink_unicast+0x628/0x6d0
  Mar 16 16:21:24 localhost kernel: [c0000014d4253960] [c00000000087da00] netlink_sendmsg+0x3b0/0x830
  Mar 16 16:21:24 localhost kernel: [c0000014d4253a40] [c0000000007f6d7c] sock_sendmsg+0xec/0x140
  Mar 16 16:21:24 localhost kernel: [c0000014d4253bb0] [c0000000007f849c] ___sys_sendmsg+0x36c/0x380
  Mar 16 16:21:24 localhost kernel: [c0000014d4253d50] [c0000000007fd844] SyS_sendmsg+0xc4/0x130
  Mar 16 16:21:24 localhost kernel: [c0000014d4253dd0] [c0000000007fe5bc] SyS_socketcall+0x3ac/0x430
  Mar 16 16:21:24 localhost kernel: [c0000014d4253e30] [c00000000000a184] system_call+0x38/0xe4
  Mar 16 16:21:24 localhost kernel: Instruction dump: 
  Mar 16 16:21:24 localhost kernel: 7d4ad214 7fe4fb78 7d454838 78a50420 4bffd529 7d20e828 7d234850 7d20e92d 
  Mar 16 16:21:24 localhost kernel: 40c2fff4 60420000 813f0018 7d2907b4 <0b090000> b2df40d0 935b000c 935b0008 
  Mar 16 16:21:24 localhost kernel: ---[ end trace cc5a527518abf256 ]---
  

• The error may be preceded by iommu_free: invalid entry warnings:

  Mar 16 16:21:24 localhost kernel: iommu_free_check: 7155 callbacks suppressed
  Mar 16 16:21:24 localhost kernel: iommu_free: invalid entry
  Mar 16 16:21:24 localhost kernel: #011entry     = 0x0
  Mar 16 16:21:24 localhost kernel: #011dma_addr  = 0x0
  Mar 16 16:21:24 localhost kernel: #011Table     = 0xc000001dd17e4800
  Mar 16 16:21:24 localhost kernel: #011bus#      = 0x1
  Mar 16 16:21:24 localhost kernel: #011size      = 0x80000
  Mar 16 16:21:24 localhost kernel: #011startOff  = 0x800000000000
  Mar 16 16:21:24 localhost kernel: #011index     = 0x70040018
  Mar 16 16:21:24 localhost kernel: ------------[ cut here ]------------
  Mar 16 16:21:24 localhost kernel: WARNING: CPU: 32 PID: 14820 at arch/powerpc/kernel/iommu.c:368 iommu_unmap_page+0x24c/0x260
  Mar 16 16:21:24 localhost kernel: Modules linked in: tracedev(OE) sctp_diag sctp dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter pseries_rng sg be2net nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c dm_service_time sd_mod crc_t10dif crct10dif_generic crct10dif_common ibmvscsi scsi_transport_srp ibmveth scsi_tgt dm_multipath dm_mirror
  Mar 16 16:21:24 localhost kernel: dm_region_hash dm_log dm_mod [last unloaded: mmfslinux]
  Mar 16 16:21:24 localhost kernel: CPU: 32 PID: 14820 Comm: ip Tainted: G        W  OE  ------------   3.10.0-693.17.1.el7.ppc64le #1
  Mar 16 16:21:24 localhost kernel: task: c00000151ca19620 ti: c0000014d4250000 task.ti: c0000014d4250000
  Mar 16 16:21:24 localhost kernel: NIP: c00000000004b48c LR: c00000000004b488 CTR: 0000000000655170
  Mar 16 16:21:24 localhost kernel: REGS: c0000014d4252d60 TRAP: 0700   Tainted: G        W  OE  ------------    (3.10.0-693.17.1.el7.ppc64le)
  Mar 16 16:21:24 localhost kernel: MSR: 8000000100029033 <SF,EE,ME,IR,DR,RI,LE>  CR: 48222828  XER: 20000000
  Mar 16 16:21:24 localhost kernel: CFAR: c0000000009dfaf4 SOFTE: 1 #012GPR00: c00000000004b488 c0000014d4252fe0 c00000000120ef00 0000000000000017 #012GPR04: 0000000000000001 0000000000000000 0000000000000600 0000000000000001 #012GPR08: 0000000000000001 0000000000000007 0000000000000006 00000002000b0004 #012GPR12: 0000000000002200 c000000007b32000 00003fffeca708b8 0000000000000000 #012GPR16: 0000000000000000 c0000014fa7ae410 c000001dd0e05c78 c000001dfe0a8098 #012GPR20: 0000000000000000 c000001dd0f50000 0000000000000000 c000001dd0e05c00 #012GPR24: 00000000000005be 00000000000005c1 c000001dd0e00900 0000000000000001 #012GPR28: 0000000000000000 0000000000000000 ffff800000000000 c000001dd17e4800
  Mar 16 16:21:24 localhost kernel: NIP [c00000000004b48c] iommu_unmap_page+0x24c/0x260
  Mar 16 16:21:24 localhost kernel: LR [c00000000004b488] iommu_unmap_page+0x248/0x260
  Mar 16 16:21:24 localhost kernel: Call Trace:
  Mar 16 16:21:24 localhost kernel: [c0000014d4252fe0] [c00000000004b488] iommu_unmap_page+0x248/0x260 (unreliable)
  Mar 16 16:21:24 localhost kernel: [c0000014d4253070] [c000000000049060] dma_iommu_unmap_page+0x20/0x40
  Mar 16 16:21:24 localhost kernel: [c0000014d4253090] [d000000013a363b8] be_tx_compl_process+0x158/0x280 [be2net]
  Mar 16 16:21:24 localhost kernel: [c0000014d42530f0] [d000000013a38d44] be_close+0x884/0x990 [be2net]
  Mar 16 16:21:24 localhost kernel: [c0000014d42531d0] [c00000000082bdc4] __dev_close+0x1a4/0x260
  Mar 16 16:21:24 localhost kernel: [c0000014d4253290] [c00000000083238c] __dev_change_flags+0x3fc/0x520
  Mar 16 16:21:24 localhost kernel: [c0000014d4253330] [c0000000008324e4] dev_change_flags+0x34/0x150
  Mar 16 16:21:24 localhost kernel: [c0000014d4253380] [c00000000084a600] do_setlink+0x400/0xfe0
  Mar 16 16:21:24 localhost kernel: [c0000014d42534e0] [c00000000084e43c] rtnl_newlink+0x8ec/0xa80
  Mar 16 16:21:24 localhost kernel: [c0000014d42537d0] [c000000000848cf4] rtnetlink_rcv_msg+0xd4/0x300
  Mar 16 16:21:24 localhost kernel: [c0000014d4253850] [c00000000087c8a8] netlink_rcv_skb+0x128/0x140
  Mar 16 16:21:24 localhost kernel: [c0000014d4253890] [c000000000848bfc] rtnetlink_rcv+0x3c/0x60
  Mar 16 16:21:24 localhost kernel: [c0000014d42538c0] [c00000000087c468] netlink_unicast+0x628/0x6d0
  Mar 16 16:21:24 localhost kernel: [c0000014d4253960] [c00000000087da00] netlink_sendmsg+0x3b0/0x830
  Mar 16 16:21:24 localhost kernel: [c0000014d4253a40] [c0000000007f6d7c] sock_sendmsg+0xec/0x140
  Mar 16 16:21:24 localhost kernel: [c0000014d4253bb0] [c0000000007f849c] ___sys_sendmsg+0x36c/0x380
  Mar 16 16:21:24 localhost kernel: [c0000014d4253d50] [c0000000007fd844] SyS_sendmsg+0xc4/0x130
  Mar 16 16:21:24 localhost kernel: [c0000014d4253dd0] [c0000000007fe5bc] SyS_socketcall+0x3ac/0x430
  Mar 16 16:21:24 localhost kernel: [c0000014d4253e30] [c00000000000a184] system_call+0x38/0xe4
  Mar 16 16:21:24 localhost kernel: Instruction dump:
  Mar 16 16:21:24 localhost kernel: 48994641 60000000 e89f0028 3c62ff9a 3863e4e0 4899462d 60000000 e89f0038
  Mar 16 16:21:24 localhost kernel: 3c62ff9a 3863e4f8 48994619 60000000 <0fe00000> 4bfffe9c 60000000 60000000
  Mar 16 16:21:24 localhost kernel: ---[ end trace cc5a527518abf254 ]---