Why traces like "WARNING: at net/core/dev.c:XXXX skb_warn_bad_offload+0xc2/0xf0()" are generated and system becomes inactive when "clusvcadm" command is run to enable some vmware service ?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 6.5
  • kernel-2.6.32-431.1.2.el6.x86_64
  • HP ProLiant DL580 G7 systems.
  • netxen_nic NIC modules.
  • ixgbe NIC modules.

Issue

  • When a vmware service is started on a node with command clusvcadm -e Plato-VMWARE , system becomes inactive and following traces are generated:
Dec 19 11:37:46 HOSTNAME kernel: ------------[ cut here ]------------
Dec 19 11:37:46 HOSTNAME kernel: WARNING: at net/core/dev.c:1907 skb_warn_bad_offload+0xc2/0xf0() (Tainted: G        W  ---------------   )
Dec 19 11:37:46 HOSTNAME kernel: Hardware name: ProLiant DL580 G7
Dec 19 11:37:46 HOSTNAME kernel: netxen_nic: caps=(0x11c9b3, 0x0) len=146 data_len=0 ip_summed=1
Dec 19 11:37:46 HOSTNAME kernel: Modules linked in: vmnet(U) ppdev parport_pc parport fuse vsock(U) vmci(U) vmmon(U) nfsd exportfs autofs4 gfs2 nfs lockd fscache auth_rpcgss nfs_acl sunrpc dlm configfs cpufreq_ondemand freq_table pcc_cpufreq arpt_mangle arptable_filter arp_tables ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_recent ipt_LOG iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_round_robin dm_multipath uinput microcode iTCO_wdt iTCO_vendor_support power_meter lpc_ich mfd_core sg netxen_nic hpilo hpwdt i7core_edac edac_core shpchp ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif qla2xxx scsi_transport_fc scsi_tgt pata_acpi ata_generic ata_piix hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Dec 19 11:37:46 HOSTNAME kernel: Pid: 0, comm: swapper Tainted: G        W  ---------------    2.6.32-431.1.2.el6.x86_64 #1
Dec 19 11:37:46 HOSTNAME kernel: Call Trace:
Dec 19 11:37:46 HOSTNAME kernel: <IRQ>  [<ffffffff81071e27>] ? warn_slowpath_common+0x87/0xc0
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff81071f16>] ? warn_slowpath_fmt+0x46/0x50
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffffa0342370>] ? netxen_nic_get_drvinfo+0xc0/0xf0 [netxen_nic]
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff8145afe2>] ? skb_warn_bad_offload+0xc2/0xf0
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff814600d1>] ? __skb_gso_segment+0x71/0xc0
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff81460133>] ? skb_gso_segment+0x13/0x20
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffffa069f9bd>] ? VNetBridgeSendLargePacket+0x1d/0x90 [vmnet]
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffffa06a0c70>] ? VNetBridgeReceiveFromDev+0x310/0x420 [vmnet]
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff8145b527>] ? __netif_receive_skb+0x477/0x750
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff8145f1c8>] ? netif_receive_skb+0x58/0x60
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffffa033e523>] ? netxen_process_rcv_ring+0x913/0xb00 [netxen_nic]
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffffa0340b84>] ? netxen_process_cmd_ring+0x44/0x2a0 [netxen_nic]
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff81058d53>] ? __wake_up+0x53/0x70
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffffa0339b65>] ? netxen_nic_poll+0x45/0xc0 [netxen_nic]
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff813301f6>] ? credit_entropy_bits+0x76/0xd0
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff81460b53>] ? net_rx_action+0x103/0x2f0
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff81330356>] ? add_timer_randomness+0x106/0x110
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff8107a8e1>] ? __do_softirq+0xc1/0x1e0
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff810e6f02>] ? handle_IRQ_event+0x92/0x170
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff8100fa75>] ? do_softirq+0x65/0xa0
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff8107a795>] ? irq_exit+0x85/0x90
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff81530ff5>] ? do_IRQ+0x75/0xf0
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Dec 19 11:37:46 HOSTNAME kernel: <EOI>  [<ffffffff81426521>] ? poll_idle+0x41/0x80
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff814264f3>] ? poll_idle+0x13/0x80
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff81426707>] ? cpuidle_idle_call+0xa7/0x140
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Dec 19 11:37:46 HOSTNAME kernel: [<ffffffff81520e2c>] ? start_secondary+0x2ac/0x2ef
Dec 19 11:37:46 HOSTNAME kernel: ---[ end trace 228c1f1e9ed1b9eb ]---

Resolution

Disable Large Receive Offload (LRO) and/or Generic Recieve Offload (GRO).

This can be done during runtime with the following commands:

# ethtool -k ethX
# ethtool -K ethX lro off
# ethtool -K ethX gro off

You may persist these settings across reboot by writing /sbin/ifup-local as described here.

Note: It is incorrect to enable LRO when IP forwarding and/or bridging are in use.

Root Cause

  • This is a warning which is triggered when a GSO skb has skb->ip_summed != CHECKSUM_PARTIAL. Ideally, in order to do GSO the skb->ip_summed should be set to CHECKSUM_PARTIAL
  • skb->ip_summed could have been set to CHECKSUM_UNNECESSARY , probably by the VMware driver or by the guest.
  • skb_gso_segment() has been factored to support openvswitch which calls this function when it receives the packet. So, except for openvswitch, it shouldn't be used in the receiving path.
  • It may be possible that the LRO is providing a packet that VMware doesn't know how to handle properly. Disabling LRO provides a workaround.
  • This proves that the bug is within VMware

Diagnostic Steps

  • Check the genuineness of NIC module used.
  • Check the logs from the sosreport.
  • Collect strace logs as mentioned below for the clusvcadm command.
# strace -o /tmp/strace.txt clusvcadm -e Plato-VMWARE

  • As far as the physical machine is concerned, this seems to be a RX packet for which NetXen NIC sets skb->ip_summed = CHECKSUM_UNNECESSARY.

  • Then the packet reaches the VMware bridge and it seems to be sending it to another dev

rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: off  (Issue happens with both GSO on and off)
generic-receive-offload: on
large-receive-offload: on

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments