RHEL7 - virtio_net + docker bridges cause skb_warn_bad_offload warnings/errors

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux Server release 7.4
  • 3.10.0-693.11.1.el7.x86_64
  • Hypervisor : Nutanix
  • Openshift Container Platform 3.6

Issue

  • skb_warn_bad_offload warning message during starting openshift pod(container)
kernel: ------------[ cut here ]------------
 kernel: WARNING: CPU: 3 PID: 2696 at net/core/dev.c:2496 skb_warn_bad_offload+0xcd/0xda
kernel: : caps=(0x0000362007db58e9, 0x0000000000000000) len=2342 data_len=2214 gso_size=1398 gso_type=5 ip_summed=1
kernel: Modules linked in: nfsv3 twofish_generic twofish_x86_64 twofish_common serpent_avx_x86_64 serpent_sse2_x86_64 xts serpent_generic xt_statistic rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache veth nf_conntrack_netlink nfnetlink xt_nat xt_recent xt_mark xt_comment br_netfilter bridge stp llc vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_defrag_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd sg joydev parport_pc virtio_balloon i2c_piix4 parport pcspkr nfsd auth_rpcgss
kernel:  nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_net virtio_scsi bochs_drm drm_kms_helper syscopyarea sysfillrect ata_piix sysimgblt fb_sys_fops ttm drm libata crct10dif_pclmul crct10dif_common i2c_core crc32c_intel virtio_pci floppy serio_raw virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
kernel: CPU: 3 PID: 2696 Comm: openshift Tainted: G        W      ------------   3.10.0-693.11.1.el7.x86_64 #1
kernel: Hardware name: Red Hat KVM, BIOS seabios-1.7.5-8.el6 04/01/2014
kernel:  ffff88023fd839a0 000000008499f94c ffff88023fd83950 ffffffff816a3e61
kernel:  ffff88023fd83990 ffffffff810879d8 000009c01c6f14b2 ffff88009cf6f500
kernel:  ffff8800bae58000 0000000000000005 0000000000000001 0000000000000006
kernel: Call Trace:
kernel:  <IRQ>  [<ffffffff816a3e61>] dump_stack+0x19/0x1b
kernel:  [<ffffffff810879d8>] __warn+0xd8/0x100
kernel:  [<ffffffff81087a5f>] warn_slowpath_fmt+0x5f/0x80
kernel:  [<ffffffff81329183>] ? ___ratelimit+0x93/0x100
kernel:  [<ffffffff816a60d4>] skb_warn_bad_offload+0xcd/0xda
kernel:  [<ffffffff81588c25>] __skb_gso_segment+0x105/0x150
kernel:  [<ffffffff81589025>] validate_xmit_skb.isra.102.part.103+0x135/0x2e0
kernel:  [<ffffffff815897f0>] __dev_queue_xmit+0x4b0/0x550
kernel:  [<ffffffff815898a0>] dev_queue_xmit+0x10/0x20
kernel:  [<ffffffff815cfdae>] ip_finish_output+0x52e/0x780
kernel:  [<ffffffff815d0303>] ip_output+0x73/0xe0
kernel:  [<ffffffff815cf880>] ? __ip_append_data.isra.50+0xa50/0xa50
kernel:  [<ffffffff815cbd16>] ip_forward_finish+0x66/0x80
kernel:  [<ffffffff815cc0ac>] ip_forward+0x37c/0x480
kernel:  [<ffffffff815cbcb0>] ? ip_frag_mem+0x40/0x40
kernel:  [<ffffffff815c9cfa>] ip_rcv_finish+0x8a/0x350
kernel:  [<ffffffff815ca686>] ip_rcv+0x2b6/0x410
kernel:  [<ffffffff815c9c70>] ? inet_del_offload+0x40/0x40
kernel:  [<ffffffff81586f22>] __netif_receive_skb_core+0x572/0x7c0
kernel:  [<ffffffff81587188>] __netif_receive_skb+0x18/0x60
kernel:  [<ffffffff81587210>] netif_receive_skb_internal+0x40/0xc0
kernel:  [<ffffffff81588318>] napi_gro_receive+0xd8/0x130
kernel:  [<ffffffffc005e505>] virtnet_poll+0x265/0x750 [virtio_net]
kernel:  [<ffffffff8158799d>] net_rx_action+0x16d/0x380
kernel:  [<ffffffff81090b4f>] __do_softirq+0xef/0x280
kernel:  [<ffffffff816b6b1c>] call_softirq+0x1c/0x30
kernel:  [<ffffffff8102d3c5>] do_softirq+0x65/0xa0
kernel:  [<ffffffff81090ed5>] irq_exit+0x105/0x110
kernel:  [<ffffffff816b76b6>] do_IRQ+0x56/0xe0
kernel:  [<ffffffff816ac2ad>] common_interrupt+0x6d/0x6d
kernel:  <EOI>
kernel: ---[ end trace bc1fd42939b400c6 ]---
kernel: ------------[ cut here ]------------
  • Trying to disable(ethtool -K) offloads in VM didn't appear to help.

Resolution

Update to RHEL 7.6 - kernel-3.10.0-957 via RHSA-2018:3083 or later.

Workaround

There is no workaround for this issue, however the message is harmless and can be ignored, no network traffic is lost.

Root Cause

Upstream commit which resolves this issue: https://patchwork.kernel.org/patch/9882493/

commit b2504a5dbef3305ef41988ad270b0e8ec289331c upstream.

Dmitry reported warnings occurring in __skb_gso_segment() [1]

All SKB_GSO_DODGY producers can allow user space to feed
packets that trigger the current check.

We could prevent them from doing so, rejecting packets, but
this might add regressions to existing programs.

It turns out our SKB_GSO_DODGY handlers properly set up checksum
information that is needed anyway when packets needs to be segmented.

By checking again skb_needs_check() after skb_mac_gso_segment(),
we should remove these pesky warnings, at a very minor cost.

This issue is addressed by Red Hat under Red Hat Private Bug 1544920 - virtio_net + docker bridges cause skb_warn_bad_offload warnings/errors.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments