Why am I seeing the rx_dropped ifconfig counter increase on rt_kernel/RHEL7?

Solution Verified - Updated -

Environment

  • Red Hat enterprise MRG realtime 2.3
    • kernel-rt-3.8.13-rt14.25.el6rt
  • Red Hat Enterprise Linux 7

Issue

  • We are using bond interface with "FC Solarflare 10G network card" and also their driver installed on MRG 2.3.
    Suddenly, dropped packets occurs on slave's one of bond interfaces, eth4.
  • After installing the RT kernel/RHEL7 we observed the rx_dropped counter increasing for ifconfig.

Resolution

  • Working as designed. These errors can safely be ignored provided they are not resulting in TCP retransmissions or lowered throughput. Use a program such as iperf to measure and ensure throughput is acceptable.
  • Use "netstat -s" to check that the retransmission rate is acceptable.
Tcp:
    10007784 active connections openings
    7067290 passive connection openings
    2976883 failed connection attempts
    26797 connection resets received
    291 connections established
    530001538 segments received
    522569216 segments send out
    31901 segments retransmited <----
    0 bad segments received.
    8720 resets sent

Root Cause

The RT and RHEL7 kernels contains code that updates the rx_dropped counter for other non-error conditions.

  • The softnet backlog full
  • Bad VLAN tags
  • Packets received with unknown or unregistered protocols
  • IPv6 frames when the server is configured only for ipv4

See the following changes in the rt_kernel core/dev.c:

static int __netif_receive_skb(struct sk_buff *skb)

.
.
.
        if (sk_memalloc_socks() && skb_pfmemalloc(skb)
                                && !skb_pfmemalloc_protocol(skb))
                goto drop;

.
.
.

       if (pt_prev) {
                if (unlikely(skb_orphan_frags(skb, GFP_ATOMIC)))
                        goto drop;
                else
                        ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
        } else {
drop:
                atomic_long_inc(&skb->dev->rx_dropped); <---- Counter is incremented
                kfree_skb(skb);

But the RHEL6 kernel does not have these changes:

       if (pt_prev) {
                ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
        } else {
                kfree_skb(skb);
                /* Jamal, now you will not able to escape explaining
                 * me how you were going to use this. :-)
                 */
                ret = NET_RX_DROP; <---- silently dropped
        }

Diagnostic Steps

The rx_dropped error counter will be increasing as shown by the following ifconfig output:

 bond0     Link encap:Ethernet  HWaddr 00:0F:xx:xx:xx:xx 
          inet addr:10.1.xx.xx  Bcast:10.1.xx.255  Mask:255.255.255.0
          inet6 addr: fe80::20f:53ff:yyyy:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:237375681 errors:0 dropped:295847 overruns:0 frame:0
          TX packets:201849259 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:28084942473 (26.1 GiB)  TX bytes:30785252323 (28.6 GiB)
  • Use ethtool -S to confirm that there are no actual errors on the interfaces.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

3 Comments

I do not see any way of configuring the bonding driver to specifically ignore and not do increment the dropped counter for unknown VLAN packets?

The physical interfaces do not report any dropped packets due to VLAN packets not configured on the host, so why should be bonding driver do so?

"Working as designed. These errors can safely be ignored .." This change makes the ifconfig RX dropped counters unreliable for actual problem detection. The redesign obfuscates errors.

These counters have been misused from their original purpose and made useless long before this. They were intended to be used for protocol-only drops, so the above usage is actually correct, but various drivers put various hardware stats into these counters in inconsistent places as well, rendering the counter a vague indication at the best of times. If you wish to see protocol drops, use netstat -s. If you wish to see hardware drops, use ethtool -S.