Why am I seeing the rx_dropped ifconfig counter increase on rt_kernel/RHEL7?
Environment
- Red Hat enterprise MRG realtime 2.3
- kernel-rt-3.8.13-rt14.25.el6rt
- Red Hat Enterprise Linux 7
Issue
- We are using bond interface with "FC Solarflare 10G network card" and also their driver installed on MRG 2.3.
Suddenly, dropped packets occurs on slave's one of bond interfaces, eth4. - After installing the RT kernel/RHEL7 we observed the rx_dropped counter increasing for ifconfig.
Resolution
- Working as designed. These errors can safely be ignored provided they are not resulting in TCP retransmissions or lowered throughput. Use a program such as iperf to measure and ensure throughput is acceptable.
- Use "netstat -s" to check that the retransmission rate is acceptable.
Tcp:
10007784 active connections openings
7067290 passive connection openings
2976883 failed connection attempts
26797 connection resets received
291 connections established
530001538 segments received
522569216 segments send out
31901 segments retransmited <----
0 bad segments received.
8720 resets sent
Root Cause
The RT and RHEL7 kernels contains code that updates the rx_dropped counter for other non-error conditions.
- The softnet backlog full
- Bad VLAN tags
- Packets received with unknown or unregistered protocols
- IPv6 frames when the server is configured only for ipv4
See the following changes in the rt_kernel core/dev.c:
static int __netif_receive_skb(struct sk_buff *skb)
.
.
.
if (sk_memalloc_socks() && skb_pfmemalloc(skb)
&& !skb_pfmemalloc_protocol(skb))
goto drop;
.
.
.
if (pt_prev) {
if (unlikely(skb_orphan_frags(skb, GFP_ATOMIC)))
goto drop;
else
ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
} else {
drop:
atomic_long_inc(&skb->dev->rx_dropped); <---- Counter is incremented
kfree_skb(skb);
But the RHEL6 kernel does not have these changes:
if (pt_prev) {
ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
} else {
kfree_skb(skb);
/* Jamal, now you will not able to escape explaining
* me how you were going to use this. :-)
*/
ret = NET_RX_DROP; <---- silently dropped
}
Diagnostic Steps
The rx_dropped error counter will be increasing as shown by the following ifconfig output:
bond0 Link encap:Ethernet HWaddr 00:0F:xx:xx:xx:xx
inet addr:10.1.xx.xx Bcast:10.1.xx.255 Mask:255.255.255.0
inet6 addr: fe80::20f:53ff:yyyy:xxxx/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:237375681 errors:0 dropped:295847 overruns:0 frame:0
TX packets:201849259 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:28084942473 (26.1 GiB) TX bytes:30785252323 (28.6 GiB)
- Use ethtool -S to confirm that there are no actual errors on the interfaces.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
