Packet loss when multiple processes subscribe to the same multicast group

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux
  • Multicast UDP traffic
  • Multiple processes subscribing to the same multicast group

Issue

  • Packet loss when multiple processes subscribe to the same multicast group
  • ksoftirqd consumes 100% cpu
  • There multiple instances of application listening to multicast at same address and port. We see ksoftirqd consumes 100% cpu and high softirq context switching (%SI). We see lots of packet drops at this moment.

Resolution

The following items may help:

Ultimately, as processes are added, it will become necessary to re-design the application to use one "listening" thread which pulls data from the multicast socket, and distributes the received data between multiple "processing" threads using an IPC method such as POSIX Shared Memory.

Root Cause

A good explanation is given in the product documentation:

Diagnostic Steps

Capture perf of a predictable workload (eg: iperf or mdump) with the receiving processes running under various conditions, such as:

  • Single listener on same CPU as softirq
  • Multiple listeners on same CPU as softirq
  • Single listener on different CPU but same NUMA Node as softirq
  • Multiple listeners on different CPUs but same NUMA Node as softirq

Add listeners until packet loss is observed.

As the problem gets worse, you should see the ksoftirqd comm spending more and more time in scheduler spinlock code such as try_to_wake_up, and less time delivering to UDP socket buffers.

Observing the output of ethtool -S and netstat -s with monitor.sh should show packet loss in the ring buffer of the NIC (various statistics depending on driver) and in the UDP socket buffer ("packet receive errors" under the "Udp" heading).

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments