Red Hat Training

A Red Hat training course is available for RHEL 8

Chapter 39. Monitoring network activity with SystemTap

You can use helpful example SystemTap scripts available in the /usr/share/systemtap/testsuite/systemtap.examples/ directory, upon installing the systemtap-testsuite package, to monitor and investigate the network activity of your system.

39.1. Profiling network activity with SystemTap

You can use the nettop.stp example SystemTap script to profile network activity. The script tracks which processes are generating network traffic on the system, and provides the following information about each process:

PID
The ID of the listed process.
UID
User ID. A user ID of 0 refers to the root user.
DEV
Which ethernet device the process used to send or receive data (for example, eth0, eth1).
XMIT_PK
The number of packets transmitted by the process.
RECV_PK
The number of packets received by the process.
XMIT_KB
The amount of data sent by the process, in kilobytes.
RECV_KB
The amount of data received by the service, in kilobytes.

Prerequisites

Procedure

  • Run the nettop.stp script:

    # stap  --example nettop.stp

    The nettop.stp script provides network profile sampling every 5 seconds.

    Output of the nettop.stp script looks similar to the following:

    [...]
      PID   UID DEV     XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
        0     0 eth0          0       5       0       0 swapper
    11178     0 eth0          2       0       0       0 synergyc
      PID   UID DEV     XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
     2886     4 eth0         79       0       5       0 cups-polld
    11362     0 eth0          0      61       0       5 firefox
        0     0 eth0          3      32       0       3 swapper
     2886     4 lo            4       4       0       0 cups-polld
    11178     0 eth0          3       0       0       0 synergyc
      PID   UID DEV     XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
        0     0 eth0          0       6       0       0 swapper
     2886     4 lo            2       2       0       0 cups-polld
    11178     0 eth0          3       0       0       0 synergyc
     3611     0 eth0          0       1       0       0 Xorg
      PID   UID DEV     XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
        0     0 eth0          3      42       0       2 swapper
    11178     0 eth0         43       1       3       0 synergyc
    11362     0 eth0          0       7       0       0 firefox
     3897     0 eth0          0       1       0       0 multiload-apple

39.2. Tracing functions called in network socket code with SystemTap

You can use the socket-trace.stp example SystemTap script to trace functions called from the kernel’s net/socket.c file. This helps you identify, in finer detail, how each process interacts with the network at the kernel level.

Prerequisites

Procedure

  • Run the socket-trace.stp script:

    # stap  --example socket-trace.stp

    A 3-second excerpt of the output of the socket-trace.stp script looks similar to the following:

    [...]
    0 Xorg(3611): -> sock_poll
    3 Xorg(3611): <- sock_poll
    0 Xorg(3611): -> sock_poll
    3 Xorg(3611): <- sock_poll
    0 gnome-terminal(11106): -> sock_poll
    5 gnome-terminal(11106): <- sock_poll
    0 scim-bridge(3883): -> sock_poll
    3 scim-bridge(3883): <- sock_poll
    0 scim-bridge(3883): -> sys_socketcall
    4 scim-bridge(3883):  -> sys_recv
    8 scim-bridge(3883):   -> sys_recvfrom
    12 scim-bridge(3883):-> sock_from_file
    16 scim-bridge(3883):<- sock_from_file
    20 scim-bridge(3883):-> sock_recvmsg
    24 scim-bridge(3883):<- sock_recvmsg
    28 scim-bridge(3883):   <- sys_recvfrom
    31 scim-bridge(3883):  <- sys_recv
    35 scim-bridge(3883): <- sys_socketcall
    [...]

39.3. Monitoring network packet drops with SystemTap

The network stack in Linux can discard packets for various reasons. Some Linux kernels include a tracepoint, kernel.trace("kfree_skb")`, which tracks where packets are discarded.

The dropwatch.stp SystemTap script uses kernel.trace("kfree_skb") to trace packet discards; the script summarizes what locations discard packets in every 5-second interval.

Prerequisites

Procedure

  • Run the dropwatch.stp script:

    # stap  --example dropwatch.stp

    Running the dropwatch.stp script for 15 seconds results in output similar to the following:

    Monitoring for dropped packets
    51 packets dropped at location 0xffffffff8024cd0f
    2 packets dropped at location 0xffffffff8044b472
    51 packets dropped at location 0xffffffff8024cd0f
    1 packets dropped at location 0xffffffff8044b472
    97 packets dropped at location 0xffffffff8024cd0f
    1 packets dropped at location 0xffffffff8044b472
    Stopping dropped packet monitor
    Note

    To make the location of packet drops more meaningful, see the /boot/System.map-$(uname -r) file. This file lists the starting addresses for each function, enabling you to map the addresses in the output of the dropwatch.stp script to a specific function name. Given the following snippet of the /boot/System.map-$(uname -r) file, the address 0xffffffff8024cd0f maps to the function unix_stream_recvmsg and the address 0xffffffff8044b472 maps to the function arp_rcv:

    [...]
    ffffffff8024c5cd T unlock_new_inode
    ffffffff8024c5da t unix_stream_sendmsg
    ffffffff8024c920 t unix_stream_recvmsg
    ffffffff8024cea1 t udp_v4_lookup_longway
    [...]
    ffffffff8044addc t arp_process
    ffffffff8044b360 t arp_rcv
    ffffffff8044b487 t parp_redo
    ffffffff8044b48c t arp_solicit
    [...]