How net.ipv4.tcp_mem value calculated?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux( All Versions )
  • tcp_mem

Issue

  • How /proc/sys/net/ipv4/tcp_mem parameter value calculated?
  • Why /proc/sys/net/ipv4/tcp_mem parameter different in the system?
  • Why /proc/sys/net/ipv4/tcp_mem parameter value is half when compared to other systems?

Resolution

  • The parameter /proc/sys/net/ipv4/tcp_mem is calculated based on the available memory in the system.

  • The /proc/sys/net/ipv4/tcp_mem parameter defaults are calculated at boot time from the amount of available memory. At boot time, the kernel will check the available memory in the system, and based on the available memory kernel will assign the /proc/sys/net/ipv4/tcp_mem value.

tcp_mem - vector of 3 INTEGERs: min, pressure, max
    min: below this number of pages TCP is not bothered about its
    memory appetite.

    pressure: when the amount of memory allocated by TCP exceeds this number
    of pages, TCP moderates its memory consumption and enters memory
    pressure mode, which is exited when memory consumption falls
    under "min".

    max: number of pages allowed for queueing by all TCP sockets.

    Defaults are calculated at boot time from amount of available
    memory.
  • The source calculations that all of the settings are based on nr_free_buffer_pages().The nr_free_buffer_pages() can be calculated from values found in /proc/zoneinfo. Basically, nr_free_buffer_pages is the sum of managed pages for each zone subtracting the zone's high mark. (Note: when the system boots the calculations will not match perfectly because during boot the zoneinfo will be slightly different than what we view in /proc/zoneinfo after the boot is complete.)

Diagnostic Steps

The value of /proc/sys/net/ipv4/tcp_mem shown below:

[root@servera ~]# /proc/sys/net/ipv4/tcp_mem
185280  247041  370560

sysctl value in the system :

[root@servera ~]# sysctl -a  |  grep net.ipv4.tcp_mem
net.ipv4.tcp_mem = 185280   247041  370560

Available memory in the system :

[root@servera ~]# grep "MemAvailable\|MemTotal" /proc/meminfo
MemTotal:       15974496 kB
MemAvailable:    4522808 kB

[root@serverb ~]#  cat /proc/sys/net/ipv4/tcp_mem
21834   29115   43668

[root@serverb  ~]#  sysctl -a  |  grep net.ipv4.tcp_mem
net.ipv4.tcp_mem = 21834    29115   43668

[root@serverb ~]#   grep "MemAvailable\|MemTotal" /proc/meminfo
MemTotal:        1017740 kB
MemAvailable:     384764 kB
  • The source code tcp uses to initialize the tcp_mem setting is this tcp_init_mem() function:

RHEL 7 and earlier kernels

  void tcp_init_mem(struct net *net)
  {
        unsigned long limit = nr_free_buffer_pages() / 8;
        limit = max(limit, 128UL);
        net->ipv4.sysctl_tcp_mem[0] = limit / 4 * 3;
        net->ipv4.sysctl_tcp_mem[1] = limit;
        net->ipv4.sysctl_tcp_mem[2] = net->ipv4.sysctl_tcp_mem[0] * 2;

RHEL 8 and later kernels

  static void __init tcp_init_mem(void)
  {
        unsigned long limit = nr_free_buffer_pages() / 16;
        limit = max(limit, 128UL);
        sysctl_tcp_mem[0] = limit / 4 * 3;                       /* 4.68% */
        sysctl_tcp_mem[1] = limit;                               /* 6.25% */
        sysctl_tcp_mem[2] = net->sysctl_tcp_mem[0] * 2;          /* 9.37% */

In the case of system servera RHEL8, nr_free_buffer_pages() is as follows:

[root@servera ~]$ grep -e zone -e managed -e high /proc/zoneinfo | grep -A 2 zone
Node 0, zone      DMA
        high     24
        managed  3840
--
Node 0, zone    DMA32
        high     2532
        managed  411024
--
Node 0, zone   Normal
        high     22785
        managed  3578760

nr_free_buffer_pages() = (3578760+411024+3840) - (24+2532+22785) = 3968283 pages

Based on the above nr_free_buffer_pages() tcp_mem values are as below for servera

sysctl_tcp_mem[0]  =  248017 /4 * 3 = 186012
sysctl_tcp_mem[1]  =  3968283 / 16  = 248017
sysctl_tcp_mem[2]  =  186012 * 2    = 372024

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments