How to setup SCTP In Red Hat Enterprise Linux 8

Resolution

The overall workflow requires installing kernel-modules-extra, adding the appropriate modules to /etc/modules-load.d/* to load before sysctls are set during boot, then rebooting to ensure the module and sysctls load appropriately. From there, ss and netstat along with some tool such as nc can be used to ensure the sizes are set.

1. Install kernel-modules-extra for the currently installed kernel;

   # dnf install kernel-modules-extra-`uname -r`
   

   • The latest kernel and kernel-modules-extra packages will be installed if the uname -r section is left out of the above command.

2. Add sctp to /etc/modules-load.d/* to load sctp before systemd-sysctl.service during boot. Loading sctp before systemd-sysctl.service allows the sctp sysctl.conf settings to be effective;

   # cat /etc/modules-load.d/sctp.conf
   sctp
   

3. sctp is blacklisted by default on installation. Comment out the blacklisting to enable sctp to be loaded.

    r8 # grep sctp /etc/modprobe.d/*
   /etc/modprobe.d/sctp-blacklist.conf:#blacklist sctp
   /etc/modprobe.d/sctp_diag-blacklist.conf:#blacklist sctp_diag
   

4. Reboot (or simply manually load the module, modprobe sctp)

5. Check some command such as ncat provided from the nmap-ncat package to ensure sctp sockets can be created

    r8 # lsmod | grep sctp  # checking the module is loaded
   sctp                  409600  4
   ip6_udp_tunnel         16384  1 sctp
   udp_tunnel             20480  1 sctp
   libcrc32c              16384  5 nf_conntrack,nf_nat,nf_tables,xfs,sctp
   
    r8 # ncat --sctp -k -l 127.0.0.1 8192  # creates an sctp socket on the local host at socket number 8192
   
   # In another terminal, check ss
    r8 # ss -pneomSa | grep -A 1 8192
   LISTEN 0      10         127.0.0.1:8192      0.0.0.0:*    users:(("ncat",pid=1912,fd=3)) ino:34833 sk:1 <->
        skmem:(r0,rb212992,t0,tb212992,f0,w0,o0,bl0,d0) locals:127.0.0.1
   

A few notes;

• By default, sctp_diag is not needed unless using tools like ss to see diagnostics and metrics about sctp sockets. Running the above ss command will force sctp_diag to load in order to produce the metrics to userspace.

• The sctp module contains hard-coded predefined values for its sysctls. As such, if sctp is manually loaded (e.g. with modprobe or insmod) before sysctls are re-read (e.g. sysctl -p), the values of the set sctp sysctls will likely differ from the sctp sysctls defined in a sysctl configuration file. For example;

  # grep sctp /etc/sysctl.conf  # configured sctp sysctls
  net.sctp.sctp_mem = 16384   24037   36054
  net.sctp.sctp_rmem = 8192   865500  865500
  net.sctp.sctp_wmem = 8192   16384   769184
  
  # sysctl -ar sctp | grep mem  # the current sctp sysctls values
  net.sctp.sctp_mem = 18027   24037   36054
  net.sctp.sctp_rmem = 4096   865500  865500
  net.sctp.sctp_wmem = 4096   16384   769184
  

  • Adding sctp to /etc/modules-load.d/* will ensure the sctp module is loaded before systemd sets sysctls with systemd-sysctl.service ensuring the sctp sysctls defined in a sysctl configuration file are set after loading the module and thus overriding the sctp module default values.

• At the time of writing this article, the sysctls net.core.rmem_default and net.core.wmem_default set the default values for socket sizes, not the net.sctp.sctp_rmem and net.sctp.sctp_wmem sysctls. In fact, net.sctp.sctp_wmem is not used at all and only the min value of net.sctp.sctp_rmem is used from these tunables;

  commit a6e1204b82d3e8d7ba1baca35bdb595c722105eb
  From 2011
  
  Update description of net.sctp.sctp_rmem and net.sctp.sctp_wmem tunables
  
  sctp does not use second and third ("default" and "max") values
  of sctp_rmem tunable. The format is the same as tcp_rmem
  but the meaning is different so make the documentation explicit to
  avoid confusion.
  

  • For example,

     r8 # sysctl -q net.core.wmem_default
    net.core.wmem_default = 212992       <--- default socket size for writes
    
     r8 # sysctl -q net.sctp.sctp_wmem
    net.sctp.sctp_wmem = 4096   16384   1792896
    
     r8 # ss -pneomSa | grep -A 1 8192   # ncat and ss example from step 4 above
    LISTEN 0      10         127.0.0.1:8192      0.0.0.0:*    users:(("ncat",pid=1912,fd=3)) ino:34833 sk:1 <->
         skmem:(r0,rb212992,t0,tb212992,f0,w0,o0,bl0,d0) locals:127.0.0.1    <--- note rb212992
    

  • In the above output the write buffer size for the socket is net.core.wmem_default and none of the values in net.sctp.sctp_wmem.

• The size of socket receive buffers for sockets will be doubled by the kernel when set via setsocktop(). The doubling of size is to accommodate the extra space needed to manage and maintain the sockets. From socket(7);

  SO_RCVBUF
        Sets  or  gets the maximum socket receive buffer in bytes.  The kernel doubles this value 
        (to allow space for bookkeeping overhead) when it is set using setsockopt(2), and this doubled 
        value is returned by getsockopt(2).  The default value is set by the /proc/sys/net/core/rmem_default 
        file, and the maximum allowed value is set by the /proc/sys/net/core/rmem_max file.  The minimum 
        (doubled) value for this option is 256.
  

  • This value is doubled after being limited by the net.core.rmem_max sysctl. As such, sizes can seem to exceed the sysctl limit on the surface.