"nf_conntrack: falling back to vmalloc" messages in /var/log/messages

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 7 and older version

Issue

  • Several messages "kernel: nf_conntrack: falling back to vmalloc" logged in /var/log/messages file.

    kernel: nf_conntrack: falling back to vmalloc.
kernel: nf_conntrack: falling back to vmalloc.

Resolution

  1. Increase vm.min_free_kbytes sysctl value.

    You may view the current value of this tunable with:

     $ cat /proc/sys/vm/min_free_kbytes 
 67584

    And double it with a setting like:

     vm.min_free_kbytes = 135168
    • The settings can be applied in /etc/sysctl.conf and loaded using #sysctl -p
    • You may keep doubling this value a few times to allow more memory to be free.
    • Depending on the usage of the system, the value should not exceed more than 0.4% or 2GB of system Memory. The kernel performs a variety of operations to ensure the memory is readily available, so setting vm.min_free_kbytes to larger values may result in the kernel erroneously working to reclaim memory and negatively impact performance.

  2. Scheduling a downtime and rebooting the system would solve the memory fragmentation problem and the error would disappear.

Root Cause

  • Kernel uses two methods to allocate memory: kmalloc and vmalloc. Kmalloc allocates a contiguous physical block and it fails if the contiguous block is not available and the kernel falls back to vmalloc method.
  • vmalloc means allocating memory that is only virtually contiguous. The reported error appears when memory is fragmented and the kernel is not able to get continuous free pages for allocation.
  • The vm.min_free_kbytes sysctl parameter determines the lowmem free minimum threshold value. It is used to keep the minimum amount of RAM free for the most critical kernel operations.
  • In other words, if your machine runs out of available memory, this parameter ensures that there's always at least some memory for the most critical operations.
  • Note : The warning no longer appears in RHEL 8. This is the commit that removed the warning :
    netfilter: nf_conntrack: silence warning on falling back to vmalloc()

    Since 88eab472ec21 ("netfilter: conntrack: adjust nf_conntrack_buckets default
    value"), the hashtable can easily hit this warning. We got reports from users
    that are getting this message in a quite spamming fashion, so better silence
    this.


diff --git a/net/netfilter/nf_conntrack_core.c b/net/netfilter/nf_conntrack_core.c
index 651039a..f168099 100644
--- a/net/netfilter/nf_conntrack_core.c
+++ b/net/netfilter/nf_conntrack_core.c
@@ -1544,10 +1544,8 @@ void *nf_ct_alloc_hashtable(unsigned int *sizep, int nulls)
        sz = nr_slots * sizeof(struct hlist_nulls_head);
        hash = (void *)__get_free_pages(GFP_KERNEL | __GFP_NOWARN | __GFP_ZERO,
                                        get_order(sz));
-       if (!hash) {
-               printk(KERN_WARNING "nf_conntrack: falling back to vmalloc.\n");
+       if (!hash)
                hash = vzalloc(sz);
-       }

        if (hash && nulls)
                for (i = 0; i < nr_slots; i++)

Diagnostic Steps

  • Search for "kernel: nf_conntrack: falling back to vmalloc" messages in /var/log/messages file:

    $ grep 'kernel: nf_conntrack: falling back to vmalloc' /var/log/messages*
kernel: nf_conntrack: falling back to vmalloc
kernel: nf_conntrack: falling back to vmalloc

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments