Why do we get "NetworkManager: Loop callback failed with: Cannot allocate memory" message while using teaming ?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 7
  • teaming / teamd

Issue

  • The problem is that under heavy load the system lose connectivity and reports the following message:
"NetworkManager: Loop callback failed with: Cannot allocate memory"

Resolution

Update to kernel-3.10.0-862.el7 or later (Errata RHSA-2018:1062).

Workaround

If a kernel update is not feasible in your environment, consider using bonding instead of teaming.

Root Cause

This issue was initially fixed upstream by team: fix memory leaks in upstream Linux kernel v4.11:

In functions team_nl_send_port_list_get() and team_nl_send_options_get(), pointer skb keeps the return value of nlmsg_new(). When the call to genlmsg_put() fails, the memory is not freed(). This will result in memory leak bugs.

This was included in development package kernel-3.10.0-717.el7 and later via Red Hat Bug 1448266 - backport team memory leak fix.

Diagnostic Steps

  • System reports following messages:
[...]
Jul 27 16:50:04 hostname NetworkManager: ens5f1: Changed port state: "current" -> "expired"
Jul 27 16:50:04 hostname kernel: private: Failed to send options change via netlink (err -105)
Jul 27 16:50:04 hostname NetworkManager: ens5f1: Changed port state: "expired" -> "current"
Jul 27 16:50:04 hostname NetworkManager: ens4f1: Changed port state: "current" -> "expired"
Jul 27 16:50:04 hostname NetworkManager: ens4f1: Failed to disable port.
Jul 27 16:50:04 hostname NetworkManager: Loop callback failed with: Cannot allocate memory
Jul 27 16:50:04 hostname NetworkManager: ens4f1: Failed to disable port.
Jul 27 16:50:04 hostname NetworkManager: Loop callback failed with: Cannot allocate memory
Jul 27 16:50:04 hostname NetworkManager: Loop callback failed with: Cannot allocate memory
Jul 27 16:50:04 hostname NetworkManager: ens4f1: Changed port state: "expired" -> "current"
[...]
  • Verify if system is using teaming:
# ip a | egrep '^[0-9]'
[...]
6: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master nm-team state UP qlen 1000
7: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master nm-team state UP qlen 1000
9: nm-team: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments