neutron-server high memory usage due to OVN mac_binding explosion at scale
Issue
- In a large-scale environment, growth of OVN southbound database MAC_Binding table can cause excessive Neutron RSS memory consumption and trigger out-of-memory killer (OOM killer).
- For example, the neutron server used 73.5G of RSS memory in a deployment with 4k VMs, 1.45K networks, 2.4K floating IPs, 800 routers, 820 security groups and 3.3K security groups rules. The 800 routers were connected to a /16 external network to serve floating IPs from this network for the VMs. The OVN southbound database had 1.15 million MAC_binding rows.
- Memory consumption did not come down after cleanup of all routers, VMs, and other neutron resources.
Environment
- Red Hat OpenStack Platform 16.2
- Red Hat OpenStack Platform 16.1
- ML2/OVN mechanism driver
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.