neutron-dnsmasq-qdhp keeps on restarting

Issue

• We have a problem with neutron-dnsmasq-qdhcp service for about 8 months now, its containers continously become down on controller nodes and new containers are created.

• We can see the following in /var/log/containers/neutron/dhcp-agent.log where neutron-dhcp-agent keeps on resynchronizing dhcp namespaces:

2022-02-23 10:59:52.203 394035 INFO neutron.agent.dhcp.agent [req-aac502f9-0e80-4bf1-9c71-301e303a89d9 - - - - -] Synchronizing state complete
2022-02-23 11:01:25.204 394035 INFO neutron.agent.dhcp.agent [req-64f1784e-cb96-462f-9c60-bbce6db6a69d - - - - -] Agent has just been revived. Scheduling full sync
2022-02-23 11:01:52.205 394035 INFO neutron.agent.dhcp.agent [req-aac502f9-0e80-4bf1-9c71-301e303a89d9 - - - - -] Synchronizing state
2022-02-23 11:01:53.695 394035 INFO neutron.agent.dhcp.agent [req-89b7b919-62b7-4e52-8e4a-fdd0d5581531 - - - - -] All active networks have been fetched through RPC.
2022-02-23 11:01:53.698 394035 INFO neutron.agent.dhcp.agent [-] Starting network 149e91fa-e5a5-4de4-b764-090b9f1cf749 dhcp configuration
2022-02-23 11:01:53.699 394035 INFO neutron.agent.dhcp.agent [-] Finished network 149e91fa-e5a5-4de4-b764-090b9f1cf749 dhcp configuration
2022-02-23 11:01:53.699 394035 INFO neutron.agent.dhcp.agent [-] Starting network 164ec21d-79bc-426f-97cb-79af47d59fc5 dhcp configuration
2022-02-23 11:01:53.701 394035 INFO neutron.agent.dhcp.agent [-] Starting network 1e8faef6-9885-4b3a-be72-e0fd96567da6 dhcp configuration
2022-02-23 11:01:53.702 394035 INFO neutron.agent.dhcp.agent [-] Starting network 20740e20-4d45-44d2-88a7-2ba19e64b948 dhcp configuration
2022-02-23 11:01:53.703 394035 INFO neutron.agent.dhcp.agent [-] Starting network 24a67298-1100-4e5a-a2e6-21ab6bc38145 dhcp configuration
2022-02-23 11:01:55.519 394035 INFO neutron.agent.dhcp.agent [-] Finished network 164ec21d-79bc-426f-97cb-79af47d59fc5 dhcp configuration
2022-02-23 11:01:55.520 394035 INFO neutron.agent.dhcp.agent [-] Starting network 2eb730cb-cd43-4cbc-af51-e17c553ba6ec dhcp configuration
2022-02-23 11:01:55.520 394035 INFO neutron.agent.dhcp.agent [-] Finished network 2eb730cb-cd43-4cbc-af51-e17c553ba6ec dhcp configuration
2022-02-23 11:01:55.520 394035 INFO neutron.agent.dhcp.agent [-] Starting network 40c0d2cb-cded-4647-97ad-a30a1afc8027 dhcp configuration
2022-02-23 11:01:55.830 394035 INFO neutron.agent.dhcp.agent [-] Finished network 1e8faef6-9885-4b3a-be72-e0fd96567da6 dhcp configuration
2022-02-23 11:01:55.831 394035 INFO neutron.agent.dhcp.agent [-] Starting network 5929825f-5304-4931-a74c-dec85fcd7ad4 dhcp configuration
2022-02-23 11:01:55.831 394035 INFO neutron.agent.dhcp.agent [-] Finished network 5929825f-5304-4931-a74c-dec85fcd7ad4 dhcp configuration
2022-02-23 11:01:55.832 394035 INFO neutron.agent.dhcp.agent [-] Starting network 5db0c874-d19f-440d-8844-de079262d615 dhcp configuration
2022-02-23 11:01:56.121 394035 INFO neutron.agent.dhcp.agent [-] Finished network 20740e20-4d45-44d2-88a7-2ba19e64b948 dhcp configuration
2022-02-23 11:01:56.132 394035 INFO neutron.agent.dhcp.agent [-] Starting network 64216273-c064-4f64-ad60-16881823f4e7 dhcp configuration
2022-02-23 11:01:56.142 394035 INFO neutron.agent.dhcp.agent [-] Finished network 64216273-c064-4f64-ad60-16881823f4e7 dhcp configuration
2022-02-23 11:01:56.143 394035 INFO neutron.agent.dhcp.agent [-] Starting network 6fe9d8fd-b5e5-42e0-b71f-efe7b46efc2e dhcp configuration
2022-02-23 11:01:56.144 394035 INFO neutron.agent.dhcp.agent [-] Finished network 6fe9d8fd-b5e5-42e0-b71f-efe7b46efc2e dhcp configuration
2022-02-23 11:01:56.144 394035 INFO neutron.agent.dhcp.agent [-] Starting network 760ede08-bcbe-4756-923e-4427eb8ef4c4 dhcp configuration
2022-02-23 11:01:56.433 394035 INFO neutron.agent.dhcp.agent [-] Finished network 24a67298-1100-4e5a-a2e6-21ab6bc38145 dhcp configuration
2022-02-23 11:01:56.434 394035 INFO neutron.agent.dhcp.agent [-] Starting network 7f5dde8a-cc61-470d-a6a0-cbc9e154f5a9 dhcp configuration
2022-02-23 11:01:58.352 394035 INFO neutron.agent.dhcp.agent [-] Finished network 40c0d2cb-cded-4647-97ad-a30a1afc8027 dhcp configuration
2022-02-23 11:01:58.352 394035 INFO neutron.agent.dhcp.agent [-] Starting network 81a5a6a0-e0c5-45be-97d1-0a08619fe4d8 dhcp configuration
2022-02-23 11:01:58.639 394035 INFO neutron.agent.dhcp.agent [-] Finished network 5db0c874-d19f-440d-8844-de079262d615 dhcp configuration
2022-02-23 11:01:58.640 394035 INFO neutron.agent.dhcp.agent [-] Starting network 84fb6b7e-43a5-4447-8db1-b0bd3ccbffaa dhcp configuration
2022-02-23 11:01:58.917 394035 INFO neutron.agent.dhcp.agent [-] Finished network 760ede08-bcbe-4756-923e-4427eb8ef4c4 dhcp configuration
2022-02-23 11:01:58.918 394035 INFO neutron.agent.dhcp.agent [-] Starting network 904d5803-9db0-4100-bd39-03d31abfd15a dhcp configuration
2022-02-23 11:01:59.170 394035 INFO neutron.agent.dhcp.agent [-] Finished network 7f5dde8a-cc61-470d-a6a0-cbc9e154f5a9 dhcp configuration
2022-02-23 11:01:59.171 394035 INFO neutron.agent.dhcp.agent [-] Starting network 978f16fd-e3c3-451a-af8c-04f25b77521a dhcp configuration
2022-02-23 11:02:01.195 394035 INFO neutron.agent.dhcp.agent [-] Finished network 81a5a6a0-e0c5-45be-97d1-0a08619fe4d8 dhcp configuration
2022-02-23 11:02:01.196 394035 INFO neutron.agent.dhcp.agent [-] Starting network 9e24d7cc-8817-44fb-b644-05ab01ee5e28 dhcp configuration
2022-02-23 11:02:01.494 394035 INFO neutron.agent.dhcp.agent [-] Finished network 84fb6b7e-43a5-4447-8db1-b0bd3ccbffaa dhcp configuration
2022-02-23 11:02:01.494 394035 INFO neutron.agent.dhcp.agent [-] Starting network ba52f609-cd23-45ff-a244-b212ea3719b5 dhcp configuration
2022-02-23 11:02:01.773 394035 INFO neutron.agent.dhcp.agent [-] Finished network 904d5803-9db0-4100-bd39-03d31abfd15a dhcp configuration
2022-02-23 11:02:01.774 394035 INFO neutron.agent.dhcp.agent [-] Starting network cf1815e4-9e79-401e-bc25-5590f4efdc2e dhcp configuration
2022-02-23 11:02:01.774 394035 INFO neutron.agent.dhcp.agent [-] Finished network cf1815e4-9e79-401e-bc25-5590f4efdc2e dhcp configuration
2022-02-23 11:02:01.775 394035 INFO neutron.agent.dhcp.agent [-] Starting network d55ee095-9297-4fe3-887d-4005834bd18c dhcp configuration
2022-02-23 11:02:02.067 394035 INFO neutron.agent.dhcp.agent [-] Finished network 978f16fd-e3c3-451a-af8c-04f25b77521a dhcp configuration
2022-02-23 11:02:02.068 394035 INFO neutron.agent.dhcp.agent [-] Starting network dcfa34d6-8abb-40a2-af4e-1ca3a93318f1 dhcp configuration
2022-02-23 11:02:02.068 394035 INFO neutron.agent.dhcp.agent [-] Finished network dcfa34d6-8abb-40a2-af4e-1ca3a93318f1 dhcp configuration
2022-02-23 11:02:02.069 394035 INFO neutron.agent.dhcp.agent [-] Starting network efdf555d-95ca-4b02-8869-dab15b982fc0 dhcp configuration
2022-02-23 11:02:04.066 394035 INFO neutron.agent.dhcp.agent [-] Finished network 9e24d7cc-8817-44fb-b644-05ab01ee5e28 dhcp configuration
2022-02-23 11:02:04.378 394035 INFO neutron.agent.dhcp.agent [-] Finished network ba52f609-cd23-45ff-a244-b212ea3719b5 dhcp configuration
2022-02-23 11:02:04.664 394035 INFO neutron.agent.dhcp.agent [-] Finished network d55ee095-9297-4fe3-887d-4005834bd18c dhcp configuration
2022-02-23 11:02:04.961 394035 INFO neutron.agent.dhcp.agent [-] Finished network efdf555d-95ca-4b02-8869-dab15b982fc0 dhcp configuration
2022-02-23 11:02:04.961 394035 INFO neutron.agent.dhcp.agent [req-89b7b919-62b7-4e52-8e4a-fdd0d5581531 - - - - -] Synchronizing state complete
2022-02-23 11:03:25.200 394035 INFO neutron.agent.dhcp.agent [req-046c6f39-1499-4a62-97fc-6ff4e970d243 - - - - -] Agent has just been revived. Scheduling full sync
2022-02-23 11:03:34.965 394035 INFO neutron.agent.dhcp.agent [req-89b7b919-62b7-4e52-8e4a-fdd0d5581531 - - - - -] Synchronizing state
2022-02-23 11:03:36.310 394035 INFO neutron.agent.dhcp.agent [req-9d7ba5c8-1e10-4d44-a8ac-01950e044a8a - - - - -] All active networks have been fetched through RPC.
2022-02-23 11:03:36.313 394035 INFO neutron.agent.dhcp.agent [-] Starting network 149e91fa-e5a5-4de4-b764-090b9f1cf749 dhcp configuration
2022-02-23 11:03:36.313 394035 INFO neutron.agent.dhcp.agent [-] Finished network 149e91fa-e5a5-4de4-b764-090b9f1cf749 dhcp configuration
2022-02-23 11:03:36.314 394035 INFO neutron.agent.dhcp.agent [-] Starting network 164ec21d-79bc-426f-97cb-79af47d59fc5 dhcp configuration
2022-02-23 11:03:36.316 394035 INFO neutron.agent.dhcp.agent [-] Starting network 1e8faef6-9885-4b3a-be72-e0fd96567da6 dhcp configuration
2022-02-23 11:03:36.317 394035 INFO neutron.agent.dhcp.agent [-] Starting network 20740e20-4d45-44d2-88a7-2ba19e64b948 dhcp configuration
2022-02-23 11:03:36.318 394035 INFO neutron.agent.dhcp.agent [-] Starting network 24a67298-1100-4e5a-a2e6-21ab6bc38145 dhcp configuration
2022-02-23 11:03:38.124 394035 INFO neutron.agent.dhcp.agent [-] Finished network 164ec21d-79bc-426f-97cb-79af47d59fc5 dhcp configuration
2022-02-23 11:03:38.125 394035 INFO neutron.agent.dhcp.agent [-] Starting network 2eb730cb-cd43-4cbc-af51-e17c553ba6ec dhcp configuration
2022-02-23 11:03:38.126 394035 INFO neutron.agent.dhcp.agent [-] Finished network 2eb730cb-cd43-4cbc-af51-e17c553ba6ec dhcp configuration
2022-02-23 11:03:38.126 394035 INFO neutron.agent.dhcp.agent [-] Starting network 40c0d2cb-cded-4647-97ad-a30a1afc8027 dhcp configuration
2022-02-23 11:03:38.424 394035 INFO neutron.agent.dhcp.agent [-] Finished network 1e8faef6-9885-4b3a-be72-e0fd96567da6 dhcp configuration
2022-02-23 11:03:38.425 394035 INFO neutron.agent.dhcp.agent [-] Starting network 5929825f-5304-4931-a74c-dec85fcd7ad4 dhcp configuration
2022-02-23 11:03:38.425 394035 INFO neutron.agent.dhcp.agent [-] Finished network 5929825f-5304-4931-a74c-dec85fcd7ad4 dhcp configuration
2022-02-23 11:03:38.426 394035 INFO neutron.agent.dhcp.agent [-] Starting network 5db0c874-d19f-440d-8844-de079262d615 dhcp configuration
2022-02-23 11:03:38.701 394035 INFO neutron.agent.dhcp.agent [-] Finished network 20740e20-4d45-44d2-88a7-2ba19e64b948 dhcp configuration
2022-02-23 11:03:38.712 394035 INFO neutron.agent.dhcp.agent [-] Starting network 64216273-c064-4f64-ad60-16881823f4e7 dhcp configuration
2022-02-23 11:03:38.721 394035 INFO neutron.agent.dhcp.agent [-] Finished network 64216273-c064-4f64-ad60-16881823f4e7 dhcp configuration
2022-02-23 11:03:38.723 394035 INFO neutron.agent.dhcp.agent [-] Starting network 6fe9d8fd-b5e5-42e0-b71f-efe7b46efc2e dhcp configuration
2022-02-23 11:03:38.723 394035 INFO neutron.agent.dhcp.agent [-] Finished network 6fe9d8fd-b5e5-42e0-b71f-efe7b46efc2e dhcp configuration
2022-02-23 11:03:38.724 394035 INFO neutron.agent.dhcp.agent [-] Starting network 760ede08-bcbe-4756-923e-4427eb8ef4c4 dhcp configuration
2022-02-23 11:03:39.008 394035 INFO neutron.agent.dhcp.agent [-] Finished network 24a67298-1100-4e5a-a2e6-21ab6bc38145 dhcp configuration
2022-02-23 11:03:39.009 394035 INFO neutron.agent.dhcp.agent [-] Starting network 7f5dde8a-cc61-470d-a6a0-cbc9e154f5a9 dhcp configuration
2022-02-23 11:03:41.037 394035 INFO neutron.agent.dhcp.agent [-] Finished network 40c0d2cb-cded-4647-97ad-a30a1afc8027 dhcp configuration
2022-02-23 11:03:41.038 394035 INFO neutron.agent.dhcp.agent [-] Starting network 81a5a6a0-e0c5-45be-97d1-0a08619fe4d8 dhcp configuration
2022-02-23 11:03:41.337 394035 INFO neutron.agent.dhcp.agent [-] Finished network 5db0c874-d19f-440d-8844-de079262d615 dhcp configuration
2022-02-23 11:03:41.338 394035 INFO neutron.agent.dhcp.agent [-] Starting network 84fb6b7e-43a5-4447-8db1-b0bd3ccbffaa dhcp configuration
2022-02-23 11:03:41.656 394035 INFO neutron.agent.dhcp.agent [-] Finished network 760ede08-bcbe-4756-923e-4427eb8ef4c4 dhcp configuration
2022-02-23 11:03:41.657 394035 INFO neutron.agent.dhcp.agent [-] Starting network 904d5803-9db0-4100-bd39-03d31abfd15a dhcp configuration
2022-02-23 11:03:41.927 394035 INFO neutron.agent.dhcp.agent [-] Finished network 7f5dde8a-cc61-470d-a6a0-cbc9e154f5a9 dhcp configuration
2022-02-23 11:03:41.928 394035 INFO neutron.agent.dhcp.agent [-] Starting network 978f16fd-e3c3-451a-af8c-04f25b77521a dhcp configuration
2022-02-23 11:03:43.846 394035 INFO neutron.agent.dhcp.agent [-] Finished network 81a5a6a0-e0c5-45be-97d1-0a08619fe4d8 dhcp configuration
2022-02-23 11:03:43.847 394035 INFO neutron.agent.dhcp.agent [-] Starting network 9e24d7cc-8817-44fb-b644-05ab01ee5e28 dhcp configuration
2022-02-23 11:03:44.147 394035 INFO neutron.agent.dhcp.agent [-] Finished network 84fb6b7e-43a5-4447-8db1-b0bd3ccbffaa dhcp configuration
2022-02-23 11:03:44.148 394035 INFO neutron.agent.dhcp.agent [-] Starting network ba52f609-cd23-45ff-a244-b212ea3719b5 dhcp configuration
2022-02-23 11:03:44.441 394035 INFO neutron.agent.dhcp.agent [-] Finished network 904d5803-9db0-4100-bd39-03d31abfd15a dhcp configuration
2022-02-23 11:03:44.442 394035 INFO neutron.agent.dhcp.agent [-] Starting network cf1815e4-9e79-401e-bc25-5590f4efdc2e dhcp configuration
2022-02-23 11:03:44.442 394035 INFO neutron.agent.dhcp.agent [-] Finished network cf1815e4-9e79-401e-bc25-5590f4efdc2e dhcp configuration
2022-02-23 11:03:44.442 394035 INFO neutron.agent.dhcp.agent [-] Starting network d55ee095-9297-4fe3-887d-4005834bd18c dhcp configuration
2022-02-23 11:03:44.717 394035 INFO neutron.agent.dhcp.agent [-] Finished network 978f16fd-e3c3-451a-af8c-04f25b77521a dhcp configuration
2022-02-23 11:03:44.718 394035 INFO neutron.agent.dhcp.agent [-] Starting network dcfa34d6-8abb-40a2-af4e-1ca3a93318f1 dhcp configuration
2022-02-23 11:03:44.719 394035 INFO neutron.agent.dhcp.agent [-] Finished network dcfa34d6-8abb-40a2-af4e-1ca3a93318f1 dhcp configuration
2022-02-23 11:03:44.719 394035 INFO neutron.agent.dhcp.agent [-] Starting network efdf555d-95ca-4b02-8869-dab15b982fc0 dhcp configuration
2022-02-23 11:03:46.605 394035 INFO neutron.agent.dhcp.agent [-] Finished network 9e24d7cc-8817-44fb-b644-05ab01ee5e28 dhcp configuration
2022-02-23 11:03:46.870 394035 INFO neutron.agent.dhcp.agent [-] Finished network ba52f609-cd23-45ff-a244-b212ea3719b5 dhcp configuration
2022-02-23 11:03:47.148 394035 INFO neutron.agent.dhcp.agent [-] Finished network d55ee095-9297-4fe3-887d-4005834bd18c dhcp configuration
2022-02-23 11:03:47.422 394035 INFO neutron.agent.dhcp.agent [-] Finished network efdf555d-95ca-4b02-8869-dab15b982fc0 dhcp configuration
2022-02-23 11:03:47.423 394035 INFO neutron.agent.dhcp.agent [req-9d7ba5c8-1e10-4d44-a8ac-01950e044a8a - - - - -] Synchronizing state complete
2022-02-23 11:05:25.196 394035 INFO neutron.agent.dhcp.agent [req-4c801faa-a4e5-42a3-9f08-3f5500b66e2c - - - - -] Agent has just been revived. Scheduling full sync
2022-02-23 11:05:47.427 394035 INFO neutron.agent.dhcp.agent [req-9d7ba5c8-1e10-4d44-a8ac-01950e044a8a - - - - -] Synchronizing state
2022-02-23 11:05:49.004 394035 INFO neutron.agent.dhcp.agent [req-ec18b47f-5c27-4cf4-a36e-861c00f0e559 - - - - -] All active networks have been fetched through RPC.

• dnsmasq-qdhcp sidercar containers keeps getting restarted:

$ docker ps | grep dnsmasq-qdhcp
f6236669864e        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-efdf555d-95ca-4b02-8869-dab15b982fc0
a1dd2b4f5cec        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-d55ee095-9297-4fe3-887d-4005834bd18c
1bbc8dd20e0a        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-ba52f609-cd23-45ff-a244-b212ea3719b5
ceca8854effa        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-9e24d7cc-8817-44fb-b644-05ab01ee5e28
cd615e0a4485        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-978f16fd-e3c3-451a-af8c-04f25b77521a
871e5b7ffac6        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-904d5803-9db0-4100-bd39-03d31abfd15a
b3a87319b478        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-84fb6b7e-43a5-4447-8db1-b0bd3ccbffaa
7bd1b7b14b80        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-81a5a6a0-e0c5-45be-97d1-0a08619fe4d8
17fc09047b59        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-7f5dde8a-cc61-470d-a6a0-cbc9e154f5a9
8d67a0bbd09f        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-760ede08-bcbe-4756-923e-4427eb8ef4c4
efc9bf68c1a4        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-5db0c874-d19f-440d-8844-de079262d615
11d891070c3e        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-40c0d2cb-cded-4647-97ad-a30a1afc8027
4d330ef25177        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-24a67298-1100-4e5a-a2e6-21ab6bc38145
bed713afff19        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-20740e20-4d45-44d2-88a7-2ba19e64b948
d49bb5c94c84        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-1e8faef6-9885-4b3a-be72-e0fd96567da6
673d87e45f91        registry:5000/containers-neutron-dhcp-agent:13.0-91          "ip netns exec qdh..."   About a minute ago   Up About a minute                          neutron-dnsmasq-qdhcp-164ec21d-79bc-426f-97cb-79af47d59fc5