Issues with floating IP creation and with NAT rules of metadata agent in Red Hat OpenStack Platform

Issue

Issues with floating IP creation and with NAT rules of metadata agent in Red Hat OpenStack Platform

The L3 agent shows various error messages, such as:

2017-10-31 03:12:26.877 22145 INFO neutron.agent.linux.interface [-] Device qg-d11e315e-8d already exists
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 80b26ddd-ac4d-4e16-b1c1-a40b09d868f6
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 523, in _process_router_update
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 462, in _process_router_if_compatible
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 476, in _process_updated_router
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent     ri.process(self)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 425, in process
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent     self.enable_keepalived()
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 154, in enable_keepalived
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent     self.keepalived_manager.spawn()
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 237, in wrapper
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent     return f(*args, **kwargs)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/keepalived.py", line 409, in spawn
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent     keepalived_pm.enable(reload_cfg=True)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/external_process.py", line 96, in enable
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent     self.reload_cfg()
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/external_process.py", line 102, in reload_cfg
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent     self.disable('HUP')
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/external_process.py", line 114, in disable
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent     utils.execute(cmd, run_as_root=True)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 122, in execute
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent     execute_rootwrap_daemon(cmd, process_input, addl_env))
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 108, in execute_rootwrap_daemon
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent     return client.execute(cmd, process_input)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_rootwrap/client.py", line 128, in execute
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent     res = proxy.run_one_command(cmd, stdin)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent   File "<string>", line 2, in run_one_command
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent   File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent     raise convert_to_error(kind, result)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent NoFilterMatched

2017-10-31 16:28:49.223 16441 ERROR neutron.agent.linux.utils [-] Exit code: 4; Stdin: # Generated by iptables_manager
*filter
-D neutron-l3-agent-scope 1
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*mangle
-D neutron-l3-agent-scope 1
-D neutron-l3-agent-scope 1
COMMIT
# Completed by iptables_manager
; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?

2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager [-] IPTablesManager.apply failed to apply the following set of iptables rules:
      1. # Generated by iptables_manager
      2. *filter
      3. -D neutron-l3-agent-scope 1
      4. COMMIT
      5. # Completed by iptables_manager
      6. # Generated by iptables_manager
      7. *mangle
      8. -D neutron-l3-agent-scope 1
      9. -D neutron-l3-agent-scope 1
     10. COMMIT
     11. # Completed by iptables_manager
     12. 
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager [-] Failure applying iptables rules
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager Traceback (most recent call last):
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 457, in defer_apply
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager     self.defer_apply_off()
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 468, in defer_apply_off
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager     self._apply()
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 482, in _apply
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager     return self._apply_synchronized()
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 559, in _apply_synchronized
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager     '\n'.join(log_lines))
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager     self.force_reraise()
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager     six.reraise(self.type_, self.value, self.tb)
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 538, in _apply_synchronized
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager     run_as_root=True)
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 146, in execute
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager     raise ProcessExecutionError(msg, returncode=returncode)
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager ProcessExecutionError: Exit code: 4; Stdin: # Generated by iptables_manager
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager *filter
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager -D neutron-l3-agent-scope 1
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager COMMIT
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager # Completed by iptables_manager
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager # Generated by iptables_manager
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager *mangle
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager -D neutron-l3-agent-scope 1
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager -D neutron-l3-agent-scope 1
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager COMMIT
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager # Completed by iptables_manager
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager ; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager 
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager 
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info [-] Failed to process floating IPs.
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 832, in process_external
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info     self.process_snat_dnat_for_fip()
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info     self.gen.next()
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 461, in defer_apply
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info     raise n_exc.IpTablesApplyException(msg)
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info IpTablesApplyException: Failure applying iptables rules
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info 
2017-10-31 16:28:49.227 16441 ERROR neutron.agent.linux.utils [-] Exit code: 4; Stdin: # Generated by iptables_manager
*filter
-D neutron-l3-agent-scope 1
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*mangle
-D neutron-l3-agent-scope 1
-D neutron-l3-agent-scope 1
COMMIT
# Completed by iptables_manager
; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?

2017-10-31 16:28:49.227 16441 ERROR neutron.agent.linux.utils [-] Exit code: 4; Stdin: # Generated by iptables_manager
*filter
-D neutron-l3-agent-scope 1
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*mangle
-D neutron-l3-agent-scope 1
-D neutron-l3-agent-scope 1
COMMIT
# Completed by iptables_manager
; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?

2017-10-31 16:28:49.228 16441 ERROR neutron.agent.linux.iptables_manager [-] IPTablesManager.apply failed to apply the following set of iptables rules:
      1. # Generated by iptables_manager
      2. *filter
      3. -D neutron-l3-agent-scope 1
      4. COMMIT
      5. # Completed by iptables_manager
      6. # Generated by iptables_manager
      7. *mangle
      8. -D neutron-l3-agent-scope 1
      9. -D neutron-l3-agent-scope 1
     10. COMMIT
     11. # Completed by iptables_manager
     12. 
2017-10-31 16:28:49.228 16441 ERROR neutron.agent.linux.iptables_manager [-] Failure applying iptables rules
(...)

Another symptom of this same issue is that newly created instances cannot be accessed via their associated floating IP addresses.

Specifically, it looks like the floating IP associated with newly spawned guest instances are not being added to the qg interface within the qrouter namespace on the controller node with an active routing role.