Issues with floating IP creation and with NAT rules of metadata agent in Red Hat OpenStack Platform
Issue
Issues with floating IP creation and with NAT rules of metadata agent in Red Hat OpenStack Platform
The L3 agent shows various error messages, such as:
2017-10-31 03:12:26.877 22145 INFO neutron.agent.linux.interface [-] Device qg-d11e315e-8d already exists
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 80b26ddd-ac4d-4e16-b1c1-a40b09d868f6
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 523, in _process_router_update
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 462, in _process_router_if_compatible
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent self._process_updated_router(router)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 476, in _process_updated_router
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent ri.process(self)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 425, in process
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent self.enable_keepalived()
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 154, in enable_keepalived
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent self.keepalived_manager.spawn()
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 237, in wrapper
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent return f(*args, **kwargs)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/keepalived.py", line 409, in spawn
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent keepalived_pm.enable(reload_cfg=True)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/external_process.py", line 96, in enable
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent self.reload_cfg()
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/external_process.py", line 102, in reload_cfg
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent self.disable('HUP')
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/external_process.py", line 114, in disable
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent utils.execute(cmd, run_as_root=True)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 122, in execute
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent execute_rootwrap_daemon(cmd, process_input, addl_env))
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 108, in execute_rootwrap_daemon
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent return client.execute(cmd, process_input)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_rootwrap/client.py", line 128, in execute
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent res = proxy.run_one_command(cmd, stdin)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent File "<string>", line 2, in run_one_command
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent raise convert_to_error(kind, result)
2017-10-31 03:12:26.889 22145 ERROR neutron.agent.l3.agent NoFilterMatched
2017-10-31 16:28:49.223 16441 ERROR neutron.agent.linux.utils [-] Exit code: 4; Stdin: # Generated by iptables_manager
*filter
-D neutron-l3-agent-scope 1
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*mangle
-D neutron-l3-agent-scope 1
-D neutron-l3-agent-scope 1
COMMIT
# Completed by iptables_manager
; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager [-] IPTablesManager.apply failed to apply the following set of iptables rules:
1. # Generated by iptables_manager
2. *filter
3. -D neutron-l3-agent-scope 1
4. COMMIT
5. # Completed by iptables_manager
6. # Generated by iptables_manager
7. *mangle
8. -D neutron-l3-agent-scope 1
9. -D neutron-l3-agent-scope 1
10. COMMIT
11. # Completed by iptables_manager
12.
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager [-] Failure applying iptables rules
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager Traceback (most recent call last):
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 457, in defer_apply
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager self.defer_apply_off()
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 468, in defer_apply_off
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager self._apply()
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 482, in _apply
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager return self._apply_synchronized()
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 559, in _apply_synchronized
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager '\n'.join(log_lines))
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager self.force_reraise()
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager six.reraise(self.type_, self.value, self.tb)
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 538, in _apply_synchronized
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager run_as_root=True)
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 146, in execute
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager raise ProcessExecutionError(msg, returncode=returncode)
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager ProcessExecutionError: Exit code: 4; Stdin: # Generated by iptables_manager
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager *filter
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager -D neutron-l3-agent-scope 1
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager COMMIT
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager # Completed by iptables_manager
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager # Generated by iptables_manager
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager *mangle
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager -D neutron-l3-agent-scope 1
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager -D neutron-l3-agent-scope 1
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager COMMIT
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager # Completed by iptables_manager
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager ; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager
2017-10-31 16:28:49.224 16441 ERROR neutron.agent.linux.iptables_manager
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info [-] Failed to process floating IPs.
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 832, in process_external
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info self.process_snat_dnat_for_fip()
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info self.gen.next()
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py", line 461, in defer_apply
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info raise n_exc.IpTablesApplyException(msg)
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info IpTablesApplyException: Failure applying iptables rules
2017-10-31 16:28:49.225 16441 ERROR neutron.agent.l3.router_info
2017-10-31 16:28:49.227 16441 ERROR neutron.agent.linux.utils [-] Exit code: 4; Stdin: # Generated by iptables_manager
*filter
-D neutron-l3-agent-scope 1
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*mangle
-D neutron-l3-agent-scope 1
-D neutron-l3-agent-scope 1
COMMIT
# Completed by iptables_manager
; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
2017-10-31 16:28:49.227 16441 ERROR neutron.agent.linux.utils [-] Exit code: 4; Stdin: # Generated by iptables_manager
*filter
-D neutron-l3-agent-scope 1
COMMIT
# Completed by iptables_manager
# Generated by iptables_manager
*mangle
-D neutron-l3-agent-scope 1
-D neutron-l3-agent-scope 1
COMMIT
# Completed by iptables_manager
; Stdout: ; Stderr: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
2017-10-31 16:28:49.228 16441 ERROR neutron.agent.linux.iptables_manager [-] IPTablesManager.apply failed to apply the following set of iptables rules:
1. # Generated by iptables_manager
2. *filter
3. -D neutron-l3-agent-scope 1
4. COMMIT
5. # Completed by iptables_manager
6. # Generated by iptables_manager
7. *mangle
8. -D neutron-l3-agent-scope 1
9. -D neutron-l3-agent-scope 1
10. COMMIT
11. # Completed by iptables_manager
12.
2017-10-31 16:28:49.228 16441 ERROR neutron.agent.linux.iptables_manager [-] Failure applying iptables rules
(...)
Another symptom of this same issue is that newly created instances cannot be accessed via their associated floating IP addresses.
Specifically, it looks like the floating IP associated with newly spawned guest instances are not being added to the qg
interface within the qrouter
namespace on the controller node with an active routing role.
Environment
Red Hat OpenStack Platform 9
Red Hat OpenStack Platform 10
Red Hat OpenStack Platform 11
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.