Floating IP not working for RHOSP 11

Comments

RHOSP 11 Build with Ceph Integration.
We have three controllers and two hypercoverged Compute.

I was able to ssh to the instance using qrouter inside controller, but not able to reach the instance from outside through floating IP

Inside Controller
sudo ip netns exec qrouter-bb4d96e5-07e1-4ad6-b120-f11c6a2298eb ssh root@172.30.10.178

VLAN

172.26.11.0/24 - Provision Network ( VLAN2611 )
172.26.12.0/24 - Internal Network ( VLAN2612 )
172.26.13.0/24 - Tentant Network ( VLAN2613 )
172.26.14.0/24 - Storage Network ( VLAN2614 )
172.26.16.0/24 - Storage Managment ( VLAN2616 )
172.26.17.0/24 - Management Network ( VLAN2617 )
172.30.10.0/23 - Externak Network ( VLAN3010 )

After Loggin into the server

[stack@director ~]$ ssh heat-admin@172.26.11.21
Last login: Fri Feb 2 13:00:30 2018 from 172.26.11.254
[heat-admin@overcloud-controller-0 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether e0:db:55:24:fb:54 brd ff:ff:ff:ff:ff:ff
inet 172.26.11.21/24 brd 172.26.11.255 scope global em1
valid_lft forever preferred_lft forever
inet6 fe80::e2db:55ff:fe24:fb54/64 scope link
valid_lft forever preferred_lft forever
3: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP qlen 1000
link/ether e0:db:55:24:fb:55 brd ff:ff:ff:ff:ff:ff
inet6 fe80::e2db:55ff:fe24:fb55/64 scope link
valid_lft forever preferred_lft forever
4: em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP qlen 1000
link/ether e0:db:55:24:fb:56 brd ff:ff:ff:ff:ff:ff
inet6 fe80::e2db:55ff:fe24:fb56/64 scope link
valid_lft forever preferred_lft forever
5: em4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether e0:db:55:24:fb:57 brd ff:ff:ff:ff:ff:ff
inet 172.26.11.154/24 brd 172.26.11.255 scope global dynamic em4
valid_lft 115sec preferred_lft 115sec
inet6 fe80::e2db:55ff:fe24:fb57/64 scope link
valid_lft forever preferred_lft forever
6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether d2:61:02:d7:53:97 brd ff:ff:ff:ff:ff:ff
7: gre0@NONE: mtu 1476 qdisc noop state DOWN qlen 1
link/gre 0.0.0.0 brd 0.0.0.0
8: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
9: gre_sys@NONE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65490 qdisc pfifo_fast master ovs-system state UNKNOWN qlen 1000
link/ether a6:64:de:6b:07:e0 brd ff:ff:ff:ff:ff:ff
inet6 fe80::a464:deff:fe6b:7e0/64 scope link
valid_lft forever preferred_lft forever
10: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 82:8c:77:14:69:45 brd ff:ff:ff:ff:ff:ff
11: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue master ovs-system state UNKNOWN qlen 1000
link/ether d6:d6:07:62:99:ce brd ff:ff:ff:ff:ff:ff
12: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether ca:d6:f7:93:f7:44 brd ff:ff:ff:ff:ff:ff
13: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether e0:db:55:24:fb:55 brd ff:ff:ff:ff:ff:ff
inet6 fe80::9cba:eaff:fe6c:249/64 scope link
valid_lft forever preferred_lft forever
14: vlan2612: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether 72:5d:8d:0e:71:5b brd ff:ff:ff:ff:ff:ff
inet 172.26.12.14/24 brd 172.26.12.255 scope global vlan2612
valid_lft forever preferred_lft forever
inet6 fe80::705d:8dff:fe0e:715b/64 scope link
valid_lft forever preferred_lft forever
15: vlan2613: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether 56:a1:09:58:c9:b9 brd ff:ff:ff:ff:ff:ff
inet 172.26.13.14/24 brd 172.26.13.255 scope global vlan2613
valid_lft forever preferred_lft forever
inet6 fe80::54a1:9ff:fe58:c9b9/64 scope link
valid_lft forever preferred_lft forever
16: vlan2614: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether c2:d5:8e:c8:6c:ad brd ff:ff:ff:ff:ff:ff
inet 172.26.14.24/24 brd 172.26.14.255 scope global vlan2614
valid_lft forever preferred_lft forever
inet6 fe80::c0d5:8eff:fec8:6cad/64 scope link
valid_lft forever preferred_lft forever
17: vlan2616: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether 46:3d:55:08:09:a6 brd ff:ff:ff:ff:ff:ff
inet 172.26.16.15/24 brd 172.26.16.255 scope global vlan2616
valid_lft forever preferred_lft forever
inet6 fe80::443d:55ff:fe08:9a6/64 scope link
valid_lft forever preferred_lft forever
18: vlan2617: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether 3e:de:14:6d:2c:a9 brd ff:ff:ff:ff:ff:ff
inet 172.26.17.17/24 brd 172.26.17.255 scope global vlan2617
valid_lft forever preferred_lft forever
inet6 fe80::3cde:14ff:fe6d:2ca9/64 scope link
valid_lft forever preferred_lft forever
19: vlan3010: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether 62:17:61:6a:59:27 brd ff:ff:ff:ff:ff:ff
inet 172.30.10.14/24 brd 172.30.10.255 scope global vlan3010
valid_lft forever preferred_lft forever
inet6 fe80::6017:61ff:fe6a:5927/64 scope link
valid_lft forever preferred_lft forever

OVS Bridge for Controller 1:

[heat-admin@overcloud-controller-0 ~]$ sudo ovs-vsctl show
0604fcf8-a350-4469-92a6-89fe4ccae061
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge br-tun
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "vxlan-ac1a0d14"
Interface "vxlan-ac1a0d14"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="172.26.13.14", out_key=flow, remote_ip="172.26.13.20"}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "vxlan-ac1a0d0c"
Interface "vxlan-ac1a0d0c"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="172.26.13.14", out_key=flow, remote_ip="172.26.13.12"}
Port "gre-ac1a0d14"
Interface "gre-ac1a0d14"
type: gre
options: {df_default="true", in_key=flow, local_ip="172.26.13.14", out_key=flow, remote_ip="172.26.13.20"}
Port "gre-ac1a0d0c"
Interface "gre-ac1a0d0c"
type: gre
options: {df_default="true", in_key=flow, local_ip="172.26.13.14", out_key=flow, remote_ip="172.26.13.12"}
Port "gre-ac1a0d0f"
Interface "gre-ac1a0d0f"
type: gre
options: {df_default="true", in_key=flow, local_ip="172.26.13.14", out_key=flow, remote_ip="172.26.13.15"}
Port br-tun
Interface br-tun
type: internal
Port "vxlan-ac1a0d11"
Interface "vxlan-ac1a0d11"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="172.26.13.14", out_key=flow, remote_ip="172.26.13.17"}
Port "vxlan-ac1a0d0f"
Interface "vxlan-ac1a0d0f"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="172.26.13.14", out_key=flow, remote_ip="172.26.13.15"}
Port "gre-ac1a0d11"
Interface "gre-ac1a0d11"
type: gre
options: {df_default="true", in_key=flow, local_ip="172.26.13.14", out_key=flow, remote_ip="172.26.13.17"}
Bridge br-ex
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
Port "vlan2613"
tag: 2613
Interface "vlan2613"
type: internal
Port "vlan2617"
tag: 2617
Interface "vlan2617"
type: internal
Port "vlan3010"
tag: 3010
Interface "vlan3010"
type: internal
Port "vlan2614"
tag: 2614
Interface "vlan2614"
type: internal
Port "bond1"
Interface "em2"
Interface "em3"
Port br-ex
Interface br-ex
type: internal
Port "vlan2612"
tag: 2612
Interface "vlan2612"
type: internal
Port "vlan2616"
tag: 2616
Interface "vlan2616"
type: internal
Port "qg-e8f74c7c-58"
Interface "qg-e8f74c7c-58"
type: internal
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "ha-4ad3b415-1b"
tag: 2
Interface "ha-4ad3b415-1b"
type: internal
Port "tap7a6bb2e3-17"
tag: 1
Interface "tap7a6bb2e3-17"
type: internal
Port int-br-ex
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
Port br-int
Interface br-int
type: internal
Port "qr-6a11beee-45"
tag: 1
Interface "qr-6a11beee-45"
type: internal
ovs_version: "2.6.1"

Neutron Agent List

[heat-admin@overcloud-controller-0 ~]$ neutron agent-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.

One of the L3 Agent:

[heat-admin@overcloud-controller-0 ~]$ neutron agent-show 901c0300-5081-412d-a7e8-2e77acc098bf
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+---------------------+-------------------------------------------------------------------------------+
| Field | Value |
+---------------------+-------------------------------------------------------------------------------+
| admin_state_up | True |
| agent_type | L3 agent |
| alive | True |
| availability_zone | nova |
| binary | neutron-l3-agent |
| configurations | { |
| | "agent_mode": "legacy", |
| | "gateway_external_network_id": "", |
| | "handle_internal_only_routers": true, |
| | "routers": 1, |
| | "interfaces": 1, |
| | "floating_ips": 1, |
| | "interface_driver": "neutron.agent.linux.interface.OVSInterfaceDriver", |
| | "log_agent_heartbeats": false, |
| | "external_network_bridge": "br-ex", |
| | "ex_gw_ports": 1 |
| | } |
| created_at | 2018-02-01 06:54:56 |
| description | |
| heartbeat_timestamp | 2018-02-02 13:25:52 |
| host | overcloud-controller-2.localdomain |
| id | 901c0300-5081-412d-a7e8-2e77acc098bf |
| started_at | 2018-02-02 11:02:27 |
| topic | l3_agent |
+---------------------+-------------------------------------------------------------------------------+

Neutron Router and DHCP.

Neutron Virtual DHCP agent is available is used to ping to the SDN router gateway

[heat-admin@overcloud-controller-0 ~]$ ip netns
qrouter-bb4d96e5-07e1-4ad6-b120-f11c6a2298eb
qdhcp-2cee840e-f683-48ed-a05f-ac993f6cac10

Router Gateway using QDHCP

[heat-admin@overcloud-controller-0 ~]$ sudo ip netns exec qdhcp-2cee840e-f683-48ed-a05f-ac993f6cac10 ping 172.30.10.173
PING 172.30.10.173 (172.30.10.173) 56(84) bytes of data.
64 bytes from 172.30.10.173: icmp_seq=1 ttl=64 time=1.16 ms
64 bytes from 172.30.10.173: icmp_seq=2 ttl=64 time=0.090 ms
64 bytes from 172.30.10.173: icmp_seq=3 ttl=64 time=0.092 ms
^Z
[1]+ Stopped sudo ip netns exec qdhcp-2cee840e-f683-48ed-a05f-ac993f6cac10 ping 172.30.10.173

Floating IP of a Instance using QDHCP

[heat-admin@overcloud-controller-0 ~]$ sudo ip netns exec qdhcp-2cee840e-f683-48ed-a05f-ac993f6cac10 ping 172.30.10.178
PING 172.30.10.178 (172.30.10.178) 56(84) bytes of data.
From 172.30.10.178 icmp_seq=1 Destination Host Unreachable
From 172.30.10.178 icmp_seq=2 Destination Host Unreachable
From 172.30.10.178 icmp_seq=3 Destination Host Unreachable
From 172.30.10.178 icmp_seq=4 Destination Host Unreachable
^C
--- 172.30.10.178 ping statistics ---
6 packets transmitted, 0 received, +4 errors, 100% packet loss, time 5000ms
pipe 4

Router Gateway using QROUTER

[heat-admin@overcloud-controller-0 ~]$ sudo ip netns exec qrouter-bb4d96e5-07e1-4ad6-b120-f11c6a2298eb ping 172.30.10.173
PING 172.30.10.173 (172.30.10.173) 56(84) bytes of data.
64 bytes from 172.30.10.173: icmp_seq=1 ttl=64 time=0.115 ms
64 bytes from 172.30.10.173: icmp_seq=2 ttl=64 time=0.061 ms
64 bytes from 172.30.10.173: icmp_seq=3 ttl=64 time=0.063 ms
64 bytes from 172.30.10.173: icmp_seq=4 ttl=64 time=0.056 ms
^Z
[5]+ Stopped sudo ip netns exec qrouter-bb4d96e5-07e1-4ad6-b120-f11c6a2298eb ping 172.30.10.173

Floating IP of a Instance using QROUTER

[heat-admin@overcloud-controller-0 ~]$ sudo ip netns exec qrouter-bb4d96e5-07e1-4ad6-b120-f11c6a2298eb ping 172.30.10.178
PING 172.30.10.178 (172.30.10.178) 56(84) bytes of data.
From 172.30.10.178 icmp_seq=1 Destination Host Unreachable
From 172.30.10.178 icmp_seq=2 Destination Host Unreachable
From 172.30.10.178 icmp_seq=3 Destination Host Unreachable
From 172.30.10.178 icmp_seq=4 Destination Host Unreachable
^Z
[6]+ Stopped sudo ip netns exec qrouter-bb4d96e5-07e1-4ad6-b120-f11c6a2298eb ping 172.30.10.178

Route of QRouter

[heat-admin@overcloud-controller-0 ~]$ sudo ip netns exec qrouter-bb4d96e5-07e1-4ad6-b120-f11c6a2298eb route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 0 0 0 qg-e8f74c7c-58
30.30.30.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-6a11beee-45
link-local 0.0.0.0 255.255.255.0 U 0 0 0 ha-4ad3b415-1b
169.254.192.0 0.0.0.0 255.255.192.0 U 0 0 0 ha-4ad3b415-1b
172.30.10.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-e8f74c7c-58

[heat-admin@overcloud-controller-0 ~]$ sudo ip netns exec qrouter-bb4d96e5-07e1-4ad6-b120-f11c6a2298eb ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: gre0@NONE: mtu 1476 qdisc noop state DOWN qlen 1
link/gre 0.0.0.0 brd 0.0.0.0
3: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
21: ha-4ad3b415-1b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:08:33:4b brd ff:ff:ff:ff:ff:ff
inet 169.254.192.3/18 brd 169.254.255.255 scope global ha-4ad3b415-1b
valid_lft forever preferred_lft forever
inet 169.254.0.1/24 scope global ha-4ad3b415-1b
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe08:334b/64 scope link
valid_lft forever preferred_lft forever
22: qg-e8f74c7c-58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:90:73:04 brd ff:ff:ff:ff:ff:ff
inet 172.30.10.173/24 scope global qg-e8f74c7c-58
valid_lft forever preferred_lft forever
inet 172.30.10.178/32 scope global qg-e8f74c7c-58
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe90:7304/64 scope link
valid_lft forever preferred_lft forever
23: qr-6a11beee-45: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:cd:08:bf brd ff:ff:ff:ff:ff:ff
inet 30.30.30.254/24 scope global qr-6a11beee-45
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fecd:8bf/64 scope link
valid_lft forever preferred_lft forever

IP Route of QRouter

[heat-admin@overcloud-controller-0 ~]$ sudo ip netns exec qrouter-bb4d96e5-07e1-4ad6-b120-f11c6a2298eb ip route
default via 172.30.10.10 dev qg-e8f74c7c-58
30.30.30.0/24 dev qr-6a11beee-45 proto kernel scope link src 30.30.30.254
169.254.0.0/24 dev ha-4ad3b415-1b proto kernel scope link src 169.254.0.1
169.254.192.0/18 dev ha-4ad3b415-1b proto kernel scope link src 169.254.192.3
172.30.10.0/24 dev qg-e8f74c7c-58 proto kernel scope link src 172.30.10.173

Started 2018-02-02T14:51:17+00:00 by

Sathiyamoorthi Karunakaran

Newbie 10 points

Select Your Language

Responses

Quick Links

Help

Site Info

Related Sites

About

Red Hat legal and privacy links

Red Hat legal and privacy links

Responses

Quick Links

Help

Site Info

Related Sites

Systems Status

About

Red Hat legal and privacy links

Red Hat legal and privacy links