Any vm migrated to a specific compute node loses network connectivity
Issue
-
If any vm comes under this compute node, they are loosing the network connectivity and vm become inaccessible.
-
H/W vendor(CISCO) checked everything and did not find any h/w issue, recommended to verify os /openstack side.
-
The following errors might be seen in
/var/log/containers/nova/nova-compute.log
:
2019-11-24 22:36:54.369 29283 ERROR oslo.messaging._drivers.impl_rabbit [-] [7f36e587-b449-45a8-9ebf-4e3184ff813f] AMQP server on overcloud-controller-1.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: error: [Errno 113] EHOSTUNREACH
2019-11-24 22:36:57.367 29283 ERROR oslo.messaging._drivers.impl_rabbit [req-78f19f0a-5b0a-4d13-8096-14ff2f1c3103 - - - - -] [b09b5ad2-8746-4e79-92de-2c10819c945d] AMQP server on overcloud-controller-2.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: error: [Errno 113] EHOSTUNREACH
2019-11-24 22:36:57.368 29283 ERROR oslo.messaging._drivers.impl_rabbit [-] [7f36e587-b449-45a8-9ebf-4e3184ff813f] AMQP server on overcloud-controller-2.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 32 seconds.: error: [Errno 113] EHOSTUNREACH
2019-11-24 22:37:01.460 29283 ERROR oslo.messaging._drivers.impl_rabbit [req-78f19f0a-5b0a-4d13-8096-14ff2f1c3103 - - - - -] [b09b5ad2-8746-4e79-92de-2c10819c945d] AMQP server on overcloud-controller-1.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: error: [Errno 113] EHOSTUNREACH
2019-11-24 22:37:04.464 29283 ERROR oslo.messaging._drivers.impl_rabbit [req-78f19f0a-5b0a-4d13-8096-14ff2f1c3103 - - - - -] [b09b5ad2-8746-4e79-92de-2c10819c945d] AMQP server on overcloud-controller-0.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 32 seconds.: error: [Errno 113] EHOSTUNREACH
- The hypervisor might show as
DOWN
:
(overcloud) [stack@undercloud ~]$ openstack hypervisor list
+-----+------------------------------------+-----------------+-------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+-----+------------------------------------+-----------------+-------------+-------+
| 2 | overcloud-osd-compute-3.localdomain | QEMU | 10.10.10.16 | up |
| 5 | overcloud-osd-compute-2.localdomain | QEMU | 10.10.10.15 | up |
| 8 | overcloud-compute-28.localdomain | QEMU | 10.10.10.45 | up |
| 11 | overcloud-compute-18.localdomain | QEMU | 10.10.10.35 | up |
| 14 | overcloud-compute-24.localdomain | QEMU | 10.10.10.41 | up |
| 17 | overcloud-compute-9.localdomain | QEMU | 10.10.10.26 | up |
| 20 | overcloud-osd-compute-0.localdomain | QEMU | 10.10.10.13 | up |
| 23 | overcloud-compute-10.localdomain | QEMU | 10.10.10.27 | up |
| 26 | overcloud-compute-6.localdomain | QEMU | 10.10.10.23 | up |
| 29 | overcloud-compute-25.localdomain | QEMU | 10.10.10.42 | up |
| 32 | overcloud-osd-compute-1.localdomain | QEMU | 10.10.10.14 | up |
| 35 | overcloud-compute-29.localdomain | QEMU | 10.10.10.46 | up |
| 38 | overcloud-compute-11.localdomain | QEMU | 10.10.10.28 | up |
| 41 | overcloud-compute-23.localdomain | QEMU | 10.10.10.40 | up |
| 44 | overcloud-compute-19.localdomain | QEMU | 10.10.10.36 | up |
| 47 | overcloud-compute-1.localdomain | QEMU | 10.10.10.18 | up |
| 50 | overcloud-compute-22.localdomain | QEMU | 10.10.10.39 | up |
| 53 | overcloud-compute-13.localdomain | QEMU | 10.10.10.30 | up |
| 56 | overcloud-compute-4.localdomain | QEMU | 10.10.10.21 | down |
| 59 | overcloud-compute-0.localdomain | QEMU | 10.10.10.17 | up |
| 62 | overcloud-compute-3.localdomain | QEMU | 10.10.10.20 | up |
| 65 | overcloud-compute-15.localdomain | QEMU | 10.10.10.32 | up |
| 68 | overcloud-compute-5.localdomain | QEMU | 10.10.10.22 | up |
| 71 | overcloud-compute-16.localdomain | QEMU | 10.10.10.33 | up |
| 74 | overcloud-compute-20.localdomain | QEMU | 10.10.10.37 | up |
| 77 | overcloud-compute-8.localdomain | QEMU | 10.10.10.25 | up |
| 80 | overcloud-compute-17.localdomain | QEMU | 10.10.10.34 | up |
| 83 | overcloud-compute-12.localdomain | QEMU | 10.10.10.29 | up |
| 86 | overcloud-compute-26.localdomain | QEMU | 10.10.10.43 | up |
| 89 | overcloud-compute-21.localdomain | QEMU | 10.10.10.38 | up |
| 92 | overcloud-compute-2.localdomain | QEMU | 10.10.10.19 | up |
| 95 | overcloud-compute-7.localdomain | QEMU | 10.10.10.24 | up |
| 98 | overcloud-compute-14.localdomain | QEMU | 10.10.10.31 | up |
| 101 | overcloud-compute-27.localdomain | QEMU | 10.10.10.44 | up |
+-----+------------------------------------+-----------------+-------------+-------+
Environment
- Red Hat OpenStack Platform 13.0 (RHOSP)
- Red Hat OpenStack Platform 8.0 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.