Any vm migrated to a specific compute node loses network connectivity

Solution In Progress - Updated -

Issue

  • If any vm comes under this compute node, they are loosing the network connectivity and vm become inaccessible.

  • H/W vendor(CISCO) checked everything and did not find any h/w issue, recommended to verify os /openstack side.

  • The following errors might be seen in /var/log/containers/nova/nova-compute.log:

2019-11-24 22:36:54.369 29283 ERROR oslo.messaging._drivers.impl_rabbit [-] [7f36e587-b449-45a8-9ebf-4e3184ff813f] AMQP server on overcloud-controller-1.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: error: [Errno 113] EHOSTUNREACH
2019-11-24 22:36:57.367 29283 ERROR oslo.messaging._drivers.impl_rabbit [req-78f19f0a-5b0a-4d13-8096-14ff2f1c3103 - - - - -] [b09b5ad2-8746-4e79-92de-2c10819c945d] AMQP server on overcloud-controller-2.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: error: [Errno 113] EHOSTUNREACH
2019-11-24 22:36:57.368 29283 ERROR oslo.messaging._drivers.impl_rabbit [-] [7f36e587-b449-45a8-9ebf-4e3184ff813f] AMQP server on overcloud-controller-2.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 32 seconds.: error: [Errno 113] EHOSTUNREACH
2019-11-24 22:37:01.460 29283 ERROR oslo.messaging._drivers.impl_rabbit [req-78f19f0a-5b0a-4d13-8096-14ff2f1c3103 - - - - -] [b09b5ad2-8746-4e79-92de-2c10819c945d] AMQP server on overcloud-controller-1.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: error: [Errno 113] EHOSTUNREACH
2019-11-24 22:37:04.464 29283 ERROR oslo.messaging._drivers.impl_rabbit [req-78f19f0a-5b0a-4d13-8096-14ff2f1c3103 - - - - -] [b09b5ad2-8746-4e79-92de-2c10819c945d] AMQP server on overcloud-controller-0.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 32 seconds.: error: [Errno 113] EHOSTUNREACH
  • The hypervisor might show as DOWN:
(overcloud) [stack@undercloud ~]$ openstack hypervisor list
+-----+------------------------------------+-----------------+-------------+-------+
|  ID | Hypervisor Hostname                | Hypervisor Type | Host IP     | State |
+-----+------------------------------------+-----------------+-------------+-------+
|   2 | overcloud-osd-compute-3.localdomain | QEMU            | 10.10.10.16 | up    |
|   5 | overcloud-osd-compute-2.localdomain | QEMU            | 10.10.10.15 | up    |
|   8 | overcloud-compute-28.localdomain    | QEMU            | 10.10.10.45 | up    |
|  11 | overcloud-compute-18.localdomain    | QEMU            | 10.10.10.35 | up    |
|  14 | overcloud-compute-24.localdomain    | QEMU            | 10.10.10.41 | up    |
|  17 | overcloud-compute-9.localdomain     | QEMU            | 10.10.10.26 | up    |
|  20 | overcloud-osd-compute-0.localdomain | QEMU            | 10.10.10.13 | up    |
|  23 | overcloud-compute-10.localdomain    | QEMU            | 10.10.10.27 | up    |
|  26 | overcloud-compute-6.localdomain     | QEMU            | 10.10.10.23 | up    |
|  29 | overcloud-compute-25.localdomain    | QEMU            | 10.10.10.42 | up    |
|  32 | overcloud-osd-compute-1.localdomain | QEMU            | 10.10.10.14 | up    |
|  35 | overcloud-compute-29.localdomain    | QEMU            | 10.10.10.46 | up    |
|  38 | overcloud-compute-11.localdomain    | QEMU            | 10.10.10.28 | up    |
|  41 | overcloud-compute-23.localdomain    | QEMU            | 10.10.10.40 | up    |
|  44 | overcloud-compute-19.localdomain    | QEMU            | 10.10.10.36 | up    |
|  47 | overcloud-compute-1.localdomain     | QEMU            | 10.10.10.18 | up    |
|  50 | overcloud-compute-22.localdomain    | QEMU            | 10.10.10.39 | up    |
|  53 | overcloud-compute-13.localdomain    | QEMU            | 10.10.10.30 | up    |
|  56 | overcloud-compute-4.localdomain     | QEMU            | 10.10.10.21 | down  |
|  59 | overcloud-compute-0.localdomain     | QEMU            | 10.10.10.17 | up    |
|  62 | overcloud-compute-3.localdomain     | QEMU            | 10.10.10.20 | up    |
|  65 | overcloud-compute-15.localdomain    | QEMU            | 10.10.10.32 | up    |
|  68 | overcloud-compute-5.localdomain     | QEMU            | 10.10.10.22 | up    |
|  71 | overcloud-compute-16.localdomain    | QEMU            | 10.10.10.33 | up    |
|  74 | overcloud-compute-20.localdomain    | QEMU            | 10.10.10.37 | up    |
|  77 | overcloud-compute-8.localdomain     | QEMU            | 10.10.10.25 | up    |
|  80 | overcloud-compute-17.localdomain    | QEMU            | 10.10.10.34 | up    |
|  83 | overcloud-compute-12.localdomain    | QEMU            | 10.10.10.29 | up    |
|  86 | overcloud-compute-26.localdomain    | QEMU            | 10.10.10.43 | up    |
|  89 | overcloud-compute-21.localdomain    | QEMU            | 10.10.10.38 | up    |
|  92 | overcloud-compute-2.localdomain     | QEMU            | 10.10.10.19 | up    |
|  95 | overcloud-compute-7.localdomain     | QEMU            | 10.10.10.24 | up    |
|  98 | overcloud-compute-14.localdomain    | QEMU            | 10.10.10.31 | up    |
| 101 | overcloud-compute-27.localdomain    | QEMU            | 10.10.10.44 | up    |
+-----+------------------------------------+-----------------+-------------+-------+

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)
  • Red Hat OpenStack Platform 8.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In