Any vm migrated to a specific compute node loses network connectivity

Solution In Progress - Updated -

Issue

  • If any vm comes under this compute node, they are loosing the network connectivity and vm become inaccessible.

  • H/W vendor(CISCO) checked everything and did not find any h/w issue, recommended to verify os /openstack side.

  • The following errors might be seen in /var/log/containers/nova/nova-compute.log:

2019-11-24 22:36:54.369 29283 ERROR oslo.messaging._drivers.impl_rabbit [-] [7f36e587-b449-45a8-9ebf-4e3184ff813f] AMQP server on overcloud-controller-1.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: error: [Errno 113] EHOSTUNREACH
2019-11-24 22:36:57.367 29283 ERROR oslo.messaging._drivers.impl_rabbit [req-78f19f0a-5b0a-4d13-8096-14ff2f1c3103 - - - - -] [b09b5ad2-8746-4e79-92de-2c10819c945d] AMQP server on overcloud-controller-2.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: error: [Errno 113] EHOSTUNREACH
2019-11-24 22:36:57.368 29283 ERROR oslo.messaging._drivers.impl_rabbit [-] [7f36e587-b449-45a8-9ebf-4e3184ff813f] AMQP server on overcloud-controller-2.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 32 seconds.: error: [Errno 113] EHOSTUNREACH
2019-11-24 22:37:01.460 29283 ERROR oslo.messaging._drivers.impl_rabbit [req-78f19f0a-5b0a-4d13-8096-14ff2f1c3103 - - - - -] [b09b5ad2-8746-4e79-92de-2c10819c945d] AMQP server on overcloud-controller-1.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds.: error: [Errno 113] EHOSTUNREACH
2019-11-24 22:37:04.464 29283 ERROR oslo.messaging._drivers.impl_rabbit [req-78f19f0a-5b0a-4d13-8096-14ff2f1c3103 - - - - -] [b09b5ad2-8746-4e79-92de-2c10819c945d] AMQP server on overcloud-controller-0.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 32 seconds.: error: [Errno 113] EHOSTUNREACH
  • The hypervisor might show as DOWN:
(overcloud) [stack@undercloud ~]$ openstack hypervisor list
+-----+------------------------------------+-----------------+-------------+-------+
|  ID | Hypervisor Hostname                | Hypervisor Type | Host IP     | State |
+-----+------------------------------------+-----------------+-------------+-------+
|   2 | overcloud-osd-compute-3.localdomain | QEMU            | 10.10.10.16 | up    |
|   5 | overcloud-osd-compute-2.localdomain | QEMU            | 10.10.10.15 | up    |
|   8 | overcloud-compute-28.localdomain    | QEMU            | 10.10.10.45 | up    |
|  11 | overcloud-compute-18.localdomain    | QEMU            | 10.10.10.35 | up    |
|  14 | overcloud-compute-24.localdomain    | QEMU            | 10.10.10.41 | up    |
|  17 | overcloud-compute-9.localdomain     | QEMU            | 10.10.10.26 | up    |
|  20 | overcloud-osd-compute-0.localdomain | QEMU            | 10.10.10.13 | up    |
|  23 | overcloud-compute-10.localdomain    | QEMU            | 10.10.10.27 | up    |
|  26 | overcloud-compute-6.localdomain     | QEMU            | 10.10.10.23 | up    |
|  29 | overcloud-compute-25.localdomain    | QEMU            | 10.10.10.42 | up    |
|  32 | overcloud-osd-compute-1.localdomain | QEMU            | 10.10.10.14 | up    |
|  35 | overcloud-compute-29.localdomain    | QEMU            | 10.10.10.46 | up    |
|  38 | overcloud-compute-11.localdomain    | QEMU            | 10.10.10.28 | up    |
|  41 | overcloud-compute-23.localdomain    | QEMU            | 10.10.10.40 | up    |
|  44 | overcloud-compute-19.localdomain    | QEMU            | 10.10.10.36 | up    |
|  47 | overcloud-compute-1.localdomain     | QEMU            | 10.10.10.18 | up    |
|  50 | overcloud-compute-22.localdomain    | QEMU            | 10.10.10.39 | up    |
|  53 | overcloud-compute-13.localdomain    | QEMU            | 10.10.10.30 | up    |
|  56 | overcloud-compute-4.localdomain     | QEMU            | 10.10.10.21 | down  |
|  59 | overcloud-compute-0.localdomain     | QEMU            | 10.10.10.17 | up    |
|  62 | overcloud-compute-3.localdomain     | QEMU            | 10.10.10.20 | up    |
|  65 | overcloud-compute-15.localdomain    | QEMU            | 10.10.10.32 | up    |
|  68 | overcloud-compute-5.localdomain     | QEMU            | 10.10.10.22 | up    |
|  71 | overcloud-compute-16.localdomain    | QEMU            | 10.10.10.33 | up    |
|  74 | overcloud-compute-20.localdomain    | QEMU            | 10.10.10.37 | up    |
|  77 | overcloud-compute-8.localdomain     | QEMU            | 10.10.10.25 | up    |
|  80 | overcloud-compute-17.localdomain    | QEMU            | 10.10.10.34 | up    |
|  83 | overcloud-compute-12.localdomain    | QEMU            | 10.10.10.29 | up    |
|  86 | overcloud-compute-26.localdomain    | QEMU            | 10.10.10.43 | up    |
|  89 | overcloud-compute-21.localdomain    | QEMU            | 10.10.10.38 | up    |
|  92 | overcloud-compute-2.localdomain     | QEMU            | 10.10.10.19 | up    |
|  95 | overcloud-compute-7.localdomain     | QEMU            | 10.10.10.24 | up    |
|  98 | overcloud-compute-14.localdomain    | QEMU            | 10.10.10.31 | up    |
| 101 | overcloud-compute-27.localdomain    | QEMU            | 10.10.10.44 | up    |
+-----+------------------------------------+-----------------+-------------+-------+

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)
  • Red Hat OpenStack Platform 8.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content