Network issues on one compute after reboot

Solution In Progress - Updated -

Issue

  • Network issues on one compute after reboot

  • /var/log/containers/nova/nova-compute.log shows it's unable to connect to rabbitmq:

2019-12-24 11:53:34.390 1 ERROR oslo.messaging._drivers.impl_rabbit [req-f43a6d24-e40f-42fa-be6d-dc4368439258 - - - - -] [e6e26e33-03f1-4fa8-97fb-3e2815413de0] AMQP server on overcloud-compute-0.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
2019-12-24 11:54:10.422 1 ERROR oslo.messaging._drivers.impl_rabbit [req-f43a6d24-e40f-42fa-be6d-dc4368439258 - - - - -] [e6e26e33-03f1-4fa8-97fb-3e2815413de0] AMQP server on overcloud-controller-2.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
  • /var/log/containers/neutron/openvswitch-agent.log shows it's unable to connect to rabbitmq:
2019-12-24 11:53:38.125 16451 ERROR oslo.messaging._drivers.impl_rabbit [-] [0cf064ec-4967-4fcd-8399-0f8aed38684c] AMQP server on overcloud-controller-0.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
2019-12-24 11:54:14.138 16451 ERROR oslo.messaging._drivers.impl_rabbit [-] [0cf064ec-4967-4fcd-8399-0f8aed38684c] AMQP server on overcloud-controller-2.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
  • `/var/log/containers/sriov-nic-agent.log shows it's unable to connect to rabbitmq:
2019-12-24 11:54:08.147 16080 ERROR oslo.messaging._drivers.impl_rabbit [req-56173a1e-86f9-4a65-a147-b225203d8826 - - - - -] [2825c358-c029-4457-a9db-7ae37d4ef858] AMQP server on overcloud-controller-2.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
2019-12-24 11:54:08.159 16080 ERROR oslo.messaging._drivers.impl_rabbit [-] [5bc01a7b-7b98-4539-92a5-62523ad0d963] AMQP server on overcloud-controller-1.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
  • systemctl status network shows that the network service failed to start:
* network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
   Active: failed (Result: timeout) since Tue 2019-12-24 10:36:48 IST; 1h 19min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 32471 ExecStart=/etc/rc.d/init.d/network start (code=killed, signal=TERM)
    Tasks: 0
   Memory: 20.0K

Dec 24 10:36:08 overcloud-compute-0.localdomain dhclient[34264]: DHCPDISCOVER on enp55s3f6 to 255.255.255.255 port 67 interval 10 (xid=0x134890da)
Dec 24 10:36:18 overcloud-compute-0.localdomain dhclient[34264]: DHCPDISCOVER on enp55s3f6 to 255.255.255.255 port 67 interval 16 (xid=0x134890da)
Dec 24 10:36:34 overcloud-compute-0.localdomain dhclient[34264]: DHCPDISCOVER on enp55s3f6 to 255.255.255.255 port 67 interval 19 (xid=0x134890da)
Dec 24 10:36:48 overcloud-compute-0.localdomain systemd[1]: network.service start operation timed out. Terminating.
Dec 24 10:36:48 overcloud-compute-0.localdomain systemd[1]: Failed to start LSB: Bring up/down networking.
Dec 24 10:36:48 overcloud-compute-0.localdomain systemd[1]: Unit network.service entered failed state.
Dec 24 10:36:48 overcloud-compute-0.localdomain systemd[1]: network.service failed.
Dec 24 10:36:53 overcloud-compute-0.localdomain dhclient[34264]: DHCPDISCOVER on enp55s3f6 to 255.255.255.255 port 67 interval 1 (xid=0x134890da)
Dec 24 10:36:54 overcloud-compute-0.localdomain dhclient[34264]: No DHCPOFFERS received.
Dec 24 10:36:54 overcloud-compute-0.localdomain network[32471]: Determining IP information for enp55s3f6... failed.
  • Routes are missing and some interfaces are not configured. Manually running ifup vlan301 and ifup vlan302 brings the interfaces up successfully.

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In