Network issues on one compute after reboot

Issue

• Network issues on one compute after reboot

• /var/log/containers/nova/nova-compute.log shows it's unable to connect to rabbitmq:

2019-12-24 11:53:34.390 1 ERROR oslo.messaging._drivers.impl_rabbit [req-f43a6d24-e40f-42fa-be6d-dc4368439258 - - - - -] [e6e26e33-03f1-4fa8-97fb-3e2815413de0] AMQP server on overcloud-compute-0.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
2019-12-24 11:54:10.422 1 ERROR oslo.messaging._drivers.impl_rabbit [req-f43a6d24-e40f-42fa-be6d-dc4368439258 - - - - -] [e6e26e33-03f1-4fa8-97fb-3e2815413de0] AMQP server on overcloud-controller-2.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out

• /var/log/containers/neutron/openvswitch-agent.log shows it's unable to connect to rabbitmq:

2019-12-24 11:53:38.125 16451 ERROR oslo.messaging._drivers.impl_rabbit [-] [0cf064ec-4967-4fcd-8399-0f8aed38684c] AMQP server on overcloud-controller-0.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
2019-12-24 11:54:14.138 16451 ERROR oslo.messaging._drivers.impl_rabbit [-] [0cf064ec-4967-4fcd-8399-0f8aed38684c] AMQP server on overcloud-controller-2.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out

• `/var/log/containers/sriov-nic-agent.log shows it's unable to connect to rabbitmq:

2019-12-24 11:54:08.147 16080 ERROR oslo.messaging._drivers.impl_rabbit [req-56173a1e-86f9-4a65-a147-b225203d8826 - - - - -] [2825c358-c029-4457-a9db-7ae37d4ef858] AMQP server on overcloud-controller-2.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
2019-12-24 11:54:08.159 16080 ERROR oslo.messaging._drivers.impl_rabbit [-] [5bc01a7b-7b98-4539-92a5-62523ad0d963] AMQP server on overcloud-controller-1.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out

• systemctl status network shows that the network service failed to start:

* network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
   Active: failed (Result: timeout) since Tue 2019-12-24 10:36:48 IST; 1h 19min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 32471 ExecStart=/etc/rc.d/init.d/network start (code=killed, signal=TERM)
    Tasks: 0
   Memory: 20.0K

Dec 24 10:36:08 overcloud-compute-0.localdomain dhclient[34264]: DHCPDISCOVER on enp55s3f6 to 255.255.255.255 port 67 interval 10 (xid=0x134890da)
Dec 24 10:36:18 overcloud-compute-0.localdomain dhclient[34264]: DHCPDISCOVER on enp55s3f6 to 255.255.255.255 port 67 interval 16 (xid=0x134890da)
Dec 24 10:36:34 overcloud-compute-0.localdomain dhclient[34264]: DHCPDISCOVER on enp55s3f6 to 255.255.255.255 port 67 interval 19 (xid=0x134890da)
Dec 24 10:36:48 overcloud-compute-0.localdomain systemd[1]: network.service start operation timed out. Terminating.
Dec 24 10:36:48 overcloud-compute-0.localdomain systemd[1]: Failed to start LSB: Bring up/down networking.
Dec 24 10:36:48 overcloud-compute-0.localdomain systemd[1]: Unit network.service entered failed state.
Dec 24 10:36:48 overcloud-compute-0.localdomain systemd[1]: network.service failed.
Dec 24 10:36:53 overcloud-compute-0.localdomain dhclient[34264]: DHCPDISCOVER on enp55s3f6 to 255.255.255.255 port 67 interval 1 (xid=0x134890da)
Dec 24 10:36:54 overcloud-compute-0.localdomain dhclient[34264]: No DHCPOFFERS received.
Dec 24 10:36:54 overcloud-compute-0.localdomain network[32471]: Determining IP information for enp55s3f6... failed.

• Routes are missing and some interfaces are not configured. Manually running ifup vlan301 and ifup vlan302 brings the interfaces up successfully.