SR-IOV VM lost IP connectivity to the GW

Issue

• We have a VM with SR-IOV VF that lost the connectivity with its GW (Physical GW). We have engaged the network vendor, but nothing suspicious from their side.

• The following error messages are seen in the logs:

Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF reset check timeout on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Setting MAC 02:c0:6d:5d:07:b0 on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF reset check timeout on VF 63
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF reset check timeout on VF 63
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Bring down and up the VF interface to make this change effective.
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Setting MAC 02:0e:c7:61:a8:71 on VF 63
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF reset check timeout on VF 63
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Bring down and up the VF interface to make this change effective.
Jun 29 11:43:51 overcloud-compute-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): tap052e4562-aa: link becomes ready
Jun 29 11:43:51 overcloud-compute-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): taped4416da-73: link becomes ready
Jun 29 11:44:32 overcloud-compute-2 kernel: vfio-pci 0000:5f:09.7: enabling device (0000 -> 0002)
Jun 29 11:44:32 overcloud-compute-2 kernel: vfio-pci 0000:5f:11.7: enabling device (0000 -> 0002)
Jun 29 11:44:32 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF reset check timeout on VF 63
Jun 29 11:44:32 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 29 11:44:32 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF reset check timeout on VF 63
Jun 29 11:44:32 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 29 11:44:34 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF reset check timeout on VF 63
Jun 29 11:44:34 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 29 11:44:34 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF reset check timeout on VF 63
Jun 29 11:44:34 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 29 11:47:16 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF reset check timeout on VF 63
Jun 29 11:47:16 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 29 11:47:16 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF reset check timeout on VF 63
Jun 29 11:47:16 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 29 11:47:19 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF reset check timeout on VF 63
Jun 29 11:47:19 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 29 11:47:19 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF reset check timeout on VF 63
Jun 29 11:47:19 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 29 11:47:54 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 29 11:47:55 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:10:34 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:11:00 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:11:00 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 30 15:11:00 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF 63 failed opcode 9, retval: -37
Jun 30 15:11:07 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:11:21 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:11:32 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:12:11 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:15:31 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:16:46 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:19:05 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:19:07 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:19:17 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:19:17 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 30 15:19:17 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF 63 failed opcode 9, retval: -37
Jun 30 15:19:22 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:19:25 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:19:59 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:20:01 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:20:56 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:21:01 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:22:03 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:22:03 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 30 15:22:03 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF 63 failed opcode 9, retval: -37
Jun 30 15:23:00 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:23:02 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:23:06 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:23:32 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:24:08 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:24:08 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 30 15:24:08 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF 63 failed opcode 9, retval: -37
Jun 30 15:24:31 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:25:51 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:25:56 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:26:20 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:26:23 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:26:32 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:26:32 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 30 15:26:32 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF 63 failed opcode 9, retval: -37
Jun 30 15:28:40 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:29:52 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:29:56 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:31:44 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:31:55 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:37:21 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:37:25 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:37:48 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:39:25 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 17:24:55 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 17:25:09 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 18:50:36 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 18:51:03 overcloud-compute-2 kernel: device p2p1 left promiscuous mode