SR-IOV VM lost IP connectivity to the GW

Solution In Progress - Updated -

Issue

  • We have a VM with SR-IOV VF that lost the connectivity with its GW (Physical GW). We have engaged the network vendor, but nothing suspicious from their side.

  • The following error messages are seen in the logs:

Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF reset check timeout on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: RX driver issue detected on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Use PF Control I/F to re-enable the VF
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Setting MAC 02:c0:6d:5d:07:b0 on VF 63
Jun 29 11:43:50 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF reset check timeout on VF 63
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF reset check timeout on VF 63
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Bring down and up the VF interface to make this change effective.
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Setting MAC 02:0e:c7:61:a8:71 on VF 63
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF reset check timeout on VF 63
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 29 11:43:51 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Bring down and up the VF interface to make this change effective.
Jun 29 11:43:51 overcloud-compute-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): tap052e4562-aa: link becomes ready
Jun 29 11:43:51 overcloud-compute-2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): taped4416da-73: link becomes ready
Jun 29 11:44:32 overcloud-compute-2 kernel: vfio-pci 0000:5f:09.7: enabling device (0000 -> 0002)
Jun 29 11:44:32 overcloud-compute-2 kernel: vfio-pci 0000:5f:11.7: enabling device (0000 -> 0002)
Jun 29 11:44:32 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF reset check timeout on VF 63
Jun 29 11:44:32 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 29 11:44:32 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF reset check timeout on VF 63
Jun 29 11:44:32 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 29 11:44:34 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF reset check timeout on VF 63
Jun 29 11:44:34 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 29 11:44:34 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF reset check timeout on VF 63
Jun 29 11:44:34 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 29 11:47:16 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF reset check timeout on VF 63
Jun 29 11:47:16 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 29 11:47:16 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF reset check timeout on VF 63
Jun 29 11:47:16 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 29 11:47:19 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF reset check timeout on VF 63
Jun 29 11:47:19 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 29 11:47:19 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF reset check timeout on VF 63
Jun 29 11:47:19 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 29 11:47:54 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 29 11:47:55 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:10:34 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:11:00 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:11:00 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 30 15:11:00 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF 63 failed opcode 9, retval: -37
Jun 30 15:11:07 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:11:21 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:11:32 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:12:11 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:15:31 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:16:46 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:19:05 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:19:07 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:19:17 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:19:17 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 30 15:19:17 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF 63 failed opcode 9, retval: -37
Jun 30 15:19:22 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:19:25 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:19:59 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:20:01 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:20:56 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:21:01 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:22:03 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:22:03 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 30 15:22:03 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF 63 failed opcode 9, retval: -37
Jun 30 15:23:00 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:23:02 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:23:06 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:23:32 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:24:08 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:24:08 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VSI seid 455 Tx ring 317 disable timeout
Jun 30 15:24:08 overcloud-compute-2 kernel: i40e 0000:5f:00.1: VF 63 failed opcode 9, retval: -37
Jun 30 15:24:31 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:25:51 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:25:56 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:26:20 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:26:23 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:26:32 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 15:26:32 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VSI seid 519 Tx ring 317 disable timeout
Jun 30 15:26:32 overcloud-compute-2 kernel: i40e 0000:5f:00.0: VF 63 failed opcode 9, retval: -37
Jun 30 15:28:40 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:29:52 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:29:56 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:31:44 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:31:55 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:37:21 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:37:25 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 15:37:48 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 15:39:25 overcloud-compute-2 kernel: device p2p1 left promiscuous mode
Jun 30 17:24:55 overcloud-compute-2 kernel: i40e 0000:5f:00.0: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 17:25:09 overcloud-compute-2 kernel: i40e 0000:5f:00.1: Unprivileged VF 63 is attempting to configure promiscuous mode
Jun 30 18:50:36 overcloud-compute-2 kernel: device p2p1 entered promiscuous mode
Jun 30 18:51:03 overcloud-compute-2 kernel: device p2p1 left promiscuous mode

Environment

  • Red Hat OpenStack Platform 17.1 (RHOSP)
  • Red Hat OpenStack Platform 16.2 (RHOSP)
  • Red Hat OpenStack Platform 10.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content