openvswitch logging extensively: netdev_tc_offloads(revalidator241)|ERR|dump_create: failed to get ifindex for qr-d3cafd87-41: Operation not supported

Issue

• We are applying latest updates to our clouds and we see increasing pattern of error logs at /var/log/openvswitch/ovs-vswitchd.log:

2020-03-06T09:56:04.479Z|3761807|netdev_tc_offloads(revalidator241)|ERR|dump_create: failed to get ifindex for qr-d3cafd87-41: Operation not supported
2020-03-06T09:56:05.480Z|3761808|netdev_tc_offloads(revalidator241)|ERR|Dropped 21 log messages in last 1 seconds (most recently, 1 seconds ago) due to excessive rate

• Here's count of log lines from logstash from last hour:

Cloud,Count
"est-cloud-24",985265
"est-cloud-64",843042
"north-cloud-42",839478

• Example from one random compute node:

[root@overcloud-compute-0 neutron]# grep dump_create /var/log/openvswitch/ovs-vswitchd.log | wc -l
31809

• Sample logs:

[root@overcloud-compute-0 neutron]# head -2 /var/log/openvswitch/ovs-vswitchd.log
2020-03-06T01:19:02.088Z|22471|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2020-03-06T01:19:02.512Z|3699762|netdev_tc_offloads(revalidator241)|ERR|Dropped 32 log messages in last 1 seconds (most recently, 0 seconds ago) due to excessive rate

• Sample logs:

[root@overcloud-compute-0 neutron]# tail -2 /var/log/openvswitch/ovs-vswitchd.log
2020-03-06T10:09:34.503Z|3763426|netdev_tc_offloads(revalidator241)|ERR|Dropped 21 log messages in last 1 seconds (most recently, 0 seconds ago) due to excessive rate
2020-03-06T10:09:34.503Z|3763427|netdev_tc_offloads(revalidator241)|ERR|dump_create: failed to get ifindex for qr-d3cafd87-41: Operation not supported

• Affected ports:

[root@overcloud-compute-0 neutron]# awk '/dump_create/{ print $7 }' /var/log/openvswitch/ovs-vswitchd.log | sort | uniq
fg-3278ceef-6c:
qr-0afb8487-1b:
qr-133d1c34-3e:
qr-2d4725fc-62:
qr-593e22c4-fe:
qr-5b452898-42:
qr-7d2c1d92-49:
qr-a4e48b27-8a:
qr-d3cafd87-41:

• Also there has been customer issues about floating IP's not reachable and random metadata issues, we do not know if these are related or not