openvswitch logging extensively: netdev_tc_offloads(revalidator241)|ERR|dump_create: failed to get ifindex for qr-d3cafd87-41: Operation not supported

Solution In Progress - Updated -

Issue

  • We are applying latest updates to our clouds and we see increasing pattern of error logs at /var/log/openvswitch/ovs-vswitchd.log:
2020-03-06T09:56:04.479Z|3761807|netdev_tc_offloads(revalidator241)|ERR|dump_create: failed to get ifindex for qr-d3cafd87-41: Operation not supported
2020-03-06T09:56:05.480Z|3761808|netdev_tc_offloads(revalidator241)|ERR|Dropped 21 log messages in last 1 seconds (most recently, 1 seconds ago) due to excessive rate
  • Here's count of log lines from logstash from last hour:
Cloud,Count
"est-cloud-24",985265
"est-cloud-64",843042
"north-cloud-42",839478
  • Example from one random compute node:
[root@overcloud-compute-0 neutron]# grep dump_create /var/log/openvswitch/ovs-vswitchd.log | wc -l
31809
  • Sample logs:
[root@overcloud-compute-0 neutron]# head -2 /var/log/openvswitch/ovs-vswitchd.log
2020-03-06T01:19:02.088Z|22471|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2020-03-06T01:19:02.512Z|3699762|netdev_tc_offloads(revalidator241)|ERR|Dropped 32 log messages in last 1 seconds (most recently, 0 seconds ago) due to excessive rate
  • Sample logs:
[root@overcloud-compute-0 neutron]# tail -2 /var/log/openvswitch/ovs-vswitchd.log
2020-03-06T10:09:34.503Z|3763426|netdev_tc_offloads(revalidator241)|ERR|Dropped 21 log messages in last 1 seconds (most recently, 0 seconds ago) due to excessive rate
2020-03-06T10:09:34.503Z|3763427|netdev_tc_offloads(revalidator241)|ERR|dump_create: failed to get ifindex for qr-d3cafd87-41: Operation not supported
  • Affected ports:
[root@overcloud-compute-0 neutron]# awk '/dump_create/{ print $7 }' /var/log/openvswitch/ovs-vswitchd.log | sort | uniq
fg-3278ceef-6c:
qr-0afb8487-1b:
qr-133d1c34-3e:
qr-2d4725fc-62:
qr-593e22c4-fe:
qr-5b452898-42:
qr-7d2c1d92-49:
qr-a4e48b27-8a:
qr-d3cafd87-41:
  • Also there has been customer issues about floating IP's not reachable and random metadata issues, we do not know if these are related or not

Environment

  • Red Hat OpenStack Platform 16.0 (RHOSP)
  • Red Hat OpenStack Platform 15.0 (RHOSP)
  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In