DPDK-OVS crashes with segmentation fault

Solution In Progress - Updated -

Issue

  • 1 controller and 2 compute nodes and are using OVS-DPDK. OVS is working fine until one of the VM is rebooted (VM location does not matter, problem is observed on both compute nodes). After the VM reboot, OVS DPDK crashes with segmentation fault. ovs-vswitchd.log:
2017-02-06T08:11:56.719Z|00291|dpdk(vhost_thread1)|INFO|State of queue 0 ( tx_qid 0 ) of vhost device 39;/var/run/openvswitch/vhue6e88b22-5939; 1 changed to 39;enabled39;
2017-02-06T08:11:59.373Z|00292|dpdk(vhost_thread1)|INFO|vHost Device 39;/var/run/openvswitch/vhu67e39e1b-4f39; 2 has been added
2017-02-06T08:11:59.373Z|00293|dpdk(vhost_thread1)|INFO|vHost Device 39;/var/run/openvswitch/vhue6e88b22-5939; 1 has been added
2017-02-06T08:11:59.373Z|00294|dpdk(vhost_thread1)|INFO|State of queue 0 ( tx_qid 0 ) of vhost device 39;/var/run/openvswitch/vhu67e39e1b-4f39; 2 changed to 39;enabled39;
2017-02-06T08:11:59.373Z|00295|dpdk(vhost_thread1)|INFO|State of queue 0 ( tx_qid 0 ) of vhost device 39;/var/run/openvswitch/vhue6e88b22-5939; 1 changed to 39;enabled39;
2017-02-06T08:49:08.548Z|00003|daemon_unix(monitor)|ERR|1 crashes: pid 32916 died, killed (Segmentation fault), core dumped, restarting
2017-02-06T08:49:08.589Z|00004|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2017-02-06T08:49:08.589Z|00005|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2017-02-06T08:49:08.589Z|00006|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2017-02-06T08:49:08.589Z|00007|memory|INFO|87988 kB peak resident set size after 3079.5 seconds
2017-02-06T08:49:08.589Z|00008|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2017-02-06T08:49:08.589Z|00009|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2017-02-06T08:49:08.598Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports recirculation
2017-02-06T08:49:08.598Z|00011|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3
2017-02-06T08:49:08.598Z|00012|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports unique flow ids
2017-02-06T08:49:08.598Z|00013|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_state
2017-02-06T08:49:08.598Z|00014|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_zone
2017-02-06T08:49:08.598Z|00015|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_mark
2017-02-06T08:49:08.598Z|00016|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_label
2017-02-06T08:49:08.639Z|00017|bridge|INFO|bridge ovsdpdkbr0: added interface ovsdpdkbr0 on port 65534
2017-02-06T08:49:08.639Z|00018|bridge|WARN|could not open network device dpdk3 (Cannot allocate memory)
2017-02-06T08:49:08.639Z|00019|bridge|WARN|could not open network device dpdk0 (Cannot allocate memory)
2017-02-06T08:49:08.639Z|00020|bridge|WARN|could not open network device dpdk1 (Cannot allocate memory)
2017-02-06T08:49:08.639Z|00021|bridge|INFO|bridge ovsdpdkbr0: added interface phy-ovsdpdkbr0 on port 4
2017-02-06T08:49:08.639Z|00022|bridge|INFO|bridge br-int: added interface int-ovsdpdkbr0 on port 7
2017-02-06T08:49:08.639Z|00023|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhue6e88b22-59
2017-02-06T08:49:08.639Z|00024|bridge|WARN|could not open network device vhue6e88b22-59 (Unknown error -1)
2017-02-06T08:49:08.639Z|00025|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhue1fe2f9a-29
2017-02-06T08:49:08.639Z|00026|bridge|WARN|could not open network device vhue1fe2f9a-29 (Unknown error -1)
  • ovs-vsctl show command output:
[root@compute-04 ~]# ovs-vsctl show
cfd11977-ab4a-4a73-a952-4e8ed2941ded
    Manager ;ptcp:6640:127.0.0.1;
        is_connected: true
    Bridge ;ovsdpdkbr0;
        Controller ;tcp:127.0.0.1:6633;
            is_connected: true
        fail_mode: secure
        Port ;ovsdpdkbr0;
            Interface ;ovsdpdkbr0;
                type: internal
        Port ;phy-ovsdpdkbr0;
            Interface ;phy-ovsdpdkbr0;
                type: patch
                options: {peer=;int-ovsdpdkbr0;}
        Port ;dpdkbond0;
            Interface ;dpdk0;
                type: dpdk
                error: ;could not open network device dpdk0 (Cannot allocate memory);
            Interface ;dpdk3;
                type: dpdk
                error: ;could not open network device dpdk3 (Cannot allocate memory);
            Interface ;dpdk1;
                type: dpdk
                error: ;could not open network device dpdk1 (Cannot allocate memory);
ovs-version: 2.5.0
DPDK: 2.2
RHEL 7.3
RHOSP: 10
HW: HP DL380 Gen9 (2680 Intel CPU v3)
NIC: Intel 82599

Environment

  • Red Hat OpenStack Platform 10.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content