Red Hat Training

A Red Hat training course is available for Red Hat OpenStack Platform

Chapter 6. Debugging

6.1. Locate the logs

6.1.1. Access OpenDaylight logs

The OpenDaylight logs are stored on OpenDaylight nodes, where you can find them in the /opt/opendaylight/data/log/ directory.

OpenDaylight stores its logs in the karaf.log file.

  • The latest log is named karaf.log, while any older logs are numbered, such as karaf.log.1, and so on.

6.1.2. Access OpenDaylight logs through karaf shell

Another way to access the logs is to login to the karaf shell on the OpenDaylight node and display the log files.

  1. Connect to the karaf account: 

    $ ssh -p 8101 karaf@localhost

  2. Enable trace level logging on NetVirt. 

    $ log set TRACE org.opendaylight.netvirt

  3. If you need to tail the logs inside of the karaf shell, use 

    $ log:tail

More information

  • The karaf shell helps users enable different logging levels for any OpenDaylight feature.
  • If you enable TRACE, it is possible that you will receive an extremely big number of log files.

6.1.3. Access OpenStack Networking logs

When OpenStack network-related commands fail, the first step is to examine the neutron logs:

These logs are stored in server.log, located on each neutron node in the /var/log/neutron directory.

  • The server.log file also includes errors about the communication with OpenDaylight.
  • If the neutron error originates from interacting with OpenDaylight, it is necessary to examine the OpenDaylight logs as well, to locate the cause of the failure.

6.2. Debug networking errors

If you experience network error (for example, there is no instance connectivity), but no errors are reported when issuing OpenStack commands or in the neutron logs, then it may be useful to inspect the OVS nodes for network traffic and OpenFlow flows:

  1. Login (as the superuser) to the affected node where the network error has occurred.

  2. Display the information about the br-int switch. 

    # ovs-ofctl -O openflow13 show br-int

  3. Examine the output. It will be similar to this example: 

    OFPT_FEATURES_REPLY (OF1.3) (xid=0x2): dpid:0000e4c153bdb306
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS QUEUE_STATS
OFPST_PORT_DESC reply (OF1.3) (xid=0x3):
 1(br-ex-patch): addr:ae:38:01:09:66:5b
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(tap1f0f610c-8e): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 3(tun1147c81b59c): addr:66:e3:d2:b3:b8:e3
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:e4:c1:53:bd:b3:06
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (OF1.3) (xid=0x5): frags=normal miss_send_len=0

  4. List the statistics for the br-int switch. 

    # ovs-ofctl -O openflow13 dump-ports br-int

  5. Examine the output. It will be similar to this example: 

    OFPST_PORT reply (OF1.3) (xid=0x2): 4 ports
  port LOCAL: rx pkts=101215, bytes=6680190, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=0, bytes=0, drop=0, errs=0, coll=0
           duration=90117.708s
  port  1: rx pkts=126887, bytes=8970074, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=18764, bytes=2067792, drop=0, errs=0, coll=0
           duration=90117.418s
  port  2: rx pkts=1171, bytes=70800, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=473, bytes=44448, drop=0, errs=0, coll=0
           duration=88644.819s
  port  3: rx pkts=120197, bytes=8776126, drop=0, errs=0, frame=0, over=0, crc=0
           tx pkts=119408, bytes=8727254, drop=0, errs=0, coll=0
           duration=88632.426s

More information

  • In step 3, you can see that there are three ports created on this OVS node. The first is a patch port going to the bridge br-ex, which in this scenario is used for External network connectivity. The second port is a tap port which connects to a DHCP agent instance (we know this because the host is a controller, otherwise on a Compute role it would be an instance), while the third port is a VXLAN tunnel port created for the tenant traffic. When you know what each port is, you can examine the port statistics to verify that the port is indeed receiving/sending traffic (see Step 4).
  • From the output in Step 5, you can see that each port is receiving (rx pkts) and sending packets (tx pkts).

6.2.1. Advanced debugging using OpenFlow flows

For advanced users who are familiar with OpenFlow, the next level of debugging is to examine the flows on the switch in order to detect where traffic is being dropped.

You can list the flows, as well as how many packets have hit them, by entering the following command: 

# ovs-ofctl -O openflow13 dump-flows br-int

The output of the command will provide you with the necessary information. 

OFPST_FLOW reply (OF1.3) (xid=0x2):
 cookie=0x8000000, duration=90071.665s, table=0, n_packets=126816, n_bytes=8964820, priority=1,in_port=1 actions=write_metadata:0x20000000001/0xffffff0000000001,goto_table:17
 cookie=0x8000000, duration=88967.292s, table=0, n_packets=473, n_bytes=44448, priority=4,in_port=2 actions=write_metadata:0x40000000000/0xffffff0000000001,goto_table:17
 cookie=0x8000001, duration=88954.901s, table=0, n_packets=120636, n_bytes=8807869, priority=5,in_port=3 actions=write_metadata:0x70000000001/0x1fffff0000000001,goto_table:36
 cookie=0x8000001, duration=90069.534s, table=17, n_packets=126814, n_bytes=8964712, priority=5,metadata=0x20000000000/0xffffff0000000000 actions=write_metadata:0xc0000200000222e0/0xfffffffffffff
ffe,goto_table:19
 cookie=0x8040000, duration=90069.533s, table=17, n_packets=126813, n_bytes=8964658, priority=6,metadata=0xc000020000000000/0xffffff0000000000 actions=write_metadata:0xe00002138a000000/0xffffffff
fffffffe,goto_table:48
 cookie=0x8040000, duration=88932.689s, table=17, n_packets=396, n_bytes=36425, priority=6,metadata=0xc000040000000000/0xffffff0000000000 actions=write_metadata:0xe00004138b000000/0xfffffffffffff
ffe,goto_table:48
Note

The sample output has been edited for length.

6.2.2. Packet traverse in OpenFlow

The important things to understand are that the network functions performed on a packet are broken into different OpenFlow tables, and packets traverse those tables in order, starting from zero. An incoming packet lands in table 0, and then progresses via the OpenFlow Pipeline until it is sent out of a port, to the OpenDaylight Controller, or dropped. A packet may skip one or more tables depending on which network function it may need to go to. The full diagram of tables and how they correspond to network functions is shown below:

Figure 6.1. OpenDaylight NetVirt OpenFlow Pipeline

OpenDaylight NetVirt OpenFlow Pipeline