Unable to activate FCoE

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7.1
  • Broadcom Corporation NetXtreme II BCM57810 10 Gig interface used for FCoE

Issue

  • We have two nearly identical Dell blades, both of which have the same 8-port, FCoE-capable NICs. On one of them we've been able to create FCoE interfaces, on the other we cannot.

  • fcoeadm -i reports No FCoE interfaces created. error message.

Resolution

  • Remove/uninstall third party driver, and use kernel provided drivers shipped by Red Hat.
  • Install the system with kernel provided drivers which fixed this issue.

Root Cause

  • While starting fcoe service in debug mode it was found that fcoe service was failing in VLAN discovery phase:

    fcoemon: OP: VLAN DISC em1_4
fcoemon: creating netlink socket
fcoemon: fip_socket_sanmac: no sanmac, ifindex 8
fcoemon: creating netlink socket
fcoemon: sending FIP VLAN request               <<---------- sending FCoE init request
fcoemon: Link event: 16 flags 11043 index 10 
fcoemon: em1_4: VLAN discovery TIMEOUT [1]          <<---------- did not receive a valid list of VLANs to login from FCF and got timed out.
fcoemon: creating netlink socket
fcoemon: sending FIP VLAN request
fcoemon: em1_4: VLAN discovery TIMEOUT [2]

  • The network packet capture collected while starting fcoe service showed that em1_4 interface was able to receive the periodic FIP discovery advertisement messages sent from the FCoE switch/FCF.

    No. Time        Source          Destination Protocol Length Info
34  6.401354    DellInc_00:00:00:00 All-ENode-MACs  FIP 76  Advertisement
35  6.404501    DellInc_00:00:00    All-ENode-MACs  FIP 76  Advertisement
89  14.496673   DellInc_00:00:00:00 All-ENode-MACs  FIP 76  Advertisement
90  14.497392   DellInc_00:00:00    All-ENode-MACs  FIP 76  Advertisement
136 22.610527   DellInc_00:00:00:00 All-ENode-MACs  FIP 76  Advertisement
137 22.611796   DellInc_00:00:00    All-ENode-MACs  FIP 76  Advertisement
[...]

  • The "em1_4" interface (mac address: bb:bb:bb:bb:bb:bb) had sent the VLAN Request to default FCoE switch/FCF MAC address All-FCF-MACs (00:00:00:00:00:00), but it looks that em1_4 has not received the list of available VLANs to which it could login:

    143 24.066807   DellInc_00:00:00:00 All-FCF-MACs    FIP 32  VLAN Request
    Destination: All-FCF-MACs (00:00:00:00:00:00)
    Source: DellInc_00:00:00:00 (00:00:00:00:00:00)
    Type: FCoE Initialization Protocol (0x8914)
    FIP VLAN Request
    [...]

147 25.070809   DellInc_00:00:00:00 All-FCF-MACs    FIP 32  VLAN Request
152 26.074818   DellInc_00:00:00:00 All-FCF-MACs    FIP 32  VLAN Request
159 27.078819   DellInc_00:00:00:00 All-FCF-MACs    FIP 32  VLAN Request
168 28.082827   DellInc_00:00:00:00 All-FCF-MACs    FIP 32  VLAN Request
177 29.086816   DellInc_00:00:00:00 All-FCF-MACs    FIP 32  VLAN Request
190 30.090831   DellInc_00:00:00:00 All-FCF-MACs    FIP 32  VLAN Request
193 30.641508   DellInc_00:00:00:00 All-ENode-MACs  FIP 76  Advertisement
194 30.651115   DellInc_00:00:00    All-ENode-MACs  FIP 76  Advertisement
195 31.093816   DellInc_00:00:00:00 All-FCF-MACs    FIP 32  VLAN Request
203 32.097821   DellInc_00:00:00:00 All-FCF-MACs    FIP 32  VLAN Request
216 33.101815   DellInc_00:00:00:00 All-FCF-MACs    FIP 32  VLAN Request
227 34.104812   DellInc_00:00:00:00 All-FCF-MACs    FIP 32  VLAN Request
238 35.108824   DellInc_00:00:00:00 All-FCF-MACs    FIP 32  VLAN Request
246 36.112828   DellInc_00:00:00:00 All-FCF-MACs    FIP 32  VLAN Request
252 37.116805   DellInc_00:00:00:00 All-FCF-MACs    FIP 32  VLAN Request
256 38.120823   DellInc_00:00:00:00 All-FCF-MACs    FIP 32  VLAN Request
[...]

Diagnostic Steps

Please use following steps to capture the tcpdump while VLAN discovery timeouts are being observed:

  • Stop the lldpad and fcoe services if it's running on the affected host.

  • Confirm that we have following configuration in "/etc/fcoe/cfg-emXX" file.

    $ cat cfg-emXX|grep -v "#"
FCOE_ENABLE="yes"
DCB_REQUIRED="no"
AUTO_VLAN="yes"
MODE="fabric"

  • Then start "lldpad" service:

    $ /etc/init.d/lldpad restart

  • Start tcpdump to capture the network packets from emXX` interface as shown below:

    o If the tcpdump rpm package is not already installed on the server, then please use following command to install the same:

    $ yum install tcpdump

    o To start tcpdump capture:

    $ tcpdump -vv -tttt -s 0 -ni em1_4 -w  /tmp/$HOSTNAME.pcap

  • Once the packet capture is started on host and FCoE switch, please start the fcoemon daemon in debug mode using the following command:

    $ fcoemon -d

  • Once we start getting the couple of following timeout error messages please stop the network packet capture and attach the same with service request.

    fcoemon: sending FIP VLAN request
fcoemon: em1_4: VLAN discovery TIMEOUT [2]

  • Later point we identified that QLogic NIC kernel driver is provided by Dell/QLogic , and the both identical dell blades are in different data centers.

  • Suggested to use inbox drivers which is shipped with kernel

  • With inbox derives FCoE interface created successfully.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.