USB Latency difference between RHEL and Fedora

Latest response

Is there anything that can cause drastic latency differences in USB transfer speeds between RHEL 7.5 (3.10.0-862.14.4.el7.x86_64) and Fedora 26 (4.16.11-100.fc26.x86_64)?

My company physically connects an RS422 device, through RS422 to USB, using the FTDI_SIO driver. These devices send an ASCII string of characters to our software running on the linux machine.

We have a server that we have tested with 3 HDDs, each with a separate OS: RHEL 7.5 (server), RHEL 7.5 (Desktop), Fedora (23).

When our software & RS422 device runs on Fedora, the Latency is approx. 48-51 MS per transaction. When it runs on either RHEL OS, the latency is approx 2267 MS/transaction.

Outside of the OS, all other factors are identical. We are using the same physical server, the same RS422 devices, the same FTDI driver and the same software build to talk with the FTDI.

As of this morning (10/17/18), all Operating Systems are up to date and, and have the latest versions of usbutils, libusb, usbmuxd, usb_Modeswitch and USB-reder packages install.

Logically, it seems there is something different between the RHEL & Fedora builds (USB configuration setting, USB monitoring, etc) thats causing the difference in speed. However, I haven't been able to pin point anything yet.

JH
Log in to join the conversation

Responses

JH Community Member 45 points
17 October 2018 5:05 PM

If need be I can add necessary info, as requested.

James Brigman's picture Newbie 10 points
17 October 2018 9:07 PM

Hi Joe - can you post some output from the RHEL system?

First, the output of the command "lsusb". This will provide a list of USB devices and what version of USB that RHEL thinks it's talking to.

Second, are you able to cat /proc/bus/usb/devices ? This provides a great deal of info about what Linux thinks it's seeing on the USB devices.

I won't ask you to post BOTH (from Fedora AND RHEL) here, but I would recommend performing both commands on all the systems and comparing the output.

Finally, I must recommend https://opensourceforu.com/2011/10/usb-drivers-in-linux-1/

It's a pretty spiffy article.

Good luck, post more, let us know how it goes! GroceryBoyJr (aka James Brigman)

JH Community Member 45 points
18 October 2018 5:08 PM

Here's some output for you, I'm still working on checking out that link. I'll break it up it different comments to make it easier to read.

JH Community Member 45 points
18 October 2018 5:09 PM

lsusb output from the fedora machine:

Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 005: ID 0930:6545 Toshiba Corp. Kingston DataTraveler 102/2.0 / HEMA Flash Drive 2 GB / PNY Attache 4GB Stick
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 005: ID 04b3:3025 IBM Corp. NetVista Full Width Keyboard
Bus 003 Device 004: ID 046d:c03e Logitech, Inc. Premium Optical Wheel Mouse (M-BT58)
Bus 003 Device 003: ID 0856:ac11 B&B Electronics 
Bus 003 Device 002: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

lsusb from redhat:

Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0930:6545 Toshiba Corp. Kingston DataTraveler 102/2.0 / HEMA Flash Drive 2 GB / PNY Attache 4GB Stick
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 005: ID 04b3:3025 IBM Corp. NetVista Full Width Keyboard
Bus 003 Device 004: ID 046d:c03e Logitech, Inc. Premium Optical Wheel Mouse (M-BT58)
Bus 003 Device 003: ID 0856:ac11 B&B Electronics 
Bus 003 Device 002: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
JH Community Member 45 points
18 October 2018 5:10 PM

Redhat /proc/bus/input/devices:

I: Bus=0019 Vendor=0000 Product=0001 Version=0000
N: Name="Power Button"
P: Phys=PNP0C0C/button/input0
S: Sysfs=/devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input0
U: Uniq=
H: Handlers=kbd event0 
B: PROP=0
B: EV=3
B: KEY=10000000000000 0

I: Bus=0019 Vendor=0000 Product=0001 Version=0000
N: Name="Power Button"
P: Phys=LNXPWRBN/button/input0
S: Sysfs=/devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
U: Uniq=
H: Handlers=kbd event1 
B: PROP=0
B: EV=3
B: KEY=10000000000000 0

I: Bus=0019 Vendor=0000 Product=0006 Version=0000
N: Name="Video Bus"
P: Phys=LNXVIDEO/video/input0
S: Sysfs=/devices/LNXSYSTM:00/device:00/PNP0A08:00/LNXVIDEO:00/input/input2
U: Uniq=
H: Handlers=kbd event2 
B: PROP=0
B: EV=3
B: KEY=3e000b00000000 0 0 0

I: Bus=0003 Vendor=046d Product=c03e Version=0110
N: Name="Logitech USB-PS/2 Optical Mouse"
P: Phys=usb-0000:00:14.0-3/input0
S: Sysfs=/devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3:1.0/input/input3
U: Uniq=
H: Handlers=mouse0 event3 
B: PROP=0
B: EV=17
B: KEY=70000 0 0 0 0
B: REL=103
B: MSC=10

I: Bus=0003 Vendor=04b3 Product=3025 Version=0110
N: Name="LITE-ON Technology USB NetVista Full Width Keyboard."
P: Phys=usb-0000:00:14.0-4/input0
S: Sysfs=/devices/pci0000:00/0000:00:14.0/usb3/3-4/3-4:1.0/input/input4
U: Uniq=
H: Handlers=sysrq kbd leds event4 
B: PROP=0
B: EV=120013
B: KEY=1000000000007 ff9f207ac14057ff febeffdfffefffff fffffffffffffffe
B: MSC=10
B: LED=7

I: Bus=0010 Vendor=001f Product=0001 Version=0100
N: Name="PC Speaker"
P: Phys=isa0061/input0
S: Sysfs=/devices/platform/pcspkr/input/input5
U: Uniq=
H: Handlers=kbd event5 
B: PROP=0
B: EV=40001
B: SND=6

I: Bus=0000 Vendor=0000 Product=0000 Version=0000
N: Name="HDA Intel PCH HDMI/DP,pcm=3"
P: Phys=ALSA
S: Sysfs=/devices/pci0000:00/0000:00:1b.0/sound/card0/input6
U: Uniq=
H: Handlers=event6 
B: PROP=0
B: EV=21
B: SW=140

I: Bus=0000 Vendor=0000 Product=0000 Version=0000
N: Name="HDA Intel PCH HDMI/DP,pcm=7"
P: Phys=ALSA
S: Sysfs=/devices/pci0000:00/0000:00:1b.0/sound/card0/input7
U: Uniq=
H: Handlers=event7 
B: PROP=0
B: EV=21
B: SW=140
JH Community Member 45 points
18 October 2018 5:11 PM

Fedora, lsmod:

Module                  Size  Used by
vfat                   24576  1
fat                    73728  1 vfat
uas                    28672  0
usb_storage            69632  2 uas
xt_CHECKSUM            16384  1
ipt_MASQUERADE         16384  3
nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
fuse                  122880  3
tun                    49152  1
nf_conntrack_netbios_ns    16384  1
nf_conntrack_broadcast    16384  1 nf_conntrack_netbios_ns
xt_CT                  16384  1
ip6t_rpfilter          16384  1
ip6t_REJECT            16384  2
nf_reject_ipv6         16384  1 ip6t_REJECT
xt_conntrack           16384  22
ip_set                 45056  0
nfnetlink              16384  1 ip_set
ebtable_nat            16384  1
ebtable_broute         16384  1
bridge                188416  1 ebtable_broute
stp                    16384  1 bridge
llc                    16384  2 bridge,stp
ip6table_nat           16384  1
nf_conntrack_ipv6      16384  12
nf_defrag_ipv6         36864  1 nf_conntrack_ipv6
nf_nat_ipv6            16384  1 ip6table_nat
ip6table_mangle        16384  1
ip6table_raw           16384  1
ip6table_security      16384  1
iptable_nat            16384  1
nf_conntrack_ipv4      16384  16
nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
nf_nat_ipv4            16384  1 iptable_nat
nf_nat                 36864  3 nf_nat_ipv6,nf_nat_masquerade_ipv4,nf_nat_ipv4
nf_conntrack          147456  11 nf_conntrack_ipv6,nf_conntrack_ipv4,ipt_MASQUERADE,nf_conntrack_broadcast,nf_conntrack_netbios_ns,xt_CT,nf_nat_ipv6,nf_nat_masquerade_ipv4,xt_conntrack,nf_nat_ipv4,nf_nat
libcrc32c              16384  2 nf_conntrack,nf_nat
iptable_mangle         16384  1
iptable_raw            16384  1
iptable_security       16384  1
ebtable_filter         16384  1
ebtables               36864  3 ebtable_filter,ebtable_nat,ebtable_broute
ip6table_filter        16384  1
ip6_tables             32768  5 ip6table_mangle,ip6table_filter,ip6table_security,ip6table_raw,ip6table_nat
sunrpc                401408  1
intel_rapl             24576  0
x86_pkg_temp_thermal    16384  0
intel_powerclamp       16384  0
snd_hda_codec_hdmi     57344  1
coretemp               16384  0
snd_hda_codec_realtek   110592  1
kvm                   708608  0
snd_hda_codec_generic    86016  1 snd_hda_codec_realtek
snd_hda_intel          45056  6
irqbypass              16384  1 kvm
snd_hda_codec         151552  4 snd_hda_intel,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_hda_codec_realtek
crct10dif_pclmul       16384  0
crc32_pclmul           16384  0
ghash_clmulni_intel    16384  0
snd_hda_core           94208  5 snd_hda_intel,snd_hda_codec,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_hda_codec_realtek
intel_cstate           16384  0
mei_wdt                16384  0
iTCO_wdt               16384  0
snd_hwdep              20480  1 snd_hda_codec
snd_seq                81920  0
iTCO_vendor_support    16384  1 iTCO_wdt
intel_uncore          131072  0
intel_rapl_perf        16384  0
snd_seq_device         16384  1 snd_seq
snd_pcm               118784  5 snd_hda_intel,snd_hda_codec,snd_hda_core,snd_hda_codec_hdmi
ftdi_sio               57344  0
snd_timer              36864  2 snd_seq,snd_pcm
snd                    94208  21 snd_hda_intel,snd_hwdep,snd_seq,snd_hda_codec,snd_timer,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_seq_device,snd_hda_codec_realtek,snd_pcm
mei_me                 45056  1
i2c_i801               28672  0
shpchp                 40960  0
soundcore              16384  1 snd
mei                   106496  3 mei_me,mei_wdt
lpc_ich                28672  0
i915                 1994752  17
drm_kms_helper        196608  1 i915
igb                   249856  0
e1000e                282624  0
drm                   454656  17 i915,drm_kms_helper
dca                    16384  1 igb
ata_generic            16384  0
pata_acpi              16384  0
ptp                    20480  2 igb,e1000e
crc32c_intel           24576  4
i2c_algo_bit           16384  2 igb,i915
pps_core               20480  1 ptp
video                  45056  1 i915
JH Community Member 45 points
18 October 2018 5:11 PM

Red Hat lsmod:

Module                  Size  Used by
tcp_lp                 12663  0 
vfat                   17461  1 
fat                    65950  1 vfat
fuse                   91880  3 
xt_CHECKSUM            12549  1 
ipt_MASQUERADE         12678  3 
nf_nat_masquerade_ipv4    13412  1 ipt_MASQUERADE
tun                    31665  1 
devlink                42368  0 
ip6t_rpfilter          12595  1 
ipt_REJECT             12541  4 
nf_reject_ipv4         13373  1 ipt_REJECT
ip6t_REJECT            12625  2 
nf_reject_ipv6         13717  1 ip6t_REJECT
xt_conntrack           12760  14 
ip_set                 40680  0 
nfnetlink              14490  1 ip_set
ebtable_nat            12807  1 
ebtable_broute         12731  1 
bridge                146976  1 ebtable_broute
stp                    12976  1 bridge
llc                    14552  2 stp,bridge
ip6table_nat           12864  1 
nf_conntrack_ipv6      18935  8 
nf_defrag_ipv6         35104  1 nf_conntrack_ipv6
nf_nat_ipv6            14131  1 ip6table_nat
ip6table_mangle        12700  1 
ip6table_security      12710  1 
ip6table_raw           12683  1 
iptable_nat            12875  1 
nf_conntrack_ipv4      15053  8 
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
nf_nat_ipv4            14115  1 iptable_nat
nf_nat                 26787  3 nf_nat_ipv4,nf_nat_ipv6,nf_nat_masquerade_ipv4
nf_conntrack          133053  7 nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_conntrack_ipv6
iptable_mangle         12695  1 
iptable_security       12705  1 
iptable_raw            12678  1 
ebtable_filter         12827  1 
ebtables               35009  3 ebtable_broute,ebtable_nat,ebtable_filter
ip6table_filter        12815  1 
ip6_tables             26912  5 ip6table_filter,ip6table_mangle,ip6table_security,ip6table_nat,ip6table_raw
iptable_filter         12810  1 
sunrpc                353310  1 
dm_mirror              22289  0 
dm_region_hash         20813  1 dm_mirror
dm_log                 18411  2 dm_region_hash,dm_mirror
dm_mod                123936  2 dm_log,dm_mirror
intel_powerclamp       14419  0 
coretemp               13444  0 
intel_rapl             19542  0 
snd_hda_codec_hdmi     52231  1 
snd_hda_codec_realtek    97691  1 
uas                    22442  0 
snd_hda_codec_generic    73855  1 snd_hda_codec_realtek
kvm                   578558  0 
usb_storage            66636  2 uas
snd_hda_intel          40122  4 
irqbypass              13503  1 kvm
crc32_pclmul           13133  0 
iTCO_wdt               13480  0 
iTCO_vendor_support    13718  1 iTCO_wdt
snd_hda_codec         135389  4 snd_hda_codec_realtek,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_hda_intel
ghash_clmulni_intel    13273  0 
ftdi_sio               52791  0 
mei_wdt                13426  0 
aesni_intel           189415  0 
snd_hda_core           85885  5 snd_hda_codec_realtek,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_hda_codec,snd_hda_intel
lrw                    13286  1 aesni_intel
gf128mul               15139  1 lrw
snd_hwdep              13608  1 snd_hda_codec
glue_helper            13990  1 aesni_intel
ablk_helper            13597  1 aesni_intel
cryptd                 20511  3 ghash_clmulni_intel,aesni_intel,ablk_helper
snd_seq                62785  0 
snd_seq_device         14356  1 snd_seq
sg                     40721  0 
snd_pcm               101643  4 snd_hda_codec_hdmi,snd_hda_codec,snd_hda_intel,snd_hda_core
lpc_ich                21086  0 
pcspkr                 12718  0 
i2c_i801               22550  0 
shpchp                 37047  0 
snd_timer              29810  2 snd_pcm,snd_seq
snd                    83311  18 snd_hda_codec_realtek,snd_hwdep,snd_timer,snd_hda_codec_hdmi,snd_pcm,snd_seq,snd_hda_codec_generic,snd_hda_codec,snd_hda_intel,snd_seq_device
mei_me                 32848  0 
soundcore              15047  1 snd
mei                    91099  3 mei_wdt,mei_me
ip_tables              27126  5 iptable_security,iptable_filter,iptable_mangle,iptable_nat,iptable_raw
xfs                  1003971  2 
libcrc32c              12644  3 xfs,nf_nat,nf_conntrack
sd_mod                 46322  6 
crc_t10dif             12912  1 sd_mod
crct10dif_generic      12647  0 
ata_generic            12923  0 
pata_acpi              13053  0 
i915                 1639910  19 
iosf_mbi               14990  2 i915,intel_rapl
drm_kms_helper        177166  1 i915
syscopyarea            12529  1 drm_kms_helper
sysfillrect            12701  1 drm_kms_helper
sysimgblt              12640  1 drm_kms_helper
fb_sys_fops            12703  1 drm_kms_helper
drm                   397988  7 i915,drm_kms_helper
ata_piix               35052  3 
igb                   210385  0 
e1000e                248587  0 
libata                243070  3 pata_acpi,ata_generic,ata_piix
crct10dif_pclmul       14307  1 
crct10dif_common       12595  3 crct10dif_pclmul,crct10dif_generic,crc_t10dif
dca                    15130  1 igb
i2c_algo_bit           13413  2 igb,i915
crc32c_intel           22094  1 
i2c_core               63151  6 drm,igb,i915,i2c_i801,drm_kms_helper,i2c_algo_bit
ptp                    19231  2 igb,e1000e
pps_core               19057  1 ptp
video                  24538  1 i915
James Brigman's picture Newbie 10 points
17 October 2018 9:33 PM

BTW: My thinking here is that RHEL might be loading an inappropriate USB driver. The bottom of the page (link I sent to you) has a nice little process for troubleshooting what drivers are getting loaded and used:

•Load the driver using insmod.

•List the loaded modules using lsmod.

•Unload the driver using rmmod.

Jamie Bainbridge's picture Red Hat Guru 7257 points
17 October 2018 10:42 PM

To test if this is a later kernel/driver change, you could try ELRepo's kernel-ml which is the latest upstream compiled for RHEL.

If that works, there could be some later kernel change we can backport to the earlier RHEL kernel to improve things for you.

If you have a RHEL support entitlement, feel free to log a support case for further investigation too.

JH Community Member 45 points
18 October 2018 5:15 PM

It looks like one of my customers is running this same USB device successfully on RHEL 7.2, so it might be a Kernel/driver difference. I'm getting some further info on this.

JH Community Member 45 points
19 October 2018 3:57 PM

There must be a driver/kernel change in RHEL 7.5. We tested with 7.2 and things ran like a champ.

Jamie Bainbridge's picture Red Hat Guru 7257 points
19 October 2018 11:17 PM

Good work! You could try the kernels for 7.3 and 7.4 to narrow down the regression point. The kernels are listed on the release dates page.

Beyond that you'd need a support case for us to try identify the responsible patch and find a solution.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.