TCP / IP communication of RHEL 7 is extremely slow compared with RHEL 6 and earlier.

Latest response

TCP / IP communication of RHEL 7 is extremely slow compared with RHEL 6 and earlier.
My application transferd from RHEL 5 to RHEL 7.
That application makes TCP / IP communication with another application on the same server.
That communication speed is about twice slower.

Apart from that, we compared it with "ping localhost". RHEL 7 averaged 0.04 ms, RHEL 6 and RHEL 5 average 0.02 ms. RHEL 7 is twice as slow as RHEL 6 or earlier.

The environment is the minimum installation, stop firewalld and postfix, then do "ping localhost".

Why was communication delayed like this?
Or, what is going on late?
Is not it worth it?

Responses

Hi Masamitsu,

As most members speak English here, I recommend to post in English. That rises the chance to receiving responses. Thank you ! :)

Regards,
Christian

Hi Christian,

Thank you. advice.

It was rewritten in English.

Regards, Masamitsu

Generally RHEL 7 should be faster than earlier versions, though it depends exactly what your application does.

I think there would be too many differences between RHEL versions and the systems to compare this easily with a small test. For example, I cannot reproduce your above ping result, and ICMP handling is different to TCP handling anyway.

Focus on the actual behaviour of the application, not arbitrary tests.

There has been a performance hit in many-system-call workloads with the security fixes for Spectre and Meltdown. If your application makes many small syscalls causing a lot of context switches, this could be a reason for a performance decrease.

The MTU size of the loopback interface has increased between RHEL 5 and RHEL 7, so your application could just be sending more data at once. This might appear as slightly longer reads and writes but the throughput is better overall.

Thank you for your answer.

I think so, too.

I think there would be too many differences between RHEL versions and the systems to compare this easily with a small test.

Therefore, I abandon the resolution of the performance problem of my application once.

Even ping command of ICMP, since there was a similar speed difference between RHEL 5,6 and RHEL 7, we would like to know the cause of this problem first.

Is there guess?

Guesses: differences in process scheduling, memory fragmentation, other CPU workload, timing inaccuracy, incorrect test method, firewall behaviour, system performance differences, code difference like the security vulnerability mentioned above, probably much more that I have not thought of.

A good troubleshooting path forward is to identify:

  • the specific behaviour in your application which is different
  • what you expect performance to be
  • what performance measurement you are currently getting

And then look into possible causes. I would start with perf collection during an application run, and possibly strace of the application although that can negatively affect performance too.

There are some more questions to give ideas at Initial investigation for any performance issue.

I see you have "L3 support" through your hardware vendor, possibly you bought RHEL pre-installed on your system, so the hardware vendor's tech support would be the first place to ask. The vendor will contact us if they identify a bug in RHEL.

One side note, make sure you really have your dns resolver /etc/resolv.conf set properly. The suggestions above are of course indeed good, but if your dns is not set properly, you'll have another round of slowness. Remember that /etc/resolv.conf is populated generally from the "DNSx" and "DOMAIN" directives found in the active '/etc/sysconfig/netowrk-scripts/ifcfg-XYZ" file. You can find what interfaces are actually active by using ip -o -4 a s which will reveal all IPV4 active interfaces with the interface name in the results at the very far left.

There are instances where if you have a system that is doing a lot of actions that rely on dns, you could make a dns caching server at your location that would assist with lookups and cache relevant things for your system.

Again, the other answers above are very useful, on spot, but if your /etc/resolv.conf is off, or not optimal, it could cause issue.

Another thing to review, and yes, it is exhaustive, the Red Hat tuning guide would be a good reference to double-check.

One method to test network bandwidth and latency performance is here.

I have not fully vetted this article where someone did some additional tuning and it would be good to validate what is in that article for legitimacy, and make backups of any configurations before making changes.

One last thing, using the rpm iftop can give you an idea of what systems are hitting your server, or visa versa.

Regards,

RJ

and yes, it is exhaustive, the Red Hat tuning guide

For reference, the Network Performance Tuning Guide PDF is only the original publish. We have updated the knowledgebase article a couple of times since then:

I have not fully vetted this article

Using tuned is a good idea... if the tuning profile matches your use case. Users are encouraged to think of the shipped profiles as just a starting point and develop their own more customised tuning profile.

An overview of the default profiles is at: https://access.redhat.com/solutions/369093

Jamie Bainbridge, R. Hinton. ,

Thanks for the advice.

I contacted my hardware support vendor, but I refused it as not being able to support because there was no error. Therefore I used here as another means.

I would like to check the DNS setting of my server.

My system used "/ etc / hosts" without DNS name resolution until RHEL 5, 6. I would like to do that also in RHEL 7.

I will investigate the following items from now. - / etc / hosts - /etc/host.conf - /etc/resolv.conf - nmcli - NetworkManager

I still have a lack of study around here.

If possible, please support again.

Regards, M. Nohara

The operation of the static hosts file /etc/hosts is the same between RHEL5 and RHEL7.

You can probably just copy your entries over and paste them on the bottom of the new file. Make sure you have both an IPv4 and and IPv6 localhost entry in this file always. eg:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
## your entries below ##

Masamitsu Nohara

Generally when I hear people describe a slow network, I usually initially wonder if their /etc/resolv.conf file is correctly configured to use a DNS server. If you want a slow system, don't configure your /etc/resolv.conf file. Now Some of the configurations are supposed to come from a properly configured network interface file, but if those are not configured properly, you'll not have a properly configured /etc/resolv.conf file, and then your expectation and ability to use the network will be diminished to some degree.

Do you have your /etc/resolv.conf populated with servers to provide dns to your system? Just curious. From what you are describing, I'm not sure if you do or not. If you do not, you will experience a slow network. Let's say you want to go to Red Hat's website or google. You need the IP address. But who memorizes IP addresses? Generally nobody. So that's where pointing your system to a DNS server is needed by populating the /etc/resolv.conf file with the proper directives in order to enable your system to access the Internet, or your local network.

The purpose of DNS is to see this link

(from link above)

Domain Name Servers (DNS) are the Internet's equivalent of a phone 
book. They maintain a directory of domain names and translate them 
to Internet Protocol (IP) addresses.

This is necessary because, although domain names are easy for 
people to remember, computers or machines, access websites 
based on IP addresses. 

Information from all the domain name servers across the Internet are 
gathered together and housed at the Central Registry. Host companies 
and Internet Service Providers interact with the Central Registry on a 
regular schedule to get updated DNS information. 

When you type in a web address, e.g., www.jimsbikes.com, your Internet 
Service Provider views the DNS associated with the domain name, 
translates it into a machine friendly IP address (for example 
216.168.224.70 is the IP for jimsbikes.com) and directs your Internet 
connection to the correct website. 

The purpose of /etc/resolv.conf is to point to DNS servers your system can access and therefore be able to find things on a network or on the Internet. http://man7.org/linux/man-pages/man5/resolv.conf.5.html

It is important to know the difference and role between the two files /etc/hosts and /etc/resolv.conf.

Regards

RJ

R. Hinton.

Thank you for your kind explanation.

My system is a production system in the factory local network. The system communicates own server or other server with a fixed IP address. For that, we did not use /etc/resolv.conf until RHEL 6. We only used the static host file /etc/hosts. In RHEL 7 as well, that does not use DNS is unchanged.

In response to the your advice I tried the following. · Comment out setting in /etc/resolv.conf # Nameserver xxx.xxx.xxx.xxx · /etc/NetworkManager/NetworkManager.conf plugins = ifcfg-rh dns = none · Clear DNS server configuration with nmcli · NetworkManager and network restart.

However, ping 's response was unchanged as follows.

ping localhost

64 bytes from localhost (127.0.0.1): imcp_seq = 1 ttl = 64 time = 0.013 ms 64 bytes from localhost (127.0.0.1): imcp_seq = 2 ttl = 64 time = 0.021 ms 64 bytes from localhost (127.0.0.1): imcp_seq = 3 ttl = 64 time = 0.022 ms 64 bytes from localhost (127.0.0.1): imcp_seq = 4 ttl = 64 time = 0.041 ms 64 bytes from localhost (127.0.0.1): imcp_seq = 5 ttl = 64 time = 0.036 ms 64 bytes from localhost (127.0.0.1): imcp_seq = 6 ttl = 64 time = 0.021 ms 64 bytes from localhost (127.0.0.1): imcp_seq = 7 ttl = 64 time = 0.034 ms 64 bytes from localhost (127.0.0.1): imcp_seq = 8 ttl = 64 time = 0.022 ms 64 bytes from localhost (127.0.0.1): imcp_seq = 9 ttl = 64 time = 0.021 ms 64 bytes from localhost (127.0.0.1): imcp_seq = 10 ttl = 64 time = 0.033 ms

By the way, the ping response in RHEL 6 is as follows. There was a difference of 1.5 times between RHEL 7 above and RHEL 6 below.

ping localhost

PING localhost (127.0.0.1) 56 (84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_seq = 1 ttl = 64 time = 0.010 ms 64 bytes from localhost (127.0.0.1): icmp_seq = 2 ttl = 64 time = 0.017 ms 64 bytes from localhost (127.0.0.1): icmp_seq = 3 ttl = 64 time = 0.028 ms 64 bytes from localhost (127.0.0.1): icmp_seq = 4 ttl = 64 time = 0.016 ms 64 bytes from localhost (127.0.0.1): icmp_seq = 5 ttl = 64 time = 0.019 ms 64 bytes from localhost (127.0.0.1): icmp_seq = 6 ttl = 64 time = 0.018 ms 64 bytes from localhost (127.0.0.1): icmp_seq = 7 ttl = 64 time = 0.017 ms 64 bytes from localhost (127.0.0.1): icmp_seq = 8 ttl = 64 time = 0.017 ms 64 bytes from localhost (127.0.0.1): icmp_seq = 9 ttl = 64 time = 0.018 ms 64 bytes from localhost (127.0.0.1): icmp_seq = 10 ttl = 64 time = 0.016 ms

Besides, I tried invalidating ipv6, but it did not change too.

It is a stuck situation. Is there anything you notice?

Regards, M.Nohara