Rhel services issue

Latest response

I am trying to resolve an issue with a Rhel 6.9 system. The system is provided as a solution by a vendor in an isolated enclave, so the vendor has been providing application updates and rpm updates periodically. We host their software on our servers.

The system in question acts as the gateway, and is running on a Dell R310 server. The issue occurs after it has been running for about 4 hours. If I run the command

$ sudo service --status-all 

the command takes a few seconds to start, takes about 60 seconds to run, and about 45 seconds is spent displaying the iptables rules.

If I run the command

$ sudo service iptables status

the command takes a few seconds to start, and then immediately displays the iptables rules.

If I reboot the system and repeat the commands, they respond immediately. Something starts failing as it approaches 4 hours. Strangely, this issue first occured in June a year ago. The vendor came out and reinstalled the software. It worked fine until this June, when the issue started again. The vendor said they think it might be hardware related, but I'm not sure. I ran diagnostics when the issue happened last year, and it indicated no issues with the hardware. I don't like coincidences, and it seems strange to me that it happened in June this year and last.

I've tried comparing before and after sos reports, but there is so much data there it's hard to correlate the differences. I have never experienced this issue on a rhel system before. Does anyone have a suggestion on how to isolate the issue?

Responses

When the server has been running for more than 4 hours and the problem is in effect, try running "iptables -L -vn". Does it run fast? Then try running "iptables -L -v". Does it run a lot slower?

If that is the case, it suggests a hostname resolution issue. Specifically, it sounds like the first nameserver listed in /etc/resolv.conf might not be responding properly.

Also check /etc/hosts: ideally, you should have all IP addresses configured to this system and their corresponding hostnames (in both FQDN & short forms) defined in there, so that the system can know its own name(s) in any situation, without accessing any DNS server for such basic information.

(Of course, this is assuming that you have a static IP address; if your system gets its IP address by DHCP, then you may have to rely on a DNS server for getting the server's current IP resolved to its hostname.)

If the first "iptables -L -vn" command is slow and the second "iptables -L -v" is fast, then the system might have been paging out a lot of stuff - check swap usage with "top" or "swapon -s" and consider if the system might need more RAM.

Or perhaps old data is not being purged regularly, and the system ends up processing and re-processing more and more data (consuming increasing amounts of RAM even if the amount of incoming data stays constant) as the time goes by, and one year from installation just happens to be the time when the system starts showing symptoms of being nearly at capacity. The reinstallation might clear all the old data and "reset the clock". If that's the case, you need to find out what's accumulating and decide what to do with it - archive it somehow so it won't get needlessly re-processed, or perhaps just erase it if it is no longer needed.

The command "iptables -L -v" runs much slower.

I looked at /etc/resolv.conf, and the nameserver is itself, 127.0.0.1. I'll have to work with the vendor, with an emphasis on the dns.

Part of the system has static IP addresses, and some of the servers have dhcp, so there is a mix.

Thank you for the reply.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.