Consolidation of apps onto one server with multiple nics

Latest response

Hello,
I am looking for information on best practice/approach. Recently we have consolidated 9 different applications that were running on their own respective workstations/servers. Now all the applications are hosted on one main RHEL server with 6 active network interfaces.

So having said all of that, My question is regarding RedHat server tweek or tuning ... Is there a way to tune Redhat to know that all these apps are hosted and not have to send the information down the wire and back again through another interface ? I am by no means a programmer and if this falls in the realm of rehashing application(s) so be it, but then again if I am able to define something on the server maybe in NetworkManager or kernel ... I am willing to learn and try.
Thanks.

Responses

Hello Bob Appelzoller,

The answer to your question may not be as easy as this forum may immediately offer through at minimum one mere response. I wish you and your server every success and hopefully, none of the issues I mention below will haunt you.

Here are some initial thoughts though...

  • Regarding segregating the network traffic for each service, there is some chance that if the traffic originated on a given NIC, hopefully, it might continue on that, but I note your concern, and your actual mileage may vary, and the usual "murphy's law disclaimers apply". This may be more of a manual process than you desire, and there may not be an off-the-shelf answer that is quick to apply to your immediate need. It may require some focused attention to running tcpdumps with your friendly-neighborhood on-staff Linux-administrator that you currently have in your employment or on-hand in some fashion. There is some chance you may need a round of firewalld rules that mimic what you would expect in IPTABLES, however, not knowing the specific 9 services you consolidated on your single system, this is speculation on my part, and there can be no quick answer that can resolve your immediate need.
  • In the EPEL repositories is a wonderful rpm named "iftop"... "It listens to network traffic on a named interface and displays a table of current bandwidth usage by pairs of hosts. Handy for answering the question "why is our ADSL link so slow?"."
  • Application logging is generally an afterthought, and consequently /var or /var/log gets overwhelmed and crashes things when it does (or has the potential to do so). I don't know the partition layout you have, and hopefully, in the consolidation of 9 applications into one server, there's been some sane partitioning in order to protect "/" and "/var" from filling up. Often, when systems are built, some neglect making a separate partition for "/var/log" and even "/var/lib/[name_of_some_service_such_as_docker_for_example]" and /var/log and /var/lib get overwhelmed with logs and or service data respectively.
  • It may be good to also run iotop which is also in EPEL on your server and even have "htop", again from EPEL.
  • I'd make sure there is a sane directive file in /etc/logrotate.d/directory that ought to have a file for each and every service if it is not there already. SOME services put their logging elsewhere based on a configuration file (namely third-party non-Red-Hat software). The way you'll see the issue, is at the most inopportune time, /var/log will fill up with one or more log files and the corresponding service (or another that needs /var/log) may or may not crash.
  • docker is often a culprit to consume all of /var/lib if there is not a well-defined and well-thought-out /var/lib/docker partition in existence already.
  • If you are wondering what is excessively using a particular partition, you may want to run lsof /name/of/partition - but this will not help you if you happen to have third-party services running on bare directories that are actually part of the "/" partition and not their own partition.
  • Services that are stood up in a hurry may suffer the problem where someone inappropriately placed service data (databases, data from services, etc) on partitions hanging on "/" itself and not their own individual partitions which if untamed and not resolved will fill up "/" again at most likely the most inopportune time. I'd recommend checking to make sure your nine services running consolidated on one server are not writing their data on a directory that is really hanging off of "/". One example, if you have a service with "/someservicename" - you can run mountpoint /someservicename and if it says "/service-name is not a mountpoint", then I'd recommend checking over some time to make sure your service is not filling up "/" which will crash your server. If you have nine applications on one server, this potential rises depending on the care, speed, and ability for thoughtfulness and quality is given in establishing a server designed to handle nine services long-term.

Going through each of your services and establishing them for best-practices may take some time. Consolidating nine services to one server... can obviously be done, but I'd recommend (not to be terribly obvious) someone monitor the system and its services for health for some time, to include rounds of systemctl is-system-running || systemctl | grep fail for example, and monitoring logs, disk consumption several times a week to note dangerous trends in disk consumption to avert panic maintenance when one or more services have failed.

Kind Regards,
RJ