Issue

• Development system
• We run all processes that are normally distributed to more machines (for performance reason) on this one machine.
• The system is heavily loaded, but is running quite well for most of the time. Then suddenly the load average rushed in to the 1000-3000 rage and the machine gets unusable. When that happens even a normal ps hangs and one of our nagios shell script executes a simple ps and than hangs forever. Also the later running nagios scripts keep hanging and this way creating the huge amount of processes, but this is only because we didn't catch the hanging server fast enough. If we detect it fast enough the are not so many nagios processes, but still the machine hangs.