Remote job - SSH or Ansible Forks

Latest response

Hi All,

We are trying to increase the number parallel executions for ssh or ansible remote jobs, actually we are runing Satellite 6.4.2.

On settings > RemoteExecution there`s a parameter called Workers pool size, that we tried to increase to 300, but we could not see any difference on job performance.

Is it possible to increase the parallelism for remote jobs on Satellite 6.4.2?

Responses

Did you restart the dynflowd/foreman-tasks service? This is required after you change the pool size.

Hi Christiaan!

Yes!!!! We also tried to change Concurrency Level on job execution menu, but we could not see any difference...

There can be several causes, listing them from the most probable to least:

  • Workers pool size defines number of threads inside dynflowd that will serve the tasks. But 300 threads would be served by some real CPUs so most of the threads would IO wait for most of time. Due to this, increasing this parameter over 15 brings rather performance degradation than a gain - CPUs are already "busy enough" while there is additional thread maintenance added. I recommend trying Workers pool size 15 as max.

  • Setting Concurrency level can just throttle the concurrency. Its default value is infinite / no throttling. Using this option makes sense rather to prevent some high CPU/memory/whatever usage, if that could happen.

  • you might hit https://projects.theforeman.org/issues/28095, that is not fixed anywhere

  • optionally, you can provide (via a support case) a sosreport taken a short time after the REX job is completed. A task CSV export captured there should have sufficient data to identify performance bottleneck.

Hi Pavel, thank you for the explanation!!

I'll lower workers pool size back to 15, let's see if it helps. Our Satellite is running on a server with 128Gb of RAM and 32 CPU's, more 6 capsules with 96Gb of ram, and we really wants to use all that resource to speedup remote execution jobs... We were also observed that when a remote job is executed all taks are routed to one of capsules, is that normal? I have an opened support case 02513563.

Thank you for your help.

We were also observed that when a remote job is executed all taks are routed to one of capsules, is that normal?

There are two factors that affect what Capsule will be used: Host's subnet and if not set, then Host's Puppet Master. See https://access.redhat.com/solutions/2981131 for more.

Hi Pavel, you are right! Thank you!

After we changed "Puppet Master" parameter on clients (Hosts > All Hosts > (server) Edit > Puppet Master) we could see that REX tasks happened according to this value.