Satellite remote execution jobs fail with "stack level too deep (SystemStackError)"

Solution Verified - Updated -

Issue

  • Running a remote execution job with a long search query fails with the following errors in `/var/log/foreman/production.log:
2022-04-12T08:14:28 [I|app|14c277bc] Started POST "/api/job_invocations" for 192.0.2.11 at 2022-04-12 08:14:28 +0100
2022-04-12T08:14:28 [I|app|14c277bc] Processing by Api::V2::JobInvocationsController#create as JSON
2022-04-12T08:14:28 [I|app|14c277bc]   Parameters: {"job_invocation"=>{"inputs"=>{"perform_puppet_run"=>"true", "yum_splay_in_minutes"=>"20"}, "search_query"=>"name = host1.example.com or name = host2.example.com or name = host3.example.com or name = host4.example.com or [...]

(Notice that the search query is very long ^^ )

2022-04-12T08:14:28 [I|app|14c277bc] Authorized user admin(Admin User)
2022-04-12T08:14:29 [I|aud|14c277bc] JobInvocation (9875) create event on job_category BLA
2022-04-12T08:14:29 [I|aud|14c277bc] JobInvocation (9875) create event on description Update LE hosts
2022-04-12T08:14:29 [I|aud|14c277bc] JobInvocation (9875) create event on concurrency_level 
2022-04-12T08:14:29 [I|aud|14c277bc] JobInvocation (9875) create event on time_span 
2022-04-12T08:14:29 [I|aud|14c277bc] JobInvocation (9875) create event on execution_timeout_interval 
2022-04-12T08:14:29 [I|aud|14c277bc] JobInvocation (9875) create event on password [redacted]
2022-04-12T08:14:29 [I|aud|14c277bc] JobInvocation (9875) create event on key_passphrase 
2022-04-12T08:14:29 [I|aud|14c277bc] JobInvocation (9875) create event on remote_execution_feature_id 
2022-04-12T08:14:29 [I|aud|14c277bc] JobInvocation (9875) create event on sudo_password 
2022-04-12T08:14:29 [I|bac|14c277bc] Task {label: Actions::RemoteExecution::RunHostsJob, id: 9488c0af-1268-4610-8da6-35f47392ffd6, execution_plan_id: e77fdb23-57f6-45ae-81ae-210d9b76a2ed} state changed: planning 
2022-04-12T08:14:29 [E|bac|] stack level too deep (SystemStackError)
/opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_language/ast.rb:80:in `map'
/opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_language/ast.rb:80:in `simplify'
/opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_language/ast.rb:146:in `block in simplify'
/opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_language/ast.rb:146:in `map'
/opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_language/ast.rb:146:in `simplify'
/opt/theforeman/tfm/root/usr/share/gems/gems/scoped_search-4.1.9/lib/scoped_search/query_language/ast.rb:146:in `block in simplify'
[...]
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler.rb:441:in `block (3 levels) in start_threads'
[ concurrent-ruby ]
2022-04-12T08:14:29 [I|app|] Completed 500 Internal Server Error in 529ms (ActiveRecord: 113.9ms | Allocations: 166005)
2022-04-12T08:14:29 [F|app|]   
SystemStackError (stack level too deep):

foreman_remote_execution (3.3.7.2) app/models/targeting.rb:47:in `block in resolve_hosts!'
app/models/concerns/foreman/thread_session.rb:108:in `as'
foreman_remote_execution (3.3.7.2) app/models/targeting.rb:47:in `resolve_hosts!'
foreman_remote_execution (3.3.7.2) app/lib/actions/remote_execution/run_hosts_job.rb:31:in `plan'
foreman_remote_execution (3.3.7.2) app/lib/actions/middleware/bind_job_invocation.rb:17:in `plan'
foreman-tasks (2.0.3) app/lib/actions/middleware/keep_current_request_id.rb:10:in `block in plan'
foreman-tasks (2.0.3) app/lib/actions/middleware/keep_current_request_id.rb:31:in `with_current_request_id'
[...]

Environment

  • Red Hat Satellite 6
  • Job with very long search query

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content