Red Hat Satellite upgrade failing with foreman_tasks_sync_task_timeout error.

Solution Verified - Updated -

Environment

  • Red Hat Satellite 6.x till 6.9.z

Issue

  • foreman-rake while performing the upgrade fails with the following error:

    Timeout::Error: The time waiting for task a5990590-6f26-4123-8cb1-abad98305d96 to finish exceeded the 'foreman_tasks_sync_task_timeout' (120s)

Resolution

  • Increase the timeout of Pulp workers using the following parameters with satellite-installer:

    # satellite-installer --foreman-proxy-content-pulp-worker-timeout 3600 --katello-pulp-worker-timeout 3600

For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

Diagnostic Steps

  • Full trace of errors as reported in /var/log/foreman-installer/satellite.log:

    [root@satellite ~]# cat /var/log/foreman-installer/satellite.log
Updating Puppet Repository Distributors
Updating Content View Puppet Environment Distributors
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.3/lib/foreman_tasks.rb:35: warning: constant ::TimeoutError is deprecated
rake aborted!
Timeout::Error: The time waiting for task a5990590-6f26-4123-8cb1-abad98305d96 to finish exceeded the 'foreman_tasks_sync_task_timeout' (120s)
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.3/lib/foreman_tasks.rb:35:in `block (2 levels) in trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'  
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'   
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.3/lib/foreman_tasks.rb:25:in `block in trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.3/lib/foreman_tasks.rb:45:in `block in rails_safe_trigger_task'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/dependencies/interlock.rb:46:in `block in permit_concurrent_loads'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/concurrency/share_lock.rb:185:in `yield_shares'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/dependencies/interlock.rb:45:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.3/lib/foreman_tasks.rb:44:in `rails_safe_trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.3/lib/foreman_tasks.rb:23:in `trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.3/lib/foreman_tasks.rb:54:in `sync_task'   
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.46/lib/katello/tasks/upgrades/3.0/update_puppet_repository_distributors.rake:14:in `block (5 levels) in <top (required)>'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/delegation.rb:39:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.46/lib/katello/tasks/upgrades/3.0/update_puppet_repository_distributors.rake:13:in `block (4 levels) in <top (required)>'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `block in execute'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `each'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `execute'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:194:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:187:in `invoke_with_call_chain'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:180:in `invoke'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:152:in `invoke_task'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block (2 levels) in top_level'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `each'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block in top_level'   
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:117:in `run_with_threads'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:102:in `top_level'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:80:in `block in run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:77:in `run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/bin/rake:23:in `load'
/opt/rh/rh-ruby24/root/usr/bin/rake:23:in `<main>'

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments