Red Hat Satellite 6 repository sync pending on Actions::Pulp::Repository::RegenerateApplicability

Solution Verified - Updated -

Environment

  • Red Hat Satellite 6

Issue

  • Synchronizing a repository runs for hours without finishing
  • Task details show there is suspended subtask Actions::Pulp::Repository::RegenerateApplicability
  • In tasks list, there is a task Actions::Katello::Repository::NodeMetadataGenerate that finished with a warning
  • All Repos are stuck in Pending status.
  • Satellite hitting the below error on various tasks:
Sep 16 03:16:28 <satellite.hostname> pulp: pulp.server.event.http:WARNING: Error response from HTTP notifier: {"displayMessage":"cannot accept execution_plan_id:997f2132-7b81-4590-bc2b-6724e9811517 core is terminating","errors":["cannot accept execution_plan_id:997f2132-7b81-4590-bc2b-6724e9811517 core is terminating"]}

Resolution

For Satellite 6.1 or newer:

If the problem appeared in Satellite 6.1 or newer, please open a support case to identify the root cause.

For Satellite 6.0:

There is a bug in Satellite causing this, which is fixed in Satellite 6.1. To prevent the issue from reoccurring, it is recommended to upgrade to Satellite 6.1. If that is not possible, the steps below serve as a workaround for the issue:

To workaround :

  • First reboot the machine to force pulp to put everything in a paused state that is currently running. The reason for the reboot is there is currently a bug with restarting the katello-service and the workers not being released.

  • Then get rid of the tasks:

# systemctl stop foreman-tasks

# foreman-rake console  #(let it load all the way)
ForemanTasks::Task.where(:state => :paused).where(:label => "Actions::Katello::Repository::Sync").destroy_all
ForemanTasks::Task.where(:state => :paused).where(:label => "Actions::Pulp::Consumer::GenerateApplicability").destroy_all
ForemanTasks::Task.where(:state => :paused).where(:label => "Actions::Katello::System::GenerateApplicability").destroy_all

# systemctl start foreman-tasks

This will kill all paused tasks.

  • Install pulp-admin by following this solution

  • Then run this script to make sure all pulp tasks are killed:

#!/bin/bash

pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)

STATE=""
for TASK in `pulp-admin -u admin -p $pulpAdminPassword tasks list | egrep '^Task Id:|^State:' | sed -e 's,^Task Id: ,,' -e 's,^State: ,,'`; do
        if [ "$STATE" = "" ]; then
                STATE=$TASK
        else
                if [ $STATE != Successful ] && [ $STATE != Cancelled ] && [ $STATE != Failed ]; then
                        pulp-admin -u admin -p $pulpAdminPassword tasks details --task-id=$TASK
                        pulp-admin -u admin -p $pulpAdminPassword tasks cancel --task-id=$TASK
                fi
                STATE=""
        fi
done
  • Reboot again and then do the following to clear orphaned entries in katello database and reindex tables:
# foreman-rake katello:delete_orphaned_content --trace
# foreman-rake katello:reindex --trace

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

15 Comments

ran the reindex and I am getting the following error :

rake aborted!
undefined method `merge' for nil:NilClass

I then reran with the --trace flag and got the following

# foreman-rake katello:reindex --trace
** Invoke katello:reindex (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke katello:reset_backends:elasticsearch (first_time)
** Invoke environment
** Execute katello:reset_backends:elasticsearch
Elasticsearch Indices cleared.
** Execute katello:reindex
Re-indexing User
Re-indexing Organization
Re-indexing Katello::ActivationKey
Re-indexing Katello::TaskStatus
Re-indexing Katello::ContentView
Re-indexing Katello::ContentViewFilter
Re-indexing Katello::Repository
rake aborted!
undefined method `merge' for nil:NilClass
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue/pulp/repo.rb:94:in `to_hash'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:195:in `block in bulk'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:176:in `map'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:176:in `bulk'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:273:in `bulk_store'
/opt/rh/ruby193/root/usr/share/gems/gems/tire-0.6.2/lib/tire/index.rb:300:in `import'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/lib/katello/tasks/reindex.rake:19:in `block (3 levels) in <top (required)>'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/lib/katello/tasks/reindex.rake:7:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/lib/katello/tasks/reindex.rake:7:in `block (2 levels) in <top (required)>'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `call'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `block in execute'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `each'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `execute'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:in `block in invoke_with_call_chain'
/opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:in `invoke_with_call_chain'
/opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:in `invoke'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:in `invoke_task'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block (2 levels) in top_level'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `each'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block in top_level'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in `standard_exception_handling'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:in `top_level'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:in `block in run'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in `standard_exception_handling'
/opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:in `run'
/opt/rh/ruby193/root/usr/bin/rake:32:in `<main>'
Tasks: TOP => katello:reindex

any suggestions would be helpful.

It seems that the problem is not fixed in 6.1.1 and the issue was raised also on https://bugzilla.redhat.com/show_bug.cgi?id=1203050 . Does anyone has an workaround for this issues because right now we cannot sync our repos?

Thanks

I have same Problem in 6.1.1. Any help is appreciated.

Hello,

even with --trace parameter the "foreman-rake katello:delete_orphaned_content --trace" will execute in background. It is possible to check status (finish) in /var/log/messages. Its something like:

pulp: celery.worker.job:INFO: Task pulp.server.managers.content.orphan.delete_all_orphans[8a4e52cf-fa3e-4f15-8e17-04bb472216d7] succeeded in 90.118128971s: None

Now I get the following? rake aborted! Please install the sqlite3 adapter: gem install activerecord-sqlite3-adapter (Could not find sqlite3 (~> 1.3.5) amongst [actionmailer-3.2.8, actionpack-3.2.8, ........

Same problem here, using Satellite 6.1.8

Following the steps listed above has not affected anything.

it happened again after update with latest patches on May 30, 2016.

The article clearly states 6.1 or higher log a call. I am using 6.2.11 and 6.2 was logged in the call. This article should not have been presented to waste my time.

foreman-rake katello:reimport --trace

Having the same problem on 6.3.1

Do you really see the error/warning message Error response from HTTP notifier: {"displayMessage":"cannot accept execution_plan_id:997f2132-7b81-4590-bc2b-6724e9811517 core is terminating"..? Or is the reg.app. task "just" taking tooo much time (where I would suspect https://bugzilla.redhat.com/show_bug.cgi?id=1573892)?

Thanks. In my case was because the satellite server subscription expired.

I ran into this problem using satellite 6.4, still evaluating what is wrong. I will answer to this as I workaround the problem.

Maybe pulp workers are too busy or somehow stuck or missing?

To check if busy: trye e.g. pulp-admin tasks list (or qpid-stat -q --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 resource_manager and queue depth says number of waiting tasks).

Stuck or no workers: probably /var/log/messages (where also pulp writes to) has some hint.

Pavel I have workarounded this issue before receiving your message, but I think you are right.

After cleaning all tasks by using a processess very similar to the above I was able to get my syncs tasks back on track.

thanks!!!!