Red Hat Satellite 6 repository sync pending on Actions::Pulp::Repository::RegenerateApplicability

Solution Verified - Updated -

Environment

  • Red Hat Satellite 6

Issue

  • Synchronizing a repository runs for hours without finishing
  • Task details show there is suspended subtask Actions::Pulp::Repository::RegenerateApplicability
  • In tasks list, there is a task Actions::Katello::Repository::NodeMetadataGenerate that finished with a warning
  • All Repos are stuck in Pending status.
  • Satellite hitting the below error on various tasks:
Sep 16 03:16:28 <satellite.hostname> pulp: pulp.server.event.http:WARNING: Error response from HTTP notifier: {"displayMessage":"cannot accept execution_plan_id:997f2132-7b81-4590-bc2b-6724e9811517 core is terminating","errors":["cannot accept execution_plan_id:997f2132-7b81-4590-bc2b-6724e9811517 core is terminating"]}

Resolution

For Satellite 6.1 or newer:

If the problem appeared in Satellite 6.1 or newer, please open a support case to identify the root cause.

For Satellite 6.0:

There is a bug in Satellite causing this, which is fixed in Satellite 6.1. To prevent the issue from reoccurring, it is recommended to upgrade to Satellite 6.1. If that is not possible, the steps below serve as a workaround for the issue:

To workaround :

  • First reboot the machine to force pulp to put everything in a paused state that is currently running. The reason for the reboot is there is currently a bug with restarting the katello-service and the workers not being released.

  • Then get rid of the tasks:

# systemctl stop foreman-tasks

# foreman-rake console  #(let it load all the way)
ForemanTasks::Task.where(:state => :paused).where(:label => "Actions::Katello::Repository::Sync").destroy_all
ForemanTasks::Task.where(:state => :paused).where(:label => "Actions::Pulp::Consumer::GenerateApplicability").destroy_all
ForemanTasks::Task.where(:state => :paused).where(:label => "Actions::Katello::System::GenerateApplicability").destroy_all

# systemctl start foreman-tasks

This will kill all paused tasks.

  • Install pulp-admin by following this solution

  • Then run this script to make sure all pulp tasks are killed:

#!/bin/bash

pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)

STATE=""
for TASK in `pulp-admin -u admin -p $pulpAdminPassword tasks list | egrep '^Task Id:|^State:' | sed -e 's,^Task Id: ,,' -e 's,^State: ,,'`; do
        if [ "$STATE" = "" ]; then
                STATE=$TASK
        else
                if [ $STATE != Successful ] && [ $STATE != Cancelled ] && [ $STATE != Failed ]; then
                        pulp-admin -u admin -p $pulpAdminPassword tasks details --task-id=$TASK
                        pulp-admin -u admin -p $pulpAdminPassword tasks cancel --task-id=$TASK
                fi
                STATE=""
        fi
done
  • Reboot again and then do the following to clear orphaned entries in katello database and reindex tables:
# foreman-rake katello:delete_orphaned_content --trace
# foreman-rake katello:reindex --trace

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.