Actions::Pulp::Repository::RegenerateApplicability (DynFlow) stuck at 63% for disconnected Satellite Content View Import

Latest response

Issue: Actions::Pulp::Repository::RegenerateApplicability in DynFlow fails at 63% for various channels upon ingest on disconnected satellite with a KNOWN-GOOD Content view export being used to import into (again) a disconnected Satellite Server.

I'm using a content view import method that Red Hatter Rich Jerrido made which generally works magnificently (the documentation has one do one channel at a time which is terribly tedious).

I have one public facing Satellite that acquires (successfully) all channels. I then use (link will come later) the method Rich Jerrido made to export as a Default Organizational View Content View of all applicable channels for my disconnected satellite servers.

So far it works great. Now I have one (at the time of this writing) Satellite 6.2.11 disconnected satellite that didn't ingest the Content view correctly (but the IDENTICAL Content view on a DIFFERENT 6.2.11 disconnected satellite worked fine, so the issue is ==not== the Content View export).

Now I opened this discussion even though at this link https://access.redhat.com/discussions/2065363 there's a related discussion, but it's different enough to warrant this new discussion.

In that discussion I linked to above, is one small comment by Stephen Wadeley of Red Hat who mentioned the issue I'm facing which is a bit atypical... "..but not always. See Red Hat Satellite6 repository sync pending on Actions::Pulp::Repository::RegenerateApplicability on the Red Hat Customer Portal."

That specific solution for Satellite version 6.0 has one do some "fun" things to kill off pulp tasks, then clear orphaned entries. However the solution ID says to call Red Hat and submit a ticket if you are running Satellite 6.1 or higher.

I submitted that case with Red Hat, but am posting this for community reference, and will update it further.

Here's more of my own conditions:

There are 11 channels left in this content repository that have hung at 63% since last week.
Reboots do not help, the solution IDs I've found have not resolved this.

The 11 channels affected are:
Red Hat Enterprise Linux Workstation
     ->Red Hat Enterprise LInux 7 Workstation - Optional RPMs x86_64 7Workstation  EDITED (New Packages 35 (335 MB)
     ->Red Hat Enterprise LInux 7 Workstation - RH Common RPMs x86_64 7Workstation EDITED "No new packages"
     ->Red Hat Enterprise LInux 7 Workstation - Supplementary RPMs X86_64 7Workstation EDITED "No new packages"
     ->Red Hat Enterprise LInux 7 Workstation RPMs x86_64 7Workstation EDITED (New Packages 232 (572 MB)
        ->Red Hat Enterprise LInux 7 Workstation - Extras RPMs x86_64 EDITED "No new packages"
        ->Red Hat Satellite Tools 6.2 for RHEL 7 Workstation RPMs x86_64 EDITED (New Packages 8 (435KB)

Oracle Java for RHEL Workstation
   -> Red Hat Enterprise Linux 7 Workstation - Oracle Java RPMs x86_64 7Workstation EDITED "No new packages"

Red Hat Enterprise LInux Server
     -> Red Hat Enterprise Linux 6 Server - RH Common RPMs x86_64 6Server EDITED "No new packages"
     -> Red Hat Enterprise Linux 6 Server - Supplementary RPMs x86_64 6Server EDITED "No new packages"

7Server
   -> RHN Tools for Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server EDITED "No new packages"
   -> Red Hat Satellite Tools 6.2 for RHEL 7 Server RPMs x86_64 EDITED "No new packages"

It is still hung on 63%  Here's an example DynFlow output:

################################## DynFlow output

Go to the "Run" tab in Dynaflow
I see the following:
"3: Actions::Pulp::Repository::Sync(success){time-in-seconds]"
"6: Actions::Katello::Repository::IndexContent (success) [time-in-seconds]"
"14: Actions::Katello::REpsitory::ErrataMail (success) [time-in-seconds]"
"17: Actions::PUlp::Repository::RegenerateApplicability (suspended) [350626.28s/2095.07s]"   SMOKING GUN?????????????
"18: Actions::Katello::Repository::Sync (pending)"  ("pending" probably because the line above is still in "suspended state")
"20: Actions::Katello::Repository::ImportApplicability (pending)"  ("pending" again, probably because the thing 2 lines up is in a "suspended" state)

Note: this is pretty much what I get for all failed channels listed earlier, the only thing that differs is the "pulp_id", the "group_id" and "total" under "Actions::Katello::Repository::ERegenerateApplicability (suspended) portion.

Also here's some more...

If you are facing this issue, run on your satellite
katello-service stop
then
katello-service start
This places a list of applicable satellite services into an array running systemctl stop [@array1]  then systemctl start [@array1]  (where "@array1" is really the applicable service in question. )

The recommendation after was to run
        katello-service status 

That takes the same array and runs systemctl status @array1 and so forth.  

in ==my== specific case, this seems likely to be a related issue, this one service that was not functioning compared to my known-good satellite server.

For pulp_resource_manager.service - Pulp Resource Manager, I see the following that is not consistent with my FUNCTIONING satellite server:

[root@redacted.someplace.com] # katello-service status (OUTPUT IS ISOLATED TO pulp_resource_manager.service]
● pulp_resource_manager.service - Pulp Resource Manager
   Loaded: loaded (/usr/lib/systemd/system/pulp_resource_manager.service; enabled; vendor preset: disabled)
   Active: active (running) since MOn 2017-11-20 15:09:10 EST; 2min 16s ago
 Main PID: 22352 (celery)
   CGroup: /system.slice/pulp_resource_manager.service
           ├─2047 /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid --heartbeat-interval=30
           └─2937 /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid --heartbeat-interval=30
Nov 20 15:90:12 redacted.someplace.com celery[22352]: - ** ---------- .> transport     qpid://redacted.someplace.com:5671//
Nov 20 15:90:12 redacted.someplace.com celery[22352]: - ** ---------- .>  disabled
Nov 20 15:90:12 redacted.someplace.com celery[22352]: - *** --- * --- .>  concurrency: 1 (prefork)
Nov 20 15:90:12 redacted.someplace.com celery[22352]:  -- ******* ----
Nov 20 15:90:12 redacted.someplace.com celery[22352]:  --- ***** ----- [queues]
Nov 20 15:90:12 redacted.someplace.com celery[22352]:  -------------- .>  resource_manager exchange=resource_manager(direct)key=resource_manager@redacted.someplace.com
Nov 20 15:90:12 redacted.someplace.com pulp[22352]: kombu.transport.qpid:info: Connected to qpid with SASL mechanism ANONYMOUS
Nov 20 15:90:12 redacted.someplace.com pulp[22352]: celery.worker.consumer:INFO: Connected to qpid with SASL mechanism ANONYMOUS

This seems to follow other issues I've noted with journalctl output:
Nov 20 09:54:53 redacted.someplace.com pulp[1310]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.service.controllers.repository.queue_download_deferred)
Nov 20 09:54:53 redacted.someplace.com pulp[1460]: celery.worker.strategy:INFO Received task: pulp.server.controllers.repository.queue_download_deferred[b40a71ae-f0f1-488d-b680-57aedbc61041]
Nov 20 09:54:53 redacted.someplace.com pulp[2207] kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Nov 20 09:54:53 redacted.someplace.com pulp[1448]: celery.worker.strategy:INFO Received task: pulp.server.controllers.repository.download_deferred[a2c11b76-0cf9-4ac-8c69-8691b5ff76b7]
Nov 20 09:54:53 redacted.someplace.com pulp[2207]: py.warnings:WARNING: (2207-75936) /usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py:536: DeprecationWarning: update is deprecated. use replace_one, update_one or update_many instead.

I also see pulp errors about MongoClient opened before fork, and it says to "Create MongoClient with connect=False, or create client after forking.

There are other complaints of "MongoClient opened before fork. Create MongoClient "

Responses