pulp_workers are stuck at 1% and are not progressing

Solution Unverified - Updated -

Environment

  • Red Hat Satellite 6.5+
  • Red Hat Satellite Capsule 6.5+

Issue

  • Capsule sync is stucked at 1% and is not progressing. Current running tasks shows:

    # sudo su - postgres -c "psql -d foreman -c 'select label,count(label),state,result from foreman_tasks_tasks where state <> '\''stopped'\'' group by label,state,result ORDER BY label;'"
                     label                      | count |   state   | result
------------------------------------------------+-------+-----------+---------
 Actions::Katello::CapsuleContent::Sync         |     1 | running   | pending
 Actions::Katello::OrphanCleanup::RemoveOrphans |     1 | running   | pending
 CreatePulpDiskSpaceNotifications               |     1 | scheduled | pending
 CreateRssNotifications                         |     1 | scheduled | pending
 SendExpireSoonNotifications                    |     1 | scheduled | pending
 StoredValuesCleanupJob                         |     1 | scheduled | pending
(6 rows)

  • qpid-stat in the capsule shows that pulp_workers are not picking up the tasks:

      pulp.task                                                              Y                      0     0      0       0      0                                                                         0         3     1
  reserved_resource_worker-0@sat6.example.com.celery.pidbox       Y                 0     0      0       0      0                                                                         0         1     2
  reserved_resource_worker-0@sat6.example.com.dq2            Y                      0     0      0       0      0                                                                         0         1     2
  reserved_resource_worker-1@sat6.example.com.celery.pidbox       Y                 0     0      0       0      0                                                                         0         1     2
  reserved_resource_worker-1@sat6.example.com.dq2            Y                      0     0      0       0      0                                                                         0         1     2
  reserved_resource_worker-2@sat6.example.com.celery.pidbox       Y                 0     0      0       0      0                                                                         0         1     2
  reserved_resource_worker-2@sat6.example.com.dq2            Y                      0     0      0       0      0                                                                         0         1     2
  reserved_resource_worker-3@sat6.example.com.celery.pidbox       Y                 0     0      0       0      0                                                                         0         1     2
  reserved_resource_worker-3@sat6.example.com.dq2            Y                      0     0      0       0      0                                                                         0         1     2
  resource_manager                                                       Y                      0     0      0       0      0                                                                         0         1     2
  resource_manager@sat6.example.com.celery.pidbox                 Y                 0     0      0       0      0                                                                         0         1     2
  resource_manager@sat6.example.com.dq2                      Y                      0     0      0       0      0                                                                         0         1     2

Resolution

  • Stop all capsule sync tasks including remove_orphan tasks running on the Capsule.

    SatWEBUI > Monitor > Tasks >

    Cancel running CapsuleSync and related tasks:

       # foreman-rake foreman_tasks:cleanup TASK_SEARCH='label = Actions::Katello::CapsuleContent::Sync' STATES='running' VERBOSE=true
   # foreman-rake foreman_tasks:cleanup TASK_SEARCH='label = Actions::Katello::OrphanCleanup::RemoveOrphans' STATES='running' VERBOSE=true

  • Apply cleanup task with the attached script after download and untar:

    • clear_pulp_task.tar clear_waiting_pulp_tasks.rake
       tar -xf clear_pulp_task.tar
   clear_waiting_pulp_tasks.rake README.md

  • Copy the script to Satellite's "/usr/share/foreman/lib/tasks/" directory then change permission/ownership/group to:

    # chmod 544
# chown root:root

  • Restart pulp services from the Capsule:

     # systemctl restart pulp_workers pulp_resource_manager pulp_celerybeat

  • Run capsule sync

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments