Repo sync job causes high load and eventually system issies

Posted on

We've been having weekly issues that I've traced to via SAR starting immediately following the scheduled weekly CDN repo sync of 6Server and 6Workstation product repositories. Note that a separate repo sync job of 6Capsule doesn't exhibit the same issue.

What happens is load is averaging around 2-4 for the week leading up to the sync, when the sync kicks off load spikes and continues to grow until it hits over 50 and causes then issues with other processes for resources.

The top consumers of CPU and RAM are postgres processes during this time. We've been having to stop katello-services, reindex elasticsearch indices (6.1.8 satellite), then vacuum and reindex postgres. Note if we don't vacuum and reindex postgres just doing the elasticsearch reindex still causes the spike to come back after katello-services are restarted.

We have an open ticket, for one month now, but was looking for any input or suggestions to the tables, processes, or ideas to look for WRT the repo sync jobs and what may be the culprit?

Thanks

Responses