Unable to restart the services on the Satellite server and getting error "Some services are not running (pulp_celerybeat, pulp_resource_manager, pulp_streamer)".

Solution Verified - Updated -

Environment

  • Red Hat Satellite 6.

Issue

  • Unable to restart pulp services on the Satellite server, getting below error:
Some services are not running (pulp_celerybeat, pulp_resource_manager, pulp_streamer)
--------------------------------------------------------------------------------
Scenario [Status Services] failed.
  • Satellite upgrade failing on pulp-manage-db step:
 Upgrade Step: migrate_pulp...
grep -qe '7.[[:digit:]]' /etc/redhat-release finished successfully!
sed -i -e 's?/var/run/mongodb/mongodb.pid?/var/run/mongodb/mongod.pid?g' /etc/mongod.conf finished successfully!
pgrep mongod finished successfully!
Traceback (most recent call last):
File "/bin/pulp-manage-db", line 9, in <module>
load_entry_point('pulp-server==2.13.4.6',
raise TypeError("option values must be strings")
TypeError: option values must be strings
 su - apache -s /bin/bash -c pulp-manage-db failed! Check the output for error!
Upgrade step migrate_pulp failed. Check logs for more information.
Exit with status code: 1 (signal was 1)
 Errors encountered during run:
su - apache -s /bin/bash -c pulp-manage-db failed! Check the output for error!
Upgrade step migrate_pulp failed. Check logs for more information.

Resolution

  • Check the permission of directories and files under /etc/pki/ using below command:
# tree -puf /etc/pki/
  • If permission is mismatched or wrong, change the permission following the below example:
# ls -ald /etc/pki/ca-trust/
drwx------. 4 root root 73 Aug 14  2017 /etc/pki/ca-trust/
# chmod 755 /etc/pki/ca-trust
# ls -ald /etc/pki/ca-trust/
drwxr-xr-x. 4 root root 73 Aug 14  2017 /etc/pki/ca-trust/

Root Cause

  • Insufficient permissions on /etc/pki/ directory or the files under /etc/pki/ directory.

Diagnostic Steps

  • Following error observed while Satellite upgrade:
ser.py", line 743, in set
2022-04-18 17:21:55 [INFO  ] [configure] /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns:     raise TypeError("option values must be strings")
2022-04-18 17:21:55 [INFO  ] [configure] /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns: TypeError: option values must be strings
2022-04-18 17:21:55 [ERROR ] [configure] 'pulp-manage-db' returned 1 instead of one of [0]
2022-04-18 17:21:55 [ERROR ] [configure] /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns: change from 'notrun' to ['0'] failed: 'pulp-manage-db' returned 1 instead of one of [0]
  • Following error observed while running journalctl:
# journalctl -u pulp_resource_manager
-- Logs begin at Mon 2022-04-18 13:56:21 +03, end at Mon 2022-04-18 16:15:59 +03. --
Apr 18 13:56:32 satellite.example.com systemd[1]: Started Pulp Resource Manager.
Apr 18 13:56:36 satellite.example.com celery[1206]: Traceback (most recent call last):
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/bin/celery", line 9, in <module>
Apr 18 13:56:36 satellite.example.com celery[1206]: load_entry_point('celery==4.0.2', 'console_scripts', 'celery')()
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/celery/__main__.py", line 14, in main
Apr 18 13:56:36 satellite.example.com celery[1206]: _main()
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 326, in main
Apr 18 13:56:36 satellite.example.com celery[1206]: cmd.execute_from_commandline(argv)
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 488, in execute_from_comma
Apr 18 13:56:36 satellite.example.com celery[1206]: super(CeleryCommand, self).execute_from_commandline(argv)))
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 279, in execute_from_command
Apr 18 13:56:36 satellite.example.com celery[1206]: argv = self.setup_app_from_commandline(argv)
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 481, in setup_app_from_comma
Apr 18 13:56:36 satellite.example.com celery[1206]: self.app = self.find_app(app)
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 503, in find_app
Apr 18 13:56:36 satellite.example.com celery[1206]: return find_app(app, symbol_by_name=self.symbol_by_name)
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/celery/app/utils.py", line 358, in find_app
Apr 18 13:56:36 satellite.example.com celery[1206]: sym = imp(app)
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
Apr 18 13:56:36 satellite.example.com celery[1206]: return imp(module, package=package)
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
Apr 18 13:56:36 satellite.example.com celery[1206]: __import__(name)
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/pulp/server/async/app.py", line 17, in <module>
Apr 18 13:56:36 satellite.example.com celery[1206]: from pulp.server import initialization
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/pulp/server/initialization.py", line 6, in <module>
Apr 18 13:56:36 satellite.example.com celery[1206]: from pulp.plugins.loader import api as plugin_api
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/pulp/plugins/loader/api.py", line 7, in <module>
Apr 18 13:56:36 satellite.example.com celery[1206]: from pulp.plugins.loader.manager import PluginManager
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/pulp/plugins/loader/manager.py", line 9, in <module>
Apr 18 13:56:36 satellite.example.com celery[1206]: from pulp.server.db.model import ContentUnit
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py", line 20, in <module
Apr 18 13:56:36 satellite.example.com celery[1206]: from pulp.plugins.model import Repository as plugin_repo
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/pulp/plugins/model.py", line 7, in <module>
Apr 18 13:56:36 satellite.example.com celery[1206]: from pulp.server import constants
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/pulp/server/constants.py", line 12, in <module>
Apr 18 13:56:36 satellite.example.com celery[1206]: PULP_PROCESS_HEARTBEAT_INTERVAL = int(config.getint('tasks', 'worker_timeout') / 5)
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib64/python2.7/ConfigParser.py", line 359, in getint
Apr 18 13:56:36 satellite.example.com celery[1206]: return self._get(section, int, option)
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib64/python2.7/ConfigParser.py", line 356, in _get
Apr 18 13:56:36 satellite.example.com celery[1206]: return conv(self.get(section, option))
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib64/python2.7/ConfigParser.py", line 604, in get
Apr 18 13:56:36 satellite.example.com celery[1206]: sectiondict = self._sections[section]
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/pulp/server/config.py", line 25, in _sections
Apr 18 13:56:36 satellite.example.com celery[1206]: self._load_config()
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib/python2.7/site-packages/pulp/server/config.py", line 40, in _load_config
Apr 18 13:56:36 satellite.example.com celery[1206]: self.set(section, option, value)
Apr 18 13:56:36 satellite.example.com celery[1206]: File "/usr/lib64/python2.7/ConfigParser.py", line 743, in set
Apr 18 13:56:36 satellite.example.com celery[1206]: raise TypeError("option values must be strings")
Apr 18 13:56:36 satellite.example.com celery[1206]: TypeError: option values must be strings
Apr 18 13:56:36 satellite.example.com systemd[1]: pulp_resource_manager.service: main process exited, code=exited, status=1/FAILURE
Apr 18 13:56:36 satellite.example.com systemd[1]: Unit pulp_resource_manager.service entered failed state.
Apr 18 13:56:36 satellite.example.com systemd[1]: pulp_resource_manager.service failed.
Apr 18 14:01:00 satellite.example.com systemd[1]: Started Pulp Resource Manager.
Apr 18 14:01:00 satellite.example.com celery[4346]: Traceback (most recent call last):
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/bin/celery", line 9, in <module>
Apr 18 14:01:00 satellite.example.com celery[4346]: load_entry_point('celery==4.0.2', 'console_scripts', 'celery')()
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/celery/__main__.py", line 14, in main
Apr 18 14:01:00 satellite.example.com celery[4346]: _main()
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 326, in main
Apr 18 14:01:00 satellite.example.com celery[4346]: cmd.execute_from_commandline(argv)
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 488, in execute_from_comma
Apr 18 14:01:00 satellite.example.com celery[4346]: super(CeleryCommand, self).execute_from_commandline(argv)))
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 279, in execute_from_command
Apr 18 14:01:00 satellite.example.com celery[4346]: argv = self.setup_app_from_commandline(argv)
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 481, in setup_app_from_comma
Apr 18 14:01:00 satellite.example.com celery[4346]: self.app = self.find_app(app)
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 503, in find_app
Apr 18 14:01:00 satellite.example.com celery[4346]: return find_app(app, symbol_by_name=self.symbol_by_name)
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/celery/app/utils.py", line 358, in find_app
Apr 18 14:01:00 satellite.example.com celery[4346]: sym = imp(app)
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
Apr 18 14:01:00 satellite.example.com celery[4346]: return imp(module, package=package)
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
Apr 18 14:01:00 satellite.example.com celery[4346]: __import__(name)
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/pulp/server/async/app.py", line 17, in <module>
Apr 18 14:01:00 satellite.example.com celery[4346]: from pulp.server import initialization
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/pulp/server/initialization.py", line 6, in <module>
Apr 18 14:01:00 satellite.example.com celery[4346]: from pulp.plugins.loader import api as plugin_api
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/pulp/plugins/loader/api.py", line 7, in <module>
Apr 18 14:01:00 satellite.example.com celery[4346]: from pulp.plugins.loader.manager import PluginManager
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/pulp/plugins/loader/manager.py", line 9, in <module>
Apr 18 14:01:00 satellite.example.com celery[4346]: from pulp.server.db.model import ContentUnit
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py", line 20, in <module
Apr 18 14:01:00 satellite.example.com celery[4346]: from pulp.plugins.model import Repository as plugin_repo
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/pulp/plugins/model.py", line 7, in <module>
Apr 18 14:01:00 satellite.example.com celery[4346]: from pulp.server import constants
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/pulp/server/constants.py", line 12, in <module>
Apr 18 14:01:00 satellite.example.com celery[4346]: PULP_PROCESS_HEARTBEAT_INTERVAL = int(config.getint('tasks', 'worker_timeout') / 5)
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib64/python2.7/ConfigParser.py", line 359, in getint
Apr 18 14:01:00 satellite.example.com celery[4346]: return self._get(section, int, option)
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib64/python2.7/ConfigParser.py", line 356, in _get
Apr 18 14:01:00 satellite.example.com celery[4346]: return conv(self.get(section, option))
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib64/python2.7/ConfigParser.py", line 604, in get
Apr 18 14:01:00 satellite.example.com celery[4346]: sectiondict = self._sections[section]
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/pulp/server/config.py", line 25, in _sections
Apr 18 14:01:00 satellite.example.com celery[4346]: self._load_config()
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib/python2.7/site-packages/pulp/server/config.py", line 40, in _load_config
Apr 18 14:01:00 satellite.example.com celery[4346]: self.set(section, option, value)
Apr 18 14:01:00 satellite.example.com celery[4346]: File "/usr/lib64/python2.7/ConfigParser.py", line 743, in set
Apr 18 14:01:00 satellite.example.com celery[4346]: raise TypeError("option values must be strings")
Apr 18 14:01:00 satellite.example.com celery[4346]: TypeError: option values must be strings
Apr 18 14:01:00 satellite.example.com systemd[1]: pulp_resource_manager.service: main process exited, code=exited, status=1/FAILURE
Apr 18 14:01:00 satellite.example.com systemd[1]: Unit pulp_resource_manager.service entered failed state.
Apr 18 14:01:00 satellite.example.com systemd[1]: pulp_resource_manager.service failed.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.