Chapter 13. Logging and Reporting Problems
This chapter provides information on how to log and report problems in Satellite, including information on relevant log files, how to enable debug logging, how to open a support case and attach the relevant log tar files, and how to access support cases within the Satellite web UI.
You can use the log files and other information described in this chapter to do your own troubleshooting, or you can capture these and many more files, as well as diagnostic and configuration information, to send to Red Hat Support if you need further assistance.
For more information about Satellite logging settings, use satellite-installer
with the --full-help
option:
# satellite-installer --full-help | grep logging
13.1. Enabling Debug Logging
Debug logging provides the most detailed log information and can help with troubleshooting issues that can arise with Satellite and its components. In the Satellite CLI, enable debug logging to log detailed debugging information for Satellite.
Procedure
To enable debug logging, enter the following command:
# satellite-installer --foreman-logging-level debug
After you complete debugging, reset the logging level to the default value:
# satellite-installer --reset-foreman-logging-level
13.2. Increasing the Logging Levels to Help with Debugging
By default, Satellite comes with :INFO
level logging enabled. You can increase or decrease the log levels on your Satellite.
Enabling debug level logging on all components
# hammer admin logging --all --level-debug # satellite-maintain service restart
Enabling debug level logging for a specific component
# hammer admin logging --components "Component" --level-debug
Reverting debug level logging to INFO
# hammer admin logging --all --level-production # satellite-maintain service restart
Listing all components and changed configuration files
# hammer admin logging --list -----------|-------------------------------------|------------------------------------- COMPONENT | AUTO-DETECTED BY EXISTENCE OF | DESTINATIONS -----------|-------------------------------------|------------------------------------- dhcpd | /etc/dhcp/dhcpd.conf | syslog /var/log/dhcpd-debug.log postgresql | /var/lib/pgsql/data/postgresql.conf | syslog /var/lib/pgsql/data/pg_log/ proxy | /etc/foreman-proxy/settings.yml | /var/log/foreman-proxy/proxy.log qpidd | /etc/qpid/qpidd.conf | syslog rails | /etc/foreman/settings.yaml | /var/log/foreman/production.log tomcat | /etc/candlepin/candlepin.conf | /var/log/candlepin/ /var/log/tomcat/ virt-who | /etc/sysconfig/virt-who | syslog -----------|-------------------------------------|-------------------------------------
13.2.1. Increasing the Logging Level For Hammer
You can find the log for Hammer in ~/.hammer/log/hammer.log
. Edit /etc/hammer/cli_config.yml
and set the :log_level:
:
:log_level: 'debug'
13.2.2. Increasing the Logging Level On Capsule
You can find the log for Capsule in /var/log/foreman-proxy/proxy.log
. Uncomment the DEBUG
line in /etc/foreman-proxy/settings.yml
:
:log_level: DEBUG
Ensure to restart the foreman-proxy
service afterwards:
# systemctl restart foreman-proxy
Running the installer will revert this change back.
13.2.3. Increasing the Logging Level For Candlepin
You can find the log for Candlepin in /var/log/candlepin/candlepin.log
. Errors are also logged to a separate file for easier debugging /var/log/candlepin/error.log
.
Extend /etc/candlepin/candlepin.conf
:
log4j.logger.org.candlepin=DEBUG
Ensure to restart the tomcat
service afterwards:
# systemctl restart tomcat
If the candlepin log files are too verbose, you can decrease the default debug level:
log4j.logger.org.candlepin.resource.ConsumerResource=WARN log4j.logger.org.candlepin.resource.HypervisorResource=WARN
13.2.4. Increasing the Logging Level On Satellite
You can find the log for Satellite in /var/log/foreman/production.log
.
Satellite stores logs for Apache in:
-
/var/log/httpd/foreman_error.log
-
/var/log/httpd/foreman_access.log
-
/var/log/httpd/foreman_ssl_error.log
-
/var/log/httpd/foreman_ssl_access.log
Procedure
Set the logging level in
/etc/foreman/settings.yaml
::logging: :production: :type: file :layout: pattern :level: debug
Enable selected loggers in
/etc/foreman/settings.yaml
::loggers: :ldap: :enabled: true :permissions: :enabled: true :sql: :enabled: true
Note that to see logging from some area, debug logging has to be set.
Restart Satellite services:
# satellite-maintain service restart
You can find the complete list of loggers with their default values in /usr/share/foreman/config/application.rb
in the Foreman::Logging.add_loggers
command.
13.2.5. Increasing the Logging Level For Qpid Dispatch Router
Qpid logs to syslog and can be viewed in /var/log/messages
or with journalctl
. Enable debug logging in /etc/qpid-dispatch/qdrouterd.conf
:
enable: debug+
Ensure to restart the Qpid Dispatch Router afterwards:
# systemctl restart qdrouterd
Running the installer will revert this change back.
13.2.6. Increasing the Logging Level For Qpid Broker
Qpid logs to syslog and can be viewed in /var/log/messages
or with journalctl
. Set the log level in /etc/qpid/qpidd.conf
:
log-enable=debug+
Ensure to restart the Qpid Broker afterwards:
# systemctl restart qpidd
Running the installer will revert this change.
13.2.7. Increasing the Logging Level For Redis
You can find the log for Redis in /var/log/redis/redis.log
. Set the log level in /etc/opt/rh/rh-redis5/redis.conf
:
loglevel debug
Ensure to restart the Redis service afterwards:
# systemctl restart rh-redis5-redis
13.2.8. Increasing the Logging Level For Postgres
You can find the log for Postgres in /var/opt/rh/rh-postgresql12/lib/pgsql/data/log/
. Uncomment the log_statement
in /var/opt/rh/rh-postgresql12/lib/pgsql/data/postgresql.conf
:
log_statement = 'all'
Ensure to restart Satellite services afterwards:
# satellite-maintain service restart
Based on the size of your Satellite installation, this can cause disk space to fill up very quickly. Only turn this on if absolutely needed.
For more debug log settings, refer to the Postgresql documentation.
13.2.9. Increasing the Logging Level For Satellite Installer
You can find the log files in /var/log/foreman-installer/
. To increase the log level of the Satellite Installer during an install:
# satellite-installer --verbose-log-level debug
13.2.10. Increasing the Logging Level For Pulp
By default, Pulp logs to syslog and can be viewed in /var/log/messages
or with journalctl
. Add the following config to the /etc/pulp/settings.py
file:
LOGGING = {"dynaconf_merge": True, "loggers": {'': {'handlers': ['console'], 'level': 'DEBUG'}}}
Ensure to restart the Pulp services afterwards:
# systemctl restart \ pulpcore-api \ pulpcore-content \ pulpcore-resource-manager \ pulpcore-worker@1 \ pulpcore-worker@2 \ rh-redis5-redis
13.2.11. Increasing the Logging Level For Puppet Agent
You can increase the logging level for Puppet agent on your Satellite Server.
Procedure
Add the following line to the
[agent]
block in the/etc/puppetlabs/puppet/puppet.conf
file:[agent] log_level = debug
You can find the logs in /var/log/puppetlabs/puppet/
13.2.12. Increasing the Logging Level For Puppet Server
You can increase the logging level for Puppet server on your Satellite Server.
Prerequisite
- Puppet must be enabled in your Satellite. For more information, see Enabling Puppet Integration with Satellite in Managing Configurations Using Puppet Integration in Red Hat Satellite.
Procedure
Add the following line to the
[master]
block in/etc/puppetlabs/puppet/puppet.conf
file:[master] log_level = debug
Restart the Puppet server:
# satellite-maintain service restart --only puppetserver
You can find the logs in /var/log/puppetlabs/puppetserver/
.
13.3. Retrieving the Status of Services
Procedure
- In the Satellite web UI, navigate to Administer > About.
- On the Smart Proxies tab, you can view the status of all Capsules.
- On the Compute Resources tab, you can view the status of attached compute resource providers.
- In the Backend System Status table, you can view the status of all back-end services.
CLI procedure
Run
hammer ping
to get information from the database and Satellite services:# hammer ping
Use
satellite-maintain
to check the status of the services running in systemd:# satellite-maintain service status
Use
satellite-maintain
to perform a health check:$ satellite-maintain health check
13.4. Restarting Services
Satellite uses a set of back-end services to perform tasks. You you experience an issue with your Satellite, check the status of Satellite services.
Procedure
Use
satellite-maintain
to restart Satellite services:# satellite-maintain service restart
Run foreman-maintain --help
for more information.
13.5. Enabling Individual Loggers
You can enable individual loggers for selective logging. Satellite uses the following loggers:
app
- Logs web requests and all general application messages. Default value: true.
audit
- Logs additional fact statistics, numbers of added, updated, and removed facts. Default value: true.
background
- Logs information from the background processing component.
blob
Logs contents of rendered templates for auditing purposes.
ImportantThe
blob
logger might contain sensitive data.dynflow
- Logs information from the Dynflow process.
ldap
- Logs high level LDAP queries and LDAP operations. Default value: false.
notifications
- Logs information from the notifications component.
permissions
- Logs queries to user roles, filters, and permissions when loading pages. Default value: false.
sql
- Logs SQL queries made through Rails ActiveRecord. Default value: false.
telemetry
- Logs debugging information from telemetry.
templates
- Logs information from the template renderer component.
Procedure
Enable the individual loggers that you want. For example, to enable
sql
andldap
loggers, enter the following command:# satellite-installer \ --foreman-loggers ldap:true \ --foreman-loggers sql:true
Optional: To reset loggers to their default values, enter the following command:
# satellite-installer --reset-foreman-loggers
13.6. Configuring Logging to Journal or File-based Logging
Satellite uses file-based logging by default. You can use the satellite-installer
command to reconfigure logging.
Procedure for configuring logging with Journal
Enter the following
satellite-installer
command to configure logging to thejournald
service:# satellite-installer \ --foreman-logging-layout pattern \ --foreman-logging-type journald \ --foreman-proxy-log JOURNAL
Optional: To inspect the log messages, use the
journalctl
utility. For example:-
journalctl --unit foreman
andjournalctl --unit foreman-proxy
show messages for theforeman
andforeman-proxy
units -
journalctl REQUEST=request_ID
shows messages for a specified request
-
Procedure for configuring file-based logging
Enter the following
satellite-installer
command to configure file-based logging:# satellite-installer \ --reset-foreman-logging-layout \ --reset-foreman-logging-type \ --reset-foreman-proxy-log
Optional: To inspect the log messages, view these files:
-
/var/log/foreman/production.log
-
/var/log/foreman-proxy.log
-
Additional resources
For more information about Journal, see Viewing logs using the command line in the Red Hat Enterprise Linux 8 Configuring Basic System Settings Guide.
13.7. Log File Directories Provided by Satellite
Red Hat Satellite provides system information in the form of notifications and log files.
Table 13.1. Log File Directories for Reporting and Troubleshooting
Log File Directories | Description of Log File Content |
---|---|
| Subscription management |
| Installer |
| Foreman maintain |
| Foreman proxy |
| Foreman |
| Apache HTTP server |
| Various other log messages |
| Configuration management |
| Subscription management |
| Candlepin webservice logs |
You can also use the foreman-tail
command to follow many of the log files related to Satellite. You can run foreman-tail -l
to list the processes and services that it follows.
13.8. Utilities for Collecting Log Information
You can collect information from log files to troubleshoot Satellite.
- sosreport
-
The
sosreport
command collects configuration and diagnostic information from a Linux system, such as the running Kernel version, loaded modules, running services, and system and service configuration files. This output is stored in a tar file located at/var/tmp/sosreport-XXX-20171002230919.tar.xz
. For more information, runsosreport --help
or see What is a sosreport and how can I create one?.
The collection process removes security information such as passwords, tokens, and keys while collecting information. However, the tar files can still contain sensitive information about the Satellite Server. Red Hat recommends that you send this information directly to the intended recipient and not to a public target.
13.9. System Journal Metadata
The following table lists metadata that the journald
service uses in Satellite. You can use this metadata to filter your queries.
Table 13.2. System Journal Metadata
Name | Description |
---|---|
AUDIT_ACTION | Audit action performed Example: Create, update, or delete |
AUDIT_TYPE | Audit resource type Example: Host, Subnet, or ContentView |
AUDIT_ID | Audit resource database ID as a number |
AUDIT_ATTRIBUTE | Audit resource field or an updated database column |
AUDIT_FIELD_OLD | Old audit value of an update action |
AUDIT_FIELD_NEW | New audit value of an update action |
AUDIT_ID | Record database ID of the audit subject |
AUDIT_ATTRIBUTE | Attribute name or column on which an action was performed Example: Name or description |
EXCEPTION_MESSAGE | Exception message when error is logged |
EXCEPTION_CLASS | Exception Ruby class when error is logged |
EXCEPTION_BACKTRACE | Exception backtrace as a multiline string when error is logged |
LOC_ID | Location database ID |
LOC_NAME | Location name |
LOC_LABEL | Location label |
LOGGER | Logger name To see the current list of loggers enabled by default, enter this command: # awk '/add_loggers/,/^$/' /usr/share/foreman/config/application.rb |
ORG_ID | Organization database ID |
ORG_NAME | Organization name |
ORG_LABEL | Organization label |
REMOTE_IP | Remote IP address of a client |
REQUEST | Request ID generated by the Action Dispatch module |
SESSION | Random ID generated per session or a request for a sessionless request |
TEMPLATE_NAME | Template name |
TEMPLATE_DIGEST | Digest (SHA256) of rendered template contents |
TEMPLATE_HOST_NAME | Host name for a rendered template if present |
TEMPLATE_HOST_ID | Host database ID for a rendered template if present |
USER_LOGIN | User login name |