Chapter 17. Logging and Reporting Problems

This chapter provides information on how to log and report problems in Satellite, including information on relevant log files, how to enable debug logging, how to open a support case and attach the relevant log tar files, and how to access support cases within the Satellite web UI.

You can use the log files and other information described in this chapter to do your own troubleshooting, or you can capture these and many more files, as well as diagnostic and configuration information, to send to Red Hat Support if you need further assistance.

For more information about Satellite logging settings, use satellite-installer with the --full-help option:

# satellite-installer --full-help | grep logging

17.1. Enabling Debug Logging

Debug logging provides the most detailed log information and can help with troubleshooting issues that can arise with Satellite and its components. In the Satellite CLI, enable debug logging to log detailed debugging information for Satellite.

Procedure

  1. To enable debug logging, enter the following command:

    # satellite-installer --foreman-logging-level debug
  2. After you complete debugging, reset the logging level to the default value:

    # satellite-installer --reset-foreman-logging-level

17.2. Increasing the Logging Levels to Help with Debugging

By default, Satellite comes with :INFO level logging enabled. You can increase or decrease the log levels on your Satellite.

Enabling debug level logging on all components

# hammer admin logging --all --level-debug
# satellite-maintain service restart

Enabling debug level logging for a specific component

# hammer admin logging --components "Component" --level-debug

Reverting debug level logging to INFO

# hammer admin logging --all --level-production
# satellite-maintain service restart

Listing all components and changed configuration files

# hammer admin logging --list
-----------|-------------------------------------|-------------------------------------
COMPONENT  | AUTO-DETECTED BY EXISTENCE OF       | DESTINATIONS
-----------|-------------------------------------|-------------------------------------
dhcpd      | /etc/dhcp/dhcpd.conf                | syslog /var/log/dhcpd-debug.log
postgresql | /var/lib/pgsql/data/postgresql.conf | syslog /var/lib/pgsql/data/pg_log/
proxy      | /etc/foreman-proxy/settings.yml     | /var/log/foreman-proxy/proxy.log
qpidd      | /etc/qpid/qpidd.conf                | syslog
rails      | /etc/foreman/settings.yaml          | /var/log/foreman/production.log
tomcat     | /etc/candlepin/candlepin.conf       | /var/log/candlepin/ /var/log/tomcat/
virt-who   | /etc/sysconfig/virt-who             | syslog
-----------|-------------------------------------|-------------------------------------

17.2.1. Increasing the Logging Level For Hammer

You can find the log for Hammer in ~/.hammer/log/hammer.log. Edit /etc/hammer/cli_config.yml and set the :log_level::

:log_level: 'debug'

17.2.2. Increasing the Logging Level On Capsule

You can find the log for Capsule in /var/log/foreman-proxy/proxy.log. Uncomment the DEBUG line in /etc/foreman-proxy/settings.yml:

:log_level: DEBUG

Ensure to restart the foreman-proxy service afterwards:

# systemctl restart foreman-proxy
Caution

Running the installer will revert this change back.

17.2.3. Increasing the Logging Level For Candlepin

You can find the log for Candlepin in /var/log/candlepin/candlepin.log. Errors are also logged to a separate file for easier debugging /var/log/candlepin/error.log.

Extend /etc/candlepin/candlepin.conf:

log4j.logger.org.candlepin=DEBUG

Ensure to restart the tomcat service afterwards:

# systemctl restart tomcat

If the candlepin log files are too verbose, you can decrease the default debug level:

log4j.logger.org.candlepin.resource.ConsumerResource=WARN
log4j.logger.org.candlepin.resource.HypervisorResource=WARN

17.2.4. Increasing the Logging Level On Satellite

You can find the log for Satellite in /var/log/foreman/production.log.

Satellite stores logs for Apache in:

  • /var/log/httpd/foreman_error.log
  • /var/log/httpd/foreman_access.log
  • /var/log/httpd/foreman_ssl_error.log
  • /var/log/httpd/foreman_ssl_access.log

Procedure

  1. Set the logging level in /etc/foreman/settings.yaml:

    :logging:
      :production:
        :type: file
        :layout: pattern
        :level: debug
  2. Enable selected loggers in /etc/foreman/settings.yaml:

    :loggers:
      :ldap:
        :enabled: true
      :permissions:
        :enabled: true
      :sql:
        :enabled: true

    Note that to see logging from some area, debug logging has to be set.

  3. Restart Satellite services:

    # satellite-maintain service restart

You can find the complete list of loggers with their default values in /usr/share/foreman/config/application.rb in the Foreman::Logging.add_loggers command.

17.2.5. Increasing the Logging Level For Qpid Dispatch Router

Qpid logs to syslog and can be viewed in /var/log/messages or with journalctl. Enable debug logging in /etc/qpid-dispatch/qdrouterd.conf:

enable: debug+

Ensure to restart the Qpid Dispatch Router afterwards:

# systemctl restart qdrouterd
Caution

Running the installer will revert this change back.

17.2.6. Increasing the Logging Level For Qpid Broker

Qpid logs to syslog and can be viewed in /var/log/messages or with journalctl. Set the log level in /etc/qpid/qpidd.conf:

log-enable=debug+

Ensure to restart the Qpid Broker afterwards:

# systemctl restart qpidd
Caution

Running the installer will revert this change.

17.2.7. Increasing the Logging Level For Redis

You can find the log for Redis in /var/log/redis/redis.log. Set the log level in /etc/opt/rh/rh-redis5/redis.conf:

loglevel debug

Ensure to restart the Redis service afterwards:

# systemctl restart rh-redis5-redis

17.2.8. Increasing the Logging Level For Postgres

You can find the log for Postgres in /var/lib/pgsql/data/log. Uncomment the log_statement in /var/lib/pgsql/data/postgresql.conf:

log_statement = 'all'

Ensure to restart Satellite services afterwards:

# satellite-maintain service restart
Caution

Based on the size of your Satellite installation, this can cause disk space to fill up very quickly. Only turn this on if absolutely needed.

For more debug log settings, refer to the Postgresql documentation.

17.2.9. Increasing the Logging Level For Satellite Installer

You can find the log files in /var/log/foreman-installer/. To increase the log level of the Satellite Installer during an install:

# satellite-installer --verbose-log-level debug

17.2.10. Increasing the Logging Level For Pulp

By default, Pulp logs to syslog and can be viewed in /var/log/messages or with journalctl. Add the following config to the /etc/pulp/settings.py file:

LOGGING = {"dynaconf_merge": True, "loggers": {'': {'handlers': ['console'], 'level': 'DEBUG'}}}

Ensure to restart the Pulp services afterwards:

# systemctl restart \
pulpcore-api \
pulpcore-content \
pulpcore-resource-manager \
pulpcore-worker@1 \
pulpcore-worker@2 \
rh-redis5-redis

17.2.11. Increasing the Logging Level For Puppet Agent

You can increase the logging level for Puppet agent on your Satellite Server.

Procedure

  1. Add the following line to the [agent] block in the /etc/puppetlabs/puppet/puppet.conf file:

    [agent]
        log_level = debug

You can find the logs in /var/log/puppetlabs/puppet/

17.2.12. Increasing the Logging Level For Puppet Server

You can increase the logging level for Puppet server on your Satellite Server.

Prerequisite

Procedure

  1. Add the following line to the [master] block in /etc/puppetlabs/puppet/puppet.conf file:

    [master]
        log_level = debug
  2. Restart the Puppet server:

    # satellite-maintain service restart --only puppetserver

You can find the logs in /var/log/puppetlabs/puppetserver/.

17.3. Retrieving the Status of Services

Procedure

  1. In the Satellite web UI, navigate to Administer > About.
  2. On the Smart Proxies tab, you can view the status of all Capsules.
  3. On the Compute Resources tab, you can view the status of attached compute resource providers.
  4. In the Backend System Status table, you can view the status of all back-end services.

CLI procedure

  • Run hammer ping to get information from the database and Satellite services:

    # hammer ping
  • Use satellite-maintain to check the status of the services running in systemd:

    # satellite-maintain service status
  • Use satellite-maintain to perform a health check:

    $ satellite-maintain health check

17.4. Restarting Services

Satellite uses a set of back-end services to perform tasks. You you experience an issue with your Satellite, check the status of Satellite services.

Procedure

  • Use satellite-maintain to restart Satellite services:

    # satellite-maintain service restart
Tip

Run foreman-maintain --help for more information.

17.5. Enabling Individual Loggers

You can enable individual loggers for selective logging. Satellite uses the following loggers:

app
Logs web requests and all general application messages. Default value: true.
audit
Logs additional fact statistics, numbers of added, updated, and removed facts. Default value: true.
background
Logs information from the background processing component.
blob

Logs contents of rendered templates for auditing purposes.

Important

The blob logger might contain sensitive data.

dynflow
Logs information from the Dynflow process.
ldap
Logs high level LDAP queries and LDAP operations. Default value: false.
notifications
Logs information from the notifications component.
permissions
Logs queries to user roles, filters, and permissions when loading pages. Default value: false.
sql
Logs SQL queries made through Rails ActiveRecord. Default value: false.
telemetry
Logs debugging information from telemetry.
templates
Logs information from the template renderer component.

Procedure

  1. Enable the individual loggers that you want. For example, to enable sql and ldap loggers, enter the following command:

    # satellite-installer \
    --foreman-loggers ldap:true \
    --foreman-loggers sql:true
  2. Optional: To reset loggers to their default values, enter the following command:

    # satellite-installer --reset-foreman-loggers

17.6. Configuring Logging to Journal or File-based Logging

Satellite uses file-based logging by default. You can use the satellite-installer command to reconfigure logging.

Procedure for configuring logging with Journal

  1. Enter the following satellite-installer command to configure logging to the journald service:

    # satellite-installer \
    --foreman-logging-layout pattern \
    --foreman-logging-type journald \
    --foreman-proxy-log JOURNAL
  2. Optional: To inspect the log messages, use the journalctl utility. For example:

    • journalctl --unit foreman and journalctl --unit foreman-proxy show messages for the foreman and foreman-proxy units
    • journalctl REQUEST=request_ID shows messages for a specified request

Procedure for configuring file-based logging

  1. Enter the following satellite-installer command to configure file-based logging:

    # satellite-installer \
    --reset-foreman-logging-layout \
    --reset-foreman-logging-type \
    --reset-foreman-proxy-log
  2. Optional: To inspect the log messages, view these files:

    • /var/log/foreman/production.log
    • /var/log/foreman-proxy.log

Additional resources

For more information about Journal, see Viewing logs using the command line in the Red Hat Enterprise Linux 8 Configuring Basic System Settings Guide.

17.7. Log File Directories Provided by Satellite

Red Hat Satellite provides system information in the form of notifications and log files.

Table 17.1. Log File Directories for Reporting and Troubleshooting

Log File DirectoriesDescription of Log File Content

/var/log/candlepin

Subscription management

/var/log/foreman-installer

Installer

/var/log/foreman-maintain

Foreman maintain

/var/log/foreman-proxy

Foreman proxy

/var/log/foreman

Foreman

/var/log/httpd

Apache HTTP server

/var/log/messages

Various other log messages

/var/log/puppetlabs/puppet

Configuration management

/var/log/rhsm

Subscription management

/var/log/tomcat

Candlepin webservice logs

You can also use the foreman-tail command to follow many of the log files related to Satellite. You can run foreman-tail -l to list the processes and services that it follows.

17.8. Utilities for Collecting Log Information

You can collect information from log files to troubleshoot Satellite.

sosreport
The sosreport command collects configuration and diagnostic information from a Linux system, such as the running Kernel version, loaded modules, running services, and system and service configuration files. This output is stored in a tar file located at /var/tmp/sosreport-XXX-20171002230919.tar.xz. For more information, run sosreport --help or see What is a sosreport and how can I create one?.
Important

The collection process removes security information such as passwords, tokens, and keys while collecting information. However, the tar files can still contain sensitive information about the Satellite Server. Red Hat recommends that you send this information directly to the intended recipient and not to a public target.

17.9. System Journal Metadata

The following table lists metadata that the journald service uses in Satellite. You can use this metadata to filter your queries.

Table 17.2. System Journal Metadata

NameDescription

AUDIT_ACTION

Audit action performed

Example: Create, update, or delete

AUDIT_TYPE

Audit resource type

Example: Host, Subnet, or ContentView

AUDIT_ID

Audit resource database ID as a number

AUDIT_ATTRIBUTE

Audit resource field or an updated database column

AUDIT_FIELD_OLD

Old audit value of an update action

AUDIT_FIELD_NEW

New audit value of an update action

AUDIT_ID

Record database ID of the audit subject

AUDIT_ATTRIBUTE

Attribute name or column on which an action was performed

Example: Name or description

EXCEPTION_MESSAGE

Exception message when error is logged

EXCEPTION_CLASS

Exception Ruby class when error is logged

EXCEPTION_BACKTRACE

Exception backtrace as a multiline string when error is logged

LOC_ID

Location database ID

LOC_NAME

Location name

LOC_LABEL

Location label

LOGGER

Logger name

To see the current list of loggers enabled by default, enter this command:

# awk '/add_loggers/,/^$/' /usr/share/foreman/config/application.rb

ORG_ID

Organization database ID

ORG_NAME

Organization name

ORG_LABEL

Organization label

REMOTE_IP

Remote IP address of a client

REQUEST

Request ID generated by the Action Dispatch module

SESSION

Random ID generated per session or a request for a sessionless request

TEMPLATE_NAME

Template name

TEMPLATE_DIGEST

Digest (SHA256) of rendered template contents

TEMPLATE_HOST_NAME

Host name for a rendered template if present

TEMPLATE_HOST_ID

Host database ID for a rendered template if present

USER_LOGIN

User login name