Red Hat Training

A Red Hat training course is available for Red Hat Satellite

Chapter 9. Maintaining Satellite Server

This chapter provides information on how to maintain a Red Hat Satellite Server, including information on relevant log files, how to enable debug logging, how to open a support case and attach the relevant log tar files, and how to use Red Hat Insights to proactively diagnose systems.

9.1. Logging and Reporting

Red Hat Satellite provides system information in the form of notifications and log files.

Table 9.1. Log Files for Reporting and Troubleshooting

Log FileDescription of Log File Content

/var/log/candlepin

Subscription management

/var/log/foreman

Foreman

/var/log/foreman-proxy

Foreman proxy

/var/log/httpd

Apache HTTP server

/var/log/foreman-installer/satellite

Satellite installer

/var/log/foreman-installer/capsule

Capsule Server installer

/var/log/libvirt

Virtualization API

/var/log/mongodb

Satellite database

/var/log/pulp

Celerybeat and Celery startup request messages. After startup is complete, messages are logged to /var/log/messages.

/var/log/puppet

Configuration management

/var/log/rhsm

Subscription management

/var/log/tomcat6 and /var/log/tomcat

Apache web server messages for Red Hat Enterprise Linux 6 and Red Hat Enterprise Linux 7, respectively.

/var/log/messages

Various other log messages related to pulp, rhsm, and goferd.

You can also use the foreman-tail command to follow many of the log files related to Satellite. You can run foreman-tail -l to list the processes and services that it follows.

On Red Hat Enterprise Linux 7, you can use the journal for more extensive logging information. See Using the Journal[1] for more information.

9.2. Enabling Debug Logging

This section describes how to enable debug logging to provide detailed debugging information for Satellite 6.3. Debug logging provides the most detailed log information and can help with troubleshooting issues that can arise with Satellite 6.3 and its components. It is also possible to enable or disable individual loggers for selective logging.

To enable debug logging, modify the /etc/foreman/settings.yaml file.

  1. Set the Logging Level to "debug"

    By default, the logging level is set to info, as in the following: 

    :logging:
  :level: info

    Alter these lines so that they look like this: 

    :logging:
  :level: debug

  2. Select Individual Logging Types

    By default, the end of /etc/foreman/settings.yaml looks like this: 

    # Individual logging types can be toggled on/off here
:loggers:

    Alter the /etc/foreman/settings.yaml file so that it looks like this: 

    :loggers:
  :ldap:
    :enabled: true
  :permissions:
    :enabled: true
  :sql:
    :enabled: true

  3. Restart Katello services: 

    # katello-service restart

Complete List of Loggers with their Default Values 

:app:
    :enabled: true
:ldap:
    :enabled: false
:permissions:
    :enabled: false
:sql:
    :enabled: false

9.3. Collecting Information from Log Files

There are two utilies available to collect information from log files.

Table 9.2. Log Collecting Utilities

CommandDescription

foreman-debug

The foreman-debug command collects configuration and log file data for Red Hat Satellite, its back-end services, and system information. This information is collected and written to a tar file. By default, the output tar file is located at /tmp/foreman-debug-xxx.tar.xz.

Additionally, the foreman-debug command exports tasks run during the last 60 days. By default, the output tar file is located at /tmp/task-export-xxx.tar.xz. If the file is missing, see the file /tmp/task-export.log to learn why task export was unsuccessful.

For more information, run foreman-debug --help.

There is no timeout when running this command.

sosreport

The sosreport command is a tool that collects configuration and diagnostic information from a Red Hat Enterprise Linux system, such as the running kernel version, loaded modules, and system and service configuration files. The command also runs external programs (for example: foreman-debug -g) to collect Satellite-specific information, and stores this output in a tar file.

By default, the output tar file is located at /var/tmp/sosreport-XXX-20171002230919.tar.xz. For more information, run sosreport --help or see https://access.redhat.com/solutions/3592: What is a sosreport and how can I create one?.

The sosreport command calls the foreman-debug -g and times out after 500 seconds. If your Satellite Server has large log files or many Satellite tasks, support engineers may require the output of sosreport and foreman-debug when you open a support case.

Important

Both foreman-debug and sosreport remove security information such as passwords, tokens, and keys while collecting information. However, the tar files can still contain sensitive information about the Red Hat Satellite Server. Red Hat recommends that you send this information directly to the intended recipient and not to a public target.

9.4. Using Log Files in Support Cases

You can use the log files and other information described in this chapter to do your own troubleshooting, or you can capture these and many more files, as well as diagnostic and configuration information, to send to Red Hat Support if you need further assistance.

There are two methods to open a support case with Red Hat Support. You can open a support case directly from the Satellite web UI or from the Customer Portal.

9.5. Cleaning Unused Tasks

Cleaning unused tasks reduces disk space in the database and limits the rate of disk growth. When you perform regular cleaning, Satellite backup completes faster and overall performance is higher.

To clean unused tasks

The installer has a feature to enable a cron job to automatically remove old tasks. This feature is not enabled by default to avoid unwanted tasks cleanup.

  1. Enable the cron job: 

    # satellite-installer --foreman-plugin-tasks-automatic-cleanup true

  2. By default, the cron job is scheduled to run every day at 19:45. To adjust to the time, change the value of the --foreman-plugin-tasks-cron-line parameter: 

    # satellite-installer --foreman-plugin-tasks-cron-line "00 15 * * *"

    The previous command schedules the cron job to run every day at 15:00, see man 5 crontab for more details on cron format.

To change the period after which to delete all the tasks and to configure further advanced settings of the cron job, change the content of the /etc/foreman/plugins/foreman-tasks.yaml file.

9.6. Recovering from a Full Disk

The following procedure describes how to resolve the situation when a logical volume (LV) with the Pulp database on it has no free space.

To recover from a full disk

  1. Let running Pulp tasks finish but do not trigger any new ones as they can fail due to the full disk.

  2. Ensure that the LV with the /var/lib/pulp directory on it has sufficient free space. Here are some ways to achieve that:

    1. Remove orphaned content: 

      # foreman-rake katello:delete_orphaned_content RAILS_ENV=production

      This is run weekly so it will not free much space.

    2. Change the download policy from Immediate to On Demand for as many repositories as possible and remove already downloaded packages. See the Red Hat Knowledgebase solution How to change syncing policy for Repositories on Satellite 6.2 from "Immediate" to "On-Demand" on the Red Hat Customer Portal for instructions.

    3. Grow the file system on the LV with the /var/lib/pulp directory on it. For more information, see Growing a File System on a Logical Volume in the Red Hat Enterprise Linux 7 Logical Volume Manager Administration Guide.

      Note

      If you use an untypical file system (other than for example ext3, ext4, or xfs), you might need to unmount the file system so that it is not in use. In that case:

      • Stop Katello services: 
      # katello-service stop
      • Grow the file system on the LV.
      • Start Katello services: 
      # katello-service start
  3. If some Pulp tasks failed due to the full disk, run them again.

9.7. Reclaiming Disk Space from MongoDB

The MongoDB database can use a large amount of disk space especially in heavily loaded deployments. The following procedure describes how to reclaim some of this disk space.

Prerequisites

To reclaim disk space from MongoDB

  1. Access the MongoDB shell: 

    # mongo pulp_database

  2. Check the amount of disk space used by MongoDB before a repair: 

    > db.stats()
  3. Ensure that you have free disk space equal to the size of your current MongoDB database plus 2 GB. If the volume containing the MongoDB database lacks sufficient space, you can mount a separate volume and use that for the repair.

  4. Enter the repair command: 

    > db.repairDatabase()

    Note that the repair command blocks all other operations and can take a long time to complete, depending on the size of the database.

  5. Check the amount of disk space used by MongoDB after a repair: 

    > db.stats()

  6. Start Pulp services: 

    # systemctl start goferd httpd pulp_workers pulp_celerybeat \
pulp_resource_manager pulp_streamer

9.8. Accessing Customer Portal Services from Red Hat Satellite

The Red Hat Access pre-installed plug-in lets you access several Red Hat Customer Portal services from within the Satellite web UI.

The Red Hat Access plug-in provides the following services:

  • Search: Search solutions in the Customer Portal from within the Satellite web UI.
  • Logs: Send specific parts (snippets) of the log files to assist in problem solving. Send these log snippets to the Red Hat Customer Portal diagnostic tool chain.
  • Support: Access your open support cases, modify an open support case and open a new support case from within the Satellite web UI.
Note

To access Red Hat Customer Portal resources, you must log in with your Red Hat Customer Portal user identification and password.

9.8.1. Searching for Solutions in the Red Hat Access Plug-in

The Red Hat Access plug-in provides search capabilities that look through the solutions database available in the Red Hat Customer Portal.

To Search for Solutions from the Red Hat Satellite Server:

  1. In the upper right, click Red Hat Access > Search.

  2. If necessary, log in to the Red Hat Customer Portal. In the main panel on the upper right, click Log In.

    Note

    To access Red Hat Customer Portal resources, you must log in with your Red Hat Customer Portal user identification and password.

  3. In the Red Hat Search field, enter your search query. Search results display in the left-hand Recommendations list.
  4. In the Recommendations list, click a solution. The solution article displays in the main panel.

9.8.2. Using Logs in the Red Hat Access Plug-in

The log file viewer lets you view the log files and isolate log snippets. You can also send the log snippets through the Customer Portal diagnostic tool chain to assist with problem solving.

To Use the Logs Diagnostic Tool from the Red Hat Satellite Server:

  1. In the upper right, click Red Hat Access > Logs.

  2. If necessary, log in to the Red Hat Customer Portal. In the main panel on the upper right, click Log In.

    Note

    To access Red Hat Customer Portal resources, you must log in with your Red Hat Customer Portal user identification and password.

  3. In the left file tree, select a log file and click the file name.
  4. Click Select File. A pop-up window displays the log file contents.
  5. In the log file, highlight any text sections you want diagnosed. The Red Hat Diagnose button displays.
  6. Click Red Hat Diagnose. The system sends the highlighted information to the Red Hat Customer Portal, and provides solutions that closely match the provided log information.

  7. If a solution does the following:

9.8.3. Viewing Existing Support Cases Using the Red Hat Access Plug-in

You can view your existing support case from your Red Hat Satellite Server using the Red Hat Access Plug-in.

To View Existing Support Cases from the Red Hat Satellite Server:

  1. In the upper right, click Red Hat Access > Support > My Cases.

  2. If necessary, log in to the Red Hat Customer Portal. In the main panel on the upper right, click Log In.

    Note

    To access Red Hat Customer Portal resources, you must log in with your Red Hat Customer Portal user identification and password.

  3. To search for a specific support case from existing cases, do any of the following:

    • In the Search field, provide a key word or phrase.
    • From the drop-down list, choose a specific Case Group. Your organization has defined Case Groups inside the Red Hat Customer Portal.
    • Choose a Case Status.
  4. From the results, choose a specific support case and click the Case ID. The support case is ready to view.

9.8.4. Modifying Support Cases Using the Red Hat Access Plug-in

You can update your existing support cases from your Red Hat Satellite Server using the Red Hat Access Plug-in.

To Update Support Cases from the Red Hat Satellite Server Web UI:

  1. Complete the instructions from Section 9.8.3, “Viewing Existing Support Cases Using the Red Hat Access Plug-in”

  2. In the support case, scroll down to the marked sections to do the following:

    • Attachments: - Attach a local file from the system. Add a file name to make it easier to identify.

      Note

      File names must be less than 80 characters and the maximum file size for attachments uploaded using the web UI is 250 MB. Use FTP for larger files.

    • Case Discussion: - Add any updated information about the case you wish to discuss with Global Support Services. After adding information, click Add Comment.

9.8.5. Creating Support Cases Using the Red Hat Access Plug-in

You can create a new support case from your Red Hat Satellite Server using the Red Hat Access Plug-in.

To Create a New Support Case Using the Red Hat Satellite Server:

  1. In the upper right, click Red Hat Access > Support > New Case.

  2. If necessary, log in to the Red Hat Customer Portal. In the main panel on the upper right, click Log In.

    Note

    To access Red Hat Customer Portal resources, you must log in with your Red Hat Customer Portal user identification and password.

  3. The Product and Product Version fields are automatically populated. Complete the other relevant fields, as follows:

    • Summary — Provide a brief summary of the issue.

    • Description — Write a detailed description of the issue.

      Based on the summary provided, recommendations for possible solutions display in the main panel.

  4. Click Next.

  5. Choose the appropriate options, as follows:

    • Severity — Select the ticket urgency as 4 (low), 3 (normal), 2 (high), or 1 (urgent).
    • Case Group — Based on who needs to be notified, create case groups associated with the support case. Select Case Groups in Red Hat Satellite. Create Case Groups within the Customer Portal.

  6. Attach the output of sosreport and any required files. Add a file description and click Attach.

    Note
    • If you have large log files or many Satellite tasks, it is recommended to also attach the output of foreman-debug.
    • File names must be less than 80 characters and the maximum file size for attachments uploaded using the web UI is 250 MB. Use FTP for larger files.
  7. Click Submit. The system uploads the case to the Customer Portal, and provides a case number for your reference.

The Red Hat Knowledgebase article https://access.redhat.com/articles/445443: Red Hat Access: Red Hat Support Tool has additional information, examples, and video tutorials.

9.9. Using Red Hat Insights with Satellite Server

Red Hat Insights enables you to proactively diagnose systems and downtime related to security exploits, performance degradation and stability failures. You can use the dashboard to quickly identify key risks to stability, security, or performance. You can sort by category, view details of the impact and resolution, and then determine what systems are affected.

Red Hat Insights is installed by default on Satellite Server. Before using Insights with Satellite Server, go to Red Hat Insights and click Satellite 6 for the pre-installation checks and to register your Satellite Servers.

9.10. Monitoring Satellite Server in the Web UI

From the About page in the Satellite Server web UI, you can find an overview of the following:

  • System Status, including Capsules, Available Providers, Compute Resources, and Plug-ins
  • Support information
  • System Information
  • Backend System Status
  • Installed packages

To navigate to the About page:

  • In the upper right corner of the Satellite Server web UI, click Administer > About.
Note

After Pulp failure, the status of Pulp might show OK instead of Error for up to 10 minutes due to synchronization delay.