Chapter 6. Troubleshooting

6.1. Debug Logging

By default, virt-who logs all its activity to the file /var/log/rhsm/rhsm.log. When troubleshooting, check the log file as this might reveal useful information. To enable more detailed logging, change the debugging line in the global configuration file /etc/sysconfig/virt-who to VIRTWHO_DEBUG=1. If virt-who is running as a service, you must restart it for the configuration change to take effect. When you have resolved the underlying issue, disable diagnostic logging by changing the debugging line back to VIRTWHO_DEBUG=0 and restarting the virt-who service.

6.2. Duplicate Configuration Lines

Since there can be multiple configuration files, both global and hypervisor-specific, duplicate configuration lines might result in virt-who behaving differently to what you intend.

To detect duplicate lines in the virt-who configuration files, use the following command. The output of this command is a list of all lines in the specified files, prefixed by the number of times it occurs. Check all instances where the same line is listed as occurring twice or more, remove the duplicate line and restart the virt-who service. For instructions see Section 5.8.1, “Restarting the virt-who Service”.

# cat /etc/sysconfig/virt-who /etc/virt-who.d/* | sort | uniq -c

6.3. Credentials

Incorrect credentials can be a source of virt-who failure. If possible, test the credentials configured for use by virt-who by logging in to the virtualization manager or hypervisor. For example, if you can log in to the VMware vSphere management console and the expected hosts are visible, then credentials are correct.

6.4. Testing Configuration Options

When troubleshooting, a common method of determining the root cause of a problem is to make a change and test the result, repeating as needed. The virt-who agent provides an option to help with this technique.

Run the command virt-who --one-shot which reads all configuration files, retrieves the list of virtual machines from all sources, then exits immediately. This tests the configuration files, credentials and connectivity to configured virtualization platforms.

# virt-who --one-shot

The output you can expect is a list of hypervisors and the hosted guest virtual machines, in JSON format. The following is an extract from virt-who output from a VMware vSphere instance. The output from all hypervisors follows the same structure.

{
    "guestId": "422f24ed-71f1-8ddf-de53-86da7900df12",
    "state": 5,
    "attributes": {
        "active": 0,
        "virtWhoType": "esx",
        "hypervisorType": "vmware"
    }
},

6.5. Example Scenarios

6.5.1. Virt-who fails to connect with the virtualization platform

Check the Red Hat Subscription Manager log file - /var/log/rhsm/rhsm.log - if virt-who fails to connect with the virtualization platform. If you find the message No route to host, one possible reason is that the hypervisor is listening on a port other than what you expect. For example, Red Hat Virtualization Manager defaults to port 8443 for backward compatibility, but virt-who defaults to using port 443. In this case, you would edit the hypervisor’s configuration file in /etc/virt-who.d/ and append :443 to the value for the server line, resulting in the line: server=https://rhevmhost1.example.com:443.

  • You can also consider to configure virt-who to use no proxy by adding NO_PROXY=* to /etc/sysconfig/virt-who. Note that the values in /etc/sysconfig/virt-who are environment variables, and are sourced during daemon runs. If running virt-who in one-shot mode, export the values in /etc/sysconfig/virt-who first. Note that the required package versions are: python-rhsm >= 1.17.9-1 and virt-who >= 0.17-11.

    # set -a
    # source /etc/sysconfig/virt-who
    # virt-who -o

6.5.2. Configure virt-who to use an internal proxy

To configure virt-who to use an internal proxy instead of the external proxy Subscription Management Server uses to connect to the external networks(for example, the CDN), add rhsm_proxy_hostname and rhsm_proxy_port to the virt-who configuration file in /etc/virt-who.d/. Note that the virt-who version must be >= 0.14. For example:

# vi /etc/virt-who.d/fabric-1.conf

rhsm_proxy_hostname = internal-proxy.example.com
rhsm_proxy_port = 3128

Optionally specify rhsm_proxy_user and rhsm_proxy_password in the same configuration file if required.

6.6. Preventing auto-attach from being applied

Complete this procedure to prevent unintended subscriptions from being attached using the auto-attach feature.

  1. Determine if auto-attach is enabled:
# subscription-manager auto-attach --show
  1. Disable auto-attach:
# subscription-manager auto-attach --disable

6.7. Renewing Host Subscriptions

This section covers three methods to reattach subscriptions for multiple hosts. The following use cases apply:

  • When host subscriptions have expired and you need to attach new valid subscriptions.
  • When host subscriptions are still valid but you need to attach additional subscriptions.

If host subscriptions have expired, but you have configured auto-attach and virt-who previously, subscription manager will attempt to reattach a valid subscription that covers the host and its virtual machines based on a set of criteria. No action is required.

6.7.1. Using Web UI

The web UI method allows you to attach multiple subscriptions to multiple hosts at the same time.

  1. Click HostsContent Hosts. If prompted, select the desired organization.
  2. Select the desired hosts. You can use the filter function to narrow down the list of hosts you want to attach subscriptions to. Use the check box at the top to select all hosts listed.
  3. Click Select ActionManage Subscriptions.
  4. Select the desired subscriptions, and click Add Selected.

When all selected subscriptions have been attached, the task result displays success. To confirm, go to HostsContent Hosts and select the desired host. Click SubscriptionsSubscriptions, and verify that the newly attached subscriptions are listed.

6.7.2. Using Hammer CLI

The Hammer CLI method allows you to update the subscriptions iteratively per host, or script and automate the action for multiple hosts.

  1. List available subscriptions in the organization.

    # hammer --output json subscription list --organization example
    
    [
    {
      "ID": 192,
      "UUID": "2c918093561eaa39015630f5cd841d56",
      "Name": "Red Hat Enterprise Linux Server, Premium (Physical or Virtual Nodes)",
       ...
    }]
  2. Search for hosts that do not have a valid subscription.

    # hammer host list --search "subscription_status = invalid"
    
    ---|---------------------------|------------------|---------------
    ID | NAME                      | OPERATING SYSTEM | HOST GROUP
    ---|---------------------------|------------------|---------------
    45 | cloudforms.example.com    | RedHat 7.2       | Infrastructure
    84 | devnode-146.example.com   | RedHat 7.2       | Wordpress
    82 | virt-testing.example.com  | RedHat 7.1       | Development
    ---|---------------------------|------------------|---------------
  3. Attach a subscription to the desired host.

    # hammer host subscription attach --host devnode-146.example.com --quantity 2 --subscription-id 192
    
    Subscription attached to the host successfully
  4. Confirm the subscription has been successfully attached.

    # hammer host list --search "subscription_status = invalid"
    
    ---|---------------------------|------------------|---------------
    ID | NAME                      | OPERATING SYSTEM | HOST GROUP
    ---|---------------------------|------------------|---------------
    45 | cloudforms.example.com    | RedHat 7.2       | Infrastructure
    82 | virt-testing.example.com  | RedHat 7.1       | Development
    ---|---------------------------|------------------|---------------

6.7.3. Using CSV Export and Import

The CSV method also uses the Hammer CLI tool and allows you to back up the mapping information of subscriptions, hosts, and activation keys and import it back to Satellite to ensure that each hosts gets the right subscriptions attached. To use this method for subscription renewal, you need to export the CSV file before the subscriptions expire.

  1. Export the CSV file. It is recommended to add this to a cron job so that the subscription status of all the hosts are always backed up.

    # hammer csv content-hosts --export --file content-hosts-export.csv --itemized-subscriptions --organization example
  2. Edit the CSV file to include new subscription details, for example the new contract numbers.
  3. Import the CSV file back to the host when your hosts' subscriptions expire, and you need to re-attach subscriptions.

    # hammer csv content-hosts --file content-hosts-export.csv --itemized-subscriptions --organization example