Chapter 5. Troubleshooting

5.1. Verifying virt-who Status

Verifying the status of all virt-who instances is the recommended first step in troubleshooting. All virt-who instances configured and deployed using either the Satellite web UI or hammer are reported in the Satellite web UI. Those virt-who instances which were configured manually are not available in the Satellite web UI.

5.1.1. Verifying virt-who Status in Satellite web UI

  1. Navigate to Infrastructure → Virt-who configurations
  2. View the Status value for each virt-who instance.

A status value of OK indicates the virt-who instance is successfully connecting to the Satellite Server and reporting those virtual machines managed by the hypervisor.

5.1.2. Verifying virt-who Status using Hammer

On the Satellite Server, list the status of all virt-who instances. 

# hammer virt-who-config list

The hammer output includes the date and time at which each virt-who instance reported to the Satellite Server.

5.2. Debug Logging

By default, virt-who logs all its activity to the file /var/log/rhsm/rhsm.log. When troubleshooting, check the log file as this might reveal useful information. To enable more detailed logging, change the debugging line in the global configuration file /etc/sysconfig/virt-who to VIRTWHO_DEBUG=1. If virt-who is running as a service, you must restart it for the configuration change to take effect. When you have resolved the underlying issue, disable diagnostic logging by changing the debugging line back to VIRTWHO_DEBUG=0 and restarting the virt-who service.

5.3. Duplicate Configuration Lines

Since there can be multiple configuration files, both global and hypervisor-specific, duplicate configuration lines might result in virt-who behaving differently to what you intend.

To detect duplicate lines in the virt-who configuration files, use the following command. The output of this command is a list of all lines in the specified files, prefixed by the number of times it occurs. Check all instances where the same line is listed as occurring twice or more, remove the duplicate line and restart the virt-who service. For instructions see Section 3.6, “Restarting the virt-who Service”. 

# cat /etc/sysconfig/virt-who /etc/virt-who.d/* | sort | uniq -c

5.4. Comment Character in a Configuration File

The comment character # must be the first character on the line in a configuration file. Any white space at the start of the line results in the line being included in the configuration.

5.5. Credentials

Incorrect credentials can be a source of virt-who failure. If possible, test the credentials configured for use by virt-who by logging in to the virtualization manager or hypervisor. For example, if you can log in to the VMware vSphere management console and the expected hosts are visible, then credentials are correct.

5.6. Testing Configuration Options

When troubleshooting, a common method of determining the root cause of a problem is to make a change and test the result, repeating as needed. The virt-who agent provides an option to help with this technique.

Run the command virt-who --one-shot which reads all configuration files, retrieves the list of virtual machines from all sources, then exits immediately. This tests the configuration files, credentials and connectivity to configured virtualization platforms. 

# virt-who --one-shot

The output you can expect is a list of hypervisors and the hosted guest virtual machines, in JSON format. The following is an extract from virt-who output from a VMware vSphere instance. The output from all hypervisors follows the same structure. 

{
    "guestId": "422f24ed-71f1-8ddf-de53-86da7900df12",
    "state": 5,
    "attributes": {
        "active": 0,
        "virtWhoType": "esx",
        "hypervisorType": "vmware"
    }
},

5.7. Identifying issues when using multiple virt-who configuration files

If you have multiple virt-who configuration files, move one virt-who configuration file at a time to your /root directory while testing after each file move. If the issue no longer occurs, the cause is associated with the most recently moved file. After you have resolved the issue, return the virt-who configuration files to their original location.

5.8. Example Scenarios

5.8.1. Virt-who fails to connect with the virtualization platform

If virt-who fails to connect to the virtualization platform check the Red Hat Subscription Manager log file /var/log/rhsm/rhsm.log. If you find the message No route to host, one possible reason is that the hypervisor is listening on a port other than what you expect. For example, Red Hat Virtualization Manager defaults to port 8443 for backward compatibility, but virt-who defaults to using port 443. In this case, edit the hypervisor’s configuration file in directory /etc/virt-who.d/ and append :443 to the value for the server line, resulting in the line: server=https://rhevmhost1.example.com:443.

5.8.2. Hypervisors are listed in the Satellite web UI by their UUID, not their host name

By default, hypervisors are identified in the Satellite web UI by their UUID. It is possible to change this so that they are identified by host name, but this configuration change must be made before Satellite is started, otherwise the hypervisors will be duplicated. If you need to change this, raise a Support Ticket with Red Hat Support.

5.8.3. Virt-who attempts to connect to virtualization manager or hypervisor via an HTTP proxy on the local network fails

There are three workarounds:

  • Configure the proxy to allow local traffic to pass through. (Recommended)
  • If allowing local traffic to pass through is not possible, install a Squid proxy server on the Satellite Server. For further details, see the Red Hat Knowledgebase solution How to bypass proxy for certain repository URLs on Satellite 6.

  • You can also configure virt-who to ignore proxy network settings by adding NO_PROXY=* to /etc/sysconfig/virt-who. Note that the values in /etc/sysconfig/virt-who are environment variables, and are sourced during daemon runs. If running virt-who in one-shot mode, export the values in /etc/sysconfig/virt-who first. Note that the required package versions are: python-rhsm >= 1.17.9-1 and virt-who >= 0.17-11. 

    # set -a
# source /etc/sysconfig/virt-who
# virt-who -o

5.8.4. Configure virt-who to use an internal proxy

To configure virt-who to use an internal proxy, add options rhsm_proxy_hostname and rhsm_proxy_port to the virt-who configuration file in /etc/virt-who.d/. Note that the virt-who version must be >= 0.14. For example: 

# vi /etc/virt-who.d/fabric-1.conf

rhsm_proxy_hostname = internal-proxy.example.com
rhsm_proxy_port = 3128

Optionally specify rhsm_proxy_user and rhsm_proxy_password in the same configuration file if required.