Chapter 6. Virt-who troubleshooting methods

Verifying virt-who status

Verify the status of the virt-who service: 

# systemctl status virt-who.service

Debug logging

Check the /var/log/rhsm/rhsm.log file, where virt-who logs all its activity by default.

For more detailed logging, enable the debugging option in the /etc/virt-who.conf file: 

[global]
debug=True

Restart the virt-who service for the change to take effect.

When the underlying issue is resolved, modify the /etc/virt-who.conf file to disable debugging, then restart the virt-who service.

Testing configuration options

Make a change and test the result, repeating as needed. Virt-who provides three options to help test the configuration files, credentials, and connectivity to the virtualization platform:

  • The virt-who --one-shot command reads the configuration files, retrieves the list of virtual machines and sends it to the subscription management system, such as Red Hat Satellite or Red Hat Subscription Management, then exits immediately.
  • The virt-who --print command reads the configuration files and prints the list of virtual machines, but does not send it to the subscription management system.

  • Starting with RHEL 9 Beta, the virt-who --status command reads the configuration files and outputs a summary of the connection status for both the source and destination systems.

    • The virt-who --status command with the --json option provides additional connectivity data, in JSON format, for each configuration.

The expected output of the virt-who --one-shot and virt-who --print commands is a list of hypervisors and their virtual machines, in JSON format. The following is an extract from a VMware vSphere instance. The output from all hypervisors follows the same structure. 

{
    "guestId": "422f24ed-71f1-8ddf-de53-86da7900df12",
    "state": 5,
    "attributes": {
        "active": 0,
        "virtWhoType": "esx",
        "hypervisorType": "vmware"
    }
},

The expected output for the virt-who --status command is a plain-text summary of the connection status for each configuration in virt-who. 

+-------------------------------------------+
           Configuration Status
+-------------------------------------------+
Configuration Name: esx_config1
Source Status: success
Destination Status: success

Configuration Name: hyperv-55
Source Status: failure
Destination Status: failure

The expected output for the virt-who --status command with the --json option provides additional information about each configuration, including its last successful run, in JSON format. This output also includes details about the success or failure status of each configuration.

  • When the status report indicates a configuration success, the JSON output includes the number of hypervisors and guests that virt-who reported during its last successful run cycle.
  • When the status report indicates a configuration failure, the JSON output includes the associated error message. 
"configurations": [
    {
        "name":"esx-conf1",
        "source":{
            "connection":"https://esx_system.example.com",
            "status":"success",
            "last_successful_retrieve":"2020-02-28 07:25:25 UTC",
            "hypervisors":20,
            "guests":37
        },
        "destination":{
            "connection":"candlepin.example.com",
            "status":"success",
            "last_successful_send":"2020-02-28 07:25:27 UTC",
            "last_successful_send_job_status":"FINISHED"
        }
    },
    {
        "name":"hyperv-55",
        "source":{
            "connection":"windows10-3.company.com",
            "status":"failure",
            "message":"Unable to connect to server: invalid credentials",
            "last_successful_retrieve":null
        },
        "destination":{
            "connection":"candlepin.company.com",
            "status":"failure",
            "message":"ConnectionRefusedError: [Errno 111] Connection refused",
            "last_successful_send":null,
            "last_successful_send_job_status":null
        }
    }
]
}

The virt-who --status command can also be used with the --debug and --config options to provide additional information about the configuration files.

Identifying issues when using multiple virt-who configuration files

If you have multiple virt-who configuration files on one server, move one file at a time to a different directory while testing after each file move. If the issue no longer occurs, the cause is associated with the most recently moved file. After you have resolved the issue, return the virt-who configuration files to their original location.

Alternatively, you can test an individual file after moving it by using the --config option to specify its location. For example: 

# virt-who --debug --one-shot --config /tmp/conf_name.conf

Starting with RHEL 9 Beta, you can enter virt-who --status with the --debug and --config options to identify the configuration file causing the issue without removing any other files from the directory. For example: 

#virt-who --debug --status --config /tmp/conf_name.conf

You can also enter the command with the --json option to view more detailed information about each configuration in JSON format. For example: 

#virt-who --debug --status --json --config /tmp/conf_name.conf

Identifying duplicate hypervisors

Duplicate hypervisors can cause subscription and entitlement errors. Enter the following commands to check for duplicate hypervisors: 

# systemctl stop virt-who
# virt-who -op >/tmp/virt-who.json
# systemctl start virt-who
# cat /tmp/virt-who.json | json_reformat | grep name | sort | uniq -c | sort -nr | head -n10
  3    "name": "localhost"
  1    "name": "rhel1.example.com"
  1    "name": "rhel2.example.com"
  1    "name": "rhel3.example.com"
  1    "name": "rhel4.example.com"
  1    "name": "rhvh1.example.com"
  1    "name": "rhvh2.example.com"
  1    "name": "rhvh3.example.com"
  1    "name": "rhvh4.example.com"
  1    "name": "rhvh5.example.com"

In this example, three hypervisors have the same FQDN (localhost), and must be corrected to use unique FQDNs.

Identifying duplicate virtual machines

Enter the following commands to check for duplicate virtual machines: 

# systemctl stop virt-who
# virt-who -op >/tmp/virt-who.json
# systemctl start virt-who
# cat /tmp/virt-who.json | json_reformat | grep "guestId" | sort | uniq -c | sort -nr | head -n10

Checking the number of hypervisors

Enter the following commands to check the number of hypervisors virt-who currently reports: 

# systemctl stop virt-who
# virt-who -op >/tmp/virt-who.json
# systemctl start virt-who
# cat /tmp/virt-who.json | json_reformat | grep name | sort | uniq -c | wc -l

Starting with RHEL 9 Beta, enter the following command to check the number of hypervisors that virt-who reported during its last successful run cycle: 

# virt-who --status --json

Checking the number of virtual machines

Enter the following commands to check the number of virtual machines that virt-who currently reports: 

# systemctl stop virt-who
# virt-who -op >/tmp/virt-who.json
# systemctl start virt-who
# cat /tmp/virt-who.json | json_reformat | grep "guestId" | sort | uniq -c | wc -l

Starting with RHEL 9 Beta, enter the following command to check the number of guests that virt-who reported during its last successful run cycle: 

# virt-who --status --json