4.2. Log Files

4.2.1. Manager Installation Log Files

Table 4.2. Installation

Log FileDescription

/var/log/ovirt-engine/engine-cleanup_yyyy_mm_dd_hh_mm_ss.log

Log from the engine-cleanup command. This is the command used to reset a Red Hat Virtualization Manager installation. A log is generated each time the command is run. The date and time of the run is used in the filename to allow multiple logs to exist.

/var/log/ovirt-engine/engine-db-install-yyyy_mm_dd_hh_mm_ss.log

Log from the engine-setup command detailing the creation and configuration of the engine database.

/var/log/ovirt-engine/ovirt-engine-dwh-setup-yyyy_mm_dd_hh_mm_ss.log

Log from the ovirt-engine-dwh-setup command. This is the command used to create the ovirt_engine_history database for reporting. A log is generated each time the command is run. The date and time of the run is used in the filename to allow multiple logs to exist concurrently.

/var/log/ovirt-engine/setup/ovirt-engine-setup-yyyymmddhhmmss.log

Log from the engine-setup command. A log is generated each time the command is run. The date and time of the run is used in the filename to allow multiple logs to exist concurrently.

4.2.2. Red Hat Virtualization Manager Log Files

Table 4.3. Service Activity

Log FileDescription

/var/log/ovirt-engine/engine.log

Reflects all Red Hat Virtualization Manager GUI crashes, Active Directory lookups, Database issues, and other events.

/var/log/ovirt-engine/host-deploy

Log files from hosts deployed from the Red Hat Virtualization Manager.

/var/lib/ovirt-engine/setup-history.txt

Tracks the installation and upgrade of packages associated with the Red Hat Virtualization Manager.

/var/log/httpd/ovirt-requests-log

Logs files from requests made to the Red Hat Virtualization Manager via HTTPS, including how long each request took.

A Correlation-Id header is included to allow you to compare requests when comparing a log file with /var/log/ovirt-engine/engine.log.

/var/log/ovn-provider/ovirt-provider-ovn.log

Logs the activities of the OVN provider. For information about Open vSwitch logs, see the Open vSwitch documentation.

4.2.3. SPICE Log Files

SPICE log files are useful when troubleshooting SPICE connection issues. To start SPICE debugging, change the log level to debugging. Then, identify the log location.

Both the clients used to access the guest machines and the guest machines themselves have SPICE log files. For client-side logs, if a SPICE client was launched using the native client, for which a console.vv file is downloaded, use the remote-viewer command to enable debugging and generate log output.

4.2.3.1. SPICE Logs for Hypervisor SPICE Servers

Table 4.4. SPICE Logs for Hypervisor SPICE Servers

Log TypeLog LocationTo Change Log Level:

Host/Hypervisor SPICE Server

/var/log/libvirt/qemu/(guest_name).log

Run export SPICE_DEBUG_LEVEL=5 on the host/hypervisor prior to launching the guest. This variable is parsed by QEMU, and if run system-wide will print the debugging information of all virtual machines on the system. This command must be run on each host in the cluster. This command works only on a per-host/hypervisor basis, not a per-cluster basis.

4.2.3.2. SPICE Logs for Guest Machines

Table 4.5. spice-vdagent Logs for Guest Machines

Log TypeLog LocationTo Change Log Level:

Windows Guest

C:\Windows\Temp\vdagent.log

C:\Windows\Temp\vdservice.log

Not applicable

Red Hat Enterprise Linux Guest

Use journalctl as the root user.

To run the spice-vdagentd service in debug mode, as the root user create a /etc/sysconfig/spice-vdagentd file with this entry: SPICE_VDAGENTD_EXTRA_ARGS=”-d -d”

To run spice-vdagent in debug mode, from the command line:

$ killall - u $USER spice-vdagent
$ spice-vdagent -x -d [-d] [ |& tee spice-vdagent.log ]

4.2.3.3. SPICE Logs for SPICE Clients Launched Using console.vv Files

For Linux client machines:

  1. Enable SPICE debugging by running the remote-viewer command with the --spice-debug option. When prompted, enter the connection URL, for example, spice://virtual_machine_IP:port.

    #  remote-viewer --spice-debug
  2. To run SPICE client with the debug parameter and to pass a .vv file to it, download the console.vv file and run the remote-viewer command with the --spice-debug option and specify the full path to the console.vv file.

    # remote-viewer --spice-debug /path/to/console.vv

For Windows client machines:

  1. In versions of virt-viewer 2.0-11.el7ev and later, virt-viewer.msi installs virt-viewer and debug-viewer.exe.
  2. Run the remote-viewer command with the spice-debug argument and direct the command at the path to the console:

    remote-viewer --spice-debug path\to\console.vv
  3. To view logs, connect to the virtual machine, and you will see a command prompt running GDB that prints standard output and standard error of remote-viewer.

4.2.4. Host Log Files

Log FileDescription

/var/log/messages

The log file used by libvirt. Use journalctl to view the log. You must be a member of the adm, systemd-journal, or wheel groups to view the log.

/var/log/vdsm/spm-lock.log

Log file detailing the host’s ability to obtain a lease on the Storage Pool Manager role. The log details when the host has acquired, released, renewed, or failed to renew the lease.

/var/log/vdsm/vdsm.log

Log file for VDSM, the Manager’s agent on the host(s).

/tmp/ovirt-host-deploy-Date.log

A host deployment log that is copied to the Manager as /var/log/ovirt-engine/host-deploy/ovirt-Date-Host-Correlation_ID.log after the host has been successfully deployed.

/var/log/vdsm/import/import-UUID-Date.log

Log file detailing virtual machine imports from a KVM host, a VMWare provider, or a RHEL 5 Xen host, including import failure information. UUID is the UUID of the virtual machine that was imported and Date is the date and time that the import began.

/var/log/vdsm/supervdsm.log

Logs VDSM tasks that were executed with superuser permissions.

/var/log/vdsm/upgrade.log

VDSM uses this log file during host upgrades to log configuration changes.

/var/log/vdsm/mom.log

Logs the activities of the VDSM’s memory overcommitment manager.

4.2.5. Setting debug-level logging for Red Hat Virtualization services

Note

Setting logging to debug-level may expose sensitive information such as passwords or internal VM data. Make sure that non-trusted or unauthorized users do not have access to debug logs.

You can set the logs of the following Red Hat Virtualization (RHV) services to debug-level by modifying the sysconfig file of each service.

Table 4.6. RHV services and sysconfig file paths

ServiceFile path

ovirt-engine.service

/etc/sysconfig/ovirt-engine

ovirt-engine-dwhd.service

/etc/sysconfig/ovirt-engine-dwhd

ovirt-fence-kdump-listener.service

/etc/sysconfig/ovirt-fence-kdump-listener

ovirt-websocket-proxy.service

/etc/sysconfig/ovirt-websocket-proxy

This modification affects logging done by the Python wrapper, not the main service process.

Setting logging to debug-level is useful for debugging issues related to start up - for example, if the main process fails to start due to a missing or incorrect Java runtime or library.

Prerequisites

  • Verify that the sysconfig file you want to modify exists. If necessary, create it.

Procedure

  1. Add the following to the sysconfig file of the service:

    OVIRT_SERVICE_DEBUG=1
  2. Restart the service:

    # systemctl restart <service>

The sysconfig log file of the service is now set to debug-level.

Logging caused by this setting goes to the system log, so the logs it generates can be found in /var/log/messages, not in the service-specific log file, or by using the journalctl command.

4.2.6. Main configuration files for Red Hat Virtualization services

In addition to a sysconfig file, each of these Red Hat Virtualization (RHV) services has another configuration file that is used more often.

Table 4.7. RHV services and configuration files

Servicesysconfig file pathMain configuration file

ovirt-engine.service

/etc/sysconfig/ovirt-engine

/etc/ovirt-engine/engine.conf.d/*.conf

ovirt-engine-dwhd.service

/etc/sysconfig/ovirt-engine-dwhd

/etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/*.conf

ovirt-fence-kdump-listener.service

/etc/sysconfig/ovirt-fence-kdump-listener

/etc/ovirt-engine/ovirt-fence-kdump-listener.conf.d/*.conf

ovirt-websocket-proxy.service

/etc/sysconfig/ovirt-websocket-proxy

/etc/ovirt-engine/ovirt-websocket-proxy.conf.d/*.conf

4.2.7. Setting Up a Host Logging Server

Hosts generate and update log files, recording their actions and problems. Collecting these log files centrally simplifies debugging.

This procedure should be used on your centralized log server. You could use a separate logging server, or use this procedure to enable host logging on the Red Hat Virtualization Manager.

Procedure

  1. Check to see if the firewall allows traffic on the UDP 514 port, and is open to syslog service traffic:

    # firewall-cmd --query-service=syslog

    If the output is no, allow traffic on the UDP 514 port with:

    # firewall-cmd --add-service=syslog --permanent
    # firewall-cmd --reload
  2. Create a new .conf file on the syslog server, for example, /etc/rsyslog.d/from_remote.conf, and add the following lines:

    template(name="DynFile" type="string" string="/var/log/%HOSTNAME%/%PROGRAMNAME%.log")
    RuleSet(name="RemoteMachine"){  action(type="omfile" dynaFile="DynFile")  }
    Module(load="imudp")
    Input(type="imudp" port="514" ruleset="RemoteMachine")
  3. Restart the rsyslog service:

    # systemctl restart rsyslog.service
  4. Log in to the hypervisor, and in the /etc/rsyslog.conf add the following line:

    *.info;mail.none;authpriv.none;cron.none @<syslog-FQDN>:514
  5. Restart the rsyslog service on the hypervisor.

    # systemctl restart rsyslog.service

Your centralized log server is now configured to receive and store the messages and secure logs from your virtualization hosts.

4.2.8. Enabling SyslogHandler to pass RHV Manager logs to a remote syslog server

This implementation uses the JBoss EAP SyslogHandler log manager and enables passing log records from the engine.log and server.log to a syslog server.

Note

RHV versions earlier than RHV 4.4.10 featured similar functionality provided by ovirt-engine-extension-logger-log4j. That package was removed in RHV 4.4.10 and replaced by a new implementation using the JBoss EAP SyslogHandler log manager. If you have been using ovirt-engine-extension-logger-log4j in earlier RHV versions, following an upgrade to RHV 4.4.10, perform following steps:

  • Manually configure sending log records to a remote syslog server using the guidelines provided in this chapter.
  • Manually remove the ovirt-engine-extension-logger-log4j configuration files (remove the /etc/ovirt-engine/extensions.d/Log4jLogger.properties configuration file).

Use this procedure on the central syslog server. You can use a separate logging server, or use this procedure to pass the engine.log and server.log files from the Manager to the syslog server. See also the configuration procedure Setting up a Host Logging Server.

Configuring the SyslogHandler implementation

  1. Create the configuration file 90-syslog.conf in the /etc/ovirt-engine/engine.conf.d directory and add the following content:

    SYSLOG_HANDLER_ENABLED=true
    SYSLOG_HANDLER_SERVER_HOSTNAME=localhost
    SYSLOG_HANDLER_FACILITY=USER_LEVEL
  2. Install and configure rsyslog.

    # dnf install rsyslog
  3. Configure SELinux to allow rsyslog traffic.

    # semanage port -a -t syslogd_port_t -p udp 514
  4. Create the configuration file /etc/rsyslog.d/rhvm.conf and add the following content:

    user.* /var/log/jboss.log
    module(load="imudp") # needs to be done just once
    input(type="imudp" port="514")
  5. Restart the rsyslog service.

    # systemctl restart rsyslog.service
  6. If the firewall is enabled and active, run the following command to add the necessary rules for opening the rsyslog ports in Firewalld:

    # firewall-cmd --permanent --add-port=514/udp
    # firewall-cmd --reload
  7. Restart Red Hat Virtualization Manager.

    # systemctl restart ovirt-engine

The syslog server can now receive and store the engine.log files.