Virtual Machine cannot be started or migrated to particular Red Hat Virtualization Host

Solution Unverified - Updated -

Environment

Red Hat Virtualization 4.1

Issue

Starting or Migrating Virtual machine or specific Red Hat Virtaulziation Host fails with qemu error:

Failed to bind socket to /var/lib/libvirt/qemu/channels/cc10a13e-b21f-4de4-9ed6-3326b7043838.com.redhat.rhevm.vdsm: Permission denied

Resolution

Reinstall the host to clean up HE configurations from another setup.

Root Cause

They host was brought up to this environment from another environment without reinstalling and some leftover HE configs prevented the host from joining the new setup successfully.

Diagnostic Steps

  • Engine logs:

    • Failed Migration attempt

      2017-08-24 08:16:15,307-03 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VM_MIGRATION_TRYING_RERUN(128), Correlation ID: 6c279a5a-f317-4e87-a6f1-f990765f3ac7, Job ID: fe323732-a590-4c00-bf2f-e971c2ca930a, Call Stack: null, Custom Event ID: -1, Message: Failed to migrate VM <VM-name> to Host <Host-name> . Trying to migrate to another Host.
      
    • Failed VM start attempt:

      2017-08-30 12:25:38,907-03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-8) [] EVENT_ID: VM_DOWN_ERROR(119), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM <VM-name> is down with error. Exit message: internal error: process exited while connecting to monitor: 2017-08-30T15:25:35.833697Z qemu-kvm: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/cc10a13e-b21f-4de4-9ed6-3326b7043838.com.redhat.rhevm.vdsm,server,nowait: Failed to bind socket to /var/lib/libvirt/qemu/channels/cc10a13e-b21f-4de4-9ed6-3326b7043838.com.redhat.rhevm.vdsm: Permission denied.
      
  • qemu logs

    2017-08-24T11:16:01.687145Z qemu-kvm: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/cc10a13e-b21f-4de4-9ed6-3326b7043838.com.redhat.rhevm.vdsm,server,nowait: Failed to bind socket to /var/lib/libvirt/qemu/channels/cc10a13e-b21f-4de4-9ed6-3326b7043838.com.redhat.rhevm.vdsm: Permission denied
    
  • vdsm logs:

    2017-08-24 08:16:00,119-0300 ERROR (jsonrpc/2) [virt.vm] (vmId='cc10a13e-b21f-4de4-9ed6-3326b7043838') Alias not found for device type lease during migration at destination host (vm:4587)
    
  • Changing the owner and group of /var/lib/libvirt/qemu/channels/ due to permission denied error in the logs did not helped to resolve the issue:

      # chown vdsm /var/lib/libvirt/qemu/channels
      # ls -ld /var/lib/libvirt/qemu/channels
      drwxrwxr-x. 2 vdsm qemu 6 Jul 25 05:01 /var/lib/libvirt/qemu/channels
      ## put host in maintenance and perform:
      # systemctl restart libvirtd
      # systemctl restart vdsmd
    
  • Unable to reproduce this issue at this moment.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.