Lost all VM configuration after yum update.

Latest response

After I performed a redhat yum update, it turns out that all my configuration from the Redhat Virtualization console are gone / disappeared from the web management console.

https://zyx:9090/machines

  • VMs gone
  • Storage pool gone
  • Network gone

Everything is completely invisible. But command line shows all the VM still exists behind the scene.

Starting a domain works, but it is not visible in the web console.
- virsh start dom00

List shows all VM exists.
└# virsh list --all
Please enter your authentication name: admin
Please enter your password:

Id Name State

  • abc00 shut off
  • pol01 shut off

Another problem, it suddenly started asking me for authentication for each time the command run, which is annoying. Any way to stop this. please help.

Thanks,

Responses

Sunderrajan Narasimhan,

Considering the dire severity of this, please submit a priority case with Red Hat since Red Hat Virtualization is a supported product. We are overwhelmingly volunteers here in the discussion area.

Kind Regards,
RJ

Apologies, this is not a dire issue. I only testing this product on my home computer : Dell Precision 7750. I am using free developer subscription.
Why did yum update blow away all my settings on the web console is a mystery, hence I posted for folks with similar issues.

I'm not familiar with Red Hat Virtualization - but check some basic things. I caveat this below to say I'm not familiar with RHV and have not used it.

2nd update: NOTE: I've asked to see if someone else can assist with this. I'm definitely not familiar with RHV and can't find anything definitive to directly address your situation.

  • UPDATE/EDITED Please see this documentation to see if it is relevant, namely the part after upgrade.
  • Consider one-time booting into the previous kernel
  • Are you able to mount any of the expected storage devices?
  • Is there anything useful in the logs under /var/log/ directory? Please scour the logs for clues
  • Consider one-time boot into non-SELinux using this Red Hat solution, I'm not saying turn off SELinux permanently, just one time, and if your system comes up as expected, you have an SELinux issue
  • Examine your system for filled partitions df -kh and look for anything at 100%
  • Run systemctl | grep fail as root, and for failed services, run journalctl -xeu name_of_service_goes_here
  • You said your network is down, please also check the basics with that, physical network connections etc.
  • This example article has an example here is OVIRTRHV where someone may have fixed a similar issue? Please evaluate for yourself.
  • Again, it will be good to evaluate the logs and mounts.

I hope this helps, else, we can attempt to re-approach, or I can ask some of other members in the Accelerator group.

Regards,
RJ

Thanks RJ for the responses. - I already saw the documentation and didn't find anything relevant. - Yes, all the devices are mountable and it does show in terminal but not in the web console storage pool. - virsh start ab00 works well, but it does not show up on the web management console under VM section. - Nothing useful in /var/log/messages or anywhere else. - RH Virt Version: 4.4 and RHEL is : 8.3 - This is a brand new system, nothing is at 100% disk usage. - systemctl |grep fail as root, nothing has failed. - Network is not down, it is up but as I said this can't be seen in the web console under the VM section. - System > Storage and System > Network in the web console, we can see all the settings. - I don't think yum update failed for me.

Sunderrajan Narasimhan

I've asked to see if someone else can assist with this. I'm definitely not familiar with RHV and can't find anything definitive to directly address your situation.

What version of RHV are you using? Also, what version of RHEL are you using?

When you upgraded, did your version of RHV go to a higher version too? Again, I'm not familiar with RHV, however there are potential other steps if your upgrade also did an upgrade of RHV

Regards,
RJ

RHV is at 4.4 and RHEL at 8.3. I think RHV was already at 4.4 before the upgrade. This is a new install, less than a month ago.

Sunderrajan Narasimhan

I've asked some others to look at this, please be prepared to examine the logs and some of the other details above. Please consider these posting tips because there's more detail needed, and you are our eyes/ears to help you.

Kind Regards,
RJ

I do have a corporate account with RedHat, in order for me to test it out in a sandbox in corporate account, I am trying to find out if this is the right product for our use case.

[I deleted this post]

Hi Sunderrajan Narasimhan,

I've asked to see if someone else can assist with this. I'm definitely not familiar with RHV and can't find anything definitive to directly address your situation.

Please see the link in the last bit I posted in my original post where I mention "This example article has an example here is OVIRTRHV where someone may have fixed a similar issue? Please evaluate for yourself." However, you seem to have stayed at the same version?

To answer your reply a little higher in this discussion -- You can certainly call them, they'll review your subscriptions.

I hope the info above helps you, else I'll attempt to see if I can get someone else to attempt to help you, perhaps I can summon a bona fide Red-Hatter, or another member of the Accelerator group.

Kind Regards,
RJ

Sunderrajan Narasimhan,

I've asked to see if someone else can assist with this. I'm definitely not familiar with RHV and can't find anything definitive to directly address your situation.

Kind Regards,
RJ

Hello Sunder,

If VMs are not showing in the cockpit web console or even the VMs tab itself if missing out there, then I feel that it is because of missing 'cockpit-machines' plugin which is not available (dnf install cockpit-machines). Check this. I'm not sure if you've already verified this over there. As RJ tried to help you with general steps even I sail in the same boat, not expertise on RHV platform.

May be some other community member would help you , otherwise, someone from Red Hat RHV group.

All the best!

Hi Sadashiva: Cockpit machines is already installed. VMs, storage pools, and network configuration are already there and can seen from the command line. Only problem here is that it is NOT visible on the web console.

└# dnf install cockpit-machines Updating Subscription Management repositories. Last metadata expiration check: 2:44:40 ago on Thu 29 Apr 2021 05:26:28 PM EDT. Package cockpit-machines-224.2-1.el8.noarch is already installed. Dependencies resolved. Nothing to do. Complete!

Hello,

From your previous comment,

Apologies, this is not a dire issue. I only testing this product on my home computer : Dell Precision 7750.

I suspect that you have create KVM virtual machines on a standard RHEL8 install, then accidentally installed RHV (vdsm) over the top of this. It would explain why you are using cockpit to manage your VM's (RHV uses a separate GUI), why virsh is now requiring authentication, (part of RHV), and why the VM's have disappeared from cockpit.

If that is true, then rolling it back would be difficult. Check your yum history and see if you can undo to before you installed the vdsm package:

grep vdsm /var/log/dnf.log*
yum history
yum history undo <id>

If that doesn't work (which is likely). Best would be to just reinstall the host. You can backup the VM disks if you have space somewhere, then copy them back after re-installing the hosts. Then you don't have to re-install the VM's also - just create the VM and point it at the existing disk.

/var/lib/libvirt/images - default location for disk images
/etc/libvirt/qemu  - default location for config (if needed for reference, it's better to re-create the VM manually)

Cheers,

Marcus

Hi, Marcus thanks for the response. I didn't know that RHV is different than KVM and initially I think I had KVM setup and cockpit had all the VMs perfectly in order. https://zyx:9090/. All was good. After yum update, it got all messed up. Probably, vdsm got installed and I didn't even know it nor did I intend to install it. Nonetheless, Rollbacking (undo) didn't work. Only best way to recover is to painfully recreate the VM, storage pool and network and then probably tweak the xml files to existing VMs. It may or may not work, let us see. There is no way to import the XML files from the /etc/libvirt/qemu and get all the settings back. If there is, I will be searching for it.

Appreciate and thanks for the response.

Ok, Just found out that RHV is not installed. RHV is a self-hosted engine that can be setup using the cockpit and that was never setup or installed. Not sure what happened, why it is asking for virsh authentication each time virsh command is run.

Did you find any evidence of vdsm being installed/removed? Check if you can list vm's with the authentication user: vdsm@ovirt, password: shibboleth . If that works, then it's probably vdsm that made that change. Installation makes a number of libvirt authentication changes, and they don't get rolled back - even if you remove vdsm (hence, OS re-install is recommended).

Regarding xml files, you can create the VM again using 'virsh define ' It will try to recreate all the same devices obviously, so might not work as well if you are moving from one host to another, should be ok after a host re-install though.

Generally, i prefer to just re-create the VM (memory, cpu, and mac address), but usually i'm working with fairly generic virtual machines.

Marcus,

I was able to recreate the problem. This time was I watching the cockpit behavior each time I did something. Guess what, as soon as I installed vdsm and vdsm-cli ( yum install vdsm vdsm-cli ), all my machines, networks, storage pools disappeared from the cockpit (web console) after a reboot. And then, I thought I have re-install RHEL and RHV all over again from scratch but as a last ditch effort I tried to remove vdsm and vdsm-cli package then reboot to see if the cockpit was able to restore my settings back. Exactly as I thought, I got all the machines, network, storage pools back where it was before. Even yum install ovirt-hosted-engine-setup installs vdsm which will get rid of all configurations from the cockpit as it also install vdsm packages. I need to read the documentation from page 1 to get a full idea of what is going on. Finally, no more authentication prompt on virsh commands, I don't know I got that before and it does not seem to do with vdsm. That annoying password prompt for each virsh command that was run could have been caused by something else entirely.

Thank you much Marcus!

Regards,
RJ