Virt-who takes a long time because it sends all hosts-to-guest mappings, and cannot do two things at the same time

Latest response

Using Satellite 6.2.2.

We have two datacenters that we typically provision to in pairs (for redundancy). We have problems with virt-who taking too long sending the hosts-to-guest mapping for vSphere in data center 1 to Satellite, so when the vm in datacenter 2 is built, registers (successfully) , and then tries to enable repos, it failes with "Status: Not Subscribed".

I see in the logs that as the vm in dc1 is started up we get the message

 [INFO] @subscriptionmanager.py:195 - Sending update in hosts-to-guests mapping for config "vsphere-dc1": 88 hypervisors and 1606 guests found

In Satellite I see the task Actions::Katello::Host::Hypervisors start, and it takes 4 minutes !
In the meantime the vm in dc2 is started up and tries to run the following:

  # add subscription manager
   yum -t -y -e 0 install subscription-manager
   rpm -ivh http://satellite.example.com/pub/katello-ca-consumer-latest.noarch.rpm

   echo "Registering the System"
   subscription-manager register --org="ACME" --name="deploytest-dc2" --activationkey="acme-rhel7"

    echo "Enabling Satellite Tools Repo"
    echo "DEPRECATED: This may be removed in a future version of Satellite, please add Satellite Tools to your activation key(s)."
    subscription-manager repos --enable=rhel-*-satellite-tools-*-rpms

     echo "Installing Katello Agent"
     yum -t -y -e 0 install katello-agent

... and fails to enable the repos and install since it doesn't have a valid subscription.

A minute after the Katello task was finished I see virt-who log:

 [INFO] @subscriptionmanager.py:195 - Sending update in hosts-to-guests mapping for config "vsphere-dc2": 34 hypervisors and 357 guests found

and a new Katello task is started, for the hosts-to-guests mapping from the vSphere in dc2.

Now it is possible to run the commands manually.

So, what believe I see is this (After I've tried to read the code, having tcpdump going, etc.):
- There are two constant TCP connections from virt-who, one to each vSphere,
- virt-who gets the mappings from the vSpheres when the VMs are provisioned (and queues them)
- There is only one connection set up from virt-who to the Satellite server, when there are updates.
- virt-who sends the mapping over (quickly), but then waits until Satellite has handled all the mappings, and is done, then disconnects (And that takes 4 minutes for 88 hypers and 1606 guests).
- Then it waits another minute (there is a 60 sec timeout specified in virtwho/init.py)
- First now it sends over the mappings for the next vSphere. To late....

Is there anything I can do to make this go quicker? Including getting a "Request for enhancement" made?

Obviously Satellite (Candlepin) has performance problems. (E.g. The 60 sec timeout between seems to have been added to prevent it being overloaded).

Is it really necessary to send the full hosts-to-guests mapping everytime one of these more than a thousand hosts are changed?

TT

Responses