Unable to get valid subscription for VMs
We have two datacenters, each with their own VMware hypervisors that we need to subscribe RHEL 7 VM's on. For the one datacenter, everything is working fine. However, in the other one, things are a bit strange. I was able to register & subscribe the VM on which virt-who is running, but any other VM I spin-up and register/subscribe only gets a 24hr temporary subscription. I've tried:
subscription-manager remove --all
subscription-manager register
subscription-manager attach --auto
... but the VMs still come up with a temporary (24hr) subscription. If I run "subscription-manager list --consumed" I see this (other lines showing account details removed):
Service Level: Standard
Service Type: L1-L3
Status Details: Guest has not been reported on any host and is using a temporary unmapped guest subscription.
Subscription Type: Stackable (Temporary)
Starts: 08/27/2016
Ends: 12/20/2016
System Type: Virtual
The VMs are put on the same hypervisor as the one that is running virt-who and was successfully registeres/subscribed, so I'm not really sure why additional VMs won't subscribe. We have a total of 24 subscriptions and only 4-5 are currently being used, so not enough subscriptions isn't the issue. Any idea? I can post the /var/log/rhsm/rhsm.log contents if need be. Thanks.
Responses
I'm (still somewhat) new to Satellite and virt-who, but I had to learn about it to diagnose something similar (but probably not the same, see https://access.redhat.com/discussions/2808161 for that).
I might be able to give some guesses. It is useful to know what is logged also to see what you do not have to investigate further.
2016-12-19 09:49:22,324 [DEBUG] @esx.py:142 - Waiting for ESX changes
Virt-who connects to the vCenter/vSphere on https port and communicates with it using the vSphere api calls in order to find out what hypervisors each vm runs on. Not sure exactly how it works (if it polls at intervals, if it is triggered when there is a change and asks for a full list, or if it just gets it. The call is described at WaitForUpdatesEx I believe). The result is a hosts-to-guest mapping (in json format) that it sends to the subscription manager component in Satellite 6, which is called Candlepin.
You can easily check if the username/password is correct using a browser and the address https://vsphereserver.example.com/vsphere-client/
2016-12-19 09:49:22,332 [DEBUG] @subscriptionmanager.py:112 - Authenticating with certificate: /etc/pki/consumer/cert.pem
The certificate that identifies the consumer (subscriber), the machine virt-who is running on. If the machine is correctly registered to the satellite server it should work. In recent versions there was added config options rhsm_username, rhsm_password that can be used. (Tip: To see the info in the certificate, run "rct cat-cert /etc/pki/consumer/cert,pem")
2016-12-19 09:49:22,664 [DEBUG] @subscriptionmanager.py:146 - Checking if server has capability 'hypervisor_async'
2016-12-19 09:49:22,935 [DEBUG] @subscriptionmanager.py:158 - Server does not have 'hypervisors_async' capability
I think this is normal when connecting to VMware/vSphere hypervisors.
2016-12-19 09:49:22,936 [INFO] @subscriptionmanager.py:165 - Sending update in hosts-to-guests mapping: {
"31363636-3735-584d-5132-323430343639": [],
"31363636-3735-584d-5132-323430343634": [{
"guestId": "420b1fba-43f7-d054-c389-68ffd18c025e",
"state": 1,
"attributes": {
"active": 1,
"virtWhoType": "esx",
"hypervisorType": "vmware"
}
}
]
}
This is the info sent to Candlepin. It is an array of hypervisors (hosts) and the VMs (guests) running on each of them. The first hypervisor is empty, the second has one guest.
It is using and UUID and not hostnames, but there now is an option (hypervisor_id=hostname) to use proper hostnames for the hypervisors which makes things a bit easier for us humans. Not sure if using UUIDs for hypervisors is wrong now, but the docs shows it set (see https://access.redhat.com/documentation/en/red-hat-satellite/6.2/single/virtual-instances-guide/ and go to section 5.1). And I'm absolutely not sure what happens if you change it on an existing setup.
Anyway: What is happening next?
If there is an error it should be in the next (few) lines.
This is ok:
2016-11-30 11:32:15,976 [virtwho.main DEBUG] MainProcess(76159):MainThread @executor.py:send_report:101 - Report for config "vcenter1" sent
This is an example of when something failed.
2016-12-09 11:51:48,976 [virtwho.main ERROR] MainProcess(24310):MainThread @executor.py:send:156 - Error in communication with subscription manager:
You can also look in the /var/log/candlepin/candlepin.log on the satellite server for entries at the same timestamp and later. I see things like:
INFO org.candlepin.resource.HypervisorResource - Syncing virt host: esx113.example.com (42 guest IDs)
INFO org.candlepin.resource.ConsumerResource - Updating 42 guest IDs.
INFO org.candlepin.resource.ConsumerResource - removing IDs.
etc. etc.
Still curious.... what is the first line(s) after :
2016-12-19 09:49:22,936 [INFO] @subscriptionmanager.py:165 - Sending update in hosts-to-guests mapping: {
"31363636-3735-584d-5132-323430343639": [],
"31363636-3735-584d-5132-323430343634": [{
"guestId": "420b1fba-43f7-d054-c389-68ffd18c025e",
"state": 1,
"attributes": {
"active": 1,
"virtWhoType": "esx",
"hypervisorType": "vmware"
}
}
]
}
That should be either an error message one can hope leads one in the correct direction, or say something like "Report for config "myconfig" sent"
Stephen,
Can you provide some more details here, as I don't fully understand your configuration. From what you've stated so far, you have two data centres, each with a VMware hypervisor. In each data centre you have virt-who installed on one of the virtual machines. I'll use the naming DC1 for the first data centre and DC2 for the second data centre.
Everything's working as expected in DC1, with virt-who installed and virtual machines provisioned on the hypervisor are properly subscribed. In DC2 you also have virt-who running but only the virtual machine on which virt-who is installed is correctly subscribed. For all virtual machines provisioned in DC2, they are only granted temporary subscriptions.
Can you please provide the virt-who configuration files used in both data centres, with confidential details removed?
The fact that virtual machines are being granted temporary subscriptions is not alone an indication of a problem. Subscriptions for virtual machines are associated with the hypervisor. For Red Hat Satellite to grant a subscription, it must know on which hypervisor the virtual machine is hosted. When a virtual machine is first provisioned, it is granted a temporary subscription so that it can access content from the Satellite instance. Within the 24-hour period, the virt-who daemon generally obtains information about which hypervisors host which virtual machines and reports this to the Satellite server. Once this is known, the Satellite server can then grant a permanent subscription to the virtual machine.
The subscription process for virtual machines is described in more detail in the Virtual Instances Guide.
When you say "Here are the two virt-who.conf files", do you really mean /etc/sysconfig/virt-who (note: without an extension) or /etc/virt-who.conf (with extension)? Or is something mixed up here?
What you show here looks like the typical "environment variables to be read by the start up scripts in /etc/init.d" file that the Red Hat family of Linuxes use that are found in /etc/sysconfig/.
What the /etc/virt-who.conf and /etc/virt-who.d/* files should look like is described in man virt-who-config. Here is an example from that page:
[test-esx]
type=esx
server=1.2.3.4
username=admin
password=password
owner=test
env=staging
rhsm_username=admin
rhsm_password=password
Virt-who also reads in these environment variables (starting with VIRTWHO_) from its environment if they exist. So it can use the config file /etc/virt-who.conf and files in /etc/virt-who.d/*, command line parameters, and environment variables. "Choice is good", seems to be the theme...
To me is seem that the cleanest way is to use /etc/virt-who.conf for generic setup and connection to the Satellite, then put one separate file for each vCenter one wants to connect to in /etc/virt-who.d/. Then it is easy to add new, and to test them one by one by moving the others files out and restarting virt-who (or run a one-shot with debug messages to a file: virt-who -o -d > logfile.txt).
I don't like to use the environment variables, especially not put passwords in there.
By the way: For anyone using systemd (as RHEL7 does), and not sysvinit or upstart (as RHEL6 does) the variables in /etc/sysconfig/virt-who are available because in the /usr/lib/systemd/system/virt-who.service we have:
EnvironmentFile=-/etc/sysconfig/virt-who
Stephen,
Thanks for that extra information. The cause for failed subscriptions of hosts in DC2 is still a mystery. The virt-who configurations look to be the same. Examining the RHSM log file (/var/log/rhsm/rhsm.log) is the best source of diagnostic information for subscription management. However, that log may contain sensitive information, so I would not recommend posting its content here in the discussion.
I have some final suggestions, not in order of importance.
1. From the information you have included, the virt-who configuration files look to be the same where they should be the same. However, it would we worthwhile to do a "diff" of them in case there's something not obvious.
2. Check that the specified VMware user account and password are valid. You included an extract of the rhsm.log file, but I'm not sure from which DC's instance of virt-who that was obtained.
3. If there is a proxy between the host running virt-who and VMware ESX instance, the virt-who configuration must be amended to account for that. For details, see the following section of the Virtual Instances Guide.
Escalate it.
My experience is that for the first-line support-technicians virt-who is a black box they don't really understand, and the documentation doesn't really tell enough to troubleshoot effectively (and it is a moving target, for good and bad (good mostly)).
The /etc/virt-who.d/template.conf can of course be used, but that isn't the purpose of it. As installed it has all(?) the possible config statements with comments, but everything is commented out. So it is for copying and editing into a real config. At least that is my theory for calling it "template". YMMV.
Where did you get the login error? Against vmware or satellite?
Note that if you configuring by putting VIRTWHO_* environment variables in the /etc/sysconfig/virt-who source script and have login names in AD format (like "DOMAIN\username") you have to escape the backslash (so: "DOMAIN\username"), because that is the way scripts work, the backslash is an escape character.
Stephen,
It's great to know you got this sorted out. One benefit of having this discussion is that it may benefit other customers.
Regarding permissions, I should have asked you to check that. I made mention of testing credentials in the Virtual Instances Guide's Troubleshooting chapter, Credentials.
A handy troubleshooting parameter for virt-who is "--one-shot", as described in Configure and Start virt-who Service. This runs virt-who once, and outputs the virtual servers found, in JSON data format.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
