Provisioning VMWare using userdata via Satellite 6.3-6.6
Will McDonald wmcdonal@redhat.com
Senior Consulting Architect
Red Hat UKI Services
Background
There are a number of different Red Hat technologies that can all be used to provision and configure VMware virtual machines.
- Cloudforms
- Ansible
- Satellite
Each has its own strengths, and scenarios where one may be favoured over another, framed by technical requirements, environmental constraints and local preferences.
During a recent engagement, we wanted to use Satellite to bulk-provision relatively large numbers of VMs to VMware Vcenter/ESX. Due to environmental constraints, we could not use Ansible, REX or SSH finish templates for post-provisioning configuration.
As a result we ended up with a solution using the upstream Foreman Userdata plug-in to further customise provisioned hosts via userdata injection for VMware customisation specification and cloud-init to ‘phone-home’. This plugin is currently being merged into Foreman core, it will likely appear in future Satellite 6 releases. It’s not currently a supported Red Hat component so be aware of this if you plan to use it.
Prerequisites
- Red Hat Satellite 6.3 (we used 6.3.3)
- VMware vCenter Server Appliance 6.7
- VMware ESXi 6.7
It is possible to reproduce this in a lab-like environment using nested virtualisation but this requires:
- 8 CPU cores
- 32GB
- 200GB of disk, thin provisioned (> 670 GB if thick provisioned)
- Vmware Workstation 14 or
- A nested ESX host with enough memory to run the VCSA in addition to VMs
Note: for those familiar with earlier releases of VCSA, the memory footprint for even a Tiny VCSA is substantial.
Setup Sequence of Events
In order to set up the initial environment, there are a few moving parts to set up. At a high-level we need to:
- Deploy a Satellite
- Deploy VCSA
- Deploy ESXi
- Satellite initial configuration
- Add manifest
- Configure and sync repositories
- Create a Lifecycle Environment
- Create a Content View and add a Repository to it
- Publish and promote the Content View
- Create Activation Keys and add Subscription
- Create a Subnet
- Create a Compute Resource
- Create a Compute Profile
- Create a Hostgroup
- Configure virt-who (optional)
- Install the Foreman Userdata Plug-in and restart Satellite
- Prepare provisioning templates (userdata and cloud-init)
- Configure the Operating System to use the userdata and cloud-init templates
- Prepare a VM template
- Create an empty VM
- Upload an ISO
- Install RHEL7
- Tweak networking (if required)
- Connect to Satellite temporarily
- Configure a public key (optional)
- Install open-vm-tools
- Install Perl
- Yum update
- Create copy (optional but recommended)
- Install cloud-init
- Configure cloud-init
- Create and run clean-up script
- Bake template
- Create Image in Satellite
- Test provisioning
Provisioning Sequence of Events
Once the environment is prepared, the provisioning workflow at a high-level is:
- User provisions VM(s) via the UI, API or hammer
- Satellite calls Vcenter to clone VM template
- Satellite userdata provisioning template injects customisation specification identity information
- Satellite cloud-init provisioning template instructs the VM to callback to Satellite when cloud-init runs post-provision
- Vcenter clones template to VM
- Vcenter applies customisation specification (VM identity including hostname, IP and DNS)
- VM builds, cloud-init is invoked and calls-back to Satellite on port 80 which then redirects to 443
Note: Even if registering the VM to a Capsule, the VM’s cloud-init phone-home always calls the Satellite, not its corresponding capsule. Make sure you take this into consideration in terms of firewall rules.
Network Configuration
The hosts were configured with the following FQDNs, IPs, operating systems and services. Forward and reverse DNS were correctly configured in DNS for all hosts.
Hostname | IP | OS | Notes |
---|---|---|---|
dns.example.com | 192.168.0.1 | RHEL7 | External DNS service |
satellite.example.com | 192.168.0.3 | RHEL7 | Yum Content, Provisioning, Subscription Management |
vcenter.example.com | 192.168.0.4 | Vmware Photon | vCenter |
esxi1.example.com | 192.168.0.5 | ESXi | ESX Hypervisor |
Management Endpoints
As this is just a demonstration lab, all SSL certificates are self-signed.
URL | Purpose |
---|---|
https://satellite.example.com | Satellite management UI |
https://vcenter.example.com:4580/ | VCSA management UI |
https://vcenter.example.com/ | Vcenter management UI |
https://esxi1.example.com/ | Note: the ESX host's Self-signed cert must be trusted in order to upload ISOs. |
VMware Installation
Installing vCenter and an ESX host or two is not covered in-depth but isn't terribly onerous.
Vmware Workstation 14 (or a suitably sized ESX host capable of running the VCSA) is recommended in order to correctly install the Vmware 6.7 VCSA. In theory it should be able to be run nested on other platforms but the installer and the OVA have extensive customisation which would need to be worked through.
Satellite Preparation
Satellite Installation
This walkthrough assumes you have a basic Satellite deployed with an Organization and Location configured, a manifest installed but no other objects created (no products or repositories, content views, activation keys etc.).
See the following documents for further information on planning for, and installing, Red Hat Satellite:
Red Hat professional services will obviously always be more than happy to help here, too. :D
This demonstration used Satellite 6.3.3 but this should work for earlier releases too.
Satellite Initial Preparation
Hammer Setup
Hammer has already been configured with sensible defaults for organization and location.
Repositores
Configure upstream Red Hat Repositories, in the Satellite UI under Content and Red Hat Repositories, enable the following
- RPMS / Red Hat Enterprise Linux Server / Red Hat Enterprise Linux 7 Server (RPMs) / Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server
Sync the upstream content in the UI, under Content and Sync Status, Expand All, Select All, and Synchronize Now.
Lifecycle Environments
Create a lifecycle environment (LE):
[root@satellite ~]# hammer lifecycle-environment create --description le-eng-rhel7-std-server --prior Library --name le-eng-rhel7-std-server
Content View
Create a content view (CV):
[root@satellite ~]# hammer content-view create --description cv-rhel7-std-server --name cv-rhel7-std-server
List Repository IDs
List the repository IDs of the synchronised content for the subsequent step:
[root@satellite ~]# hammer repository list --order id | awk -F '|' '{ print $1, $2 }'
Add Repositories to Content View
Add the required repositories from the previous step into the Content View:
[root@satellite ~]# hammer content-view update --repository-ids 1,2 --name "cv-rhel7-std-server"
Publish Content to Library
Publish a new version of the Content View into the initial Library:
[root@satellite ~]# hammer content-view publish --name "cv-rhel7-std-server" --async
Promote Content into Content View
Now promote the newly published Content View from the Library into the first Lifecycle Environment:
[root@satellite ~]# hammer content-view version promote --content-view "cv-rhel7-std-server" --to-lifecycle-environment "le-eng-rhel7-std-server" --async
Create Activation Key
Create an Activation Key (AK) to be used subsequently to register newly provisioned hosts:
[root@satellite ~]# hammer activation-key create --content-view "cv-rhel7-std-server" --lifecycle-environment "le-eng-rhel7-std-server" --name "ak-eng-rhel7-std-server"
List Available Subscriptions
List the available subscriptions (those loaded into the Satellite through its manifest):
[root@satellite ~]# hammer subscription list | awk -F'|' '{ print $1, $3, $9, $10 }'
--- ------------------------------------ ---------- ---------
ID NAME QUANTITY CONSUMED
--- ------------------------------------ ---------- ---------
1 Red Hat Satellite (Self-Supported) 1 0
2 Red Hat Enterprise Linux 1 0
--- ------------------------------------ ---------- ---------
Add Subscriptions to the Activation Key
Add the desired subscriptions to the Activation Key:
[root@satellite ~]# hammer activation-key add-subscription --name "ak-eng-rhel7-std-server" --subscription-id 2
Configure the Domain
A domain will already have been created during Satellite installation. Add this domain to the default location and organization through the UI.
Create a subnet
Create a new subnet for the network to be provisioned to:
[root@satellite ~]# hammer subnet create --name "sn-default-servers" --network "192.168.0.0" --mask "255.255.255.0" --gateway "192.168.0.1" --dns-primary "192.168.0.8" --domains example.com --locations loc-example --organizations org-example
Create a Compute Resource
Either username@domain and domain\username both work from a compute resource creation perspective.
[root@satellite ~]# hammer compute-resource create --caching-enabled 1 --datacenter "dc-example" --name "cr-vcenter" --password "password" --provider "Vmware" --server "vcenter.example.com" --user 'vsphere.local\administrator' --locations "loc-example" --organizations "org-example"
[root@satellite ~]# hammer compute-resource create --caching-enabled 1 --datacenter "dc-example" --name "cr-vcenter" --password "password" --provider "Vmware" --server "vcenter.example.com" --user 'administrator@vsphere.local' --locations "loc-example" --organizations "org-example"
Create a Compute profile
Note: This can’t currently be done via hammer, see the following articles for more information:
- Satellite 6: How to create compute-profile through hammer cli?
- Bug 1278917 - RFE Commands for read operations around compute profiles and attributes
- Commands for read operations around compute profiles and attributes
In the UI select Infrastructure > Compute profile > Create Compute Profile
Populate the indicated fields following values, select Compute Resource (cr-vcenter):
Field | Value |
---|---|
Name | vmware-small |
Cluster | esxi1.example.com |
Resource pool | auto populated ‘Resources’ |
Folder | vm |
Guest OS | Red Hat Enterprise Linux 7 (64-bit) |
Image | img-rhel7-std-server |
NIC type | VMXNET3 |
Network | VM Network |
Storage | See below... |
Data store | datastore1 |
Thin provision | uncheck |
Create a Hostgroup
Note: This can be accomplished in hammer but for a one-off it's quicker to create in the UI to save having to enumerate each value to illustrate the process.
A hostgroup was created in the UI with the following values:
[root@satellite ~]# hammer hostgroup info --id 1
Id: 1
Name: hg-rhel7-std-server
Title: hg-rhel7-std-server
Operating System: RedHat 7.5
Subnet: sn-default-servers
Domain: example.com
Architecture: x86_64
Puppet CA Proxy Id:
Puppet Master Proxy Id:
ComputeProfile: vmware-small
Puppetclasses:
Parameters:
kt_activation_keys => ak-eng-rhel7-std-server
Locations:
loc-example
Organisations:
org-example
Parent Id:
OpenSCAP Proxy:
Content View:
ID: 2
Name: cv-rhel7-std-server
Lifecycle Environment:
ID: 2
Name: le-eng-rhel7-std-server
Content Source:
ID: 1
Name: satellite.example.com
Kickstart Repository:
ID:
Name:
virt-who Configuration
Given the target compute in this walkthrough is Vmware, it’s likely you will need to configure virt-who for VDC subscription.
In this example, we are installing virt-who on the Satellite and this approach is generally fine for small to medium virtual estates. For very large estates, please refer to Red Hat documentation, support or consulting.
Install virt-who
Install the virt-who package and its dependencies
[root@satellite ~]# yum install -y virt-who
Create virt-who configuration
Create a virt-who configuration for each vCenter to be targeted
[root@satellite ~]# hammer virt-who-config create --name vcenter.example.com --organization org-example --interval 60 --filtering-mode none --hypervisor-id hostname --hypervisor-type esx --hypervisor-server vcenter.example.com --hypervisor-username 'administrator@vsphere.local' --hypervisor-password 'password' --satellite-url satellite.example.com
Deploy the virt-who configuration
Deploy the virt-who configuration
[root@satellite ~]# hammer virt-who-config deploy --id 1
Debugging virt-who
In the event of problems registering the vCenter, virt-who can be debugged by stopping the service and running in debug mode in the foreground.
[root@satellite ~]# systemctl stop virt-who.service
[root@satellite ~]# virt-who -d -o
Foreman Userdata Plugin Configuration
Next we install the upstream Foreman Userdata plugin appropriate for the target release.
Note: this plugin is not currently part of core Foreman, nor is it shipped by Red Hat. As a result this is NOT currently a supported solution. The plugin is on the roadmap for future inclusion.
Check Foreman Version
Check the version of Foreman on which the installed Satellite release is based:
[root@satellite ~]# rpm -q foreman
foreman-1.15.6.48-1.el7sat.noarch
Note: foreman-*1.15*.6.48-1
. The 1.15 value is important for the subsequent step.
Enabled Foreman Plugin Repository
Create a Yum repository configuration stub. For Satellite 6.3:
[root@satellite ~]# cat > /etc/yum.repos.d/foreman-plugins.repo <<EOF
[foreman-plugins]
name=Foreman plugins
baseurl=http://yum.theforeman.org/plugins/1.15/el7/x86_64/
enabled=1
gpgcheck=0
EOF
For Satellite 6.4:
[root@satellite ~]# cat > /etc/yum.repos.d/foreman-plugins.repo <<EOF
[foreman-plugins]
name=Foreman plugins
baseurl=http://yum.theforeman.org/plugins/1.18/el7/x86_64/
enabled=1
gpgcheck=0
EOF
For Satellite 6.5:
[root@satellite ~]# cat > /etc/yum.repos.d/foreman-plugins.repo <<EOF
[foreman-plugins]
name=Foreman plugins
baseurl=http://yum.theforeman.org/plugins/1.20/el7/x86_64/
enabled=1
gpgcheck=0
EOF
For Satellite 6.6:
[root@satellite ~]# cat > /etc/yum.repos.d/foreman-plugins.repo <<EOF
[foreman-plugins]
name=Foreman plugins
baseurl=http://yum.theforeman.org/plugins/1.22/el7/x86_64/
enabled=1
gpgcheck=0
EOF
The plugin code has been merged into core functionality in Foreman 1.23, therefore future Satellite updates will contain the feature out of box and this step can be skipped completely.
Install the Foreman Userdata plugin
Now we can 'yum install' the foreman_userdata plugin from the upstream repository:
[root@satellite ~]# yum -y install tfm-rubygem-foreman_userdata
Restart Satellite
Restart the Satellite services to pick up the newly installed plugin.
[root@satellite ~]# katello-service restart
Configure Provisioning Templates
Now we need to create and load some additional provisioning templates to be used by the userdata injection and cloud-init client.
Create provisioning template files
vmware-cloud-init-template
Create a cloud-init template file:
[root@satellite ~]# cat > ~/vmware-cloud-init-template.erb <<EOF
#cloud-config
hostname: <%= @host.name %>
fqdn: <%= @host %>
manage_etc_hosts: true
users: {}
runcmd:
- touch ~/cloud-init
phone_home:
url: <%= foreman_url('built') %>
post: []
tries: 10
EOF
Note: the ‘#cloud-config’ line is required in the cloud-init template, similar to a magic number of shebang.
Note: the runcmd illustrated here is merely a simple test to ensure cloud-init is functioning. More complex logic can be used in the template for registration, subscription management or other bespoke callback or finish tasks.
vmware-userdata-template
Create a userdata template file:
[root@satellite ~]# cat > ~/vmware-userdata-template.erb <<EOF
# Template for VMWare customization via open-vm-tools
identity:
LinuxPrep:
domain: <%= @host.domain %>
hostName: <%= @host.shortname %>
hwClockUTC: true
timeZone: <%= @host.params['time-zone'] || 'UTC' %>
globalIPSettings:
dnsSuffixList: [<%= @host.domain %>]
<%- @host.interfaces.each do |interface| -%>
<%- next unless interface.subnet -%>
dnsServerList: [<%= interface.subnet.dns_primary %>, <%= interface.subnet.dns_secondary %>]
<%- end -%>
nicSettingMap:
<%- @host.interfaces.each do |interface| -%>
<%- next unless interface.subnet -%>
- adapter:
dnsDomain: <%= interface.domain %>
dnsServerList: [<%= interface.subnet.dns_primary %>, <%= interface.subnet.dns_secondary %>]
gateway: [<%= interface.subnet.gateway %>]
ip: <%= interface.ip %>
subnetMask: <%= interface.subnet.mask %>
<%- end -%>
EOF
Install provisioning templates
Now install the newly created templates:
[root@satellite ~]# hammer template create --name vmware-cloud-init --file ~/vmware-cloud-init-template.erb --locations loc-example --organizations org-example --operatingsystem-ids 1 --type cloud-init
[root@satellite ~]# hammer template create --name vmware-userdata --file ~/vmware-userdata-template.erb --locations loc-example --organizations org-example --operatingsystem-ids 1 --type user_data
Configure Templates in the Operating System
Now the templates are created and loaded into the Satellite, we can update the Operating System definition to use the desired templates during provisioning requests.
In the UI, switch the Red Hat 7.5 Operating System to use these templates:
Go to Hosts > Operating systems > Red Hat 7.5 > Templates
- User data template: vmware-userdata
- Cloud-init: vmware-cloud-init
Template Preparation
Create Empty Virtual Machine
In vCenter, create an empty RHEL7 VM container with the following properties:
- 1 vCPU
- 2GB vMem
- 16GB VMDK
Upload a Red Hat Enterprise Linux ISO
Download rhel-server-7.5-x86_64-dvd.iso from the [Red Hat Customer Portal | https://access.redhat.com/downloads ], then upload into vCenter to a datastore ‘isos’ folder
Note: log in to the ESXi host directly in a browser to accept its self-signed certificate or ISO upload won’t work.
Install Red Hat Enterprise Linux
Install RHEL7 minimal from the ISO, consult Red Hat Enterprise Linux 7 documentation for more details.
Virtual Machine Network Configuration
The virtual machine will require some temporary network identity settings in order to prepare the template. Tweak interface connection name and DNS as appropriate for the environment:
[root@localhost ~]# nmcli con modify 'System ens192' connection.id ens192
[root@localhost ~]# nmcli con mod ens192 ipv4.method auto
[root@localhost ~]# nmcli con edit ens192
nmcli connection edit ens192
nmcli> remove ipv4.dns
nmcli> set ipv4.ignore-auto-dns yes
nmcli> set ipv4.dns 192.168.0.2
nmcli> save
nmcli> quit
nmcli device reapply ens192
[root@localhost ~]# hostnamectl set-hostname "localhost.localdomain"
Subscribe to Satellite
Now we need to temporarily subscribe to the Satellite for some template prerequisite packages and a few other bits and pieces:
[root@localhost ~]# rpm -ivh http://satellite.example.com/pub/katello-ca-consumer-latest.noarch.rpm
[root@localhost ~]# subscription-manager register --org=org-example --activationkey=ak-eng-rhel7-std-server
Drop in SSH Keys
This is an opportune moment to drop in any SSH public keys if desired:
[root@localhost ~]# mkdir ~/.ssh
[root@localhost ~]# chmod 700 ~/.ssh
[root@localhost ~]# cat > ~/.ssh/authorized_keys <<EOF
ssh-rsa AAAABlongkeyfingerprint will@example.com
EOF
[root@localhost ~]# chmod 600 ~/.ssh/authorized_keys
Note: keys can also be injected into the template during provisioning time via cloud-init.
Install open-vm-tools
Install open-vm-tools to ensure the userdata customisation specification is properly handled:
[root@localhost ~]# yum -y install open-vm-tools.x86_64
Install Perl
Perl is also required for open-vm-tools to correctly parse and handle the customisation specification injection at provisioning time:
[root@localhost ~]# yum -y install perl
Yum Update Virtual Machine
While the VM template is temporarily registered to the Satellite, it will reduce future updates if we bring it up to the latest patch level:
[root@localhost ~]# yum -y update
Note: At this stage, it can be wise to power off and create a copy of the template without cloud-init. cloud-init can be quite invasive, if the template preparation requires multiple iterations this is a good place to restart from.
Install cloud-init
Power the virtual machine back on and install cloud-init:
[root@localhost ~]# yum -y install cloud-init
Configure cloud-init to Skip Networking
Drop in a configuration stub to instruct cloud-init to skip network configuration:
[root@localhost ~]# cat << EOF > /etc/cloud/cloud.cfg.d/01_network.cfg
network:
config: disabled
EOF
Note: Satellite will do this via the Vmware customisation specification).
Configure cloud-init to Callback to Satellite Post-build
Drop in a configuration stub to instruct cloud-init to call back to Satellite post-provision. This informs Satellite of the build's completion, and allows additional post-provisioning behaviour to be injected via cloud-init configuration (in the form of the cloud-init provisioning template).
[root@localhost ~]# cat << EOF > /etc/cloud/cloud.cfg.d/10_foreman.cfg
datasource_list: [NoCloud]
datasource:
NoCloud:
seedfrom: http://satellite.example.com/userdata/
EOF
Replace Default cloud-init Configuration
The default cloud-init configuration may be considered quite invasive by the standards of some environments (YMMV). Back-up the default cloud-init configuration and replace with a more minimal set of configuration that does just what we require:
[root@localhost ~]# cp /etc/cloud/cloud.cfg ~/cloud.cfg.`date -I`
[root@localhost ~]# cat << EOF > /etc/cloud/cloud.cfg
cloud_init_modules:
- bootcmd
cloud_config_modules:
- runcmd
cloud_final_modules:
- scripts-per-once
- scripts-per-boot
- scripts-per-instance
- scripts-user
- phone-home
system_info:
distro: rhel
paths:
cloud_dir: /var/lib/cloud
templates_dir: /etc/cloud/templates
ssh_svcname: sshd
# vim:syntax=yaml
EOF
Unregister Virtual Machine from Satellite
Now the template is fully prepared bar the clean/sysprep phase, we can unregister it from the Satellite:
[root@localhost ~]# subscription-manager unregister
Unregistering from: satellite.example.com:443/rhsm
System has been unregistered.
[root@localhost ~]# subscription-manager clean
All local data removed
Create Clean Script
Now we drop in a simple clean/sysprep script to remove the virtual machine identity prior to converting back to a template:
[root@localhost ~]# cat > ~/clean.sh <<EOF
#!/bin/bash
# stop logging services
/usr/bin/systemctl stop rsyslog
/usr/bin/systemctl stop auditd
# remove old kernels
# /bin/package-cleanup -oldkernels -count=1
#clean yum cache
/usr/bin/yum clean all
#force logrotate to shrink logspace and remove old logs as well as truncate logs
/usr/sbin/logrotate -f /etc/logrotate.conf
/bin/rm -f /var/log/*-???????? /var/log/*.gz
/bin/rm -f /var/log/dmesg.old
/bin/rm -rf /var/log/anaconda
/bin/cat /dev/null > /var/log/audit/audit.log
/bin/cat /dev/null > /var/log/wtmp
/bin/cat /dev/null > /var/log/lastlog
/bin/cat /dev/null > /var/log/grubby
#remove udev hardware rules
/bin/rm -f /etc/udev/rules.d/70*
#remove uuid from ifcfg scripts
/bin/cat > /etc/sysconfig/network-scripts/ifcfg-ens192 <<EOM
DEVICE=ens192
ONBOOT=yes
EOM
#remove SSH host keys
/bin/rm -f /etc/ssh/*key*
#remove root users shell history
/bin/rm -f ~root/.bash_history
unset HISTFILE
#remove root users SSH history
/bin/rm -rf ~root/.ssh/known_hosts
EOF
Run Clean Script
Clean the system:
[root@localhost ~]# sh ~/clean.sh
Power Off Virtual Machine
Power off the VM:
[root@localhost ~]# systemctl poweroff
Create Virtual Machine Template
Now convert the VM to a template in vCenter ‘template-rhel7-cloudinit’
Finalising Satellite Preparation
Now we have our Satellite and virtual machine template prepared, there's just one final step before moving to testing the provisioning flow.
List Operating System, Architecture and Compute Resource
List the Operating System, Architecture and Compute Resource configuration in Satellite to identify the corresponding object IDs:
[root@satellite ~]# hammer os list; hammer architecture list; hammer compute-resource list
---|------------|--------------|-------
ID | TITLE | RELEASE NAME | FAMILY
---|------------|--------------|-------
1 | RedHat 7.5 | | Redhat
---|------------|--------------|-------
---|-------
ID | NAME
---|-------
1 | x86_64
2 | i386
---|-------
---|------------|---------
ID | NAME | PROVIDER
---|------------|---------
1 | cr-vcenter | VMware
---|------------|---------
Create Image in Satellite
Now we create an Image in Satellite, linking the template in vCenter with the required resources in Satellite:
[root@satellite ~]# hammer compute-resource image create --operatingsystem-id 1 --architecture-id 1 --compute-resource-id 1 --user-data true --uuid template-rhel7-cloudinit --username root --name img-rhel7-std-server
Test Guest Provisioning
We should now be in a position to test provisioning of guest VMs via the Satellite UI or hammer.
For example:
[root@satellite ~]# hammer host create --name test-host --organization org-example --location loc-example --hostgroup hg-rhel7-std-server --compute-resource cr-vcenter --provision-method image --image img-rhel7-std-server --enabled true --managed true --compute-profile vmware-small --operatingsystem-id 1 --compute-attributes="start=1" --interface "managed=true,compute_type=VirtualVmxnet3,type=interface,domain_id=1,identifier=ens192,ip=192.168.0.50,subnet_id=1,primary=true,compute_network='VM Network'"
Next Steps
Once provisioning works, guests are being cloned, allotted the correct hostname, domain and IP, and the ~/cloud-init file is being created, the cloud-init provisioning template can be extended to perform any other post/finish tasks as required. For example:
- Installing the katello-consumer-ca from the appropriate content host
- Registering with a specific activation key
- Installing the katello-agent
Once host provisioning works for a single host, it’s relatively trivial to scale this up to 10s or 100s of hosts.
Even in a small, nested environment, the end-to-end process takes just a couple of minutes per-VM at most. However, care should be taken around concurrent host registration so it may be wise to batch bulk provisioning into groups of 10s or 20s of VMs and wait for completion of each batch.
References
Red Hat Documentation
- https://access.redhat.com/documentation/en-us/red_hat_satellite/6.2/html-single/hammer_cli_guide/index#sect-CLI_Guide-Authentication
- https://access.redhat.com/documentation/en-us/red_hat_satellite/6.3/html/installation_guide/installing_satellite_server#performing_initial_configuration_sat_server_parent
- https://access.redhat.com/documentation/en-us/red_hat_satellite/6.3/
- https://access.redhat.com/documentation/en-us/red_hat_satellite/6.3/html/virtual_instances_guide/
Useful Links
- https://translate.google.co.uk/translate?hl=en&sl=fr&u=https://www.mchelgham.com/installer-vcsa-6-7-dans-vmware-workstation-fusion/&prev=search
- https://www.virten.net/2017/02/homelab-downsizing-vcenter-server-appliance-6-5/
- https://www.starwindsoftware.com/blog/vmware-vcenter-server-appliance-vcsa-and-after-install-tricks
- http://everything-virtual.com/2016/05/06/creating-a-centos-7-2-vmware-gold-template/
Comments
Awesome! Will this be in the Satellite 6.4 release?
Hello, this is still work in progress in upstream Foreman project. Eventually I expect it to be part of Satellite.
Is there anything simillar for 6.2.x? What are the steps for the cases where we can use ssh/ansible?
Hi, Thanks for publishing this, really helpful piece!
I've just run this with Satellite 6.4 against vSphere 6.7, can confirm that it worked for me providing that you disable the 'safe mode rendering' mode setting in Satellite, otherwise you will get an error when rendering the templates.
The foreman-proxy repo should be updated too, to 1.18:
[root@satellite ~]# cat > /etc/yum.repos.d/foreman-plugins.repo <<EOF [foreman-plugins] name=Foreman plugins baseurl=http://yum.theforeman.org/plugins/1.18/el7/x86_64/ enabled=1 gpgcheck=0 EOF
I was getting error: undefined method '#params' for Host::Managed::Jail (Host::Managed) when trying to provision from VMware template. Disabling safe mode rendering fixed it for me. Thanks!
Hi Lukas,
Great information! Do these steps only apply to servers using DHCP? I've been struggling to get this working correctly with static addresses. Any advice on how to implement this would be greatly appreciated!
Thank you very much in advance for your assistance! Greg
I use static IP addresses, but my vmware-userdata-template looks a bit different. I was not able to get Lukas' to work for me. The LinuxPrep for VMware is very picky and very limited in its capabilities. This cloud-init approach was a significant improvement for me over my prior custom first boot scripted process.
You will either need to enter an IP address in the network interface manually when provisioning or configure a pool of addresses to pull from for the selected network.
Is there still the case for you that cloud-init doesn't work with static IP addresses? There used to be a known issue on reboot that would nullify any static IP address settings.
has this been integrated in 6.4 ? do I still need to install the userdata rpm also do I need to setup cloud-init and install cloud init in the vm ?
Yes, you need the tfm-rubygem-foreman_userdata RPM installed on Satellite. This is not an "official" solution from Red Hat (at least not yet, probably not ever). It should also be be noted that you need both a "User data template" for the VMware LinuxSysprep AND a cloud-init template made possible by the userdata RPM. This is a little confusing because the "User data" templates are typically used for "real" cloud-init based deployments, i.e. OpenStack. When it is a vmware template deployment it uses the VMware API and LinuxSysprep which is not at all cloud-init like.
Yes, you need to have cloud-init installed and configured in the VMware template in order for this to work correctly. Note, it must use the NoCloud datasource type which basically just makes a web service call back to Satellite to read from the "cloud-init" template (not to be confused with the user data template). There are some frustrating limitations to the NoCloud type. It's basically just a single URL datasource. You only get one go at it. You can't provide an array of cloud-init meta data endpoints for it to try.
I recommend something like Packer for creating and maintaining your vmware server templates.
Hi,
I would like to point that the VMware template should be made on the same cluster where the new machines will be created (some kind of staging cluster). If the machine needs to run on another cluster, a manual VMotion should occur to place the vmguest on the right cluster.
Otherwise you could get an error as next: Unable to save Failed to create a compute DC1-Prod (VMware) instance xxxxx.mydomian.com: CannotAccessVmConfig: Unable to access the virtual machine configuration: Unable to access file [DS1111-F1P-Prod-050DF]
It comes with the fact that a group of datastores are available only on one specific VMware cluster. But, I think that on VMware is also allowed to publish all datastores on all clusters. Probably it is not a good practice.
Hi there, Satellite 6.4 against vSphere 6.5, I am getting from UI: "ERF42-5701 [Foreman::Exception]: The user-data template must be a hash in YAML format for VM customization to work." Any ideas?
From hammer host create, getting a the same. Thoughts?
Thanks!
I very quick look through our Knowledge Centered Support documents didn't turn anything up. I would encourage you to open a support case so we can investigate it.
Dean,
This is an issue that I ran into myself as well. What I had to do is run a api call to satellite and list the computer profile ID. It does not parse out the profile by name but currently only by ID. Here is what I did:
curl -X GET -s -k -u admin https://satellite.example.com/api/v2/compute_profiles | python -m json.tool Enter host password for user 'admin':{ "page": 1, "per_page": 100, "results": [ { "created_at": "2019-02-22 20:21:12 UTC", "id": 8, "name": "vmware-os-apps", "updated_at": "2019-02-22 20:21:12 UTC" }, { "created_at": "2019-02-22 19:38:55 UTC", "id": 4, "name": "vmware-os-master-infra", "updated_at": "2019-02-22 20:10:22 UTC" } ], "search": null, "sort": { "by": null, "order": null }, "subtotal": 2, "total": 2 }
As you can see, the ID's are 8 and 4, so I had to use --compute-profile-id 4
On the side note, I am still getting the Nil error for myself:[DEBUG 2019-03-05T13:37:31 API] Headers: {} [DEBUG 2019-03-05T13:37:31 API] Using authenticator: HammerCLIForeman::Api::InteractiveBasicAuth [ERROR 2019-03-05T13:37:36 API] 422 Unprocessable Entity [DEBUG 2019-03-05T13:37:36 API] { "error" => { "id" => nil, "errors" => { "conflict" => [], "interfaces.conflict" => [], "base" => [ [0] "Failed to create a compute CS_vSphere_Non (VMware) instance satellite.example.com: undefined method []' for nil:NilClass\n " ] }, "full_messages" => [ [0] "Failed to create a compute vSphere_Non (VMware) instance satellite.example.com: undefined method[]' for nil:NilClass\n " ] } } [DEBUG 2019-03-05T13:37:36 Exception] Using exception handler HammerCLIForeman::ExceptionHandler#handle_unprocessable_entity [ERROR 2019-03-05T13:37:36 Exception] Failed to create a compute vSphere_Non (VMware) instance satellite.example.com: undefined method `[]' for nil:NilClass ~~~
HI Michael, thanks for that. I managed to fix the issue and can create the VM now. Will report back and other issues. BR
See Coment
Hi guys, Satellite 6.4 against vSphere 6.5 went well. We can boot the VM's now and Network is working as expected. One issue tho, /var/log/messages:
The Satellite is still showing: Build Pending installation
Any thoughts? Thanks
Hello, Me too, i'm trying to make "deploying VM" using RH Satellite 6.4 and using Vsphere 6.0. VM's deploys correctly + registers the VM into RHS and runs yum update. However, here too, I keep getting "Build Pending installation". Post Phone Home doesn't seem to work. Extract of provisioning template (vmware-cloud-init) in Red HAt Satellite: ... phone_home: url: <%= foreman_url('built') %> post: [] tries: 3 ...
Any idea's
Yes, it does look like "Build Pending installation" can be canceled.
I have no idea where to put this /etc/ssh/ssh_host_dsa_key.pub on the Satellite? Or how to create one...
SUCCESS. I just created a dsa key in template with above name. All logs are clear of errors.
Thanks for the effort in writing this doco!
Thanks for getting back to us. We are looking into including this into Satellite next-next.
at the risk of sounding pessimistic:
1) Why is cloning a VM from within Satellite only being introduced in Satellite 6.4, and 2) Only supports SDRS now (for goodness sake, it's been out EIGHT years), and 3) This above procedure is 50 steps long. Cloning a VM in vCenter is a couple of clicks. Really.
Actually, image based provisioning for vmware has been available in Satellite 6.x for quite some time if not the very beginning. The fundamental difference between simple image cloning and the foreman user data plugin is that it brings cloud-init type functionality (OpenStack like) to the vmware platform via Satellite. The user data template for deploying vmware images via Satellite is limited to the functionality of the vmware LinuxPrep Customization API which is very, very limited. It's pretty much only good for pushing down a static IP address when not using DHCP for server provisioning. Vmware does not have a cloud-init meta data service like OpenStack. The foreman user data plugin adds a new template type (cloud-init) and web service to render the resulting server specific cloud-init content to the client.
I found this addition very useful. I was initially disappointed with the limited capability of image based provisioning for vmware from Satellite. I was expecting the user data templates to be more OpenStack like, but this disappointment was fundamentally due to the limited nature of the vmware LinuxPrep Customization API than anything specific in Satellite. I was able to work around the limitation via custom first-boot init scripts baked into the image templates, but that always felt like a hack and required cutting a new image anytime I wanted to modify it. I have since deprecated my custom init scripts in favor of the method outlined in this blog post and I am much happier with the result. Cloud-init has become a fairly common and well understood methodology in cloud based server provisioning. I prefer to use common standards over a custom scripting whenever possible.
Cloning a VM should be simple. this method certainly isn't that. And for that matter, the step entitled "Virtual Machine Network Configuration" is just not going to be scalable. (ie "The virtual machine will require some temporary network identity settings in order to prepare the template").
We have multiple subnets (20?) on multiple VXLANS where Linux hosts can exist. We would then have to have MULTIPLE templates each with a different temporary IP address that Satellite would use to SSH into the system to change the IP to the final IP address.
Moreover, there are further issues:
1) The host key would get cached in ~foreman-proxy/.ssh/known_hosts, and foreman proxy would fail 2) We would be unable to build more than one host at a time (due to IP address conflicts). 3) Managing 20 provisioning templates PER OS would be a nightmare.
It's probably worth pointing out that this temporary network config really is only used to make one-off template preparation easier. You only do this once (typically), then you can ship your template around anywhere you need it.
(Ongoing template maintenance is another topic entirely and that's why tools like Packer exist. This is just a how-to, explaining the basic steps so they can be improved.)
Trying to configure the template into an appropriate state using just the console is painful and error-prone (I'm speaking from experience), especially if you take a few iterations to get your template in the state you need it.
It's also worth pointing out that the identity config (IP and hostname) is removed again before the template is baked. At this point, the template could be considered portable across clusters/environments.
We don't SSH into the cloned VM during provisioning to configure its new identity. The new identity is injected into the cloned VM via userdata.
And finally, this is just one possible provisioning mechanism/flow. There's always more than one way to do something and this was useful in a specific scenario, I wouldn't suggest it as a catch-all approach. It won't work for every environment, or every scenario.
Hope this helps clarify a few things.
actually it's even worse than that. Because we have two sites and two domains, would have to have 4 VM templates (to accommodate DNS differences) for every single OS release PER VXLAN
So : 80 VM tempates of RHEL 7.6, 80 templates of RHEL 7.5, 80 templates of RHEL 7.4.....
and then there's RHEL 6.*
This is true. We have 40 subnets on vmware. Plus Azure. No dhcp for now. As I see it, Sat6 is just a licensing and patching tool for us. Not to mention cmdb and Infoblox integration when we will start using dhcp. Which integration? ;) (CASE 02335634)
So RH (IBM) need to wake up, and give us complete Azure solution as well, and drop the atrocities with MS and RH, or we will drop the Satelite and RH all together, I am afraid. Since we are forced more and more to use Azure.
Btw CASE 02337878 is interesting, the deployed VM's are not showing Type (vmware) and Virtual Host (vmware host name). Thoughts? None from RH...
and I just tried this, and it fails at the provisioning step with:
[root@lprsat102 ~]# hammer host create --name test-host --organization org_SOMETHING --location loc_SOMETHING --hostgroup hg_RHEL7_BASE_DEV --compute-resource VCENTER --provision-method image --image rhel74template --enabled true --managed true --compute-profile "Small RHEL" --operatingsystem-id 6 --compute-attributes="start=1" --interface "managed=true,compute_type=VirtualVmxnet3,type=interface,domain_id=2,identifier=ens192,ip=10.156.154.7,subnet_id=2,primary=true,compute_network='vxw-dvs-360-virtualwire-29-sid-7020-DC2_Dev-Test2_BR3154'" Could not create the host: Error: Found more than one compute_profile.
and this bug too:
https://bugzilla.redhat.com/show_bug.cgi?id=1584785
Even If I substitute the "--compute-profile-id" number instead of "--compute-profile", I get:
Could not create the host: Failed to create a compute WPRVCS102 (VMware) instance test-host.powerdev.dev.int: undefined method `[]' for nil:NilClass
It's a bit of a crap error message, don't you think?
Please create case or/and bugzilla for this, a VMWare plugin engineer must take a look into this.
Indeed this is an excellent guide and very informative . I upgraded satellite from 6.4 to 6.6.2, I thought it would be a breeze to use cloudinit and userdate to create a VM but it still failed to create.
I am suing compute-profile-id as someone suggested above.
Hi guys,
unfortunately, the tfm-rubygem-foreman_userdata broke the Satellite upgrade to 6.5 and needs to be removed.
The repo for 6.5 is http://yum.theforeman.org/plugins/1.20/el7/x86_64/
Enjoj
Thanks, this will be incorporated a future Satellite version, it's been merged upstream already.
With Satellite 6.5 and Infoblox DNS, the above does not work any more if creation of the DNS records on Infoblox is expected. Case 02391995 opened.
Hello, I don't believe that Infoblox integration has anything to do with this workflow. Let's see in the case.
We have a report that Satellite redirects the nocloud request to HTTPS, it looks like HTTPS must be used:
seedfrom: https://satellite.example.com/userdata/
Server (Satellite) CA certificate must be added to system list of trusted certificates for the image.
https://github.com/theforeman/foreman/pull/6845
Patch to support HTTP as well, will be part of Satellite (later version)
Try to install tfm-rubygem-foreman_userdata on Satellite 6.4 , getting errors
--> Finished Dependency Resolution Error: Package: tfm-rubygem-foreman_userdata-0.0.1-1.fm1_15.el7.noarch (foreman-plugins) Requires: rh-ruby22-ruby Available: rh-ruby22-ruby-2.2.2-11.el7.x86_64 (rhel-server-rhscl-7-rpms) rh-ruby22-ruby = 2.2.2-11.el7 Available: rh-ruby22-ruby-2.2.2-12.el7.x86_64 (rhel-server-rhscl-7-rpms) rh-ruby22-ruby = 2.2.2-12.el7 Available: rh-ruby22-ruby-2.2.2-15.el7.x86_64 (rhel-server-rhscl-7-rpms) rh-ruby22-ruby = 2.2.2-15.el7 Available: rh-ruby22-ruby-2.2.2-16.el7.x86_64 (rhel-server-rhscl-7-rpms) rh-ruby22-ruby = 2.2.2-16.el7 Available: rh-ruby22-ruby-2.2.9-19.el7.x86_64 (rhel-server-rhscl-7-rpms) rh-ruby22-ruby = 2.2.9-19.el7
Than try to change
baseurl=http://yum.theforeman.org/plugins/1.15/el7/x86_64/ to 1.16-1.20 its complaining rub25
Error: Package: tfm-rubygem-foreman_userdata-0.1.0-1.fm1_20.el7.noarch (foreman-plugins) Requires: rh-ruby25-ruby(release) Error: Package: tfm-rubygem-foreman_userdata-0.1.0-1.fm1_20.el7.noarch (foreman-plugins) Requires: rh-ruby25-ruby Error: Package: tfm-rubygem-foreman_userdata-0.1.0-1.fm1_20.el7.noarch (foreman-plugins)
Any help
Update to 6.5.3? All updates went well from 6.4 to 6.5.3 for us.
I am trying to enable redhat subscription using cloud init template so that VM provisioned on vCenter can fetch the repos from the satellite server. Can anyone help me in creating the cloud init template?
Thanks in advance!
You can run arbitrary commands on first (or every) boot. Some handy examples are at https://cloudinit.readthedocs.io/en/latest/topics/examples.html#run-commands-on-first-boot.
Hi, success with RHEL8. To confirm, this is working well: - Installing the katello-consumer-ca from the appropriate content host - Registering with a specific activation key - Installing the katello-agent
The question is how can we specify a specific activation key in the cloud-init provisioning template?
About cloud init, if we look at my comment from March 8 2019 at 12:28 PM (I removed the server name), the Sat Cloud init is using some strange host name http://xxxxx... Let me explain, this is on satellite: [root@rhnsat6dev ~]# hostname -f rhnsat6dev.is.uwa.edu.au [root@rhnsat6dev ~]# hostname rhnsat6dev.is.uwa.edu.au
and the client /var/log/cloud-init.log is showing: . url_helper.py[DEBUG]: [0/11] open 'http://rhnsat6dev.uni..... . 2019-08-13 00:43:30,853 - util.py[DEBUG]: Failed to post phone home data to http://rhnsat6dev.uni.../unattended/built?token=386cd1e7-c24d-4a1e-b09d-cf5b3dd9499e in 10 tries Traceback (most recent call last): . .
No idea why NOT using the DNS name which is rhnsat6dev.is... but rhnsat6dev.uni...? I'll raise an CASE. Cheers
What do you have defined in /etc/cloud/cloud.cfg.d/10_foreman.cfg?
Check the rest of your cloud-init configuration for errant seedfrom entries.
It might also be worth double-checking the Apache config on whichever host you're phoning home to in case there are redirects or a misconfigured vhost, ServerName or ServerAlias?
Thx. foreman.cfg is set correctly. We did not set Apache, RedHat Consultant did and is the same for everything as per hostname command on Satellite;) Cheers
Hello Community
What is the meaning of statement "Likely bad things to come!". Is it worry some ?
[ 150.848951] cloud-init[8731]: Can not apply stage final, no datasource found! Likely bad things to come!
Yeah, I see that everywhere, not a concern atm.
Unable to save Failed to create a compute host-esx (VMware) instance host.ca.example.com: unexpected path class NilClass
I am trying to delpoy VM with fix IP using Robert's LinuxPrep template
globalIPSettings: dnsServerList: [<%= @host.primary_interface.subnet.dns_primary %>, <%= @host.primary_interface.subnet.dns_secondary %>] dnsSuffixList: [<%= @host.domain %>]
nicSettingMap: <% @host.interfaces.each do |interface| %> - adapter: ip: <%= interface.ip %> subnetMask: <%= interface.subnet.mask %> <% if interface.primary -%>gateway: [<%= interface.subnet.gateway %>]<% end %> <% end %>
using hammer create host command with interface --interface "managed=true,compute_type=VirtualVmxnet3,type=interface,domain_id=2,identifier=ens192,ip=10.50.61.99,subnet_id=1,primary=true,compute_network='MY-TEST-561'" but no luck, still getting IP from dhcp. Any help will be appreciated
You could used Formatting help as the start :) Other than that, where is the LinuxPrep Templete on this Doco? So no, not supported. Start with basics plain, simple template.
On RH Satellite 6.6.1 attempting to install the plugin fails:
Pages