Discovering bare metal with Satellite 6.1

Satellite 6.1 will ship with the discovery plug-in pre-installed. The discovery plug-in enables automatic bare-metal discovery of unknown nodes on the provisioning network. These new nodes register to the Satellite Server and upload system facts such as serial ID, network interfaces, memory, and disks, as collected by Facter. After registration, the nodes show up on the Discovered Hosts page and you can initiate provisioning either manually (using the web UI or CLI, or the API) or automatically, using predefined discovery rules.

All communication occurs through the Satellite Capsule, which has direct access both to the provisioning network and Satellite Server instance:

Discovered Host (Satellite Discovery Image) <--> Satellite Capsule <--> Satellite Server

There are three different components:

1. Satellite Server Discovery plug-in. The plug-in runs on the Satellite Server and provides API and UI functionality for working with discovered nodes. The ruby193-rubygem-foreman_discovery package contains this component.

2. Satellite Capsule Discovery plug-in. This plug-in represents a communication proxy between discovered hosts on a provisioning network and Satellite Server. The rubygem-smart_proxy_discovery package contains this component.

3. Satellite Discovery Image. This is the minimal operating system based on Red Hat Enterprise Linux that is PXE booted on hosts to acquire initial hardware information and to check-in to the Satellite Server. Discovered hosts keep running Satellite Discovery image until they are rebooted into the anaconda installation program, which then initiates the provisioning process. The foreman-discovery-image package contains this image. It must be installed on the Satellite Capsule that provides TFTP services.

Configuration

Deploying the Satellite Discovery Image

Install the package containing the Satellite discovery image on the Satellite Capsule node that provides TFTP services (not on the Satellite Server itself):

# yum install foreman-discovery-image

The package contains the Linux kernel and initial RAM disk image wrapped as a bootable ISO file which is used for PXE-booting discovered nodes. To investigate the contents of the package, use the following command (the package version, time stamp, and release number can be different):

$ rpm -ql foreman-discovery-image
/usr/share/foreman-discovery-image
/usr/share/foreman-discovery-image/fdi-image-rhel_7-2.1.0-20150212.1.iso

During RPM installation, the package extracts the kernel and image from the ISO file into the TFTP directory and creates symbolic links pointing to the latest versions of the image/kernel. Use the symbolic links to make sure that you do not need to change the version in the template every time the foreman-discovery-image package is upgraded. For example:

$ find /var/lib/tftpboot/boot
/var/lib/tftpboot/boot
/var/lib/tftpboot/boot/fdi-image-rhel_7-2.1.0-20150212.1-img
/var/lib/tftpboot/boot/fdi-image-rhel_7-2.1.0-20150212.1-vmlinuz
/var/lib/tftpboot/boot/fdi-image-rhel_7-img
/var/lib/tftpboot/boot/fdi-image-rhel_7-vmlinuz

Note: Currently, only RHEL7 images are provided (even for Satellite 6 RHEL 6 installations). Red Hat may provide errata for the Satellite Discovery Image. If there are discovered hosts running during the upgrade of the foreman-discovery-image package, reboot them all to load into the updated version of the image as soon as possible. This can be done via the Satellite 6 web interface or command line interface or API.

Configuring PXE Booting

When an unknown host is booted on the provisioning network, the Satellite Server normally provides a PXELinux boot menu with only one option to boot from local hard drive. To enable hardware discovery, we need to change this behavior.

In the Satellite 6 user interface, navigate to Provisioning Templates and edit the PXELinux global default template. Add a new menu entry:

LABEL discovery
MENU LABEL Foreman Discovery
MENU DEFAULT
KERNEL boot/fdi-image-rhel_7-vmlinuz
APPEND initrd=boot/fdi-image-rhel_7-img rootflags=loop root=live:/fdi.iso rootfstype=auto ro rd.live.image acpi=force rd.luks=0 rd.md=0 rd.dm=0 rd.lvm=0 rd.bootif=0 rd.neednet=0 nomodeset proxy.url=https://SATELLITE_CAPSULE_URL:9090 proxy.type=proxy
IPAPPEND 2

The option "proxy.type" can be set to either proxy or foreman. When the option is set to "proxy", all communication goes through the Satellite Capsule. When the option is set to "foreman", the communication goes directly to Satellite Server.

The proxy.url specifies the Satellite Capsule or Satellite Server's URL, depending on the previous setting. Both http and https schemes are supported.

Note: When using proxy type, the default port on the Satellite Capsule is 9090, but for direct communication with Satellite Server, manually set the port to 80.

Note: Satellite 6.1 currently only provides one CAPSULE_URL for all subnets where hosts can be discovered. Until a configuration template that is per-subnet is supported, use a DNS alias name on all the networks or use SRV record (explained in this document).

After you have correctly modified the APPEND line, set the entry to default via the ONTIMEOUT option. This sets the discovery entry as the default option on timeout:

ONTIMEOUT discovery

Alternatively, the Discovery image also searches for DNS SRV record named x-foreman.tcp. If the DNS server is configured for this (see the example for ISC BIND below), then it is not required to provide foreman.url at all, but proxy.type is still required.

Note: Here is an example of ISC BIND configuration statement to set foreman A record to be used with HTTPS protocol.

_x-foreman._tcp SRV 0 5 9090 capsule

You can mix both approaches and override SRV record with the command line.

It is important to keep IPAPPEND 2 option which is the key configuration option to detect interfaces connected to the provisioning network. Never change or remove the root option, otherwise, the image will not boot properly.

It is important to know that the DNS servers from DHCP settings are taken into account only for the interface that are specified via the BOOTIF option. This means that when a system has multiple NICs, DNS will work for the interface that it was booted from.

Updating the PXE Default Template

Once the template has been updated, click the Build PXE Default button at the top of the "Provisioning Templates" page. This will instruct the TFTP proxy to rewrite the pxelinux.cfg/default file.

Repeat this step every time a change is made to the default template otherwise the changes will not be deployed on the TFTP Satellite Capsule.

Reviewing global settings

Discovered hosts will be automatically placed under the first Organization and Location created. If hosts should be placed in some other Location/Organization, alter the default settings in Administer > Settings > Discovered.

There is also a setting called discovery_fact which defaults to discovery_bootif. It specifies which incoming fact should be used to get the MAC address. By default, the PXELinux BOOTIF kernel command line option is used which gives the MAC address of the interface which was booted from.

Note that the auto_discovery option is set to false by default. If you want to trigger provisioning automatically with rules, you need to turn this setting on. Try provisioning manually initially to test if it works before proceeding.

It is possible to add any fact reported by Facter via the discovery_fact_column global option onto the Discovered Hosts page table as a new column. To do that, set the value of this setting to the name of a fact reported. To hide the new column, set it to a blank value.

Configuring Discovery Subnets

All subnets with discovered nodes need this specified in Satellite Server so it connects via the Foreman Proxy. To do this, go to Infrastructure > Capsules and verify if the desired proxy lists Discovery feature. If not, click on Refresh features button and it will appear immediately.

In Infrastructure > Subnets select desired Smart Proxy for each Subnet that is supposed to do discovery and verify it is connected to the Discovery Proxy.

Reviewing Permissions

The plug-in will create a Role called Discovery when first started. This can be assigned to roles for non-admins to allow them to use the discovery plug-in. Alternatively assign the perform_discovery permission to an existing Role.

Usage

Satellite Server Discovery plug-in provides user interface, API and CLI.

Hardware discovery
Boot a machine in any provisioning network that was configured with the default PXE configuration above. It should register with Satellite Server and appear in Hosts > Discovered Hosts.

Manual provisioning
Select a discovered host and choose "Provision". This will redirect to the normal Edit page for a host, with the discovered data filled in where possible. Fill in the details as normal.

Once the fields are saved, Satellite modifies the host's PXELinux file on the TFTP server and reboots the discovered host. It will then boot into an installer for the chosen OS, and finally into the installed OS.

Delete a machine and reboot it to have it move back to the Discovery Pool.

Automatic Provisioning
With Satellite 6.1, it is possible to predefine provisioning rules which will assign host group to provisioned hosts and trigger provisioning automatically. To do that, head over to Configure > Discovery rules and create such a rule:

• Name represents the rule name shown to the users. It must not contain spaces or non-alphanumeric characters.
• Search statement is used to match discovered hosts for the particular rule. Use Scoped Search syntax to define it. Examples are shown below.
• Host Group is assigned to a matching host before starting the provisioning process. Make sure that the selected host group has all the required parameters set (domain, subnet, root password), otherwise the provisioning process will fail.
• Hostname defines a pattern to assign human-readable hostnames to the matching hosts. When left empty, the hostname is assigned in the format "macMACADDRESS" by default. The same syntax used for provisioning templates is used in this instance. See below for more information and examples.
• Hosts limit enables a limit to the maximum amount of provisioned hosts per rule. If a limit is reached, the rule will not take effect until one or more hosts are deleted. Typical use cases are rules per server rack or row when it is necessary to change provisioning parameters like hostname or host group per each entry.
• Priority puts the rules in order. It must be greater than zero. Low numbers go first. Rules are always matched by priority given.
  Enabled flag is used for temporary shutdown of rules.

By default, Foreman does not trigger auto discovery automatically. This must be explicitly turned on in Administer > Settings > Discovered > discovery_auto.

Once some rules are defined, the good practice is to discover a host and apply the rules using the Auto discover button on the host which will trigger the auto-provisioning without need of turning on the global option.

Search Syntax
The easiest way of testing search patterns is in the Discovered hosts list because the search box gives the same results. A typical search field is "facts" because they all start with "facts.". Auto completion can be used to browse the facts as well as discovered hosts detail screen. Typical search queries:

facts.architecture = x86_64
facts.bios_vendor ~ 'Dell*'
facts.macaddress = "aa:bb:cc:dd:ee:ff"
facts.macaddress_eth0 = "aa:bb:cc:dd:ee:ff"
facts.ipaddress_eth1 ~ "192.168.*"

In Satellite 6.1, all the facts are currently simple strings, so it is not possible to do numeric comparisons. Luckily, some important facts are extracted and converted to numbers. These are:

cpu_count - number of CPUs
disk_count - number of disks attached
disks_size - total amount of disk space (in MiB)

Possible queries are (examples):

cpu_count >= 8
disk_count < 10
disks_size > 1000000

See the searching section for more information.

Hostname patterns
The target hostname template pattern has the same syntax as the Provisioning Templates (ERB). The domain will be appended automatically. A hostname based on MAC address will be used when left blank. In addition to @host attribute, function rand for random integers is available. Examples:

application-server-<%= rand(99999) %>
load-balancer-<%= @host.facts['bios_vendor'] + '-' + rand(99999) %>
wwwsrv-<%= @host.hostgroup.name %>
minion-<%= @host.discovery_rule.name %>
db-server-<%= @host.ip.gsub('.','-') + '-' + @host.hostgroup.subnet.name %>

When creating hostname patterns, make sure the resulting host names are unique. This is very important. Hostnames must not start with numbers. A good approach is to use unique information provided by facter (MAC address, BIOS or serial ID) or to randomize the hostname somehow.

Extending the Image

It is possible to extend the Satellite Discovery Image with custom facts, software or device drivers easily. It's possible to provide a zip file containing extra code for the image to use. First, create a directory structure like:

.
├── autostart.d
│   └── 01_zip.sh
├── bin
│   └── ntpdate
├── facts
│   └── test.rb
└── lib
    ├── libcrypto.so.1.0.0
    └── ruby
        └── test.rb

autostart.d contains scripts that will be executed in POSIX order by the image as it starts up, before the host is registered to Foreman. bin is added to PATH, you can place binaries here and use them in the autostart scripts. facts is added to FACTERLIB so that custom facts may be configured and sent to Foreman. lib is added to LD_LIBRARY_PATH and lib/ruby is added to RUBYLIB, so that binaries in bin can be executed properly.

New directives and options are appended to the existing environment variables (PATH, LD_LIBRARY_PATH, RUBYLIB and FACTERLIB). If you need to specify the path to something explicitly in your scripts, the zip contents are extracted to /opt/extension on the image.

An example structure is provided in example_zip in this repo. You can zip up your structure with zip -r my_extension.zip .

You can create multiple zip files but be aware they will be extracted to the same place on the discovery image, so files in later zips will overwrite earlier ones if they have the same filename.

To inform the image of the extensions it should use, place your zip(s) on your TFTP server along with the discovery image, and then update your PXELinux APPEND line with fdi.zips where the paths are relative to the TFTP root. So if you have two zips at $TFTP/zip1.zip and $TFTP/boot/zip2.zip, you would use fdi.zips=zip1.zip,boot/zip2.zip.

Troubleshooting

If a machine does not show up correctly in Satellite Server:

• Check your default PXE Linux template and try to re-deploy it.
• Check your pxelinux.cfg/default configuration file on the TFTP Capsule.
• Check network connectivity between nodes, Capsule and Server.
• Check the proxy.url and proxy.type options.
• Check DNS is working for that image, or use an IP in proxy.url.
• Check DHCP is handing IPs to the booted image correctly.
• Make sure discovered server (or VM) has more than 500 MB of memory (can lead to various random kernel panic errors as the image needs to be extracted in-memory).

For gathering important system facts, use the script called discovery-debug. It prints out system logs, network configuration, status, list of facts and other info on the standard output. The typical use case is to redirect this and copy out via scp for further investigation.

On the discovered host, the first virtual console is reserved for logs, all systemd logging is shown there. Particularly useful system logs are tagged with:

• discover-host - initial facts upload
• foreman-discovery - facts refresh, reboot remote commands
• nm-prepare - boot script which pre-configures NetworkManager
• NetworkManager - networking information

Use tty2 console (or higher) to login onto a discovered host. The root account and ssh access are disabled by default, but you can enable ssh and set root password using the following kernel command line options (in the Default PXELinux template on the APPEND line, see above):

fdi.ssh=1 fdi.rootpw=redhat