Chapter 4. Running Jobs on Hosts

Red Hat Satellite supports the ability to run jobs with arbitrary commands on hosts using shell scripts and to run Ansible tasks and playbooks. This is referred to as remote execution.

Remote execution is enabled by default on the Satellite Server, but must be enabled manually on Capsule Servers. For custom Ansible roles that you create, or roles that you download, the package containing the roles must be installed on the Capsule’s base system where a task is executed, or a playbook is run. The roles must also be imported into Satellite from the Capsule where they are installed before Satellite can use them.

Communication occurs through the Capsule Server which means that Satellite Server does not require direct access to the target host, and can scale to control many hosts. Remote execution uses the SSH service which must be enabled and running on the target host. Ensure the Capsule has access to port 22 on the target hosts.

Satellite uses ERB syntax job templates similar to the provisioning templates. Several job templates for shell scripts and Ansible are included by default. See Section 4.4.1, “Setting up Job Templates”.

Note

Any Capsule Server’s base system is a client of Satellite Server’s internal Capsule, and therefore this section applies to any type of host connected to Satellite Server, including Capsule Servers.

You can run jobs on multiple hosts at once, and you can use variables in your commands to suit your deployment. Variable values can be filled by host fact, Smart Class Parameter, Smart Variable, or even host parameter. In addition, you can specify custom values for templates when you run the command. See Section 4.4.2, “Executing Jobs”.

By default, each Capsule is installed with the remote execution feature disabled. To enable remote execution, enter the following command:

# satellite-installer --scenario capsule \
--enable-foreman-proxy-plugin-remote-execution-ssh

To verify that remote execution is enabled on the Capsule Server, in the web UI navigate to Infrastructure > Capsules and in the Actions column, to the right of the edit icon, select Refresh features and confirm the feature is listed.

By default, Satellite Server is configured to use remote execution rather than Katello Agent. These settings can be changed by first creating custom job templates and then selecting these new templates in the web UI by going to Administer > Remote Execution Features. For each action you want to change, select the label and then select the job template to use.

4.1. Establishing a Secure Connection for Remote Commands

The SSH keys used for remote execution are created automatically when installing a Capsule and the settings are in the /etc/foreman-proxy/settings.d/remote_execution_ssh.yml file. They include the following options:

ssh_identity_file
File to load the SSH key from. By default, set to /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy.
local_working_dir
Directory used on the Satellite or Capsule to run the scripts necessary for remote execution. By default, set to /var/tmp.
remote_working_dir
Directory on the client system that is used to execute the remote execution jobs. By default, set to /var/tmp.
Note

If the client system has noexec set for the /var/ volume or file system, change the remote_working_dir as otherwise the remote execution job will fail since the script cannot be run.

If you must use an alternative directory, create the new directory, for example new_place, and then copy the SELinux context from the default directory. For example:

# chcon --reference=/var new_place

See Maintaining SELinux Labels in the SELinux User’s and Administrator’s Guide for more information on working with SELinux labels.

Distributing SSH Keys for Remote Execution

To use SSH keys for authenticating remote execution connections, distribute the public SSH key from a Capsule to its attached hosts that you want to manage. Ensure the SSH service is enabled and running on the hosts. Configure any network or host-based firewalls to enable access to port 22.

There are three ways to distribute the public key from a Capsule to target hosts:

  • To distribute SSH keys manually, enter the following command on the Capsule:

    # ssh-copy-id -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy.pub root@target.example.com

    Where target.example.com is the host name of the target host. Repeat for each target host you want to manage.

    To confirm the key was successfully copied to the target host, enter the following command on the Capsule:

    # ssh -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy root@target.example.com
  • To use the Satellite API to download the public key directly from the Capsule, enter the following command on each target host:

    # curl https://capsule.example.com:9090/ssh/pubkey >> ~/.ssh/authorized_keys

    Where capsule.example.com is the host name of the Capsule that the host is attached to.

  • To include the public key in newly-provisioned hosts, modify for example the Kickstart default finish template to include the following line:

    <%= snippet 'remote_execution_ssh_keys' %>

4.2. Configuring a keytab for Kerberos Ticket Granting Tickets

Use this procedure to configure Satellite to use a keytab to obtain Kerberos ticket granting tickets. If you do not set up a keytab, you must manually retrieve tickets.

To ensure that the foreman-proxy user on Satellite can obtain Kerberos ticket granting tickets, complete the following steps:

  1. Find the ID of the foreman-proxy user:

    # id -u foreman-proxy
  2. Modify the umask value so that new files have the permissions 600:

    # umask 077
  3. Create the directory for the keytab:

    # mkdir -p "/var/kerberos/krb5/user/USER_ID"
  4. Create a keytab or copy an existing keytab to the directory:

    # cp your_client.keytab /var/kerberos/krb5/user/USER_ID/client.keytab
  5. Change the directory owner to the foreman-proxy user:

    # chown -R foreman-proxy:foreman-proxy "/var/kerberos/krb5/user/USER_ID"
  6. Ensure that the keytab file is read-only:

    # chmod -wx "/var/kerberos/krb5/user/USER_ID/client.keytab"
  7. Restore the SELinux context:

    # restorecon -RvF /var/kerberos/krb5

4.3. Setting Up Kerberos Authentication for Remote Execution

From Satellite 6.4, you can use Kerberos authentication to establish an SSH connection for remote execution on Satellite hosts.

Prerequisites

Before you can use Kerberos authentication for remote execution on Red Hat Satellite, you must set up a Kerberos server for identity management and ensure that you complete the following prerequisites:

  • Enroll Satellite Server on the Kerberos server
  • Enroll the Satellite target host on the Kerberos server
  • Configure and initialize a Kerberos user account for remote execution
  • Ensure that the foreman-proxy user on Satellite has a valid Kerberos ticket granting ticket

To set up Satellite to use Kerberos authentication for remote execution on hosts, complete the following steps:

  1. Before installing the tfm-rubygem-net-ssh-krb package, you must temporarily set SELinux to permissive until Red Hat bug 1541481 is resolved:

    # setenforce 0
  2. To install the tfm-rubygem-net-ssh-krb package, enter the following command:

    # yum install tfm-rubygem-net-ssh-krb
  3. To install and enable Kerberos authentication for remote execution, enter the following command:

    # satellite-installer --scenario satellite \
     --foreman-proxy-plugin-remote-execution-ssh-ssh-kerberos-auth true
  4. Set SELinux to enforcing:

    # setenforce 1
  5. To edit the default user for remote execution, in the Satellite web UI, navigate to Administer > Settings and click the RemoteExecution tab. In the remote_execution_ssh_user row, edit the second column and add the user name for the Kerberos account.
  6. Navigate to remote_execution_effective_user and edit the second column to add the user name for the Kerberos account.

To confirm that Kerberos authentication is ready to use, run a remote job on the host.

4.4. Configuring and Running Remote Jobs

Any command that you want to apply to a remote host must be defined as a job template. After you have defined a job template you can execute it multiple times.

4.4.1. Setting up Job Templates

Satellite provides default job templates that you can use for executing jobs. To view the list of job templates, navigate to Hosts > Job templates. If want to use a template without making changes, proceed to Section 4.4.2, “Executing Jobs”.

You can use default templates as a base for developing your own. Default job templates are locked for editing. Clone the template and edit the clone.

  1. To clone a template, in the Actions column, select Clone.
  2. Enter a unique name for the clone and click Submit to save the changes.

Job templates use the Embedded Ruby (ERB) syntax. For more information about writing templates, see the Appendix A, Template Writing Reference.

Ansible Considerations

To create an Ansible job template, use the following procedure and instead of ERB syntax, use YAML syntax. Begin the template with --- and to the first line, you must add - hosts: all. You can embed an Ansible playbook YAML file into the job template body. You can also add ERB syntax to customize your YAML Ansible template. You can also import Ansible playbooks in Satellite. For more information, see Synchronizing Templates with Git in the Content Management Guide.

Parameter Variables

At run time, job templates can accept parameter variables that you define for a host. Note that only the parameters visible on the Parameters tab at the host’s edit page can be used as input parameters for job templates. If you do not want your Ansible job template to accept parameter variables at run time, in the Satellite web UI, navigate to Administer > Settings and click the Ansible tab. In the Top level Ansible variables row, change the Value parameter to No.

To Create a Job Template:

  1. Navigate to Hosts > Job templates.
  2. Click New Job Template.
  3. Click the Template tab, and in the Name field, enter a unique name for your job template.
  4. Select Default to make the template available for all organizations and locations.
  5. Create the template directly in the template editor or upload it from a text file by clicking Import.
  6. Optional: In the Audit Comment field, add information about the change.
  7. Click the Job tab, and in the Job category field, enter your own category or select from the default categories listed in Table 4.1, “Default Job Template Categories”.
  8. Optional: In the Description Format field, enter a description template. For example, Install package %{package_name}. You can also use %{template_name} and %{job_category} in your template.
  9. From the Provider Type menu, select SSH for shell scripts and Ansible for Ansible tasks or playbooks.
  10. Optional: In the Timeout to kill field, enter a timeout value to terminate the job if it does not complete.
  11. Optional: Click Add Input to define an input parameter. Parameters are requested when executing the job and do not have to be defined in the template. For examples, see the Help tab.
  12. Optional: Click Foreign input set to include other templates in this job.
  13. Optional: In the Effective user area, configure a user if the command cannot use the default remote_execution_effective_user setting.
  14. Optional: If this template is a snippet to be included in other templates, click the Type tab and select Snippet.
  15. Click the Location tab and add the locations where you want to use the template.
  16. Click the Organizations tab and add the organizations where you want to use the template.
  17. Click Submit to save your changes.

You can create advanced templates by including other templates in the template syntax, see ] for more information. An advanced template is required, for example, for executing jobs that perform power actions; see xref:exam-Managing_Hosts-Including_Power_Actions_in_Templates[ for information on how to include the Power Action - SSH Default template in a custom template.

Table 4.1. Default Job Template Categories

Job template categoryDescription

Packages

Templates for performing package related actions. Install, update, and remove actions are included by default.

Puppet

Templates for executing Puppet runs on target hosts.

Power

Templates for performing power related actions. Restart and shutdown actions are included by default.

Commands

Templates for executing custom commands on remote hosts.

Services

Templates for performing service related actions. Start, stop, restart, and status actions are included by default.

Katello

Templates for performing content related actions. These templates are used mainly from different parts of the Satellite web UI (for example bulk actions UI for content hosts), but can be used separately to perform operations such as errata installation.

Example 4.1. Creating a restorecon Template

This example shows how to create a template called Run Command - restorecon that will restore the default SELinux context for all files in the selected directory on target hosts.

  1. Navigate to Hosts > Job templates. Click New Job Template.
  2. Enter Run Command - restorecon in the Name field. Select Default to make the template available to all organizations. Add the following text to the template editor:

    restorecon -RvF <%= input("directory") %>

    The <%= input("directory") %> string will be replaced by a user-defined directory during job invocation.

  3. On the Job tab, set Job category to Commands.
  4. Click Add Input to allow job customization. Enter directory to the Name field. The input name must match the value specified in the template editor.
  5. Click Required so that the command cannot be executed without the user specified parameter.
  6. Select User input from the Input type drop-down list. Enter a description to be shown during job invocation, for example Target directory for restorecon.
  7. Click Submit.

See Example 4.2, “Executing a restorecon Template on Multiple Hosts” for information on how to execute a job based on this template.

4.4.2. Executing Jobs

This section shows how to run a job based on a job template against one or more hosts.

To Execute a Remote Job:

  1. Navigate to Hosts > All hosts and select the target hosts for your job. You can use the search field to filter the host list.
  2. From the Select Action menu, select Schedule Remote Job.
  3. On the Job invocation page, define the main job settings:

    1. Select the Job category and the job template you want to use.
    2. Optionally, select a stored search string in the Bookmark list to specify the target hosts.
    3. Optionally, further limit the targeted hosts by entering a Search query. The Resolves to line displays the number of hosts affected by your query. Use the refresh button to recalculate the number after changing the query. The preview icon will list the targeted hosts.
    4. The remaining settings depend on the selected job template. See To Create a Job Template: for information on adding custom parameters to a template.
  4. Clicking Display advanced fields will show advanced setting for the job. Some of the advanced settings depend on the job template, the following settings are general:

    • Effective user defines the user for executing the job, by default it is the SSH user.
    • Concurrency level defines maximum number of jobs executed at once, which can prevent overload of systems' resources in a case of executing the job on a large number of hosts.
    • Time span defines time interval in seconds after which the job should be killed, if it is not finished already. A task which could not be started during the defined interval, for example, if the previous task took too long to finish, is canceled.
    • Type of query defines when the search query is evaluated. This helps to keep the query up to date for scheduled tasks.

      Concurrency level and Time span settings enable you to tailor job execution to fit your infrastructure hardware and needs.

  5. If you want to run the job immediately, ensure that Schedule is set to Execute now. You can also define a one-time future job, or set up a recurring job. For recurring tasks, you can define start and end dates, number and frequency of runs. You can also use cron syntax to define repetition. For more information about cron, see the Automating System Tasks section of the Red Hat Enterprise Linux 7 System Administrator’s Guide.
  6. Click Submit. This displays the Job Overview page, and when the job completes, also displays the status of the job.

Example 4.2. Executing a restorecon Template on Multiple Hosts

This example shows how to run a job based on the template created in Example 4.1, “Creating a restorecon Template” on multiple hosts. The job will restore the SELinux context in all files under the /home/ directory.

  1. Navigate to Hosts > All hosts and select target hosts. Select Schedule Remote Job from the Select Action drop-down list.
  2. In the Job invocation page, select the Commands job category and the Run Command - restorecon job template.
  3. Type /home in the directory field.
  4. Set Schedule to Execute now.
  5. Click Submit. You are taken to the Job invocation page where you can monitor the status of job execution.

4.4.3. Monitoring Jobs

You can monitor the progress of the job while it is running. This can help in any troubleshooting that may be required.

To Monitor a Job:

  1. Navigate to the Job page. This page is automatically displayed if you triggered the job with the Execute now setting. To monitor scheduled jobs, navigate to Monitor > Jobs and select the job run you wish to inspect.
  2. On the Job page, click the Hosts tab. This displays the list of hosts on which the job is running.
  3. In the Host column, click the name of the host that you want to inspect. This displays the Detail of Commands page where you can monitor the job execution in real time.
  4. Click Back to Job at any time to return to the Job Details page.

4.4.4. Creating Advanced Templates

When creating a job template, you can include an existing template in the template editor field. This way you can combine templates, or create more specific templates from the general ones.

The following template combines default templates to install and start the httpd service on Red Hat Enterprise Linux systems:

<%= render_template 'Package Action - SSH Default', :action => 'install', :package => 'httpd' %>
<%= render_template 'Service Action - SSH Default', :action => 'start', :service_name => 'httpd' %>

The above template specifies parameter values for the rendered template directly. It is also possible to use the input() method to allow users to define input for the rendered template on job execution. For example, you can use the following syntax:

<%= render_template 'Package Action - SSH Default', :action => 'install', :package => input("package") %>

With the above template, you have to import the parameter definition from the rendered template. To do so, navigate to the Jobs tab, click Add Foreign Input Set, and select the rendered template from the Target template drop-down list. You can import all parameters or specify a comma separated list.

Example 4.3. Rendering a restorecon Template

This example shows how to create a template derived from the Run command - restorecon template created in Example 4.1, “Creating a restorecon Template”. This template does not require user input on job execution, it will restore the SELinux context in all files under the /home/ directory on target hosts.

Create a new template as described in Section 4.4.1, “Setting up Job Templates”, and specify the following string in the template editor:

<%= render_template("Run Command - restorecon", :directory => "/home") %>

Example 4.4. Including Power Actions in Templates

This example shows how to set up a job template for performing power actions, such as reboot. This procedure prevents Satellite from interpreting the disconnect exception upon reboot as an error, and consequently, remote execution of the job works correctly.

Create a new template as described in Section 4.4.1, “Setting up Job Templates”, and specify the following string in the template editor:

<%= render_template("Power Action - SSH Default", :action => "restart") %>

4.5. Configuring Global Settings

The Satellite remote execution feature provides numerous global settings that you can use to configure its behavior. These are listed in Table 4.2, “Global Settings for Remote Execution”. To review and update these settings, navigate to Administer > Settings and click the Remote Execution tab.

Table 4.2. Global Settings for Remote Execution

Parameter NameDescription

Default SSH key passphrase

Defines the default key passphrase to use for SSH. You can override this for each host by setting the remote_execution_ssh_key_passphrase parameter.

Enable Global Capsule

When enabled, searches the host for any Capsule with remote execution configured. This is useful when the host has no subnet configured or if the subnet does not have a Capsule with remote execution enabled.

Fallback Without Capsule

When enabled, the remote execution will try to run the commands directly, when no Capsule with remote execution feature is configured for the host.

Fallback to Any Capsule

When enabled, searches for a remote execution Capsule outside of the Capsules assigned to the host. If the location or organization is configured, the search will be limited to the host’s organization or location.

Effective User Method

Defines which method to use to set the effective user on the target host. One of sudo, dzdo, su.

Effective User

Defines the default effective user for any job. When the job is executed, the effective user of the process is changed accordingly. You can override this for each job template and job invocation.

Connect by IP

When enabled, the IP addresses on host interfaces is preferred over the FQDN. It is useful when DNS is not resolving the FQDNs properly. You can override this for each host by setting the remote_execution_connect_by_ip parameter.

SSH User

Defines the default user to use while the Capsule connects to the target using SSH. You can override this for each host by setting the remote_execution_ssh_user parameter. You can set this by host, host group, operating system, domain, location, or organization. This can also be a different user from the effective user.

Sudo password

Defines the sudo password.

Default SSH password

Defines the default password to use for SSH. You can override this for each host by setting the remote_execution_ssh_password parameter.

SSH Port

Defines the port to use for SSH communication. The default port is 22. You can override this for each host by setting the remote_execution_ssh_port parameter.

Sync Job Templates

When enabled, job templates are synchronized from disk when seeding a database.

Cleanup working directories

When enabled, working directories will be removed after task completion. You can override this for each host by setting a remote_execution_cleanup_working_dirs parameter.

Workers pool size

Defines the number of workers in the pool to handle the execution of the remote execution jobs. Restart of the dynflowd/foreman-tasks service is required.

Important

Modify these parameters in the web UI, because any manual changes that you make in the /etc/foreman/settings.yaml file are overwritten the next time you run satellite-installer. Alternatively, use the foreman-rake config command from a console.

4.6. Remote Execution and Capsules

By default, Capsules have the remote execution provider feature enabled. You can use any Capsule that you add to the host’s organization and location to perform remote execution.

In more complex environments, where not all Capsules can be used because of possible network isolation, you can set the remote_execution_global_proxy variable to false to disable remote execution on Capsules. In this configuration, you can assign a pool of Capsules to each subnet, and jobs are load balanced across them. To use remote execution for a host, any Capsule Server that you want to use for remote execution must reside within the host’s subnet. Ensure that the host’s interface for that subnet is enabled for remote execution.

Alternatively, you can set the remote_execution_fallback_proxy variable to true to enable fallback mode. In this configuration, remote execution will use any Capsule associated with the host, such as its Puppet Master, provided that Capsule also has remote execution configured.

4.7. Delegating Permissions for Remote Execution

You can control which users can run which jobs within your infrastructure, including which hosts they can target. The remote execution feature provides two built-in roles:

  • Remote Execution Manager: This role allows access to all remote execution features and functionality.
  • Remote Execution User: This role only allows running jobs; it does not provide permission to modify job templates.

You can clone the Remote Execution User role and customize its filter for increased granularity. If you adjust the filter with the view_job_templates permission, the user can only see and trigger jobs based on matching job templates. You can use the view_hosts and view_smart_proxies permissions to limit which hosts or Capsules are visible to the role.

The execute_template_invocation permission is a special permission that is checked immediately before execution of a job begins. This permission defines which job template you can run on a particular host. This allows for even more granularity when specifying permissions. For more information on working with roles and permissions see Creating and Managing Roles in the Administering Red Hat Satellite.

The following example shows filters for the execute_template_invocation permission:

name = Reboot and host.name = staging.example.com
name = Reboot and host.name ~ *.staging.example.com
name = "Restart service" and host_group.name = webservers

The first line in the above example permits the user to apply the Reboot template to one selected host. The second line defines a pool of hosts with names ending with .staging.example.com. The third line binds the template with a host group.

Note

Permissions assigned to users can change over time. If a user has already scheduled some jobs to run in the future, and the permissions have changed, this can result in execution failure because the permissions are checked immediately before job execution.