Making systems ready for Satellite 6.2's remote execution
Satellite 6.2 ships with the much anticipated remote execution feature, which allows you to run scripts and jobs on a group of systems and then gather and view the output in the Satellite interface.
Remote Execution by itself works fine out of the box for new machines, but already existing machines need to be bootstrapped by adding an SSH public key to root's ~/.ssh/authorized_keys.
You can use Puppet to do this - assuming you are using Puppet at all - in two distinct ways:
- you can use a full module for it, for example the ghoneycutt/ssh module
- you can use hash_resources to place the key generated by Satellite in root's ~/.ssh/authorized_keys
Because there is a pretty large number of ssh related modules on Puppet Forge, I'm not going to go into that in much detail. If you have chosen one of those, it pretty much rules out all the others (that's just how Puppet works), so this would become a pretty long story. It's much easier to use the hash_resources module to manage this single key.
The hash_resources module was written by Stephen Benjamin and I'm utterly in love with it. You can use it to transform a hash into a Puppet resource. This is useful for many things, from creating an Apache virtual host with the apache::vhost defined type, a MariaDB database with the mysql::db defined type, or - in this case - setting up an SSH public key for authentication and using that with remote execution.
The setup
The setup is mind-numbingly simple. Just add the hash_resources module to your Puppet CV / environment, add the hash_resources class to your system / host group and override the resources parameter with:
---
ssh_authorized_key:
'foreman-proxy@satellite.example.com':
user: root
type: <%= @host.params['remote_execution_ssh_keys'][0].split(' ')[0] %>
key: <%= @host.params['remote_execution_ssh_keys'][0].split(' ')[1] %>
Now granted, the above isn't exactly pretty, but it works and doesn't need a separate module at all. An explanation:
The first line specifies the name of an existing resource we want hash_resources to create (ssh_authorized_keys). The second line is the name we want that resource to get (the name / comment of our key, in this case). Lines 4 through 6 are parameters for that resource, starting with the name of the user, then the key type, and finally the body of the key.
I'm using some Ruby to split up the existing global parameter remote_execution_ssh_keys, because the contents of the remote_execution_ssh_keys variable itself isn't suitable to pass to hash_resources. I take the first element of the remote_execution_ssh_keys array, split that on spaces and take the first and second element of that resulting array for SSH key type and SSH key body, respectively.
The above snippet will work fine if you have only a single Satellite, without external capsules. If you have external capsules, you probably want to use a different snippet that distributes all of the remote execution SSH keys to all of your hosts (remember remote_execution_ssh_keys is an array!).
---
ssh_authorized_key:
<% for _key in @host.params['remote_execution_ssh_keys'] do -%>
<% key = _key.split(' ') -%>
<%= key[2] %>:
user: <%= @host.params['remote_execution_ssh_user'] %>
type: <%= key[0] %>
key: <%= key[1] %>
<% end -%>
The above code isn't any prettier, but it will work on an arbitrary number of SSH keys that need to be distributed. It'll loop over the remote_execution_ssh_keys array and generate input for hash_resources on the fly. It'll even set the user correctly, if you don't want to use the default root account.
Ups and downs
| Method | Ups | Downs |
|---|---|---|
| Using a full blown Puppet module | More control over other aspects of the SSH setup, like Match clauses and PermitRootLogin. | Either introduces a new module into the existing Puppet setup (if you aren't using Puppet to manage SSH yet), which can be a slow process, or you'll need to use the module already available (because generally, you can use only one module to manage SSH). Also, as there are at this time 65 modules that involve SSH on Puppet Forge, documenting a generic way to do this is impossible. |
| Using hash_resources only | Really easy to introduce, as it's likely you will use hash_resources at some point anyway. Doesn't require introducing a (new) SSH module, that might interfere with the existing setup. | Requires that you already use or don't mind starting to use hash_resources. |

Comments
Max,
I have a fairly simple solution using IdM, but I have two problems (neither are IdM related).
Steps
1) Created a user called remote-exec using Red Hat Identity Management (IdM). 2) Added the ssh key and setup sudo & hbac rules for user in IdM. 3) Changed the Settings from the Administrator pull-down menu in Satellite to use the new user "remote-exec". 4) Make sure that remote execution is enabled on the subnet that I want to use. 5) Run the job. It works! hurray!
My problems. 1) How do I changed the ssh key used? By default it uses the one at /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy.pub. Is this a configurable setting in Satellite somewhere? 2) I don't want to execute the script from /var/tmp. Is there a way to change the execution location on the remote host?
Hi Justin,
you should be able to change both settings through the
/etc/foreman-proxy/settings.d/remote_execution_ssh.ymlconfiguration file.M
Max,
Several tests later, I can confirm that satellite totally ignores the contents of this file when using remote execution and instead uses the defaults.
Are changing these settings preserved during future upgrades?
Justin,
The satellite-installer command has flags to override those properties as well (like --foreman-proxy-plugin-remote-execution-ssh-remote-working-dir, which is probably the longest command-line flag in existence). If you use those, they should survive upgrades.
Hey Justin, I checking internally and found there is a quirk in where the remote execution process gets it's configuration from.
To work around this, please do the following:
This puts the configuration in the place expected by the remote execution process. Now run the installer with the --foreman-proxy-plugin-remote-execution-ssh-remote-working-dir and --foreman-proxy-plugin-remote-execution-ssh-local-working-dir to update the working directories remotely and locally. Restart your Satellite processes and you should be good to go.
Updating the settings this way will survive an upgrade.
Hope this help,
M
That worked... with one small note, restarting the katello-service by itself was not enough. I had to shut everything down and then start it back up. Thanks!
Hi Max, What parameter type should be used for the resources for those ruby snippets in the puppet class? I tried "string", but I keep getting parser error when executing puppet on the client side. I am still Satellite 6.1.x though. Not sure if this is the problem. Thanks in advance!
puppet agent -t run:
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: uninitialized constant Puppet::ParserError at /etc/puppet/environments/KT_XXXXX_Library_rhel6_111/modules/hash_resources/manifests/init.pp:58 on node aaa.bbb.ccc
Hi Yuri,
It needs to be a hash. I just noticed a typo in the second snippet (will try and fix today), where the I didn't indent the code correctly. The correct snippet is (notice how I indented user, type and key here) below. This is required to create a correct YAML hash.
Thanks, Max. Still not working. I guess I'll wait for Satellite 6.2 to give this another try. Thanks again!
-- deleted, wrong redhat account ---
I should note that it doesn´t work to set "Merge overrides" for the resources parameter due to this foreman bug: http://projects.theforeman.org/issues/12122 foreman-tail gives error: "NoMethodError: undefined method `each_pair'"
Finally I was able to get it to work with the following. Please note, "[0]" is removed.
ssh_authorized_key: 'foreman-proxy@satellite.example.com': user: root type: <%= @host.params['remote_execution_ssh_keys'].split(' ')[0] %> key: <%= @host.params['remote_execution_ssh_keys'].split(' ')[1] %>
any idea why this would not work with bootstrap.py script and using the -rex option? I'm migrating client from satellite5 and the server fails authentication everytime when trying to run a job after a sucessful migration?
Todd, I don't know why your setup does not work, but we can test by checking the following: - is the public key in /usr/share/foreman-proxy/.ssh installed into /root/.ssh/authorized_keys on your client systems? - can you try to log into your client systems from the Satellite using the private key in /usr/share/foreman-proxy/.ssh? - can you log into you client systems as 'root' in the first place?
Maxim, I opted to use ghoneycott/ssh module but from Satellite I'm not able to see any detailed smart class parameter, only generic "client options, "server options", etc. Ideas?
Hi Alberto,
If you want to use that module, you pass a hash to the ssh::keys parameter. The hash needs to contain your key; refer to the module doc at https://github.com/ghoneycutt/puppet-module-ssh on the exact layout for the hash. It's described all the way at the bottom of that page.
Hi, Maxim, another question: can I use ERB syntax in a ghoneycutt/ssh module parameter? I'm having a LOT of troubles
To use this feature, i will have to compromise my existing security config not allow root login unless on console.
I've tried using --rex-user and setting to local account but that fails as well.
Is there way to use this and still have PermitRootLogin set to no in sshd_config?
Todd, You should be able to keep your security intact. Does this link help at all?
https://access.redhat.com/documentation/en/red-hat-satellite/6.2/paged/host-configuration-guide/113-configuring-global-settings
Hey Todd, go to https://$YOURSATELLITE/settings#RemoteExecution and change remote_execution_ssh_user to whatever user you want to use. Make sure the public key for foreman-proxy is installed in ~that_user/.ssh/authorized_keys and add the users to the sudoers file for NOPASSWD.
I just tested this and it works fine for me. Please report back whether these instructions make it work for you. Mind you, this settings is global. You can override it on a host-by-host basis by setting the remote_execution_ssh_user as a host parameter.
I'm getting this trying to execute remote SSH:
Output: Error initializing command # ArgumentError Could not parse PKey: no start line Exit status: EXCEPTION
Errors: Script execution failed
Under Settings > RemoteExecution I'm using another_user for remote_execution_ssh_user and root for remote_execution_effective_user. I've verified I can SSH using keys from another host and issue sudo commands without a password.
What's going on here?
Thanks
Hi Osborne,
Could you try running this from the capsule (or satellite server, if you use it for accessing the remote hosts):
ssh -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy REMOTE_USER@SOME_REMOTE_HOST
to check, if the issue is in the private key or somewhere else…
If I create a new key pair and do not use a passphrase, I get prompted for the user's password:
ssh -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy another_user@hostfqdnUnauthorized access is prohibited. another_user@hostfqdn's password:
Have you added the public key of the newly-generated pair into the remote host authorized_keys?
Yes. Using the SSH client SecureCRT, it connects without issue or prompts.
Here's the tail-end of the verbose output.
"missing begin marker" sounds similar to the original error, "no start line."
It seems like the issue is with the private key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy, I was able to reproduce this, when I tried to use the pub key for authentication
I suggest verifying the format of the private key, it should start with
-----BEGIN RSA PRIVATE KEY-----followed by around 30 lines of base64 characters. I suspect your private key has different format.I recreated the key pair without using a passphrase and it works--both the "ssh -i" command and the job task within Satellite 6.2.
Ivan,
I get...
ssh -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy another_user@hostfqdnUnauthorized access is prohibited. Enter passphrase for key '/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy': Enter passphrase for key '/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy': Enter passphrase for key '/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy':
And it won't accept the passphrase... When I try using an SSH client to the same host with the same user, I can enter the private key and it accepts.
Monkey-patching in of SSH key passphrase support:
Define your SSH key passphrase in /etc/foreman-proxy/settings.d/remote_execution_ssh.yml as ssh_identity_key_passphrase
Add lines to the ruby code in /opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_remote_execution_ssh_core-0.1.2/lib/smart_proxy_remote_execution_ssh_core to handle it:
connector.rb:18: @client_private_key_passphrase = options[:client_private_key_passphrase]
connector.rb:147: ssh_options[:passphrase] = @client_private_key_passphrase if @client_private_key_passphrase
dispatcher.rb:44: :client_private_key_passphrase => Settings.instance.ssh_identity_key_passphrase,
session.rb:18: @client_private_key_passphrase = Settings.instance.ssh_identity_key_passphrase
session.rb:135: options[:client_private_key_passphrase] = @client_private_key_passphrase
Finally restart the smart_proxy_dynflow_core service.
If you haven't already, you should definitely submit this upstream!
I've got an RFE #1437538 open and it's been updated with those changes, so should make it's way through in due course.
Andrew, is your RFE to attempt to make remote execution possible without having to inherit the security vulnerabilities of 1) an ssh-key going to root (and thereby also allowing ssh to root as well) or 2) a sudoer rule that can actually not have "NOPASSWD" which is another unacceptable security risk (to a variety of unique security-conscious customers)?
Ok, So I know i'm new to the whole puppet thing, but I've followed everything, and it does not even appear that the puppet client even sees this module to attempt to run it. Can someone point me to some quick, plain english documentation on how to get a system to see a puppet module on a sat 6.2 install.
How is this?
https://access.redhat.com/documentation/en-us/red_hat_satellite/6.2/html/puppet_guide/
You may also consider this, if you do not want to have to load to module in manually. https://access.redhat.com/solutions/1290513
I've followed both of those. I have the repository already set. I can see the modules, I have added them to My content view, have assigned my test server to that content view, ran the puppet client to update the configuration, but it never shows any indication that it even sees the module to execute anything. Not sure how to debug(and yes I have opened a ticket with support as well )
Have you enabled the puppet class for this system, or the hostgroup that it is part of? Also ensure that the host and/or host group is using the correct puppet environment.
So I now see the module running on my client, but I do not see that it creates the key. I assume that this writes to the root users .sh directory correct? What happens if the directory does not exist?
We're interested in a form of remote execution of Satellite version 6.2.current that can have two primary characteristics.
1) not allow ssh directly to root because it violates mandatory security controls we are forced to follow, and no exception is possible 2) not allow a sudoer rule that contains "NOPASSWD" for the same reason in #1 above.
It seems Todd Walters submitted an RFE and separately, maybe Red-Hatter James Mills believes there is a way to achieve this without these risks?
Does anyone (or has anyone) made a remote execution solution that --does-not--- allow direct ssh to root, or a sudoer rule that --omits-- the use of "NOPASSWD" directive?
Thanks to anyone who might be interested in helping
EDITED: RFE number cited earlier is 1437538
I know the first requirement is possible. In your Satellite GUI go to Administer -> Settings -> RemoteExecution. There you can change what user is used to ssh, the effective user and the sudo method:
However the second requirement is not possible at the moment as far as I know. I have to use a sudo rule in IdM to allow my user to use sudo without password. Do you have the ID of that RFE you mention? I think it is a very valid RFE to have another setting e.g (encrypted of course):
Ture Karlsson, thanks for chiming in, sorry, the RFE # cited earlier in this discussion is 1437538 And my dumb question of the day, you mentioned:
remote_execution_ssh_user: my-rex-userIf I had a user (let's say) of "elvis" would I simply enter "elvis" or would I enter something else because of "my-rex-user"?Thanks Ture Karlsson
R. Hinton, thanks for the RFE number!
1437538 refers to being able to have a SSH key with a passphrase, which does not solve your problem regarding the NOPASSWD directive.
If I understand your problem correctly, you need a way to escalate the user privilege of the ssh user without using the NOPASSWD setting in your sudoers config. Hence my suggestion with the "remote_execution_sudo_password" setting above.
And yes, replace my-rex-user with the username you want to use.
Thanks Ture Karlsson - yes, we wish to be able to have a means to do this without the use of the NOPASSWD directive. The suggestion for "remote_execution_sudo_password" would be great if it could be made available.
R. Hinton, I've opened a Bugzilla for this: https://bugzilla.redhat.com/show_bug.cgi?id=1515888
If you have an open case with the support regarding this issue, make sure it is connected to that Bugzilla.