Chapter 2. Building Puppet Modules from Scratch

This chapter explores how to build and test your own Puppet modules. This includes a basic tutorial on creating a Puppet module that deploys a simple web server configuration.

2.1. Examining the Anatomy of a Puppet Module

Before creating our module, we need to understand the components that create a Puppet module.

Manifests

Manifests are files that contain code to define a set of resources and their attributes. A resource is any configurable part of a system. Examples of resources include packages, services, files, users and groups, SELinux configuration, SSH key authentication, cron jobs, and more. A manifest defines each required resource using a set of key-value pairs for their attributes. For example:

package { 'httpd':
  ensure => installed,
}

This declaration checks if the httpd package is installed. If not, the manifest executes yum and installs it.

Manifests are located in the manifest directory of a module.

Puppet modules also use a test directory for test manifests. These manifests are used to test certain classes contained in your official manifests.

Static Files

Modules can contain static files that Puppet can copy to certain locations on your system. These locations, and other attributes such as permissions, are defined through file resource declarations in manifests.

Static files are located in the files directory of a module.

Templates

Sometimes configuration files require custom content. In this situation, users would create a template instead of a static file. Like static files, templates are defined in manifests and copied to locations on a system. The difference is that templates allow Ruby expressions to define customized content and variable input. For example, if you wanted to configure httpd with a customizable port then the template for the configuration file would include:

Listen <%= @httpd_port %>

The httpd_port variable in this case is defined in the manifest that references this template.

Templates are located in the templates directory of a module.

Plug-ins

Plug-ins allow for aspects that extend beyond the core functionality of Puppet. You can use plug-ins to define custom facts, custom resources, or new functions. For example, a database administrator might need a resource type for PostgreSQL databases. This could help the database administrator populate PostgreSQL with a set of new databases after installing PostgreSQL. As a result, the database administrator need only create a Puppet manifest that ensures PostgreSQL installs and the databases are created afterwards.

Plug-ins are located in the lib directory of a module. This includes a set of subdirectories depending on the plug-in type. For example:

  • /lib/facter — Location for custom facts.
  • /lib/puppet/type — Location for custom resource type definitions, which outline the key-value pairs for attributes.
  • /lib/puppet/provider — Location for custom resource providers, which are used in conjunction with resource type definitions to control resources.
  • /lib/puppet/parser/functions — Location for custom functions.

2.2. Setting up a Puppet Development System

A Puppet development system is useful for creating and testing your own modules. It is recommended to use a new system with a Red Hat Enterprise Linux 6 or 7 subscription.

After installing the new system and registering your version of Red Hat Enterprise Linux, enable the Red Hat Satellite 6 Tools repository. For example, for Red Hat Enterprise Linux 7:

# subscription-manager repos --enable=rhel-7-server-satellite-tools-6.2-rpms

After enabling the repository, install the puppet package:

# yum install puppet

2.3. Generating a New Module Boilerplate

The first step in creating a new module is to change to the Puppet module directory and create a basic module structure. Either create this structure manually or use Puppet to create a boilerplate for your module:

# cd /etc/puppet/modules
# puppet module generate user_name-module_name

An interactive wizard appears and guides you through populating the module’s metadata.json file with metadata.

Important

The puppet module generate command requires the module name take the format of user_name-module_name to comply with Puppet Forge specifications. However, to test our tutorial module and use it with Satellite 6 we need to rename the module directory to remove the user name. For example, for user_name-mymodule you would run:

# puppet module generate user_name-mymodule
# mv user_name-mymodule mymodule

When the module generation process completes, the new modules contains some basic files, including a manifests directory. This directory already contains a manifest file called init.pp, which is the module’s main manifest file. View the file to see the empty class declaration for the module:

class mymodule {


}

The module also contains a tests directory containing a manifest also named init.pp. This test manifest contains a reference to the mymodule class within manifests/init.pp:

include mymodule

Puppet will use this test manifest to test our module.

We are now ready to add our system configuration to our module.

2.4. Installing a HTTP Server

Our Puppet module will install the packages necessary to run an HTTP server. This requires a resource definition that defines configurations for the httpd package.

In the module’s manifests directory, create a new manifest file called httpd.pp:

# touch mymodule/manifests/httpd.pp

This manifest will contain all HTTP configuration for our module. For organizational purposes, we will keep this manifest separate from the init.pp manifest.

Add the following content to the new httpd.pp manifest:

class mymodule::httpd {
  package { 'httpd':
    ensure => installed,
  }
}

This code defines a subclass of mymodule called httpd, then defines a package resource declaration for the httpd package. The ensure ⇒ installed attribute tells Puppet to check if the package is installed. If it is not installed, Puppet executes yum to install it.

We also need to include this subclass in our main manifest file. Edit the init.pp manifest:

class mymodule {
  include mymodule::httpd
}

It is now time to test the module. Run the following command:

# puppet apply mymodule/tests/init.pp --noop

The puppet apply command applies the configuration in the manifest to your system. We use the test init.pp manifest, which refers to the main init.pp manifest. The --noop performs a dry run of the configuration, which shows only the output but does not actually apply the configuration. The output should resemble the following:

Notice: Compiled catalog for puppet.example.com in environment production in 0.59 seconds
Notice: /Stage[main]/Mymodule::Httpd/Package[httpd]/ensure: current_value absent, should be present (noop)
Notice: Class[Mymodule::Httpd]: Would have triggered 'refresh' from 1 events
Notice: Stage[main]: Would have triggered 'refresh' from 1 events
Notice: Finished catalog run in 0.67 seconds

The highlighted line is the result of the ensure ⇒ installed attribute. The current_value absent means that Puppet has detected the httpd package is not installed. Without the --noop option, Puppet would install the httpd package.

2.5. Running the HTTP Server

After installing the httpd package, we start the service using another resource declaration: service.

Edit the httpd.pp manifest and add the highlighted lines:

class mymodule::httpd {
  package { 'httpd':
    ensure => installed,
  }
  service { 'httpd':
    ensure => running,
    enable => true,
    require => Package["httpd"],
  }
}

This achieves a couple of things:

  • The ensure ⇒ running attribute checks if the service is running. If not, Puppet starts it.
  • The enable ⇒ true attribute sets the service to run when the system boots.
  • The require ⇒ Package["httpd"] attribute defines an ordering relationship between one resource declaration and another. In this case, it ensures the httpd service starts after the httpd package installs. This creates a dependency between the service and its respective package.

Run the puppet apply command again to test the changes to our module:

# puppet apply mymodule/tests/init.pp --noop
Notice: Compiled catalog for puppet.example.com in environment production in 0.56 seconds
Notice: /Stage[main]/Mymodule::Httpd/Package[httpd]/ensure: current_value absent, should be present (noop)
Notice: /Stage[main]/Mymodule::Httpd/Service[httpd]/ensure: current_value stopped, should be running (noop)
Notice: Class[Mymodule::Httpd]: Would have triggered 'refresh' from 2 events
Notice: Stage[main]: Would have triggered 'refresh' from 1 events
Notice: Finished catalog run in 0.41 seconds

The highlighted line is the result of our new resource definition for the httpd service.

2.6. Configuring the HTTP Server

The HTTP Server is now installed and enabled. The next step is to provide some configuration. The HTTP server already provides some default configuration in /etc/httpd/conf/httpd.conf, which provides a web server on port 80. We will add some additional configuration to provide an additional web server on a user-specified port.

We use a template file to store our configuration content because the user-defined port requires variable input. In our module, create a directory called templates and add a file called myserver.conf.erb in the new directory. Add the following contents to the file:

Listen <%= @httpd_port %>
NameVirtualHost *:<%= @httpd_port %>
<VirtualHost *:<%= @httpd_port %>>
  DocumentRoot /var/www/myserver/
  ServerName <%= @fqdn %>
  <Directory "/var/www/myserver/">
    Options All Indexes FollowSymLinks
    Order allow,deny
    Allow from all
  </Directory>
</VirtualHost>

This template follows the standard syntax for Apache web server configuration. The only difference is the inclusion of Ruby escape characters to inject variables from our module. For example, httpd_port, which we use to specify the web server port.

Notice also the inclusion of fqdn, which is a variable that stores the fully qualified domain name of the system. This is known as a system fact. System facts are collected from each system prior to generating each respective system’s Puppet catalog. Puppet uses the facter command to gather these system facts and you can also run facter to view a list of these facts.

Edit the httpd.pp manifest and add the highlighted lines:

class mymodule::httpd {
  package { 'httpd':
    ensure => installed,
  }
  service { 'httpd':
    ensure => running,
    enable => true,
    require => Package["httpd"],
  }
  file {'/etc/httpd/conf.d/myserver.conf':
  notify => Service["httpd"],
    ensure => file,
    require => Package["httpd"],
    content => template("mymodule/myserver.conf.erb"),
  }
  file { "/var/www/myserver":
    ensure => "directory",
  }
}

This achieves the following:

  • We add a file resource declaration for the server configuration file /etc/httpd/conf.d/myserver.conf.
  • We add a relationship between the configuration file and the httpd service using the notify ⇒ Service["httpd"] attribute. This checks our configuration file for any changes. If the file has changed, Puppet restarts the service.
  • We check the httpd package is installed before adding this file.
  • The content for this /etc/httpd/conf.d/myserver.conf file is the myserver.conf.erb template we created earlier.
  • We add a second file resource declaration. This one creates a directory /var/www/myserver/ for our web server.

We also need to include the httpd_port parameter in our main manifest file. Edit the init.pp manifest and add the following text shown in bold:

class mymodule (
  $httpd_port = 8120
) {
  include mymodule::httpd
}

This sets the httpd_port parameter to a default value of 8120. You can override this value with the Satellite Server.

Run the puppet apply command again to test the changes to our module:

# puppet apply mymodule/tests/init.pp --noop
Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
Notice: Compiled catalog for puppet.example.com in environment production in 0.84 seconds
Notice: /Stage[main]/Mymodule::Httpd/File[/var/www/myserver]/ensure: current_value absent, should be directory (noop)
Notice: /Stage[main]/Mymodule::Httpd/Package[httpd]/ensure: current_value absent, should be present (noop)
Notice: /Stage[main]/Mymodule::Httpd/File[/etc/httpd/conf.d/myserver.conf]/ensure: current_value absent, should be file (noop)
Notice: /Stage[main]/Mymodule::Httpd/Service[httpd]/ensure: current_value stopped, should be running (noop)
Notice: Class[Mymodule::Httpd]: Would have triggered 'refresh' from 4 events
Notice: Stage[main]: Would have triggered 'refresh' from 1 events
Notice: Finished catalog run in 0.51 seconds
Note

The warning for the hiera.yaml file is safe to ignore.

The highlighted lines show the creation of the configuration file and our web server directory.

2.7. Configuring the Firewall

The web server requires an open port so people can access the pages hosted on our web server. The open problem is that different versions of Red Hat Enterprise Linux uses different methods for controlling the firewall. For Red Hat Enterprise Linux 6 and below, we use iptables. For Red Hat Enterprise Linux 7, we use firewalld.

This decision is something Puppet handles using conditional logic and system facts. For this step, we add a statement to check the operating system and run the appropriate firewall commands.

Add the following code inside your mymodule::httpd class:

  if versioncmp($::operatingsystemmajrelease, '6') <= 0 {
    exec { 'iptables':
      command => "iptables -I INPUT 1 -p tcp -m multiport --ports ${httpd_port} -m comment --comment 'Custom HTTP Web Host' -j ACCEPT &amp;&amp; iptables-save > /etc/sysconfig/iptables",
      path => "/sbin",
      refreshonly => true,
      subscribe => Package['httpd'],
    }
    service { 'iptables':
      ensure => running,
      enable => true,
      hasrestart => true,
      subscribe => Exec['iptables'],
    }
  }
  elsif $operatingsystemmajrelease == 7 {
    exec { 'firewall-cmd':
      command => "firewall-cmd --zone=public --add-port=${httpd_port}/tcp --permanent",
      path => "/usr/bin/",
      refreshonly => true,
      subscribe => Package['httpd'],
    }
    service { 'firewalld':
      ensure => running,
      enable => true,
      hasrestart => true,
      subscribe => Exec['firewall-cmd'],
    }
  }

This code performs the following:

  • Use the operatingsystemmajrelease fact to determine whether the operating system is Red Hat Enterprise Linux 6 or 7.
  • If using Red Hat Enterprise Linux 6, declare an executable (exec) resource that runs iptables and iptables-save to add a permanent firewall rule. The httpd_port variable is used inline to define the port to open. After the exec resource completes, we trigger a refresh of the iptables service. To achieve this, we define a service resource that includes the subscribe attribute. This attribute checks if there are any changes to another resource and, if so, performs a refresh. In this case, it checks the iptables executable resource.
  • If using Red Hat Enterprise Linux 7, declare a similar executable resource that runs firewall-cmd to add a permanent firewall rule. The httpd_port variable is also used inline to define the port to open. After the exec resource completes, we trigger a refresh of the firewalld service but with a subscribe attribute pointing to the firewall-cmd executable resource.
  • The code for both firewall executable resources contains refreshonly ⇒ true and subscribe ⇒ Package['httpd'] attributes. This ensures the firewall commands only run after the httpd installs. Without these attributes, subsequent runs will add multiple instances of the same firewall rule.

Run the puppet apply command again to test the changes to our module. The following example is a test of Red Hat Enterprise Linux 6:

# puppet apply mymodule/tests/init.pp --noop
Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
Notice: Compiled catalog for puppet.example.com in environment production in 0.82 seconds
Notice: /Stage[main]/Mymodule::Httpd/Exec[iptables]/returns: current_value notrun, should be 0 (noop)
Notice: /Stage[main]/Mymodule::Httpd/Service[iptables]: Would have triggered 'refresh' from 1 events
...

The highlighted lines show the execution of the firewall rule creation and the subsequent service refresh as a result of the subscribe attribute.

Important

This configuration serves only as an example of using conditional statements. If you aim to manage multiple firewall rules for your system in the future, it is recommended to create a custom resource for firewalls. It is inadvisable to use executable resources to constantly chain many Bash commands.

2.8. Configuring SELinux

SELinux restricts non-standard access to the HTTP server by default. If we define a custom port, we need to add configuration that allows SELinux to grant access.

Puppet contains resource types to manage some SELinux functions, such as Booleans and modules. However, we need to execute the semanage command to manage port settings. This tool is a part of the policycoreutils-python package, which is not installed on Red Hat Enterprise Linux systems by default.

Add the following code inside your mymodule::httpd class:

  exec { 'semanage-port':
    command => "semanage port -a -t http_port_t -p tcp ${httpd_port}",
    path => "/usr/sbin",
    require => Package['policycoreutils-python'],
    before => Service['httpd'],
    subscribe => Package['httpd'],
    refreshonly => true,
  }
  package { 'policycoreutils-python':
    ensure => installed,
  }

This code performs the following:

  • The require ⇒ Package['policycoreutils-python'] attribute makes sure the policycoreutils-python is installed prior to executing the command.
  • Puppet executes semanage, using httpd_port as a variable, to add the custom port to the list of TCP ports Apache is allowed to listen on.
  • The before ⇒ Service ['httpd'] makes sure to execute this command before the httpd service starts. If httpd starts before the SELinux command, SELinux denies access to the port and the service fails to start.
  • The code for the SELinux executable resource contains refreshonly ⇒ true and subscribe ⇒ Package['httpd'] attributes. This ensures the SELinux commands only run after the httpd installs. Without these attributes, subsequent runs result in failure. This is because SELinux detects the port is already enabled and reports an error.

Run the puppet apply command again to test the changes to our module.

# puppet apply mymodule/tests/init.pp --noop
...
Notice: /Stage[main]/Mymodule::Httpd/Package[policycoreutils-python]/ensure: current_value absent, should be present (noop)
...
Notice: /Stage[main]/Mymodule::Httpd/Exec[semanage-port]/returns: current_value notrun, should be 0 (noop)
...
Notice: /Stage[main]/Mymodule::Httpd/Service[httpd]/ensure: current_value stopped, should be running (noop)
...

Puppet installs policycoreutils-python first, then configures port access before starting the httpd service.

2.9. Copying a HTML file to the Web Host

The HTTP server configuration is now complete. This provides a platform for installing a web-based application, which Puppet can also configure. For this example, however, we will only copy over a simple index web page to our web server.

Create a file named index.html in the files directory. Add the following content to this file:

<html>
  <head>
    <title>Congratulations</title>
  <head>
  <body>
    <h1>Congratulations</h1>
    <p>Your puppet module has correctly applied your configuration.</p>
  </body>
</html>

Create a manifest named app.pp in the manifests directory. Add the following content to this file:

class mymodule::app {
  file { "/var/www/myserver/index.html":
    ensure => file,
    mode   => '755',
    owner  => root,
    group  => root,
    source => "puppet:///modules/mymodule/index.html",
    require => Class["mymodule::httpd"],
  }
}

This new class contains a single resource declaration. This declaration copies a file from the module’s file directory from the Puppet Server to the system and sets its permissions. Additionally, the require attribute ensures the mymodule::httpd class completes the configuration successfully before we apply mymodule::app.

Finally, include this new manifest in our main init.pp manifest:

class mymodule (
  $httpd_port = 8120
) {
  include mymodule::httpd
  include mymodule::app
}

Run the puppet apply command again to test the changes to our module. The output should resemble the following:

# puppet apply mymodule/tests/init.pp --noop
Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
Notice: Compiled catalog for puppet.example.com in environment production in 0.66 seconds
Notice: /Stage[main]/Mymodule::Httpd/Exec[iptables]/returns: current_value notrun, should be 0 (noop)
Notice: /Stage[main]/Mymodule::Httpd/Package[policycoreutils-python]/ensure: current_value absent, should be present (noop)
Notice: /Stage[main]/Mymodule::Httpd/Service[iptables]: Would have triggered 'refresh' from 1 events
Notice: /Stage[main]/Mymodule::Httpd/File[/var/www/myserver]/ensure: current_value absent, should be directory (noop)
Notice: /Stage[main]/Mymodule::Httpd/Package[httpd]/ensure: current_value absent, should be present (noop)
Notice: /Stage[main]/Mymodule::Httpd/File[/etc/httpd/conf.d/myserver.conf]/ensure: current_value absent, should be file (noop)
Notice: /Stage[main]/Mymodule::Httpd/Exec[semanage-port]/returns: current_value notrun, should be 0 (noop)
Notice: /Stage[main]/Mymodule::Httpd/Service[httpd]/ensure: current_value stopped, should be running (noop)
Notice: Class[Mymodule::Httpd]: Would have triggered 'refresh' from 8 events
Notice: /Stage[main]/Mymodule::App/File[/var/www/myserver/index.html]/ensure: current_value absent, should be file (noop)
Notice: Class[Mymodule::App]: Would have triggered 'refresh' from 1 events
Notice: Stage[main]: Would have triggered 'refresh' from 2 events
Notice: Finished catalog run in 0.74 seconds

The highlighted line shows that the index.html file would be copied to the web server.

2.10. Finalizing the Module

Our module is ready for use. To export the module into an archive for Red Hat Satellite 6 to use, enter the following command:

# puppet module build mymodule

This creates an archive file at mymodule/pkg/mymodule-0.1.0.tar.gz, which contains the contents of our mymodule directory. We upload this module to our Red Hat Satellite 6 Server to provision our own HTTP server.

If any changes are required, edit the files within the modules directories and rebuild the module using the puppet module build command. The changes will only be reflected in Satellite if the module version is increased. To increase the version number, edit the /etc/puppet/modules/mymodule/metadata.json file and then rebuild the module. Upload and publish the new version in Satellite Server.