Chapter 17. Managing containers with Ansible

Red Hat OpenStack Platform 17.1 uses the tripleo_container_manage Ansible role to perform management operations on containers. You can also write custom playbooks to perform specific container management operations:

  • Collect the container configuration data that heat generates. The tripleo_container_manage role uses this data to orchestrate container deployment.
  • Start containers.
  • Stop containers.
  • Update containers.
  • Delete containers.
  • Run a container with a specific configuration.

Although director performs container management automatically, you might want to customize a container configuration, or apply a hotfix to a container without redeploying the overcloud.

Note

This role supports only Podman container management.

17.1. tripleo-container-manage role defaults and variables

The following excerpt shows the defaults and variables for the tripleo_container_manage Ansible role.

# All variables intended for modification should place placed in this file.
tripleo_container_manage_hide_sensitive_logs: '{{ hide_sensitive_logs | default(true)
  }}'
tripleo_container_manage_debug: '{{ ((ansible_verbosity | int) >= 2) | bool }}'
tripleo_container_manage_clean_orphans: true

# All variables within this role should have a prefix of "tripleo_container_manage"
tripleo_container_manage_check_puppet_config: false
tripleo_container_manage_cli: podman
tripleo_container_manage_concurrency: 1
tripleo_container_manage_config: /var/lib/tripleo-config/
tripleo_container_manage_config_id: tripleo
tripleo_container_manage_config_overrides: {}
tripleo_container_manage_config_patterns: '*.json'
# Some containers where Puppet is run, can take up to 10 minutes to finish
# in slow environments.
tripleo_container_manage_create_retries: 120
# Default delay is 5s so 120 retries makes a timeout of 10 minutes which is
# what we have observed a necessary value for nova and neutron db-sync execs.
tripleo_container_manage_exec_retries: 120
tripleo_container_manage_healthcheck_disabled: false
tripleo_container_manage_log_path: /var/log/containers/stdouts
tripleo_container_manage_systemd_teardown: true

17.2. tripleo-container-manage molecule scenarios

Molecule is used to test the tripleo_container_manage role. The following shows a molecule default inventory:

hosts:
  all:
    hosts:
      instance:
        ansible_host: localhost
        ansible_connection: local
        ansible_distribution: centos8

Usage

Red Hat OpenStack 17.1 supports only Podman in this role. Docker support is on the roadmap.

The Molecule Ansible role performs the following tasks:

  • Collect container configuration data, generated by the TripleO Heat Templates. This data is used as a source of truth. If a container is already managed by Molecule, no matter its present state, the configuration data will reconfigure the container as needed.
  • Manage systemd shutdown files. It creates the TripleO Container systemd service, required for service ordering when shutting down or starting a node. It also manages the netns-placeholder service.
  • Delete containers that are nonger needed or that require reconfiguration. It uses a custom filter, named needs_delete() which has a set of rules to determine if the container needs to be deleted.

    • A container will not be deleted if, the container is not managed by tripleo_ansible or the container config_id does not match the input ID.
    • A container will be deleted, if the container has no config_data or the container has config_data which does not match data in input. Note that when a container is removed, the role also disables and removes the systemd services and healtchecks.
  • Create containers in a specific order defined by start_order container config, where the default is 0.

    • If the container is an exec, a dedicated playbook for execs is run, using async so multiple execs can be run at the same time.
    • Otherwise, the podman_container is used, in async, to create the containers. If the container has a restart policy, systemd service is configured. If the container has a healthcheck script, systemd healthcheck service is configured.
Note

tripleo_container_manage_concurrency parameter is set to 1 by default, and putting a value higher than 2 can expose issues with Podman locks.

Example of a playbook:

- name: Manage step_1 containers using tripleo-ansible
  block:
    - name: "Manage containers for step 1 with tripleo-ansible"
      include_role:
        name: tripleo_container_manage
      vars:
        tripleo_container_manage_config: "/var/lib/tripleo-config/container-startup-config/step_1"
        tripleo_container_manage_config_id: "tripleo_step1"

17.3. tripleo_container_manage role variables

The tripleo_container_manage Ansible role contains the following variables:

Table 17.1. Role variables

NameDefault valueDescription

tripleo_container_manage_check_puppet_config

false

Use this variable if you want Ansible to check Puppet container configurations. Ansible can identify updated container configuration using the configuration hash. If a container has a new configuration from Puppet, set this variable to true so that Ansible can detect the new configuration and add the container to the list of containers that Ansible must restart.

tripleo_container_manage_cli

podman

Use this variable to set the command line interface that you want to use to manage containers. The tripleo_container_manage role supports only Podman.

tripleo_container_manage_concurrency

1

Use this variable to set the number of containers that you want to manage concurrently.

tripleo_container_manage_config

/var/lib/tripleo-config/

Use this variable to set the path to the container configuration directory.

tripleo_container_manage_config_id

tripleo

Use this variable to set the ID of a specific configuration step. For example, set this value to tripleo_step2 to manage containers for step two of the deployment.

tripleo_container_manage_config_patterns

*.json

Use this variable to set the bash regular expression that identifies configuration files in the container configuration directory.

tripleo_container_manage_debug

false

Use this variable to enable or disable debug mode. Run the tripleo_container_manage role in debug mode if you want to run a container with a specific one-time configuration, to output the container commands that manage the lifecycle of containers, or to run no-op container management operations for testing and verification purposes.

tripleo_container_manage_healthcheck_disable

false

Use this variable to enable or disable healthchecks.

tripleo_container_manage_log_path

/var/log/containers/stdouts

Use this variable to set the stdout log path for containers.

tripleo_container_manage_systemd_order

false

Use this variable to enable or disable systemd shutdown ordering with Ansible.

tripleo_container_manage_systemd_teardown

true

Use this variable to trigger the cleanup of obsolete containers.

tripleo_container_manage_config_overrides

{}

Use this variable to override any container configuration. This variable takes a dictionary of values where each key is the container name and the parameters that you want to override, for example, the container image or user. This variable does not write custom overrides to the JSON container configuration files and any new container deployments, updates, or upgrades revert to the content of the JSON configuration file.

tripleo_container_manage_valid_exit_code

[]

Use this variable to check if a container returns an exit code. This value must be a list, for example, [0,3].

17.4. tripleo-container-manage healthchecks

Until Red Hat OpenStack 17.1, container healthcheck was implemented by a systemd timer which would run podman exec to determine if a given container was healthy. Now, it uses the native healthcheck interface in Podman which is easier to integrate and consume.

To check if a container (for example, keystone) is healthy, run the following command:

$ sudo podman healthcheck run keystone

The return code should be 0 and “healthy”.

"Healthcheck": {
    "Status": "healthy",
    "FailingStreak": 0,
    "Log": [
        {
            "Start": "2020-04-14T18:48:57.272180578Z",
            "End": "2020-04-14T18:48:57.806659104Z",
            "ExitCode": 0,
            "Output": ""
        },
        (...)
    ]
}

17.5. tripleo-container-manage debug

The tripleo_container_manage Ansible role allows you to perform specific actions on a given container. This can be used to:

  • Run a container with a specific one-off configuration.
  • Output the container commands to manage containers lifecycle.
  • Output the changes made on containers by Ansible.
Note

To manage a single container, you need to know two things:

  • At which step during the overcloud deployment was the container deployed.
  • The name of the generated JSON file containing the container configuration.

The following is an example of a playbook to manage HAproxy container at step 1 which overrides the image setting:

- hosts: localhost
  become: true
  tasks:
    - name: Manage step_1 containers using tripleo-ansible
      block:
        - name: "Manage HAproxy container at step 1 with tripleo-ansible"
          include_role:
            name: tripleo_container_manage
          vars:
            tripleo_container_manage_config_patterns: 'haproxy.json'
            tripleo_container_manage_config: "/var/lib/tripleo-config/container-startup-config/step_1"
            tripleo_container_manage_config_id: "tripleo_step1"
            tripleo_container_manage_clean_orphans: false
            tripleo_container_manage_config_overrides:
              haproxy:
                image: quay.io/tripleomastercentos9/centos-binary-haproxy:hotfix

If Ansible is run in check mode, no container is removed or created, however at the end of the playbook run a list of commands is displayed to show the possible outcome of the playbook. This is useful for debugging purposes.

$ ansible-playbook haproxy.yaml --check

Adding the diff mode will show the changes that would have been made on containers by Ansible.

$ ansible-playbook haproxy.yaml --check --diff

The tripleo_container_manage_clean_orphans parameter is optional. It can be set to false meaning orphaned containers, with a specific config_id, will not be removed. It can be used to manage a single container without impacting other running containers with same config_id.

The tripleo_container_manage_config_overrides parameter is optional and can be used to override a specific container attribute, for example the image or the container user. The parameter creates dictionary with container name and the parameters to override. These parameters have to exist and they define the container configuration in TripleO Heat Templates.

Note the dictionary does not update the overrides in the JSON file so if an update or upgrade is executed, the container will be re-configured with the configuration in the JSON file.