Red Hat Training

A Red Hat training course is available for RHEL 8

Chapter 21. Monitoring containers

Use Podman commands to manage a Podman environment. With that, you can determine the health of the container, by displaying system and pod information, and monitoring Podman events.

21.1. Using a health check on a container

You can use the health check to determine the health or readiness of the process running inside the container.

If the health check succeeds, the container is marked as "healthy"; otherwise, it is "unhealthy". You can compare a health check with running the podman exec command and examining the exit code. The zero exit value means that the container is "healthy".

Health checks can be set when building an image using the HEALTHCHECK instruction in the Containerfile or when creating the container on the command line. You can display the health-check status of a container using the podman inspect or podman ps commands.

A health check consists of six basic components:

  • Command
  • Retries
  • Interval
  • Start-period
  • Timeout
  • Container recovery

The description of health check components follows:

Command (--health-cmd option)
Podman executes the command inside the target container and waits for the exit code.

The other five components are related to the scheduling of the health check and they are optional.

Retries (--health-retries option)
Defines the number of consecutive failed health checks that need to occur before the container is marked as "unhealthy". A successful health check resets the retry counter.
Interval (--health-interval option)
Describes the time between running the health check command. Note that small intervals cause your system to spend a lot of time running health checks. The large intervals cause struggles with catching time outs.
Start-period (--health-start-period option)
Describes the time between when the container starts and when you want to ignore health check failures.
Timeout (--health-timeout option)
Describes the period of time the health check must complete before being considered unsuccessful.
Note

The values of the Retries, Interval, and Start-period components are time durations, for example “30s” or “1h15m”. Valid time units are "ns," "us," or "µs", "ms," "s," "m," and "h".

Container recovery (--health-on-failure option)

Determines which actions to perform when the status of a container is unhealthy. When the application fails, Podman restarts it automatically to provide robustness. The --health-on-failure option supports four actions:

  • none: Take no action, this is the default action.
  • kill: Kill the container.
  • restart: Restart the container.

  • stop: Stop the container.

    Note

    The --health-on-failure option is available in Podman version 4.2 and later.

Warning

Do not combine the restart action with the --restart option. When running inside of a systemd unit, consider using the kill or stop action instead, to make use of systemd restart policy.

Health checks run inside the container. Health checks only make sense if you know what the health state of the service is and can differentiate between a successful and unsuccessful health check.

Additional resources

21.2. Performing a health check using the command line

You can set a health check when creating the container on the command line.

Prerequisites

  • The container-tools module is installed.

Procedure

  1. Define a health check: 

    $ podman run -dt --name=hc-container -p 8080:8080 --health-cmd='curl http://localhost:8080 || exit 1' --health-interval=0 registry.access.redhat.com/ubi8/httpd-24
    • The --health-cmd option sets a health check command for the container.
    • The --health-interval=0 option with 0 value indicates that you want to run the health check manually.

  2. Check the health status of the hc-container container:

    • Using the podman inspect command: 

      $ podman inspect --format='{{json .State.Health.Status}}' hc-container
healthy

    • Using the podman ps command: 

      $ podman ps
CONTAINER ID  IMAGE                 COMMAND               CREATED      STATUS          PORTS       NAMES
a680c6919fe  localhost/hc-container:latest  /usr/bin/run-http...  2 minutes ago  Up 2 minutes (healthy) hc-container

    • Using the podman healthcheck run command: 

      $ podman healthcheck run hc-container
healthy

Additional resources

21.3. Performing a health check using a Containerfile

You can set a health check by using the HEALTHCHECK instruction in the Containerfile.

Prerequisites

  • The container-tools module is installed.

Procedure

  1. Create a Containerfile: 

    $ cat Containerfile
FROM registry.access.redhat.com/ubi8/httpd-24
EXPOSE 8080
HEALTHCHECK CMD curl http://localhost:8080 || exit 1
    Note

    The HEALTHCHECK instruction is supported only for the docker image format. For the oci image format, the instruction is ignored.

  2. Build the container and add an image name: 

    $ podman build --format=docker -t hc-container .
STEP 1/3: FROM registry.access.redhat.com/ubi8/httpd-24
STEP 2/3: EXPOSE 8080
--> 5aea97430fd
STEP 3/3: HEALTHCHECK CMD curl http://localhost:8080 || exit 1
COMMIT health-check
Successfully tagged localhost/health-check:latest
a680c6919fe6bf1a79219a1b3d6216550d5a8f83570c36d0dadfee1bb74b924e

  3. Run the container: 

    $ podman run -dt --name=hc-container localhost/hc-container

  4. Check the health status of the hc-container container:

    • Using the podman inspect command: 

      $ podman inspect --format='{{json .State.Health.Status}}' hc-container
healthy

    • Using the podman ps command: 

      $ podman ps
CONTAINER ID  IMAGE                 COMMAND               CREATED      STATUS          PORTS       NAMES
a680c6919fe  localhost/hc-container:latest  /usr/bin/run-http...  2 minutes ago  Up 2 minutes (healthy) hc-container

    • Using the podman healthcheck run command: 

      $ podman healthcheck run hc-container
healthy

Additional resources

21.4. Displaying Podman system information

The podman system command enables you to manage the Podman systems by displaying system information.

Prerequisites

  • The container-tools module is installed.

Procedure

  • Display Podman system information:

    • To show Podman disk usage, enter: 

      $ podman system df
TYPE           TOTAL       ACTIVE      SIZE        RECLAIMABLE
Images         3           2           1.085GB     233.4MB (0%)
Containers     2           0           28.17kB     28.17kB (100%)
Local Volumes  3           0           0B          0B (0%)

    • To show detailed information about space usage, enter: 

      $ podman system df -v
Images space usage:

REPOSITORY                                TAG         IMAGE ID      CREATED     SIZE        SHARED SIZE  UNIQUE SIZE  CONTAINERS
registry.access.redhat.com/ubi8           latest      b1e63aaae5cf  13 days     233.4MB     233.4MB      0B           0
registry.access.redhat.com/ubi8/httpd-24  latest      0d04740850e8  13 days     461.5MB     0B           461.5MB      1
registry.redhat.io/rhel8/podman           latest      dce10f591a2d  13 days     390.6MB     233.4MB      157.2MB      1

Containers space usage:

CONTAINER ID  IMAGE         COMMAND                     LOCAL VOLUMES  SIZE        CREATED     STATUS      NAMES
311180ab99fb  0d04740850e8  /usr/bin/run-httpd          0              28.17kB     16 hours    exited      hc1
bedb6c287ed6  dce10f591a2d  podman run ubi8 echo hello  0              0B          11 hours    configured  dazzling_tu

Local Volumes space usage:

VOLUME NAME                                                       LINKS       SIZE
76de0efa83a3dae1a388b9e9e67161d28187e093955df185ea228ad0b3e435d0  0           0B
8a1b4658aecc9ff38711a2c7f2da6de192c5b1e753bb7e3b25e9bf3bb7da8b13  0           0B
d9cab4f6ccbcf2ac3cd750d2efff9d2b0f29411d430a119210dd242e8be20e26  0           0B

    • To display information about the host, current storage stats, and build of Podman, enter: 

      $ podman system info
host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.29-1.module+el8.5.0+12381+e822eb26.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: 7d0fa63455025991c2fc641da85922fde889c91b'
  cpus: 2
  distribution:
    distribution: '"rhel"'
    version: "8.5"
  eventLogger: file
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.18.0-323.el8.x86_64
  linkmode: dynamic
  memFree: 352288768
  memTotal: 2819129344
  ociRuntime:
    name: runc
    package: runc-1.0.2-1.module+el8.5.0+12381+e822eb26.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.2
      spec: 1.0.2-dev
      go: go1.16.7
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.5.0+12381+e822eb26.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 3113668608
  swapTotal: 3124752384
  uptime: 11h 24m 12.52s (Approximately 0.46 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-1.module+el8.5.0+12381+e822eb26.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /run/user/1000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630360721
  BuiltTime: Mon Aug 30 23:58:41 2021
  GitCommit: ""
  GoVersion: go1.16.7
  OsArch: linux/amd64
  Version: 3.3.1

    • To remove all unused containers, images and volume data, enter: 

      $ podman system prune
WARNING! This will remove:
        - all stopped containers
        - all stopped pods
        - all dangling images
        - all build cache
Are you sure you want to continue? [y/N] y
      • The podman system prune command removes all unused containers (both dangling and unreferenced), pods and optionally, volumes from local storage.
      • Use the --all option to delete all unused images. Unused images are dangling images and any image that does not have any containers based on it.
      • Use the --volume option to prune volumes. By default, volumes are not removed to prevent important data from being deleted if there is currently no container using the volume.

Additional resources

  • podman-system-df man page
  • podman-system-info man page
  • podman-system-prune man page

21.5. Podman event types

You can monitor events that occur in Podman. Several event types exist and each event type reports different statuses.

The container event type reports the following statuses:

  • attach
  • checkpoint
  • cleanup
  • commit
  • create
  • exec
  • export
  • import
  • init
  • kill
  • mount
  • pause
  • prune
  • remove
  • restart
  • restore
  • start
  • stop
  • sync
  • unmount
  • unpause

The pod event type reports the following statuses:

  • create
  • kill
  • pause
  • remove
  • start
  • stop
  • unpause

The image event type reports the following statuses:

  • prune
  • push
  • pull
  • save
  • remove
  • tag
  • untag

The system type reports the following statuses:

  • refresh
  • renumber

The volume type reports the following statuses:

  • create
  • prune
  • remove

Additional resources

  • podman-events man page

21.6. Monitoring Podman events

You can monitor and print events that occur in Podman using the podman events command. Each event will include a timestamp, a type, a status, name, if applicable, and image, if applicable.

Prerequisites

  • The container-tools module is installed.

Procedure

  1. Run the myubi container: 

    $ podman run -q --rm --name=myubi registry.access.redhat.com/ubi8/ubi:latest

  2. Display the Podman events:

    • To display all Podman events, enter: 

      $ now=$(date --iso-8601=seconds)
$ podman events --since=now --stream=false
2023-03-08 14:27:20.696167362 +0100 CET container create d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72fe09 (image=registry.access.redhat.com/ubi8/ubi:latest, name=myubi,...)
2023-03-08 14:27:20.652325082 +0100 CET image pull  registry.access.redhat.com/ubi8/ubi:latest
2023-03-08 14:27:20.795695396 +0100 CET container init d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72fe09 (image=registry.access.redhat.com/ubi8/ubi:latest, name=myubi...)
2023-03-08 14:27:20.809205161 +0100 CET container start d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72fe09 (image=registry.access.redhat.com/ubi8/ubi:latest, name=myubi...)
2023-03-08 14:27:20.809903022 +0100 CET container attach d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72fe09 (image=registry.access.redhat.com/ubi8/ubi:latest, name=myubi...)
2023-03-08 14:27:20.831710446 +0100 CET container died d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72fe09 (image=registry.access.redhat.com/ubi8/ubi:latest, name=myubi...)
2023-03-08 14:27:20.913786892 +0100 CET container remove d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72fe09 (image=registry.access.redhat.com/ubi8/ubi:latest, name=myubi...)

      The --stream=false option ensures that the podman events command exits when reading the last known event.

      You can see several events that happened when you enter the podman run command:

      • container create when creating a new container.
      • image pull when pulling an image if the container image is not present in the local storage.
      • container init when initializing the container in the runtime and setting a network.
      • container start when starting the container.
      • container attach when attaching to the terminal of a container. That is because the container runs in the foreground.
      • container died is emitted when the container exits.
      • container remove because the --rm flag was used to remove the container after it exits.

    • You can also use the journalctl command to display Podman events: 

      $ journalctl --user -r SYSLOG_IDENTIFIER=podman
Mar 08 14:27:20 fedora podman[129324]: 2023-03-08 14:27:20.913786892 +0100 CET m=+0.066920979 container remove
...
Mar 08 14:27:20 fedora podman[129289]: 2023-03-08 14:27:20.696167362 +0100 CET m=+0.079089208 container create d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72f>

    • To show only Podman create events, enter: 

      $ podman events --filter event=create
2023-03-08 14:27:20.696167362 +0100 CET container create d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72fe09 (image=registry.access.redhat.com/ubi8/ubi:latest, name=myubi,...)

    • You can also use the journalctl command to display Podman create events: 

      $ journalctl --user -r PODMAN_EVENT=create
Mar 08 14:27:20 fedora podman[129289]: 2023-03-08 14:27:20.696167362 +0100 CET m=+0.079089208 container create d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72f>

Additional resources

21.7. Using Podman events for auditing

Previously, the events had to be connected to an event to interpret them correctly. For example, the container-create event had to be linked with an image-pull event to know which image had been used. The container-create event also did not include all data, for example, the security settings, volumes, mounts, and so on.

Beginning with Podman v4.4, you can gather all relevant information about a container directly from a single event and journald entry. The data is in JSON format, the same as from the podman container inspect command and includes all configuration and security settings of a container. You can configure Podman to attach the container-inspect data for auditing purposes.

Prerequisites

  • The container-tools module is installed.

Procedure

  1. Modify the ~/.config/containers/containers.conf file and add the events_container_create_inspect_data=true option to the [engine] section: 

    $ cat ~/.config/containers/containers.conf
[engine]
events_container_create_inspect_data=true

    For the system-wide configuration, modify the /etc/containers/containers.conf or /usr/share/container/containers.conf file.

  2. Create the container: 

    $ podman create registry.access.redhat.com/ubi8/ubi:latest
19524fe3c145df32d4f0c9af83e7964e4fb79fc4c397c514192d9d7620a36cd3

  3. Display the Podman events:

    • Using the podman events command: 

      $ now=$(date --iso-8601=seconds)
$ podman events --since $now --stream=false --format "{{.ContainerInspectData}}" | jq “.Config.CreateCommand"
[
  "/usr/bin/podman",
  "create",
  "registry.access.redhat.com/ubi8"
]
      • The --format "{{.ContainerInspectData}}" option displays the inspect data.
      • The jq ".Config.CreateCommand" transforms the JSON data into a more readable format and displays the parameters for the podman create command.

    • Using the journalctl command: 

      $ journalctl --user -r PODMAN_EVENT=create --all -o json | jq ".PODMAN_CONTAINER_INSPECT_DATA | fromjson" | jq ".Config.CreateCommand"
[
  "/usr/bin/podman",
  "create",
  "registry.access.redhat.com/ubi8"
]

      The output data for the podman events and journalctl commands are the same.

Additional resources