Red Hat Training

A Red Hat training course is available for Red Hat Enterprise Linux

Chapter 3. Signing Container Images

Signing container images on RHEL systems provides a means of validating where a container image came from, checking that the image has not been tampered with, and setting policies to determine which validated images you will allow to be pulled to your systems. Before you begin, there are a few things you should know about Red Hat container image signing:

  • Docker version: The features described here require at least Docker 1.12.3. So you can use the docker package for any RHEL and RHEL Atomic release after 7.3.2.
  • Red Hat Signed Images: As of RHEL and RHEL Atomic 7.4, image signing is fully supported (no longer tech preview). With RHEL 7.4, Red Hat has also begun signing its own container images. So you can use the instructions provided in this chapter to determine the authenticity of those images using Red Hat GPG keys.

This chapter describes tools and procedures you can use on Red Hat systems to not only sign images, but also consume images that have been signed in these ways:

  • Creating Image Signatures: By signing images with a private key and sharing a public key generated from it, others can use the public key to authenticate the images you share. The signatures needed to validate the images can be made available from an accessible location (like a Web server) in what is referred to as a "signature store" or made available from a directory on the local filesystem. The actual signature can be created from an image stored in a registry or at the time the image is pushed to a container registry.
  • Verifying Signed Images: You can check a signed image when it is pulled from a registry. This includes verifying Red Hat’s signed images.
  • Trusting Images: Besides determining that an image is valid, you can also set policies that say which valid images you trust to use on your system, as well as which registries you trust to use without validation.

For the current release Red Hat Enterprise Linux and RHEL Atomic Host, there are a limited number of tools and container registries that support image signing. Over time, however, you can expect most features on RHEL systems that pull or store images to support signing. To get you started in the mean time, however, you can use the following features in RHEL:

  • Registries: Currently, you can use a local container registry (docker-distribution package) and the Docker Hub (docker.io) from RHEL systems to push and pull signed images. For the time being, image signing features are only supported in v2 (not v1) Docker Registries.
  • Image signing tools: To create image signatures, you can use atomic sign (to create a signature from a stored image) or atomic push (to create an image signature as you push it to a registry).
  • Image verifying tools: To verify a signed image, you can use the atomic trust command to identify which image registries you trust without verification and which registries require verification. Later, when you pull an image, the atomic pull or docker pull command will validate the image as it is pulled to the local system.
  • Operating systems: The image signing and verification features described here are supported in Red Hat Enterprise Linux Server and RHEL Atomic Host systems, version 7.3.1 and later.

For a more complete description of Red Hat container image signing, see:

Container Image Signing Integration Guide

3.1. Getting Container Signing Software

If you are using a RHEL Atomic Host 7.3.2 system (or later), the tools you need to sign, trust and verify images are already included.

Most container-related software for RHEL server is in the rhel-7-server-extras-rpms yum repository. So, on your RHEL 7.3 server system, you should enable that repository, install packages, and start the docker service as follows:

# subscription-manager repos --enable=rhel-7-server-extras-rpms
# yum install skopeo docker atomic

Before you start the docker service, you must enable signature verification. To do that edit the /etc/sysconfig/docker file and change --signature-verification=false to --signature-verification=true on the OPTIONS line. It should appear as follows:

OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=true'

Then, enable and start the docker service:

# systemctl enable docker; systemctl start docker

The docker service should now be running and ready for you to use.

3.2. Creating Image Signatures

Image signing in this section is broken down into the following steps:

  • GPG keys: Create GPG keys for signing images.
  • Sign images: Choose from either creating a signature from an image already in a container registry or creating a signature as you push it to a container registry.

3.2.1. Create GPG Keys

To sign container images on Red Hat systems, you need to have a private GPG key and a public key you create from it. If you don’t already have GPG keys you want to use, you can create them with the gpg2 --gen-key command. This procedure was done from a terminal window on a GNOME desktop as the user who will sign the images:

  1. Create private key: Use the following command to interactively add information needed to create the private key. You can use defaults for most prompts, although you should properly identify your user name, email address, and add a comment. You also must add a passphrase when prompted.

    $ gpg2 --gen-key
    Please select what kind of key you want:
    Your selection? 1
    What keysize do you want? (2048) 2048
    Please specify how long the key should be valid.
             0 = key does not expire
    Key is valid for? (0) 0
    Key does not expire at all
    Is this correct? (y/N) y
    Real name: Joe Smith
    Email address: jjsmith@example.com
    Comment: Image Signing Key
    You selected this USER-ID:
        "Joe Smith (Image Signing Key) <jjsmith@example.com>"
    Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
    You need a Passphrase to protect your secret key.
    gpg: /home/jsmith/.gnupg/trustdb.gpg: trustdb created
    gpg: key D3F46FFF marked as ultimately trusted
    public and secret key created and signed.

    You will need the passphrase later, when you attempt to sign an image. Anyone with access to your private key and passphrase will be able to identify content as belonging to you.

  2. Create entropy: You need to generate activity on your system to help produce random data that will be used to produce your key. This can be done by moving windows around or opening another Terminal window and running a variety of commands, such as ones that write to disk or read lots of data. Once enough entropy is generated, the gpg2 command will exit and show information about the key just created.
  3. Create public key: Here’s an example of creating a public key from that private key:

    $ gpg2 --armor --export --output mysignkey.gpg jjsmith@example.com
  4. List keys: Use the following command to list your keys.

    $ gpg2 --list-keys
    /home/jsmith/.gnupg/pubring.gpg
    -------------------------------
    pub   2048R/D3F46FFF 2016-10-20
    uid                  Joe Smith (Image Signing Key) <jjsmith@example.com>
    sub   2048R/775A4344 2016-10-20

At this point, the private key you created is available to use from this user account to sign images and the public key (mysignkey.gpg) can be shared with others for them to use to validate your images. Keep the private key secure. If someone else has that key, they could use it to sign files as though those files belonged to you.

3.2.2. Creating an Image Signature

With your private key in place, you can begin preparing to sign your images. The steps below show two different ways to sign images:

  • From a repository: You can create a signature for an image that is already in a repository using atomic sign.
  • Image at push time: You can tag a local image and create an image signature as you push it to a registry using atomic push.

Image signing requires super user privileges to run the atomic and docker commands. However, when you sign, you probably want to use your own keys. To take that into account, when you run the atomic command with sudo, it will read keys from your regular user account’s home directory (not the root user’s directory) to do the signing.

3.3. Set up to do Image Signing

If you are going to sign a lot of images on a personal system, you can identify signing information in your /etc/atomic.conf file. Once you add that information to atomic.conf, the atomic command assumes that you want to use that information to sign any image you push or sign. For example, for a user account jjsmith with a default signer of jjsmith@example.com, you could add the following lines to the /etc/atomic.conf file so that all images you push or sign would be signed with that information by default:

default_signer: jjsmith@example.com
gnupg_homedir: /home/jjsmith/.gnupg

If you want to use a different signer or signing home directory, to override those default values, you can do that later on the atomic command line using the --sign-by and --gnupghome options, respectively. For example, to have jjsmith@example.com and /home/jjsmith/.gnupg used as the signer and default gnupg directory, type the following on the atomic command line:

$ sudo atomic push --sign-by jjsmith@example.com   \
                   --gnupghome /home/jjsmith/.gnupg \
                   docker.io/wherever/whatever

3.4. Creating a Signature for an Image in a Repository

You can create an image signature for an image that is already pushed to a registry using the atomic sign command. Use docker search to find the image, then atomic sign to create a signature for that image.

IMPORTANT: The image signature is created using the exact name you enter on the atomic sign command line. When someone verifies that image against the signature later, they must use the exact same name or the image will not be verified.

  1. Find image: Find the image for which you want to create the signature using the docker search command:

    $ sudo docker search docker.io/jjsmith/mybusybox
    INDEX       NAME                          DESCRIPTION   STARS     OFFICIAL   AUTOMATED
    docker.io   docker.io/jjsmith/mybusybox                 0....
  2. Create the image signature: Choose the image you want to sign (jjsmith/mybusybox in this example). To sign it with the default signer and home directory entered in /etc/atomic.conf, type the following:

    $ sudo atomic sign docker.io/jjsmith/mybusybox
    Created: /var/lib/atomic/sigstore/docker.io/jjsmith/mybusybox@sha256:9393222c6789842b16bcf7306b6eb4b486d81a48d3b8b8f206589b5d1d5a6101/signature-1

When you are prompted for a passphrase, enter the passphrase you entered when you created your private key. As noted in the output, you can see that the signature was created and stored in the /var/lib/atomic/sigstore directory on your local system under the registry name, user name, and image name (docker.io/jjsmith/mybusybox*sha256:…​).

3.5. Creating an Image Signature at Push Time

To create an image signature for an image at the time you push it, you can tag it with the identity of the registry and possibly the username you want to be associated with the image. This shows an example of creating an image signature at the point that you push it to the Docker Hub (docker.io). In this case, the procedure relies on the default signer and GNUPG home directory assigned earlier in the /etc/atomic.conf file.

  1. Tag image: Using the image ID of the image, tag it with the identity of the registry to which you want to push it.

    $ sudo docker tag hangman docker.io/jjsmith/hangman:latest
  2. Push and sign the image: The following command creates the signature as the image is pushed to docker.io:

    $ sudo atomic push -t docker docker.io/jjsmith/hangman:latest
    Registry Username: jjsmith
    Registry Password: *****
    Copying blob sha256:5f70bf18...
    Signing manifest
    Writing manifest to image destination
    Storing signatures

    When prompted, enter the passphrase you assigned when you created your private key. At this point, the image should be available from the repository and ready to pull.

3.6. Sharing the Signature Store

The signatures you just created are stored in the /var/lib/atomic/sigstore directory. For the purposes of trying out signatures, you can just use that signature store from the local system. However, when you begin sharing signed images with others and have them validate those images, you would typically share that signature store directory structure from a Web server or other centrally available location. You would also need to share the public key associated with the private key you used to create the signatures.

For this procedure, you could just copy your public key to the /etc/pki/containers directory and use the signature store from the local /var/lib/atomic/sigstore directory. For example:

$ sudo mkdir /etc/pki/containers
$ sudo cp mysignkey.gpg /etc/pki/containers/

As for the location of the signature store, you can assign that location when you run an atomic trust add command (shown later). Or you can edit the /etc/containers/registries.d/default.yaml file directly and identify a value for the sigstore setting (such as, sigstore: file:///var/lib/atomic/sigstore).

3.7. Validating and Trusting Signed Images

Using the atomic trust command, you can identify policies that determine which registries you trust to allow container images to be pulled to your system. To further refine the images you accept, you can set a trust value to accept all images from a registry or accept only signed images from a registry. As part of accepting signed images, you can also identify the location of the keys to use to validate the images.

The following procedure describes how to show and change trust policies related to pulling images to your system with the atomic command.

  1. Check trust values: Run this command to see the current trust value for pulling container images with the atomic command:

    $ sudo atomic trust show
    * (default)                         accept

    When you start out, the trust default allows any request to pull an image to be accepted.

  2. Set default to reject: Having the default be reject might be harsh if you are just trying out containers, but could be considered when you are ready to lock down which registries to allow. So you could leave the default as accept, if you like. To limit pulled images to only accept images from certain registries, you can start by changing the default to reject as follows:

    $ sudo atomic trust default reject
    $ sudo atomic trust show
    * (default)                         reject
    $ sudo atomic pull docker.io/centos
    Pulling docker.io/library/centos:latest ...
    FATA[0000] Source image rejected: Running image docker://centos:latest is rejected by policy.

    You can see that the default is now to reject all requests to pull images, so an attempt to pull a container image fails.

  3. Add trusted registry without signatures: This step shows how to allow to your system to pull images from the docker.io registry without requiring signature checking. You could repeat this step for every registry you want to allow (including your own local registries) for which you don’t want to require signatures. Type the following to allow docker.io images to be pulled, without signature verification:

    $ sudo atomic trust add docker.io --type insecureAcceptAnything
    $ sudo atomic pull docker.io/centos
    Pulling docker.io/library/centos:latest ...
    Copying blob ...
    $

    Notice that you were able to pull the centos image from docker.io after adding that trust line.

  4. Add trusted registry with signatures: This example identifies the Red Hat Registry (registry.access.redhat.com) as being a registry from which the local system will be able to pull signed images:

    sudo atomic trust add registry.access.redhat.com    \
        --pubkeys /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release \
        --sigstore https://access.redhat.com/webassets/docker/content/sigstore \
        --type signedBy
  5. Pull and verify image: At this point, you can verify the trust settings. Run the atomic trust show command, which shows that only signed images from registry.access.redhat.com and any image from docker.io will be accepted. All other images will be rejected.

    $ sudo atomic trust show
    * (default)                         reject
    docker.io                           accept
    registry.access.redhat.com          signed security@redhat.com,security@redhat.com

The trust policies you just set are stored in the /etc/containers/policy.json file. See the Reasonable Locked-Down System example policy.json file for an good, working example of this file. You can add your own policy files to the /etc/containers/registries.d directory. You can name those files anything you like, as long as .yaml is at the end.

3.8. Validating Signed Images from Red Hat

Red Hat signs its container images using its own GPG keys. Using the same public keys Red Hat uses to validate RPM packages, you can validate signed Red Hat container images. The following procedure describes the process of validating signed images from Red Hat.

Refer to the following articles related to validating Red Hat container images:

  • Verifying image signing for Red Hat Container Registry: Describes how to use the atomic command to indicate that you trust images from the Red Hat Registry and check the signatures of images you pull from that registry using the docker service.
  • Image Signatures: Describes how to use image signatures with OpenShift commands and the OpenShift Registry.

Follow these steps to identify how to trust the Red Hat Registry and validate the signatures of images you get from that registry:

  1. Add the Red Hat container registry (registry.access.redhat.com) as a trusted registry. Identify the location of the signature store (--sigstore) that contains the signature for each signed image in the registry and the public keys (--pubkeys) used to validate those signatures against the images.

    $ sudo atomic trust add \
        --pubkeys /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release \
        --sigstore https://access.redhat.com/webassets/docker/content/sigstore \
        registry.access.redhat.com
  2. Pull a container from the trusted registry. If the signature is valid, the image should be pulled with the output looking similar to the following:

    $ sudo atomic pull rhel7/etcd
    Pulling registry.access.redhat.com/rhel7/etcd:latest ...
    Copying blob ...
    ...
    Writing manifest to image destination
    Storing signatures

    If you had identified the wrong signature directory or no signature was found there for the image you wanted to pull, you would see output that looks as follows:

    FATA[0004] Source image rejected: A signature was required, but no signature exists

3.9. Understanding Image Signing Configuration Files

The image signing process illustrated in this chapter resulted in several configuration files being created or modified. Instead using the atomic command to create and modify those files, you could edit those files directly. Here are examples of the files created when you run commands to trust registries.

3.9.1. policy.json file

The atomic trust command modifies settings in the /etc/containers/policy.json file. Here is the content of that file that resulted from changing the default trust policy to reject, accepting all requests to pull images from the registry.access.redhat.com registry without verifying signatures, and accepting requests to pull images from the jjones account in the docker.io registry that are signed by GPGKeys in the /etc/pki/containers/mysignkey.gpg file:

    "default": [
        {
            "type": "reject"
        }
    ],
    "transports": {
        "docker": {
            "registry.access.redhat.com": [
                {
                    "type": "insecureAcceptAnything"
                }
            ],
            "docker.io/jjones": [
                {
                    "keyType": "GPGKeys",
                    "type": "signedBy",
                    "keyPath": "/etc/pki/containers/mysignkey.gpg"
                }
            ]
        }
    }
}

3.9.2. whatever.yaml

Settings added from the atomic trust add command line when adding trust settings for a registry are stored in a new file that is created in the /etc/containers/registries.d/ directory. The file name includes the registry’s name and ends in the .yaml suffix. For example, if you were adding trust settings for the user jjones at docker.io (docker.io/jjones), the file that stores the settings is /etc/containers/registries.d/docker.io-jjones.yaml. The command line could include the location of the signature store:

docker:
  docker.io/jjones:
    sigstore: file:///var/lib/atomic/sigstore/

The settings in the docker.io-jjones.yaml file override default setting on your system. Default signing settings are stored in the /etc/containers/registries.d/default.yaml file. For more information of the format of the registries.d files, see Registries Configuration Directory.