Container Image Signing Integration Guide

Updated -

Red Hat container image signing provides a path to ensure authorship, integrity and non-repudiation. This article describes the technology architecture and how to integrate signing into existing workflows.

This document is provided to help gain an understanding of Red Hat’s approach to container image signing. It provides some guidance on the leadership role Red Hat is taking in this technology. It may be used as a point of reference for integrating with the image signing solution.

Audience

This reference document is primarily intended to address the needs of:

  1. Organizations that want to sign and distribute container images, including operating system vendors and independent software vendors.
  2. Tooling developers implementing signing and/or serving image signatures, including enterprise infrastructure and management tools.
  3. Red Hat customers implementing secure workflows to cryptographically ensure provenance and integrity.

Capabilities

There are two primary capabilities in the signing technology Red Hat is introducing:

  1. Ability to generate detached signatures for container images, accounting for different registry implementations.
  2. Ability to manage host system trust policy describing signature requirements when images are pulled.

The atomic command line interface (CLI), version 1.12.5 or greater, provides a reference implementation of these capabilities using the following high-level commands:

$ atomic push [--sign-by ...] IMAGE
$ atomic sign [--sign-by ...] IMAGE
$ atomic trust show|add|delete|default ...

In addition, initial support has been added to version 3.3 of the OpenShift integrated registry. This includes ability to store signatures via REST API. Refer to Working with OpenShift Registry for a more detailed discussion.

Image Provenance and Ancestry

The goals of any digital signature scheme include:

  • authenticating authorship
  • non-repudiation
  • ensuring contents have not been altered in transit (integrity)

The reference implementation satisfies these goals for a given image. Due to the layering capabilities of container images these goals can be extended to image layers. For example, if a base image from a software vendor is used to build an image by another software vendor, the end user may want to validate authorship, non-repudiation and integrity of not just the resulting image, but also of the base image. The design of this signature scheme allows for this possibility, but the documented implementation does not currently provide this capability.

Digital signatures are a critical component to any comprehensive governance, compliance and risk (GCR) model. It is understood that none of these claims validate the functionality of the image contents over time. In other words, digital signatures do not change when a vulnerability is identified in a software package years later.

Architecture

Overview

The container image signing solution was designed for simplicity and flexibility, allowing for integration and re-implementation.

signing architecture diagram

Image signing is a process of creating a signature file that includes the image manifest digest. This file is encrypted with the signer’s private key. The file (or contents of the file) is typically pushed to a signature content server.

Image verification is a process of decrypting the signature file with the signer’s public key and comparing the manifest digest of the image with the decrypted signature file. Trust policy determines requirements for client verification, associating public key(s) with a registry or image scope. See below for a discussion on scope.

The signature content server may be a static web server with the specified file layout or a REST API.

With this design, the registry implementation is independent. This places a higher burden on client configuration but allows for a very secure and flexible architecture.

The shared library containers/image provides a common framework for the signature approach.

Generating Signatures

The signature is an encrypted JSON data structure. The file must be compatible with the signature specification. The JSON is encrypted using a private key resulting in binary data that may be stored as a file or in a REST API.

Storing and Serving Signatures

Signatures are referenced by transport type. This provides a flexible way to reference remote and local signatures and associated container formats. The library defines the following transport types:

dir

Signatures in a local filesystem.

docker

Signatures associated with images in a registry implementing the Docker Registry HTTP API V2. These are served via a web server.

atomic

Signatures stored as objects in the OpenShift API.

oci

Images in directories compliant with Open Container Image (OCI) Layout Specification.

Client Configuration

Scope

Image scope defines the level which trust is evaluated. Clients are configured with a scope associated with a signature requirement. Scope is a registry/image reference for a particular transport type. Scope may be very general (registry) or very specific (a particular image in a registry).

For example, a scope may be defined that all images from a particular registry must be verified with a certain public key. Or a wider scope may be defined for all images, regardless of registry. A more narrow scope may be defined for a particular repository or specific image.

See registry scope and search order documentation.

Signature Server

Clients must be configured with signature server endpoints. Configuration is a set of simple YAML objects identifying where signatures are stored. Each configuration item has a scope. See signature server configuration documentation for details.

Trust Policy

Trust policy provides a way to define signature rules for a scope. Policy maps a scope with one or more public keys. The policy allows for defaults, both global and for each transport type. It is defined as a JSON object. See trust policy documentation.

Signing Components and Tools

The following components are provided as implementation reference.

Atomic

The atomic CLI (python, source) is the primary workstation interface for Red Hat Enterprise Linux, including Atomic Host. Atomic wraps the skopeo CLI. Skopeo is not designed to be used directly.

OpenShift

The OpenShift API (documentation) provides a method for reading and writing image signatures. OpenShift provides an integrated registry, a.k.a. Atomic Registry. See Working With OpenShift Registry section for more details.

Security Issues and Practices

Image Identification and Reference

The image manifest digest is used as the cryptographic reference for image signatures. Container image manifests are complex. An image manifest is not created when an image is built. It is an artifact of the registry. Manifests are are not an ideal reference for local images because some clients do not retain them locally for future verification. There are two manifest versions: schema 1 and schema 2. See the registry compatibility documentation. Docker version 1.10 is the earliest docker client to support schema 2.

Under some conditions an image manifest digest may change. The manifest references the compressed hash. The docker client is designed to retain the uncompressed version of the layers. When compressing and re-compressing image layers it is possible the tar implementation used could result in a different hash value. Additionally, using schema version 1 introduces additional unreliability when tagging and/or pushing to a registry. This could result in false negatives where an image signature does not validate even though the content of the image is the same.

Recommendations

  • Use schema version 2. This is required for atomic v1.12.5, with schema version 1 support added in later releases. Schema support is determined by the registry and client. If the docker client supports schema version 2 but the registry is not configured to accept schema version 2, the client will push a manifest that is schema version 1. The OpenShift integrated Atomic Registry can be configured for schema version. See Working with Openshift Registry.
  • Minimize re-compression with unnecessary push and pull actions. Each time an image is pulled, tagged and pushed to another registry it introduces the possibility that the manifest digest may change, invalidating the signature.
  • Keep tooling and operating systems in sync. It is possible that different implementations of tar or gzip could result in different image hash. This is unexpected but possible.
  • Do not rebuild images. Rebuilding an image will most likely result in a different image hash.

Introducing Image Signing in Stages

Introducing signature content and client trust must be planned. There are low-risk steps that can be taken to stage signatures and client policy because signatures may be delivered without clients validating against them. In other words, until a client enforces security policy, signatures will be ignored. This is similar to the behavior when an RPM repo file is configured with gpgcheck=0.

Consider the following steps as a guideline for introducing image signing:

  1. Implement a process to sign all new images. This typically requires integrating signing into automated build workflows.
  2. Bulk sign existing images. The atomic sign reference command may be used for this. Custom scripting may be required to perform this task in bulk.
  3. Introduce a signedBy client policy for registries or repositories under your control, with an accept default for everything else.
  4. When all sources are signed, lock down the default with a reject policy rule. For example, using atomic CLI, atomic trust default reject.

Automating Cluster Configuration

Secure environments typically use automation to ensure proper configuration. Tools such as Ansible may be used to control the system policy, sigstore configuration and trusted keys. The following procedure may be used as a guide to automating client configuration.

  1. Login to an interactive shell on a workstation that matches the target environment.
  2. Download trusted public keys using secure transport methods. It is recommended to place these in /etc/pki/containers.
  3. Generate the trust configuration using a CLI tool. For example, using the reference atomic CLI:
# atomic trust add registry.example.com \
         --sigstore signatures.example.com/sigs/ \
         --pubkeys /etc/pki/containers/key.pub
# atomic trust default reject
  1. Generate automation script that distributes the following files and directories to the target systems:
  • /etc/pki/containers/
  • /etc/containers/policy.json
  • /etc/containers/registries.d/

If the workstation has ansible installed the following Ansible playbook snippet may be referenced as an example to get started. Run this playbook from a workstation or part of system configuration automation.

- hosts: atomic-cluster
  tasks:
  - name: Create /etc/pki/containers directory
    file: path=/etc/pki/containers state=directory
  - name: Create /etc/containers/registries.d directory
    file: path=/etc/containers/registries.d state=directory
  - name: Copy trusted public keys
          copy: src=/etc/pki/containers/ dest=/etc/pki/containers
  - name: Copy container trust policy
    copy: src=/etc/containers/policy.json
          dest=/etc/containers/policy.json
  - name: Copy signature server configuration files
    copy: src=/etc/containers/registries.d/
          dest=/etc/containers/registries.d/

Key Management

Public keys are distributed openly. However, it is critical that keys are installed from trusted sources. For example, Red Hat public keys are pre-installed on a system using RPM, which can be verified. See Product Signing (GPG) Keys for more information. Public keys may be manually verified by comparing the published fingerprint with the installed key’s fingerprint. Public keys should be transported within a secure TLS connection. They should come from from well-known locations with systems that employ secure DNS practices.

Private keys must not be shared. It is strongly recommended to protect private keys with a passphrase. The gpg-agent CLI provides a caching mechanism for passphrases. This may provide a secure way to use a passphrase-protected key in automated environments. See the gpg-agent manual page (man gpg-agent) for more details. For comprehensive documentation on gnupg, use the info formatted documentation (info gnupg). For enterprise environments with strict security requirements consider securing the signing server with a hardware security module (HSM).

See the Red Hat Security Guide for securely creating and managing private keys.

Reference Implementation

In this section we provide step-by-step instructions to try out the signing technology:

  1. Create a GPG key pair
  2. Sign an image while pushing it to a registry
  3. Publishing signatures
  4. Configure system trust
  5. Pull an image securely

Signing Images

Create a GPG Key Pair

Before signing images your system must have at least one GPG key pair. GPG signing keys are referenced using the system keyring. If necessary create a GPG key pair. There are several ways to manage GPG keys. Here we use the gpg2 command line tool.

$ gpg2 --gen-key

To publish your public key, save as a text file and upload the output file to a web server

$ gpg2 --armor --export --output /path/to/mykey.gpg mykey@example.com

See managing GPG keys for more details.

In cases where a dedicated server is used for signing, you may need to copy your private signing key to another server using the following steps as a reference.

  1. Export your private key from your workstation.

    $ gpg2 --export-secret-keys -a mykey@example.com > privatekey.asc
    
  2. Copy the .asc file to the signing server. On the signing server use the --import command to import the private key.

    $ gpg2 --import privatekey.asc
    

Push and Create a Signature

Offline signing involves creating a local signature file with the purpose of being served from a web server or "sigstore". Files will appear in their respective directories as determined by their configuration in /etc/containers/registries.d/default.yaml. Image names should be fully-qualified, i.e. in the form of registry/repository/image and include a tag.

Here we push and sign a local image using our GPG keyring. Replace MYIMAGE with a local image on the workstation:

$ sudo atomic push \
       --sign-by mykey@example.com \
      MYIMAGE:latest

Add a default signer and gnupg home directory to /etc/atomic.conf to automatically sign without using --sign-by with each command. The .gnupg directory is typically in the user’s home directory. When signing as the root user the path would typically be /root/.gnupg.

default_signer: [mykey@example.com](mailto:mykey@example.com)
gnupg_homedir: /home/USERNAME/.gnupg

With this configuration signing happens automatically with this simplified command:

$ sudo atomic push MYIMAGE:latest

To disable automatic signing for a single command, pass in an empty string to the argument:

$ sudo atomic push --sign-by="" MYIMAGE:latest

Publishing Signatures

Static Web Server

Image signatures may be served by a web server. In this example we run the web server on the signing server using a container, serving all the signatures on the machine. Alternatively, these files may be part of a file share or copied via rsync from the signing server to the web server.

  1. Run the Red Hat Apache Web Server container. Bind mount the default signature staging directory so all signatures are served.

    $ sudo docker run -d \
      -v /var/lib/atomic/sigstore/:/var/www/html/signatures:Z \
      -p 80:80 \
      --name sigstore \
      registry.access.redhat.com/rhscl/httpd-24-rhel7
    
  2. Configure a client machine to associate the registry with the sigstore

    $ sudo atomic trust add localhost:5000 \
      --sigstore http://localhost/signatures \
      --pubkeys /etc/pki/containers/pubkey
    
  3. Configure the default trust policy to reject all images so we can be sure this works

    $ sudo atomic trust default reject
    
  4. Show trust policy

    $ sudo atomic trust show
    
  5. Pull the image to test the configuration

    $ sudo atomic pull localhost:5000/MYIMAGE
    

A successful pull means the signature was validated.

AWS S3

Amazon Simple Storage Service (S3) is a convenient cloud service for distributing image signatures. Signatures are typically not protected content so web servers may be configured as world readable.

Create a bucket for signatures and make it world readable using a bucket policy. See example granting read-only permission to an anonymous user.

Install the AWS client. Obtain your credentials from the AWS web console and authenticate the aws client interactively using the configure command:

$ aws configure

Use this command to publish all signatures from the default staging directory. This command must be run each time a new signature is created.

$ aws s3 cp /var/lib/atomic/sigstore/ s3://BUCKETNAME/ --recursive

The sigstore URL for this configuration is https://s3.amazonaws.com/BUCKETNAME. This is the value to publish to users.

Serving Automated Trust Configuration Metadata

Serving trust discovery information involves putting the metadata needed for the atomic trust add command into a Dockerfile metadata labels. The community labels repository provides a description of the required values. This image is built, tagged and pushed to the corresponding registry.

You will need the sigstore URL[:PORT]/PATH of the signature server and the public key ID in the form of an email address. The public key fingerprint can be found using command gpg2 --fingerprint. The public key must be exported and served on a web server.

This workflow assumes the following:

  1. Create a Dockerfile using this Dockerfile as a reference. The GPG fingerprint can be retrieved using command gpg2 --fingerprint KEY.
  2. Build the container image
  3. Tag the image named "REGISTRY/REPOSITORY/sigstore:latest". For example, given a signature server for images in “registry.example.com” for the “production” repository, tag the image registry.example.com/production/sigstore:latest
  4. Push this image to the registry

This enables clients to be auto-configured for the required trust settings. See discussion on Automating Client Trust Configuration for an example workflow.

Pulling Images Securely

Managing Trust Policy

Trust policy determines what public keys you trust for a registry or registry/repository. Policy also defines what the default behavior is.

Manually Adding Trust Policy
  1. Download a public key from a trusted source.

    $ sudo curl --create-dirs -o /etc/pki/containers/aweiteka \
      https://s3.amazonaws.com/aweiteka-sigstore/aweiteka.gpg
    
  2. Add a policy. You will need to know the URL of the signature server (aka "sigstore")

    $ sudo atomic trust add docker.io/aweiteka \
      --sigstore https://s3.amazonaws.com/aweiteka-sigstore \
      --pubkeys /etc/pki/containers/aweiteka
    
  3. View the added policy

    $ sudo atomic trust show
    
Updating Existing Trust Policy
  1. View the current policy

    $ sudo atomic trust show
    
  2. Update the policy using the "add" command. Here we add another required public key. The sigstore URL is not altered. You will be prompted to overwrite the existing policy.

    $ sudo atomic trust add docker.io/aweiteka \
      --pubkeys /etc/pki/containers/aweiteka
    
Modifying Default Policy

The default trust policy determines behavior for images that do not have a specific policy. Here we "lock down" the environment to reject any unsigned images that are not specified in policy scope.

  1. Reject all unsigned images by default

    $ sudo atomic trust default reject
    
  2. View the policy

    $ sudo atomic trust show
    
About Trusted Public Keys

Trusted public keys for signature verification are not read from the user’s GPG keyring. They must be installed on the system, referenced in /etc/containers/policy.json as files in /etc/pki/containers/ using keyPath or directly included inline in the policy file using keyData. See policy documentation for details.

Automating Client Trust Configuration

While the most secure workflow is to explicitly add trust policy, configuration may be automated. Trust discovery searches for an image named "sigstore:latest" that has metadata labels defining the trust configuration.

  1. Pull the image, verifying the prompted information

    $ sudo atomic pull docker.io/aweiteka/hello-world
    
  2. View the updated policy

    $ sudo atomic trust show
    
  3. Automated trust discovery may be disabled in /etc/atomic.conf:

    discover_sigstores: false
    

See above for details on serving this configuration.

Working With OpenShift and Atomic Registry

OpenShift Container Platform provides an integrated container registry, a.k.a. Atomic Registry. OpenShift adds many capabilities to the registry, including authentication, authorization, quotas and other enterprise management features. OpenShift version 3.3 introduces basic support for image signatures through storing image signatures via REST API. This makes signatures easier to work with by automating serving signatures. There is no need to publish locally generated signature files. The signatures are stored and served by API.

With OpenShift version 3.3 images must be signed with external tooling using the atomic CLI.

Configuration

Due to the emerging nature of signing capabilities, some cluster administrator steps are required to enable support for image signatures.

NOTE: The registry must have same URL as openshift master API because there is no client configuration support for the atomic transport type.

  1. Install oc client. The OpenShift oc client must be installed to authenticate to the master server. Refer to the installation steps from OpenShift documentation.

  2. Login. Refer to the login steps from OpenShift documentation. A session token may be found in the OpenShift console or registry console.

    $ oc login --token TOKEN MASTER_URL:8443
    
  3. Add role. Signing images requires a cluster-scoped policy role. The system:image-signer cluster role may be assigned to a specific user or a group. Choose one of the following commands.

    $ oadm policy add-cluster-role-to-user system:image-signer USER
    $ oadm policy add-cluster-role-to-group system:image-signer GROUP
    
  4. Configure registry to accept schema version 2. It is recommended that registries be configured for schema version 2. This could break older docker clients but is considered a more robust manifest format. The OpenShift registry may be configured to accept schema version 2. Refer to registry documentation for configuration steps.

Push and Sign a Local Image

  1. Authenticate against both the registry and the OpenShift API

    $ sudo docker login -p TOKEN -u unused -e unused REGISTRY_URL
    $ sudo oc login --token TOKEN MASTER_URL:8443
    
  2. Create a project using the web console or oc client. This will be used as the registry repository name.

    $ oc new-project myproject
    
  3. Tag a local image on the workstation with the format registry/project/image

    $ docker tag myimage registry.example.com/myproject/myimage
    
  4. Push and sign an image on the local workstation

    $ sudo atomic push \
      --type atomic \
      --sign-by mykey@example.com \
      registry.example.com/myproject/myimage:latest
    

Configuration Reference

The following summarizes related configuration files and directories.

Atomic CLI general configuration file

/etc/atomic.conf

Trust policy file

/etc/containers/policy.json

Offline signature storage default path

/var/lib/atomic/sigstore

Recommended Trusted Public Keys Directory

/etc/pki/containers/

Signature Server Configuration

Signature file storage is configured arbitrary list of YAML files. Each file must have a top level transport type key, e.g. "docker", with keys for each registry.

  • sigstore: a read-only local file path or remote web server
  • sigstore-staging: a local path to write offline or local signatures

See documentation for additional details.

Directory

/etc/containers/registries.d/*.yaml

Example file:

docker:
  registry.example.com:
    sigstore: https://server.example.com
    sigstore-staging: file:///path/to/local/dir/

Was this helpful?

We appreciate your feedback. Leave a comment if you would like to provide more detail.
It looks like we have some work to do. Leave a comment to let us know how we could improve.