Red Hat Training

A Red Hat training course is available for RHEL 8

Chapter 8. Signing container images

You can use a GNU Privacy Guard (GPG) signature or a sigstore signature to sign your container image. Both signing techniques are generally compatible with any OCI compliant container registries. You can use Podman to sign the image before pushing it into a remote registry and configure consumers so that any unsigned image is rejected. Signing container images helps to prevent supply chain attacks.

Signing using GPG keys requires deploying a separate lookaside server to distribute signatures. The lookaside server can be any HTTP server. Starting with Podman version 4.2, you can use the sigstore format of container signatures. Compared to the GPG keys, the separate lookaside server is not required because the sigstore signatures are stored in the container registry.

8.1. Signing container images with GPG signatures

You can sign images using a GNU Privacy Guard (GPG) key.

Prerequisites

  • The container-tools module is installed.
  • The GPG tool is installed.

  • The lookaside web server is set up and you can publish files on it.

    • You can check the system-wide registries configuration in the /etc/containers/registries.d/default.yaml file. The lookaside-staging option references a file path for signature writing and is typically set on hosts publishing signatures. 

      # cat /etc/containers/registries.d/default.yaml
docker:
    <registry>:
        lookaside: https://registry-lookaside.example.com
        lookaside-staging: file:///var/lib/containers/sigstore
...

Procedure

  1. Generate a GPG key: 

    # gpg --full-gen-key

  2. Export the public key: 

    # gpg --output <path>/key.gpg --armor --export <username@domain.com>

  3. Build the container image using Containerfile in the current directory: 

    $ podman build -t <registry>/<namespace>/<image>

    Replace <registry>, <namespace>, and <image> with the container image identifiers. For more details, see Container registries.

  4. Sign the image and push it to the registry: 

     $  podman push \
    --sign-by <username@domain.com> \
    <registry>/<namespace>/<image>
    Note

    If you need to sign existing images while moving them across container registries, you can use the skopeo copy command.

  5. Optional: Display the new image signature: 

    # (cd /var/lib/containers/sigstore/; find . -type f)
./<image>@sha256=<digest>/signature-1

  6. Copy your local signatures to the lookaside web server: 

    # rsync -a /var/lib/containers/sigstore <user@registry-lookaside.example.com>:/registry-lookaside/webroot/sigstore

The signatures are stored in the location determined by the lookaside-staging option, in this case, /var/lib/containers/sigstore directory.

Verification

Additional resources

8.2. Verifying GPG image signatures

You can verify that a container image is correctly signed with a GPG key using the following procedure.

Prerequisites

  • The container-tools module is installed.

  • The web server for a signature reading is set up and you can publish files on it.

    • You can check the system-wide registries configuration in the /etc/containers/registries.d/default.yaml file. The lookaside option references a web server for signature reading. The lookaside option has to be set for verifying signatures. 

      # cat /etc/containers/registries.d/default.yaml
docker:
    <registry>:
        lookaside: https://registry-lookaside.example.com
        lookaside-staging: file:///var/lib/containers/sigstore
...

Procedure

  1. Update a trust scope for the <registry>: 

    $ podman image trust set -f <path>/key.gpg <registry>/<namespace>

  2. Optional: Verify the trust policy configuration by displaying the /etc/containers/policy.json file: 

    $ cat /etc/containers/policy.json
{
  ...
  "transports": {
    "docker": {
      "<registry>/<namespace>": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "<path>/key.gpg"
        }
      ]
    }
  }
}
    Note

    Typically, the /etc/containers.policy.json file is configured at a level of organization where the same keys are used. For example, <registry>/<namespace> for a public registry, or just a <registry> for a single-company dedicated registry.

  3. Pull the image: 

    # podman pull <registry>/<namespace>/<image>
...
Storing signatures
e7d92cdc71feacf90708cb59182d0df1b911f8ae022d29e8e95d75ca6a99776a

    The podman pull command enforces signature presence as configured, no extra options are required.

Note

You can edit the system-wide registry configuration in the /etc/containers/registries.d/default.yaml file. You can also edit the registry or repository configuration section in any YAML file in the /etc/containers/registries.d directory. All YAML files are read and the filename can be arbitrary. A single scope (default-docker, registry, or namespace) can only exist in one file within the /etc/containers/registries.d directory.

Important

The system-wide registries configuration in the /etc/containers/registries.d/default.yaml file allows accessing the published signatures. The sigstore and sigstore-staging options are now deprecated. These options refer to signing storage, and they are not connected to the sigstore signature format. Use the new equivalent lookaside and lookaside-staging options instead.

Additional resources

  • podman-image-trust man page
  • podman-pull man page

8.3. Signing container images with sigstore signatures using a private key

Starting with Podman version 4.2, you can use the sigstore format of container signatures.

Prerequisites

  • The container-tools module is installed.

Procedure

  1. Generate a sigstore public/private key pair: 

    $ skopeo generate-sigstore-key --output-prefix myKey

    • The public and private keys myKey.pub and myKey.private are generated.

      Note

      The skopeo generate-sigstore-key command is available from RHEL 8.8. Otherwise, you must use the upstream Cosign project to generate public/private key pair:

      • Install the cosign tool: 

        $ git clone -b v2.0.0 https://github.com/sigstore/cosign
$ cd cosign
$ make ./cosign

      • Generate a public/private key pair: 

        $ ./cosign generate-key-pair
...
Private key written to cosign.key
Public key written to cosign.pub

  2. Add the following content to the /etc/containers/registries.d/default.yaml file: 

    docker:
    <registry>:
        use-sigstore-attachments: true

    By setting the use-sigstore-attachments option, Podman and Skopeo can read and write the container sigstore signatures together with the image and save them in the same repository as the signed image.

    Note

    You can edit the system-wide registry configuration in the /etc/containers/registries.d/default.yaml file. You can also edit the registry or repository configuration section in any YAML file in the /etc/containers/registries.d directory. All YAML files are read and the filename can be arbitrary. A single scope (default-docker, registry, or namespace) can only exist in one file within the /etc/containers/registries.d directory.

  3. Build the container image using Containerfile in the current directory: 

    $ podman build -t <registry>/<namespace>/<image>

  4. Sign the image and push it to the registry: 

    $ podman push --sign-by-sigstore-private-key ./myKey.private <registry>/<namespace>/image>

    The podman push command pushes the <registry>/<namespace>/<image> local image to the remote registry as <registry>/<namespace>/<image>. The --sign-by-sigstore-private-key option adds a sigstore signature using the myKey.private private key to the <registry>/<namespace>/<image> image. The image and the sigstore signature are uploaded to the remote registry.

Note

If you need to sign existing images while moving them across container registries, you can use the skopeo copy command.

Verification

Additional resources

8.4. Verifying sigstore image signatures using a public key

You can verify that a container image is correctly signed using the following procedure.

Prerequisites

  • The container-tools module is installed.

Procedure

  1. Add the following content to the /etc/containers/registries.d/default.yaml file: 

    docker:
    <registry>:
        use-sigstore-attachments: true

    By setting the use-sigstore-attachments option, Podman and Skopeo can read and write the container sigstore signatures together with the image and save them in the same repository as the signed image.

    Note

    You can edit the system-wide registry configuration in the /etc/containers/registries.d/default.yaml file. You can also edit the registry or repository configuration section in any YAML file in the /etc/containers/registries.d directory. All YAML files are read and the filename can be arbitrary. A single scope (default-docker, registry, or namespace) can only exist in one file within the /etc/containers/registries.d directory.

  2. Edit the /etc/containers/policy.json file to enforce sigstore signature presence: 

    ...
"transports": {
            "docker": {
                "<registry>/<namespace>": [
                    {
                        "type": "sigstoreSigned",
                        "keyPath": "/some/path/to/cosign.pub"
                    }
                ]
            }
        }
...

    By modifying the /etc/containers/policy.json configuration file, you change the trust policy configuration. Podman, Buildah, and Skopeo enforce the existence of the container image signatures.

  3. Pull the image: 

    $ podman pull <registry>/<namespace>/<image>

The podman pull command enforces signature presence as configured, no extra options are required.

Additional resources

8.5. Signing container images with sigstore signatures using Fulcio and Rekor

With Fulcio and Rekor servers, you can now create signatures by using short-term certificates based on an OpenID Connect (OIDC) server authentication, instead of manually managing a private key.

Prerequisites

  • The container-tools module is installed.
  • You have Fulcio (https://<your-fulcio-server>) and Rekor (https://<your-rekor-server>) servers running and configured.
  • You have Podman v4.4 or higher installed.

Procedure

  1. Add the following content to the /etc/containers/registries.conf.d/default.yaml file: 

    docker:
    <registry>:
        use-sigstore-attachments: true

    • By setting the use-sigstore-attachments option, Podman and Skopeo can read and write the container sigstore signatures together with the image and save them in the same repository as the signed image.

      Note

      You can edit the registry or repository configuration section in any YAML file in the /etc/containers/registries.d directory. A single scope (default-docker, registry, or namespace) can only exist in one file within the /etc/containers/registries.d directory. You can also edit the system-wide registry configuration in the /etc/containers/registries.d/default.yaml file. Please note that all YAML files are read and the filename is arbitrary.

  2. Create the file.yml file: 

    fulcio:
  fulcioURL: "https://<your-fulcio-server>"
  oidcMode: "interactive"
  oidcIssuerURL: "https://<your-OIDC-provider>"
  oidcClientID: "sigstore"
rekorURL: "https://<your-rekor-server>"
    • The file.yml is the sigstore signing parameter YAML file used to store options required to create sigstore signatures.

  3. Sign the image and push it to the registry: 

    $ podman push --sign-by-sigstore=file.yml <registry>/<namespace>/<image>
    • You can alternatively use the skopeo copy command with similar --sign-by-sigstore options to sign existing images while moving them across container registries.
Warning

Note that your submission for public servers includes data about the public key and certificate, metadata about the signature.

Verification

Additional resources

  • containers-sigstore-signing-params.yaml man page
  • podman-push man page
  • container-registries.d man page

8.6. Verifying container images with sigstore signatures using Fulcio and Rekor

You can verify image signatures by adding the Fulcio and Rekor-related information to the policy.json file. Verifying container images signatures ensures that the images come from a trusted source and has not been tampered or modified.

Prerequisites

  • The container-tools module is installed.

Procedure

  1. Add the following content to the /etc/containers/registries.conf.d/default.yaml file: 

    docker:
    <registry>:
        use-sigstore-attachments: true

    • By setting the use-sigstore-attachments option, Podman and Skopeo can read and write the container sigstore signatures together with the image and save them in the same repository as the signed image.

      Note

      You can edit the registry or repository configuration section in any YAML file in the /etc/containers/registries.d directory. A single scope (default-docker, registry, or namespace) can only exist in one file within the /etc/containers/registries.d directory. You can also edit the system-wide registry configuration in the /etc/containers/registries.d/default.yaml file. Please note that all YAML files are read and the filename is arbitrary.

  2. Add the fulcio section and the rekorPublicKeyPath or rekorPublicKeyData fields in the /etc/containers/policy.json file: 

    {
    ...
    "transports": {
	"docker": {
        "<registry>/<namespace>": [
            {
            "type": "sigstoreSigned",
            "fulcio": {
                "caPath": "/path/to/local/CA/file",
                "oidcIssuer": "https://expected.OIDC.issuer/",
                "subjectEmail", "expected-signing-user@example.com",
            },
            "rekorPublicKeyPath": "/path/to/local/public/key/file",
            }
        ]
    ...
	}
  }
  ...
}
    • The fulcio section provides that the signature is based on a Fulcio-issued certificate.
    • You have to specify one of caPath and caData fields, containing the CA certificate of the Fulcio instance.
    • Both oidcIssuer and subjectEmail are mandatory, exactly specifying the expected identity provider, and the identity of the user obtaining the Fulcio certificate.
    • You have to specify one of rekorPublicKeyPath and rekorPublicKeyData fields.

  3. Pull the image: 

    $ podman pull <registry>/<namespace>/<image>

The podman pull command enforces signature presence as configured, no extra options are required.

Additional resources

  • policy.json man page
  • container-registries.d man page

8.7. Signing container images with sigstore signatures with a private key and Rekor

Starting with Podman version 4.4, you can use the sigstore format of container signatures together with Rekor servers. You can also upload public signatures to the public rekor.sigstore.dev server, which increases the interoperability with Cosign. You can then use the cosign verify command to verify your signatures without having to explicitly disable Rekor.

Prerequisites

  • The container-tools module is installed.

Procedure

  1. Generate a sigstore public/private key pair: 

    $ skopeo generate-sigstore-key --output-prefix myKey
    • The public and private keys myKey.pub and myKey.private are generated.

  2. Add the following content to the /etc/containers/registries.conf.d/default.yaml file: 

    docker:
    <registry>:
        use-sigstore-attachments: true

    • By setting the use-sigstore-attachments option, Podman and Skopeo can read and write the container sigstore signatures together with the image and save them in the same repository as the signed image.

      Note

      You can edit the registry or repository configuration section in any YAML file in the /etc/containers/registries.d directory. A single scope (default-docker, registry, or namespace) can only exist in one file within the /etc/containers/registries.d directory. You can also edit the system-wide registry configuration in the /etc/containers/registries.d/default.yaml file. Please note that all YAML files are read and the filename is arbitrary.

  3. Build the container image using Containerfile in the current directory: 

    $ podman build -t <registry>/<namespace>/<image>

  4. Create the file.yml file: 

    privateKeyFile: "/home/user/sigstore/myKey.private"
privateKeyPassphraseFile: "/mnt/user/sigstore-myKey-passphrase"
rekorURL: "https://<your-rekor-server>"
    • The file.yml is the sigstore signing parameter YAML file used to store options required to create sigstore signatures.

  5. Sign the image and push it to the registry: 

    $ podman push --sign-by-sigstore=file.yml <registry>/<namespace>/<image>
    • You can alternatively use the skopeo copy command with similar --sign-by-sigstore options to sign existing images while moving them across container registries.
Warning

Note that your submission for public servers includes data about the public key and metadata about the signature.

Verification

  • Use one of the following methods to verify that a container image is correctly signed:

    • Use the cosign verify command: 

      $ cosign verify <registry>/<namespace>/<image> --key myKey.pub

  • Use the podman pull command:

    • Add the rekorPublicKeyPath or rekorPublicKeyData fields in the /etc/containers/policy.json file: 

      {
  ...
  "transports": {
	"docker": {
        "<registry>/<namespace>": [
            {
                "type": "sigstoreSigned",
                "rekorPublicKeyPath": "/path/to/local/public/key/file",
            }
        ]
	...
	}
  }
  ...
}

    • Pull the image: 

      $ podman pull <registry>/<namespace>/<image>
      • The podman pull command enforces signature presence as configured, no extra options are required.

Additional resources