Verifying Red Hat container image signatures in OpenShift Container Platform 4

Updated -

Red Hat delivers signatures for the images in the Red Hat Container Registries. Those signatures can be automatically verified when being pulled to OpenShift Container Platform (OCP) 4 clusters using Machine Config.

Most of the images that make OCP 4 are shipped from quay.io, and only the release image is signed. The release images refers to the rest of the OCP 4 images by digest making supply chain attacks harder to pull off. However some extensions to OCP 4 such as Logging, Monitoring, and Service Mesh are shipped as operators from the Operator Lifecycle Manager. Those images ship from the Red Hat Container Registry (registry.redhat.io). In order to verify the integrity of those images between Red Hat registries and your infrastructure it's a good idea to enable signature verification by following the instructions that follow.

Enabling Signature Verification for Red Hat Container Registries

Here are the commands to enable signature verification on OCP 4:

  • Create the files which link the registry URLs to the sigstore, and specifies the key to verify the images with:
cat > policy.json <<EOF
{
  "default": [
    {
      "type": "insecureAcceptAnything"
    }
  ],
  "transports": {
    "docker": {
      "registry.access.redhat.com": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ],
      "registry.redhat.io": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ]
    },
    "docker-daemon": {
      "": [
        {
          "type": "insecureAcceptAnything"
        }
      ]
    }
  }
}
EOF

cat <<EOF > registry.access.redhat.com.yaml
docker:
     registry.access.redhat.com:
         sigstore: https://access.redhat.com/webassets/docker/content/sigstore
EOF

cat <<EOF > registry.redhat.io.yaml
docker:
     registry.redhat.io:
         sigstore: https://registry.redhat.io/containers/sigstore
EOF
  • Base64 encode those files ready to be feed into the machine config template
export ARC_REG=$( cat registry.access.redhat.com.yaml | base64 -w0 )
export RIO_REG=$( cat registry.redhat.io.yaml | base64 -w0 )
export POLICY_CONFIG=$( cat policy.json | base64 -w0 )
  • Create a machine config which writes those files to disk on the worker nodes:
cat > 51-worker-rh-registry-trust.yaml <<EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 51-worker-rh-registry-trust
spec:
  config:
    ignition:
      config: {}
      security:
        tls: {}
      timeouts: {}
      version: 2.2.0
    networkd: {}
    passwd: {}
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,${ARC_REG}
          verification: {}
        filesystem: root
        mode: 420
        path: /etc/containers/registries.d/registry.access.redhat.com.yaml
      - contents:
          source: data:text/plain;charset=utf-8;base64,${RIO_REG}
          verification: {}
        filesystem: root
        mode: 420
        path: /etc/containers/registries.d/registry.redhat.io.yaml
      - contents:
          source: data:text/plain;charset=utf-8;base64,${POLICY_CONFIG}
          verification: {}
        filesystem: root
        mode: 420
        path: /etc/containers/policy.json
  osImageURL: ""
EOF
  • Apply the worker Machine Config changes to the cluster
oc apply -f 51-worker-rh-registry-trust.yaml

Usually the master nodes won't be running Operators or other workload pods from the OLM, but it doesn't hurt to also apply the configuration to the master Machine Config. If you are using oc debug master/<node> for example that pulls an image for the Red Hat Container registry

  • Repeat a similar command on the master Machine Config
cat > 51-master-rh-registry-trust.yaml <<EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 51-master-rh-registry-trust
spec:
  config:
    ignition:
      config: {}
      security:
        tls: {}
      timeouts: {}
      version: 2.2.0
    networkd: {}
    passwd: {}
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,${ARC_REG}
          verification: {}
        filesystem: root
        mode: 420
        path: /etc/containers/registries.d/registry.access.redhat.com.yaml
      - contents:
          source: data:text/plain;charset=utf-8;base64,${RIO_REG}
          verification: {}
        filesystem: root
        mode: 420
        path: /etc/containers/registries.d/registry.redhat.io.yaml
      - contents:
          source: data:text/plain;charset=utf-8;base64,${POLICY_CONFIG}
          verification: {}
        filesystem: root
        mode: 420
        path: /etc/containers/policy.json
  osImageURL: ""
EOF
  • Apply the machine Machine Config changes to the cluster
oc apply -f 51-master-rh-registry-trust.yaml

Verifying the configuration was applied correctly

The machine-config-controller will notice the new MachineConfig and generate a new "rendered" version that looks like rendered-worker-(hash). Use oc describe machineconfigpool/worker to monitor the status of the rollout of the new rendered config to each node.

$ oc describe machineconfigpool/worker
Name:         worker
Namespace:    
Labels:       machineconfiguration.openshift.io/mco-built-in=
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1
Kind:         MachineConfigPool
Metadata:
  Creation Timestamp:  2019-12-19T02:02:12Z
  Generation:          3
  Resource Version:    16229
  Self Link:           /apis/machineconfiguration.openshift.io/v1/machineconfigpools/worker
  UID:                 92697796-2203-11ea-b48c-fa163e3940e5
Spec:
  Configuration:
    Name:  rendered-worker-f6819366eb455a401c42f8d96ab25c02
    Source:
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         00-worker
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         01-worker-container-runtime
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         01-worker-kubelet
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         51-worker-rh-registry-trust
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-worker-92697796-2203-11ea-b48c-fa163e3940e5-registries
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-worker-ssh
  Machine Config Selector:
    Match Labels:
      machineconfiguration.openshift.io/role:  worker
  Node Selector:
    Match Labels:
      node-role.kubernetes.io/worker:  
  Paused:                              false
Status:
  Conditions:
    Last Transition Time:  2019-12-19T02:03:27Z
    Message:               
    Reason:                
    Status:                False
    Type:                  RenderDegraded
    Last Transition Time:  2019-12-19T02:03:43Z
    Message:               
    Reason:                
    Status:                False
    Type:                  NodeDegraded
    Last Transition Time:  2019-12-19T02:03:43Z
    Message:               
    Reason:                
    Status:                False
    Type:                  Degraded
    Last Transition Time:  2019-12-19T02:28:23Z
    Message:               
    Reason:                
    Status:                False
    Type:                  Updated
    Last Transition Time:  2019-12-19T02:28:23Z
    Message:               All nodes are updating to rendered-worker-f6819366eb455a401c42f8d96ab25c02
    Reason:                
    Status:                True
    Type:                  Updating
  Configuration:
    Name:  rendered-worker-d9b3f4ffcfd65c30dcf591a0e8cf9b2e
    Source:
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   00-worker
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   01-worker-container-runtime
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   01-worker-kubelet
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   99-worker-92697796-2203-11ea-b48c-fa163e3940e5-registries
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   99-worker-ssh
  Degraded Machine Count:     0
  Machine Count:              1
  Observed Generation:        3
  Ready Machine Count:        0
  Unavailable Machine Count:  1
  Updated Machine Count:      0
Events:                       <none>
  • Which will be later updated:
$ oc describe machineconfigpool/worker
...
    Last Transition Time:  2019-12-19T04:53:09Z
    Message:               
    Reason:                
    Status:                False
    Type:                  Updated
    Last Transition Time:  2019-12-19T04:53:09Z
    Message:               All nodes are updating to rendered-worker-44d515bcc439808af36f591f3e3ef117
    Reason:                
    Status:                True
    Type:                  Updating
  Configuration:
    Name:  rendered-worker-f6819366eb455a401c42f8d96ab25c02
    Source:
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   00-worker
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   01-worker-container-runtime
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   01-worker-kubelet
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   51-worker-rh-registry-trust
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   99-worker-92697796-2203-11ea-b48c-fa163e3940e5-registries
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   99-worker-ssh
  Degraded Machine Count:     0
  Machine Count:              1
  Observed Generation:        4
...

Notice the Observed Generation count has been incremented and the Configuration Source now includes 51-worker-rh-registry-trust

  • To verify the files have changed on disk you can use:
$ oc debug node/<node> -- chroot /host cat /etc/containers/policy.json
Starting pod/<node>-debug ...
To use host binaries, run `chroot /host`
{
  "default": [
    {
      "type": "insecureAcceptAnything"
    }
  ],
  "transports": {
    "docker": {
      "registry.access.redhat.com": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ],
      "registry.redhat.io": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
      ]
    },
    "docker-daemon": {
      "": [
        {
          "type": "insecureAcceptAnything"
        }
      ]
    }
  }
}

$ oc debug node/<node> -- chroot /host cat /etc/containers/registries.d/registry.redhat.io.yaml
Starting pod/<node>-debug ...
To use host binaries, run `chroot /host`
docker:
     registry.redhat.io:
         sigstore: https://registry.redhat.io/containers/sigstore

$ oc debug node/<node> -- chroot /host cat /etc/containers/registries.d/registry.access.redhat.com.yaml
Starting pod/<node>-debug ...
To use host binaries, run `chroot /host`
docker:
     registry.access.redhat.com:
         sigstore: https://access.redhat.com/webassets/docker/content/sigstore

References

Verifying image signing for Red Hat Container Registry

5 Comments

Log in to comment
DB Newbie 11 points
14 June 2020 9:06 AM

Hello, Very useful instruction. There is a typo in the procedure: A file is created in the procedure - "registry.access.redhat.com.yaml" But then, when the check is performed, the file "redhat.access.redhat.com.yaml" is searched.

Regards, Dimitar

Pedro Jose Amoedo Martinez's picture Red Hat Active Contributor 358 points
15 June 2020 8:53 AM

Hi Dimitar, thanks for your comment, you are right, it was just a typo which is now fixed.

BTW, please note that the name of the file doesn't have to match with the repository configured inside, we are using the same names to be self-explanatory.

Best Regards.

pb Newbie 5 points
19 June 2020 6:37 PM

Hi, I tried this on CRC 1.11 however these instructions don't seem to work with private image repositories and signatures. Any ideas?

crc version: 1.11.0+883ca49 OpenShift version: 4.4.5 (embedded in binary)

JM Newbie 5 points
29 April 2022 2:00 PM

I believe 3 of the catalog source index images from registry.redhat.io are signed using the ISV Container Signing Key and require some additions to the image signing policy presented here. From https://bugzilla.redhat.com/show_bug.cgi?id=1903632#c52, I think this should include:

        "docker": {
            "registry.redhat.io/redhat/certified-operator-index": [
                {
                    "type": "signedBy",
                    "keyType": "GPGKeys",
                    "keyPath": "/usr/share/openshift/gpg-keys/containerisvsign.asc"
                }
            ],
            "registry.redhat.io/redhat/community-operator-index": [
                {
                    "type": "signedBy",
                    "keyType": "GPGKeys",
                    "keyPath": "/usr/share/openshift/gpg-keys/containerisvsign.asc"
                }
            ],
            "registry.redhat.io/redhat/redhat-marketplace-index": [
                {
                    "type": "signedBy",
                    "keyType": "GPGKeys",
                    "keyPath": "/usr/share/openshift/gpg-keys/containerisvsign.asc"
                }
            ],

A pointer to where to get that key would be helpful as well. The catalog source index images were added to registry.redhat.io in 4.6.

It was great to find this.

Sam Morris's picture Pro 428 points
29 June 2022 9:05 AM

How does this interact with the config.openshift.io's Image.spec.registrySources.allowedRegistries attribute, which causes some operator to create /etc/containers/policy.json on nodes?