This blog post looks at the final part of creating secure software: shipping it to users in a safe way. It explains how to use transport security and package signatures to achieve this goal.
There are two commonly used tools related to RPM package management,
rpm. (Recent Fedora versions have replaced
dnf, a rewrite with similar functionality.) The
yum tool inspects package sources (repositories), downloads RPM packages, and makes sure that required dependencies are installed along with fresh package installations and package updates.
rpm as a library to install packages.
yum repositories are defined by
.repo files in
/etc/yum.repos.d, or by
yum plugins for repository management (such as
subscription-manager for Red Hat subscription management).
rpm is the low-level tool which operates on explicit set of RPM packages.
rpm provides both a set of command-line tools, and a library to process RPM packages. In contrast to
yum, package dependencies are checked, but violations are not resolved automatically. This means that
rpm typically relies on
yum to tell it what to do exactly; the recipe for a change to a package set is called a transaction. Securing package distribution at the
yum layer resembles transport layer security. The
rpm security mechanism is more like end-to-end security (in fact,
rpm uses OpenPGP internally, which has traditionally been used for end-to-end email protection).
Transport security with
Transport security is comparatively easy to implement. The web server just needs to serve the package repository metadata (
repomd.xml and its descendants) over HTTPS instead of HTTP. On the client, a
.repo file in
/etc/yum.repos.d has to look like this:
[gnu-hello] name=gnu-hello for Fedora $releasever baseurl=https://download.example.com/dist/fedora/$releasever/os/ enabled=1
$releasever expands to the Fedora version at run time (like “
22”). By default, end-to-end security with RPM signatures is enabled (see the next section), but we will focus on transport security first.
yum will verify the cryptographic digests contained in the metadata files, so serving the metadata over HTTPS is sufficient, but offering the
.rpm files over HTTPS as well is a sensible precaution. The metadata can instruct
yum to download packages from absolute, unrelated URLs, so it is necessary to inspect the metadata to make sure it does not contain such absolute “
http://” URLs. However, transport security with a third-party mirror network is quite meaningless, particularly if anyone can join the mirror network (as it is the case with CentOS, Debian, Fedora, and others). Rather than attacking the HTTPS connections directly, an attacker could just become part of the mirror network. There are two fundamentally different approaches to achieve some degree of transport security.
Fedora provides a centralized, non-mirrored Fedora-run metalink service which provides a list if active mirrors and the expected cryptographic digest of the
yum uses this information to select a mirror and verify that it serves the up-to-date, untampered
repomd.xml. The chain of cryptographic digests is verified from there, eventually leading to verification of the
.rpm file contents. This is how the long-standing Fedora bug 998 was eventually fixed.
Red Hat uses a different option to distribute Red Hat Enterprise Linux and its RPM-based products: a content-distribution network, managed by a trusted third party. Furthermore, the repositories provided by Red Hat use a separate public key infrastructure which is managed by Red Hat, so breaches in the browser PKI (that is, compromises of certificate authorities or misissued individual certificates) do not affect the transport security checks
yum provides. Organizations that wish to implement something similar can use the
sslcacert configuration switch of
yum. This is the way Red Hat Satellite 6 implements transport security as well. Transport security has the advantage that it is straightforward to set up (it is not more difficult than to enable HTTPS). It also guards against manipulation at a lower level, and will detect tampering before data is passed to complex file format parsers such as SQLite, RPM, or the XZ decompressor. However, end-to-end security is often more desirable, and we cover that in the next section.
End-to-end security with RPM signatures
RPM package signatures can be used to implement cryptographic integrity checks for RPM packages. This approach is end-to-end in the sense that the package build infrastructure at the vendor can use an offline or half-online private key (such as one stored in hardware security module), and the final system which consumes these packages can directly verify the signatures because they are built into the
.rpm package files. Intermediates such as proxies and caches (which are sometimes used to separate production servers from the Internet) cannot tamper with these signatures. In contrast, transport security protections are weakened or lost in such an environment.
Generating RPM signatures
To add an RPM signature to a
.rpm signature, you need to generate a GnuPG key first, using
gpg --gen-key. Let's assume that this key has the user ID “
email@example.com”. We first export the public key part to a file in a special directory, otherwise
rpmsign will not be able to verify the signatures we create as it uses the RPM database as a source of trusted signing keys (and not the user GnuPG keyring):
$ mkdir $HOME/rpm-signing-keys $ gpg --export -a firstname.lastname@example.org > $HOME/rpm-signing-keys/example-com.key
The name of the directory
$HOME/rpm-signing-keys does not matter, but the name of the file containing the public key must end in “
.key”. On Red Hat Enterprise Linux 7, CentOS 7, and Fedora, you may have to install the
rpm-sign package, which contains the
rpmsign program. The
rpmsign command to create the signature looks like this:
$ rpmsign -D '_gpg_name email@example.com' --addsign hello-2.10.1-1.el6.x86_64.rpm Enter pass phrase: Pass phrase is good. hello-2.10.1-1.el6.x86_64.rpm:
(On success, there is no output after the file name on the last line, and the shell prompt reappears.) The file
hello-2.10.1-1.el6.x86_64.rpm is overwritten in place, with a variant that contains the signature embedded into the RPM header. The presence of a signature can be checked with this command:
$ rpm -Kv -D "_keyringpath $HOME/rpm-signing-keys" hello-2.10.1-1.el6.x86_64.rpm hello-2.10.1-1.el6.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID de337997: OK Header SHA1 digest: OK (b2be54480baf46542bcf395358aef540f596c0b1) V4 RSA/SHA1 Signature, key ID de337997: OK MD5 digest: OK (6969408a8d61c74877691457e9e297c6)
If the output of this command contains “
NOKEY” lines instead, like the following, it means that the public key in the directory
$HOME/rpm-signing-keys has not been loaded successfully:
hello-2.10.1-1.el6.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID de337997: NOKEY Header SHA1 digest: OK (b2be54480baf46542bcf395358aef540f596c0b1) V4 RSA/SHA1 Signature, key ID de337997: NOKEY MD5 digest: OK (6969408a8d61c74877691457e9e297c6)
Afterwards, the RPM files can be distributed as usual and served over HTTP or HTTPS, as if they were unsigned.
Consuming RPM signatures
To enable RPM signature checking in
rpm explicitly, the
yum repository file must contain a
gpgcheck=1 line, as in:
[gnu-hello] name=gnu-hello for Fedora $releasever baseurl=https://download.example.com/dist/fedora/$releasever/os/ enabled=1 gpgcheck=1
Once signature checks are enabled in this way, package installation will fail with a
NOKEY error until the signing key used by
.rpm files in the repository is added to the system RPM database. This can be achieved with a command like this:
$ rpm --import https://download.example.com/keys/rpmsign.asc
The file needs to be transported over a trusted channel, hence the use of an
https:// URL in the example. (It is also possible to instruct the user to download the file from a trusted web site, copy it to the target system, and import it directly from the file system.) Afterwards, package installation works as before.
After a key has been import, it will appear in the output of the “
rpm -qa” command:
$ rpm -qa | grep ^gpg-pubkey- … gpg-pubkey-ab0e12ef-de337997 …
More information about the key can be obtained with “
rpm -qi gpg-pubkey-ab0e12ef-de337997”, and the key can be removed again using the “
rpm --erase gpg-pubkey-ab0e12ef-de337997”, just as if it were a regular RPM package.
Note: Package signatures are only checked by
yumif the package is downloaded from a repository (which has checking enabled). This happens if the package is specified as a name or name-version-release on the
yumcommand line. If the
yumcommand line names a file or URL instead, or the
rpmcommand is used, no signature check is performed in current versions of Red Hat Enterprise Linux, Fedora, or CentOS.
Issues to avoid
When publishing RPM software repositories, the following should be avoided:
- The recommended
yumrepository configuration uses
- The recommended
yumrepository configuration explicitly disables RPM signature checking with
- There are optional instructions to import RPM keys, but these instructions do not tell the system administrator to disable the
gpgcheck=0line in the default
yumconfiguration provided by the independent software vendor.
- The recommended “
rpm --import” command refers to the public key file using an
The first three deficiencies in particular open the system up to a straightforward man-in-the-middle attack on package downloads. An attacker can replace the repository or RPM files while they are downloaded, thus gaining the ability execute arbitrary commands when they are installed. As outlined in the article on the PKI used by the Red Hat CDN, some enterprise networks perform TLS intercept, and HTTPS downloads will fail. This possibility is not sufficient to justify weakening package authentication for all customers, such as recommending to use
http:// instead of
https:// in the
yum configuration. Similarly, some customers do not want to perform the extra step involving “
rpm --import”, but again, this is not an excuse to disable verification for everyone, as long as RPM signatures are actually available in the repository. (Some software delivery processes make it difficult to create such end-to-end verifiable signatures.)
If you are creating a repository of packages you should ensure give your users a secure way to consume them. You can do this by following these recommendations:
https://URLs everywhere in configuration advice regarding RPM repository setup for yum.
- Create a signing key and use them to sign RPM packages, as outlined above.
- Make sure RPM signature checking is enabled in the
- Use an
https://URL to download the public key in the setup instructions.
We acknowledge that package signing might not be possible for everyone, but software downloads over HTTPS downloads are straightforward to implement and should always be used.