A minimal security response process

This blog post outlines a lightweight security response process for community upstream projects: What you (as a project maintainer or contributor) can do to be prepared for incoming reports of security vulnerabilities, and to eventually respond with a security update. This is purely reactive - it is not about not shipping vulnerable code in the first place. But it is an important step in the right direction, and one that requires relatively little effort.

Release engineering

Without a minimal amount of release engineering, it does not make sense to speak of security updates; therefore, an important first step is to come up with a release process. Most projects already have one, and you just need to document it and make it more explicit. Details do not matter much - for example, you could aim for time-based releases (twice per year, for example), or releases based on feature tasks. However, it is important that you attach version numbers of some kind to your releases, and that you do not change the code shipped under a specific version number. New code should always have a new version number.

You should keep your software under version control, and be able to find the precise released versions in your version control repositories. Automated regression tests help to make sure that a security fix does not accidentally break functionality.

You need to identify the way you announce new versions. Eventually, you will publish notifications about security updates through this channel. In many cases, this is just an "announce" mailing list. You should keep this channel clear of discussions, so that it is low-traffic and the volume of discussions does not deter subscribers.

Your software (whether source code or pre-compiled binaries) should be available from a canonical download location. You should be able to publish your security updates in the same manner. Serving these resources over HTTPS with a valid certificate is a simple way to provide some level of authentication, and it is easier for your users to associate with your project than some random contributor's GnuPG key.

Contacts

You should publish how you want to receive security notifications. Putting this information under "/security/" on your project website, and linking it from relevant pages (support pages, mailing lists, general contact information) is one way to do this. The contact point can either be a private mailing list (usually under the "security@" alias), or a list of email addresses for the key developers. Another alternative is to configure an existing bug tracker so that it accepts private security bug reports, and make sure that these bugs are not included in automated announcements and bug search results.

If your software is redistributed by others (such as GNU/Linux distributions), find out if they have security teams and make note of their preferred points of contacts. You can use this information for pre-notification of upcoming security issues, as described below. It is also a good idea to know the individuals who package your software for the redistributors.

If you are part of a larger community, such as Apache, Fedora, or Debian, you might already have colleagues who deal with security reports. Approach them and exchange contact information. You should also familiarize yourself with security policies and procedures of your umbrella organization, if they exist.

In some cases, it might be appropriate to prepare to send pre-notifications to end users, but it can be challenging to decide who should receive such a notification.

Handling a security report

When you finally receive a report of a potential security vulnerability, you first need to make sure that you have all the relevant information that enables you to reproduce it. You should also ask the reporter if and how they want to be credited, and you should try to reach a preliminary agreement on a disclosure date, with an understanding that it can lapse if unforeseen obstacles arise.

Once you have isolated the issue, you need to determine the affected versions, and implement a fix. In many cases, this is just a small source code patch. Handling this patch can be a bit tricky because you should avoid putting it into a version control system which is viewable by the general public. In most cases, sending the patches as email attachments (with file names that indicate the base name, a patch summary, and a revision) to those who need them during the pre-disclosure phase will be sufficient.

If possible, you should provide minimal patches which fix the vulnerability on all currently active release branches of your software. Just releasing a new version with the current development, up to including the security fix, often makes swift upgrading difficult for your users, and makes it harder for downstream
redistributors to integrate the fix (because they typically rely on minimal backports).

You should prepare a security advisory that contains at least the following information:

• project name and URL
• versions known to be affected
• versions known to be not affected (for example, the vulnerable code was introduced in a recent version, and older versions are therefore unaffected)
• versions not checked
• type of vulnerability and its impact
• CVE identifier (see below)
• the planned, coordinated release date
• mitigating factors (for example, the vulnerability is only exposed in uncommon, non-default configurations)
• workarounds (configuration changes users can make to reduce their exposure to the vulnerability)
• credits to the original reporter

If your project has downstream redistributors, you might want to provide them with pre-notification about the upcoming security fix, using a pre-release version of your security advisory, along with the source code patch(es) you have developed. This is often called an "embargo", a time period where vendors have access to details concerning the security vulnerability, with an understanding not to publish these details or the fixes they have prepared. The embargo ends with a coordinated release date ("CRD").

During the embargo, the parties involved can discuss your draft security advisory and the fix you recommend and provide peer review. Sometimes, it can be tricky to determine the actual impact of a security bug, particularly if complex systems are interacting over the network. Discussions with other security teams can help with the impact assessment

Organizations like Red Hat will prepare and test updates, and will report issues such as regressions and incompatibilities back to you. This is important because during the embargo period, any community testing (which is probably part of your regular release process) is not possible.

The coordinated release date needs to be negotiated between you and the reporter, in consultation with downstream redistributors. Releases on Monday to Wednesday are preferred, so that system administrators do not have to deal with security updates over the weekend. Try to avoid major holidays if possible, although this can be a challenge with a global user base. Do not forget to communicate the coordinated release date to the original reporter and the downstream redistributors.

You can either contact downstream redistributors individually, or rely on the distros mailing list. (Please read the submission instructions before posting.) The distros mailing list prefers a short embargo, in the order of a week, so it is best to send a pre-notification there only if you already have working patches. Keep in mind that the "distros" mailing list only reaches redistributors with open source connections. If your program is embedded by hardware manufacturers, for example, you need other ways to reach them.

If your project has release branches which receive critical bug fixes, you can quietly prepare new releases for those branches, following your existing versioning scheme. If you do not normally release maintenance updates for releases, you can still publish patch files and add a prominent note to your download page which points to them. (This does not work if you ship pre-compiled binaries.)

On the coordinated release date, you publish the security update, update your public version control repositories and your web pages, and send out your security announcement.

You might also want to obtain a CVE identifier for your vulnerability. The CVE project run by MITRE assigns unique identifiers to published vulnerabilities, so that they can be tracked more easily across vendors and redistributors. You can ask one of the CVE Numbering Authorities for an identifier. (If there are multiple vulnerabilities, be prepared to provide a rough description for each, so that the naming authority can decide how to split them between multiple CVE identifiers if necessary.) If you decide to provide pre-notification via the "distros" list, you can ask in your posting for a CVE assignment. (Only ask for an assignment in a single place, to avoid duplicate assignments.) The CVE identifier should be included in your vulnerability announcement.

Conclusion

Depending on your viewpoint, the process sketched above may seem rather convoluted (if you have not attempted to establish an embargo before), or overly simplistic (if you have to schedule a release between multiple development and quality engineering teams). But in our experience, this approach helps to reduce the actual impact of security vulnerabilities for end users. But if you get stuck at one point, do not hesitate to ask a friendly security team (perhaps from a downstream redistributor) for advice.