Reactive Product Security at Red Hat

The goal of Product Security at Red Hat is “to help protect customers from meaningful security concerns when using Red Hat products and services.” What does that really mean and how do we go about it? In this blog, we take a look at how Red Hat handles security vulnerabilities and what we do to reduce risk to our customers.

In 2001, we founded a dedicated security team within Red Hat to handle product security. Back then, we really had just one product line, the Red Hat^® Linux^® distribution. Now, 14 years later, we support well over 100 different products and versions from Red Hat Enterprise Linux, to OpenStack^®, to Docker. In addition to handling the reaction to vulnerabilities found in our products, we also proactively work on improving security for the future. An upcoming blog post will highlight some of those activities.

All software, no matter what the license, provenance, or supply-chain involved, have bugs–mistakes in the code which introduce errors. Some of those errors may cause a program to behave differently than what is expected, others may cause a program to crash. Of these errors, a small proportion are classified as vulnerabilities if they pose a security risk where an attacker can deliberately cause a program to fail.

Our products are generally made up of many different open source components; for example, Red Hat Enterprise Linux 7 is composed of several thousand different packages and each one can be a separate open source project. Red Hat Product Security is accountable for knowing every component used in every product so we can keep track of the security issues. This has become an area of expertise for us and is recognized by the industry as handling vulnerabilities in third party software is not a trivial task.

It all starts with a team which monitors a number of sources to find out about security issues in such third party components. In a previous blog post, we gave some metrics for a years worth of vulnerabilities and showed that in nearly half of the vulnerabilities we fixed, we were aware in advance of the issue being made public. The biggest source of information regarding non-public issues was through the two-way relationships we have with upstream open source projects and our peer vendors. Additionally, 17% of all the issues we fixed were found internally by Red Hat through security audits by our Quality Engineering team or by the product engineering teams themselves.

The next step is to assess whether these issues affect any of our products and determine the severity of each one. We do this based on a technical assessment from our team of skilled researchers. In addition to the nature of the vulnerability itself and the types of exploits likely to operate against it, other considerations include which specific pieces of code are impacted, the sensitivity of the applications they support, and their potential degree of exposure. For any given a vulnerability in an Open Source component, different products across different vendors could be affected in different ways depending on the versions being used, what patches are included, and even how the package is compiled.

In order to manage this workload, the Product Security team makes use of a number of tools and workflow processes all built around the principles of GTD.

Depending on the severity of the issue and the life-cycle for the product, patches get created and updates prepared. For many of our products, our policy is to back-port fixes an approach that significantly reduces the potential for compatibility issues and the introduction of additional vulnerabilities while making it easier for customers to consume updates. These updates, together with our advisory text explaining the issue, make their way to customers as security errata.

We actively monitor the time it takes for vulnerabilities to pass through this entire process. For example, Red Hat Enterprise Linux 5, since its release in 2007, has had 98% of all critical flaw fixes available to customers either the same day or next calendar day, once the issue was known to the public. We make all of our data on this available so customers can determine metrics for their particular environment.

In practice, what this means is that a Red Hat subscription provides customers with guidance, stability, and security that can only come from Red Hat. For a given product, there is a single mechanism to get updates for security issues across all components and technologies included, no matter which open source project they came from. Products are supported with long life cycles, and we maintain security updates for open source components included even beyond their upstream end of life.

We've briefly shown that we have well-established processes to effectively manage vulnerabilities in open source software, and that we are effective in getting fixes for these issues to customers, but there's more that we do on the reactive side of handling security events.

2014 will be remembered for a number of high profile vulnerabilities, including several in widely used open source components: Heartbleed, ShellShock, and Poodle. Where these affected Red Hat products, we provided fast updates to correct them. However, getting fast fixes out was only part of the value.

In September last year, serious issues were found in the UNIX-like shell, Bash, called ShellShock. During this incident, Red Hat customers also received:

• Timely advice. By the time the issue went public, we had specific knowledge base articles on the Customer Portal explaining how products were affected, how to get and install the fixes, and how to determine if you were vulnerable to the issue. Our article, linked above, was the definitive source of information about the vulnerability–being cited by most news articles, Wikipedia, and even US-CERT. The knowledge base and blog were continually updated with the latest knowledge and best practices.
• Industry-leading security expertise. After the original flaw in Bash was identified and fixed, a second issue was discovered in public. It was a Red Hat Product Security engineer who designed and wrote the comprehensive patch used by most vendors in fully addressing this issue.
• Immediate support. The Red Hat Customer Portal had an alert on every page, with notifications, and our support staff had access to the technical information. We were ready to provide immediate support to customers.
• Proactive notifications. For customers with products affected by the issue, we sent email notifications within the first few hours. This email provided a call to action and linked to our specific knowledge and fixes for this issue. Posts on our Red Hat Support social media channels also directed customers to our knowledge base articles and fixes.
• A self-detection tool: We also released a self-detection tool via Red Hat Access Labs to allow customers to easily identify whether their environment was vulnerable.

We'd like customers to hear about these major security issues from us first and then be able to install the fix for the issue. When a significant security event occurs, customers can can come to Red Hat first, safe in the knowledge that we'll be on top of the situation and be able to give specific, timely, calm, and technically-accurate advice on how the issues affect all of our products and services.