Security-based Red Hat Insights rules attempt to analyze and detect issues that impact the security of your systems in different ways:

• Detect high profile, high priority, and 0-day vulnerabilities
• Detect misconfigurations of your software which may impact security
• Detect other issues that could have security implications, such as expired certificates

The Red Hat Product Security team works closely with the Red Hat Insights team to provide current, updated, and helpful content for these security rules. In this blog, we’ll focus on the first category of the rules, which are targeted at high profile vulnerabilities and the associated background work we do.

Customer Security Awareness

The Red Hat Product Security team continuously analyzes security vulnerabilities that affect Red Hat products. Some security flaws are recognized to be of especially great concern, or are expected to generate significant media attention. These issues might be branded (with a name, logo, website), are actively used in exploits “in the wild,” or are a severe problem in core packages or in the functionality of Red Hat products.

When such a high-priority vulnerability is presented, the Red Hat Product Security team starts a process known as Customer Security Awareness (CSAw). CSAw issues are frequently kept secret (embargoed) for a period of time so that proper fixes can be developed and prepared for release by those involved, before the vulnerability is publicly disclosed. During an embargo period, the Red Hat Product Security team and engineering team(s) work with upstream package maintainers, security researchers, and security teams of other Linux vendors with the goal of creating an optimal fix. The objective is to allow stakeholders to have simultaneous access to the fix(es) so that as many end users as possible have access to it before a potential attacker gets details on how the vulnerability can be exploited.

In addition to analyzing the vulnerability in-depth, making sure we have well-tested fixes available, creating an article to explain the issue, etc., the Red Hat Product Security and Red Hat Insights product teams start another process to make sure that Red Hat Insights brings a high level of value for security-conscious customers:

Preparation work

It is often a race against the clock to develop detections and remediations for a vulnerability before the exploit goes public. We treat the data and content around these issues with the highest priority, knowing that a significant number of customers depend on it. We must keep the information confidential, but we also have to cooperate with various internal parties and subject matter experts to create a response. To that end, we create a private repository, with access restricted to peers on a “need to know” basis. Team coordination is critical.

Here are three recent examples of CSAw issues:

• Samba - Loading shared modules from any path in the system leading to Remote Code Execution (CVE-2017-7494)
• sudo: Privilege escalation via improper get_process_ttyname() parsing (CVE-2017-1000367)
• Stack Guard Page Circumvention Affecting Multiple Packages (CVE-2017-1000364 & CVE-2017-1000366)

These issues utilized different engineering and testing groups. Each issue had unique technical nuances and potential impacts that had to be evaluated and corrected … all simultaneously. Often times we’ll have up to four developers collaborating on the final solution. With all of these moving parts, good lines of communication between collaborators are essential.

Identification of vulnerable packages

Once we have the shared, embargoed workspace setup, we collaborate on the vulnerability, with each engineer bringing their unique experience and expertise to bear. The first step is to identify which of our packages contains the vulnerable software and create a list of vulnerable packages that were released in various channels.

The breadth of the Red Hat Insights coverage goes beyond our initial analysis that kicked off the CSAw event. Red Hat Insights coverage includes outdated/out-of-support versions of the packages, and the solution(s) must support those older versions.

Like most major Linux distributions, Red Hat Enterprise Linux uses a process called backporting whereby security fixes are applied to existing stable versions instead of only using new, upstream packages that might introduce new features, changes, or unexpected behavior with the package. If you are not familiar with this concept, you can read Determining your risk, which discusses why commercial security scanners are often wrong when it comes to our products.

Because of our backporting policy, we do not rely on version comparison. We must be much more precise, so we create a tree of vulnerable package versions. If a system analyzed by Red Hat Insights uses a vulnerable package version, we flag it as such.

Rule development

Close cooperation within the Red Hat Product Security team is our next step, especially between the engineers working on the Insights rule and a group of analysts who analyze the vulnerability. In the race against the clock to develop the Insights rule, we communicate with the technical analysts and package maintainers searching for updates on available information. The Insights rule development for CSAw is a parallel and iterative process.

The Red Hat Insights client can be configured to frequently gather the required data (such as installed packages, running services, or software configurations) so we can start analyzing the scope of impact. The Red Hat Product Security engineers who are working on the Insights rule develop several artifacts: rule server back-end logic, rule web UI front-end content, detection scripts, and Ansible Playbooks.

The goal is to provide the Insights rule components, scripts, and playbooks to the customers, as quickly as possible.

This is a complex task that can be interrupted many times and requires additional considerations and changes as new information emerges. To bring order and prevent missed steps, we utilize a set of extensive checklists that we update and improve based on our experience with them. These checklists assist us in delivering functionality that is up to our high quality standards, and ensures that important details are not overlooked.

We have one checklist for making sure that we follow a specific timeline so we can act quickly and correctly if the vulnerability goes public earlier than expected. We have another checklist with all tasks to make sure that any team member can see what is the state of work and cover for emergencies. And yet another checklist helps us communicate with analysts and make sure we are consistent when using their analyses.

Categorization

One of the main questions we ask ourselves when developing a rule is, “Can we break out all of the affected systems into more categories?”. This is important because we want to build in flexibility and enable our customers to take a risk-based approach to remediation. Customers can quickly recognize systems they should act on first, if they have limited resources.

A good example of this categorization is a rule for another CSAw vulnerability, DROWN - Cross-protocol attack on TLS using SSLv2 (CVE-2016-0800), with eight different categorizations. This enables customers to easily evaluate the scale of impact on various systems – from systems that have the vulnerable package merely installed, to ones that are running software listening on externally accessible ports while using an old version of OpenSSL, making the system vulnerable to very effective exploitation.

This real-world advice helps our customers prioritize their efforts and initially focus on areas where they have the most exposure, allowing them to choose to defer remediation of lower risk systems until after the greatest risks are remediated.

Mitigation

Most vulnerabilities can be fixed by applying an update to affected packages. Ultimately, this is the best solution since it corrects the vulnerability. However, sometimes that is not desirable, or the fix might not be available immediately. Scheduling downtime for critical systems can sometimes be time-consuming, complex, or not feasible at that time.

If analysts are aware of effective mitigations of the issue – like using SELinux, changing software configurations or settings, running a short script, etc. – and the Red Hat Insights framework is able to detect those when they are used, the rule will also propose them. Having options for mitigations allows customers to better plan how they want to react without having to worry about exposure, and deploy the final fix at a time of their choosing.

In some cases, when we agree that the mitigation is as good as a fix, the Insights rule will disappear from the Red Hat Insights notifications list. If we deem it to be a good, but temporary solution, its severity is toned down.

Showtime!

Before a CSAw vulnerability becomes public, we review our rule against real production data. The rule is still embargoed and is accessible to as few people as necessary, even within Red Hat. Because of this, the rule back-end is uploaded to the production server out-of-band. It is moved from one private space to another as the access to the production server is also limited.

Everything is double-checked and tested as a whole. If testing goes as planned, the rule is ready when the CSAw vulnerability goes public. We share much of our work through our Red Hat Product Security Center articles and the Red Hat Insights service itself. Simultaneously, the vulnerability is announced on the mailing lists and to the public, a vulnerability article is published on the Customer Portal, fixes are made available in the repositories, and the rule status is changed to “active”.

And at that point, the long days of work by many tireless people become available to Red Hat Insights customers – so you can be aware of the things that matter the most at the moment.

Red Hat Security Blog