Tutorial on how to process vulnerability scans

Updated -

Introduction

Security considerations are even more important today than in the past. Every day we discover new vulnerabilities that impact our computer systems, which become more complex.

The process of interpreting and addressing threats found in a vulnerability scan can often appear complex. Scan results with a greater number of findings can be tempting to be viewed as more accurate than scans with less; however, that’s not always the case. A truly accurate scan will remove duplicates, utilize vendor-disclosed data, and assist the security professional in prioritizing the elimination of risk.

False positives, false negatives, and data discrepancies are the main reasons vulnerability scanning is difficult to compare and understand. These reasons are exaggerated as the complexity of software increases. The best examples are containerized environments like Red Hat OpenShift, which package and include platform dependencies in addition to container first content.

This document intends to help customers better understand the terminology, concepts, and industry-derived practices to prioritize a risk assessment after performing a vulnerability scan.

Understanding terminology

Affected

This is simply a yes or no decision to indicate whether the vulnerable code is present. It doesn't matter if the vulnerable code can't or won’t be executed. If it is present, then we consider the component to be affected.

Impacted

If there is an impact on any of the information security triad: confidentiality, integrity, and availability (CIA), the affected component is considered impacted by the vulnerability.
There are cases when an affected component is not impacted by the vulnerability when the vulnerable artifact/code is not exploitable in the component or product. Hence, there is no impact on the CIA.

Severity rating

Red Hat Product Security rates the severity of security issues found in Red Hat products using a four-point scale (Low, Moderate, Important, and Critical). The Red Hat severity rating is intended to provide a prioritized risk assessment specific to how Red Hat builds, ships, and configures that code, and assists in scheduling your upgrades. This rating is not derived as a direct correlation with CVSS. This means that our ratings will sometimes differ from NVD based on how our software is built and deployed.

NVD may rate a flaw in a particular service as having a High Impact on the CVSS CIA Triad (Confidentiality, Integrity, Availability), where the service in question is typically run as the root user with full privileges on a system. However, in a Red Hat product, the service may be specifically configured to run as a dedicated non-privileged user running entirely in an SELinux sandbox, greatly reducing the immediate impact from compromise, resulting in Low impact. The recommendation is to always use the Red Hat severity when scanning Red Hat products.

CVSS score

Red Hat uses CVSS base metrics to convey how a particular vulnerability works and what aspects of the information security triad—confidentiality, integrity, and availability (CIA)— are impacted by the flaw. CVSS base metrics are not a measurement of risk and should be calculated together with additional temporal and environmental metrics. For open source software shipped by multiple vendors, the CVSS base scores may vary for each vendor's version depending on the version they ship, how they ship it, the platform, and even how the software is compiled. These dependencies make scoring vulnerabilities difficult for third-party vulnerability databases, such as NVD, that only provide a single CVSS base score for each vulnerability. Red Hat scores reflect how a vulnerability affects our products specifically.

Risk and prioritization

Consumers of vulnerability scans should use a combination of Red Hat Severity ratings (risk priority) and CVSS scores (base+temporal+environmental) in their own risk assessment to make correct judgments regarding priority. The risk of one vulnerability might be different in various environments. The same vulnerability in a standard bare metal system might have a high severity when the containerized environment might not be exploitable at all. Hence, the risk is reduced to the minimum. Red Hat engineering teams utilize our severity ratings to determine patching prioritization, which explains why low and moderate-rated CVEs may not always be addressed as quickly as critical and important CVEs.

A scanned artifact versus parent package

Security scanners usually provide information about the detected vulnerable artifact, which might be one library or module; however, it is always crucial to check the package providing that artifact. In one single rpm package, it may contain a bundle of many libraries or modules. When one of these modules is vulnerable, the patch is shipped in a new main rpm package version (or a new container image). Red Hat does not ship a single module or library with security fixes. That’s why knowledge about the source rpm (or container image) is very important. For more information about this for rpm packages, read https://access.redhat.com/solutions/6002741.

Versioning and backporting

To maintain code stability and compatibility, Red Hat usually does not rebase packages to entirely new versions. Instead, we backport fixes and new features to an older version of the package we distribute. This can result in some security scanners that only consider the package version to report the package as vulnerable, leading to a very different response and priority of effort.

Red Hat Security Advisories (RHSA)

Red Hat publishes several forms of advisories. Red Hat Security Advisories (RHSA) are published whenever an update to a product contains a security fix. Any RHSA can include fixes for multiple CVEs and, as such, always inherits the highest Red Hat Severity Rating of the CVEs being corrected. Many vulnerability scanners utilize RHSA data to determine whether vulnerabilities have been addressed; however, this determination can be imperfect. In some cases layered offerings or services can consume platform security fixes in a traditional bug advisory (RHBA).

Red Hat vulnerability data

All data (affected state, affected offerings, scores, and ratings) related to vulnerabilities impacting the Red Hat portfolio are publicly available after public disclosure of the vulnerability. We publish our data in the following industry-standard human and machine-readable formats:

Each of these disclosure formats has pros and cons, the latter of which can create the potential for false positives if they are not properly accounted for and utilized collectively. OVAL, for instance, only addresses vulnerabilities for components packaged as RPMs.

Recommended procedures for resolving identified vulnerabilities

Remove duplicates and false positives

There are several methods to use that can filter and prioritize findings. An explanation of each method is below.

  1. Research and understand each of the reported vulnerabilities.

    Example: CVE-2018-20225 - python-pip
    This vulnerability was not considered a real vulnerability by many vendors, including Red Hat, as indicated in the statement section on the Red Hat CVE page: “Although this issue affects versions of pip shipped with Red Hat Enterprise Linux, Red Hat Software
    Collections and Red Hat CodeReady Workspaces, Red Hat Product Security does not consider this to be a security vulnerability because per the pip documentation, this is intended behavior of pip when using the --extra-index-url flag. Therefore, this issue has been marked WONTFIX.”

  2. Understand the reported vulnerable artifact.

    • If the reported vulnerable artifact is an rpm package, you may compare the detected rpm package version with the vulnerable package version. The security scanners often match incorrect versions, and already fixed versions are reported as vulnerable.

      Example: CVE-2005-2541 - tar package below 1.15.1 is vulnerable; however, In the [rhel8/postgresql-10:1-192]
      (https://catalog.redhat.com/software/containers/rhel8/postgresql-10/5ba0ae0ddd19c70b45cbf4cd?container-tabs=packages) container image the shipped tar version is 1.30-5, which is a rebased and fixed version of tar package, even though the initial Red Hat decision for this vulnerability was “Will not fix” due to the low risk related to this flaw.

    • If the reported artifact looks like a single module or library, check the source rpm package for this artifact based on the artifact path. You can find information on how to do that in the following solution: https://access.redhat.com/solutions/6002741

      When you know the exact source rpm, you may check the Red Hat CVE page for the particular reported vulnerability and find information about the rpm package where the detected artifact is bundled.

      Example: CVE-2020-2134 - Sandbox protection in Jenkins Script Security Plugin 1.70 and earlier is vulnerable.
      When you perform a security scan against for example openshift4/ose-jenkins container image, a path like org.kohsuke:groovy-sandbox:1.16 might be flagged incorrectly as vulnerable. In checking this, the Jenkins Script Security Plugin can be found bundled in the jenkins-2-plugin rpm package and this CVE was already fixed in OpenShift 4.4 and 4.3:
      https://access.redhat.com/errata/RHSA-2020:3616
      https://access.redhat.com/errata/RHSA-2020:2737

  3. Suppose the reported artifact looks like a single module or library, and it’s confirmed as not associated with any of the installed packages. In that case, this is non-rpm content that might appear in the container’s first builds.
    In such cases, if the reported vulnerable container is the original Red Hat container image, you should check the exact name of the specific container on the Red Hat CVE page.

    Example: CVE-2021-44716
    If the scan report says that the openshift4/ose-jenkins container image is affected by this vulnerability, you may check the Red Hat CVE page and find openshift4/ose-jenkins container on the list of affected products and their components and you will find that openshift4/ose-jenkins container has been already fixed in OpenShift 4.10.3: https://access.redhat.com/errata/RHSA-2022:0056

  4. If the vulnerability scan was performed for a containerized product with many containers (like OpenShift Container Platform) and a detected CVE is related to the standard library of some programing language like Golang, check the Red Hat CVE page statement, because sometimes platform products are not affected by very common vulnerabilities, like CVE-2021-38297. In the Statement section, you can find the following information:
    “CVE-2021-38297 does not affect the OpenShift Container Platform (OCP) because it does not build anything with GOARCH=wasm GOOS=js. Hence, OCP-based services are not affected either.”

  5. Check the scanner interface for source product information. Certified security scanners sometimes provide additional security information related to the Red Hat published security advisories. Unfortunately, if the source product data for the scanned component is not detected correctly, the security scanner might use incorrect input data and then inadvertently make incorrect assessments and report false positives.

    For example, if you performed a security scan against OpenShift 4.10.17 and in the report there is information that Important RHSA (like RHSA-2021:4647) is not applied. In such a case you should:

    • Check the reported RHSA and verify the CVEs that are included in that security advisory. RHSA-2021:4647 is related to kernel package vulnerabilities "CVE-2021-20317", "CVE-2021-43267"
    • Check the package version and data source (you may check it based on the package extension) for the product being scanned.
    • Check the corresponding fixes for the specific package and product data source on the Red Hat CVE page for CVEs reported by the scanner in the suggested RHSA (in our case: "CVE-2021-20317", "CVE-2021-43267").
    • Check a CVE, for example CVE-2021-20317, and comparing the information about the currently installed kernel package in the OCP 4.10.17, which is 4.18.0-305.49.1.el8_4, you will find that for Red Hat Enterprise Linux 8.4 Extended Update Support the fixed kernel version was provided in RHSA-2021:4650 not in RHSA-2021:4647 like it was reported in the scanning report. By comparing the kernel package version from RHSA-2021:4650 to the installed version, you are able to confirm that the installed version is already fixed.

Remediate and mitigate what you can

Remediation and mitigation are the two objectives a security engineer has in mind when reviewing vulnerability scans. Remediation is pretty straightforward and usually ensures that there is a plan in place to update affected components with any newer vendor-supplied versions containing a fix. If confirmed vulnerabilities do not have a newer fixed version available, sometimes it’s possible to mitigate the risk by applying mitigation steps when available.

The first step in mitigation is to distinguish regular operating systems (bare metal or virtual environments) and containerized environments. For a regular operating system environment, it is possible to use mitigation steps available on the Red Hat CVE pages.
For example, see the Mitigation section in CVE-2021-4034.

Vulnerability mitigation in containerized environments might be more difficult. To illustrate the complexity of our computer systems, consider that in a typical OpenShift system, generally, every Linux package exists in at least four places — on the underlying host system, within container images in the control plane, within container images in the platform's worker nodes and within container images in the user-provided workloads.

Based on the knowledge from Step 1, “Remove duplicates and false positives,” it is possible to verify if mitigation or remediation procedures can be applied. For example, CVE-2022-2403 shows an example mitigation procedure that can be applied to the OpenShift product by deploying a custom webhook in the cluster. Read Containers vulnerability risk assessment for a better understanding. This blog post contains examples of mitigating the flaw related to the rpm packages in the custom, user-provided workloads in the “Mitigating affected packages” section.

Prioritize any unresolved vulnerabilities with the vendor

There are two common scenarios to describe unresolved vulnerabilities. Those marked as “will not fix” and those marked as unresolved. Issues marked as will not fix are considered resolved, while those in a pending state are unresolved.

  • How to handle “Will not fix”

    The “Will not fix” status is set when the Red Hat risk assessment (Red Hat severity rating) of the particular vulnerability is either low or moderate severity, and the offering team determines not to address the specific issue in favor of other priorities. This is directly related to the product lifecycle and support model. If addressing a vulnerability marked “Will not fix” is important to you, please open a support case to request prioritization of releasing a fix for this vulnerability by justifying why the case is important.

  • How to handle unresolved vulnerabilities

    When analyzing vulnerabilities, Red Hat Product Security notifies all affected offerings. Unresolved vulnerabilities occur when the risk assessment (Red Hat severity rating) of the particular vulnerability is either low or moderate severity, and the offering team has not made a decision on when to address the vulnerability. Low and moderate CVEs are most often addressed in major offering releases as opposed to asynchronous errata. Similar to “will not fix,” if addressing an unresolved vulnerability is important to you, please open a support case to request prioritization of releasing a fix for this vulnerability by justifying why the case is important.

Collecting necessary Red Hat Security Data

Inconsistent and inaccurate security data, scan target versioning matching and various vulnerability detection methods are the main reasons why vulnerability scanning is difficult. Detecting all the incorrect results is another huge challenge. To correctly analyze and verify the security scan report, it’s necessary to collect the necessary data, many times from different
Red Hat data sources (for example, Security Data API, Red Hat advisories, Containers Metadata).

The cve-analyser tool is an example of a public project which consolidates all Red Hat security data for the specific CVE, which might help to make the right decision about the next step in the vulnerability risk assessment process.

The cve-analyser tool evaluates Red Hat container image details (container metadata). It uses the container image details to detect and extract all necessary security information for the reported vulnerabilities. The cve-analyser tool evaluates vulnerability data related to the container RPM packages and non-RPM content if the necessary information is available in the Red Hat security data. It can also do a simple correlation between a package/artifact name and the name reported by the scanner CVE instead of a deep container image correlation validation (fuzzy searching).

The cve-analyser tool expects two arguments within an input file (in .CSV format):

  • CVE id (as a first argument) detected by the 3rd party security scanner
  • container name reported as potentially affected (the second argument)

Users can also specify the package/artifact name of the reported vulnerability as a second argument instead of the container image name.
Then the cve-analyzer tool does a simple fuzzy search and shows all possible information available in the CVE security data.

The cve-analyser tool requires the container name in the following format:
repository_name/container:tag
This container image naming convention is the same on the Red Hat container portal.
For example, the OpenShift Grafana container should be listed in the input file as
openshift4/ose-grafana:v4.10.0-202205311508.p0.g48aec35.assembly.stream.
The exact format of the container name is necessary to detect all container metadata correctly..

As an output, the user will get a list of the discovered vulnerability information about:

  • fix state (if the specific is affected not)
  • product-level impact (because each product might be impacted differently by the same vulnerability)
  • available fixes (there might be many fixes for different product streams)

If the tool is unable to detect any correlation between the reported CVE in the container image (or package) and Red Hat security data, then it shows the message “Not Found Any Information”. Such cases require manual verification as it is not good practice to assume another false positive blindly.

Example output from the cve-analyser tool:
https://github.com/p-rog/cve-analyser/blob/main/samples/mix-output.csv

Contact Red Hat Product Security

The recommended procedures within this tutorial can guide security teams in managing vulnerability scan data; however we know there will still be questions. For questions about discrepancies in data or concerns about potential data gaps, please contact Red Hat Product Security by sending an email to secalert@redhat.com

Here are some example scenarios for contacting Red Hat Product Security:

  • The Red Hat severity rating for the CVE seems incorrect and you found an exploitation patch, which should have a significant impact on the CVE severity rating.
  • The CVE is not recorded in the Red Hat CVE database, and based on the performed vulnerability analysis there is evidence that Red Hat product is affected by the particular CVE.

When contacting Red Hat Product Security, please provide the following information:

  • CVE ID
  • Details about the scanned product and component
    (like: OpenShift 4.10.2 / grafana-container or OpenShift container openshift4/ose-grafana:v4.11.0-202208291725.p0.g6773185.assembly.stream)
  • How the vulnerability was detected (scanner tool name and version)
  • Vulnerability evidence: detected vulnerable path and artifact