Security testing is an SDL practice

Updated -

Introduction

Security continues to be a high priority for our customers. To address their security needs, Red Hat’s Product Security team has proactively developed and deployed a framework and processes to enhance the security of Red Hat offerings.

Recent attacks on software running on critical systems have redefined the software security landscape. Quality assurance and security testing have gained a lot of attention, importance, and funding among software authors and integrators.

Red Hat Product Security helps ensure that Red Hat produces highly secure, quality software to meet customers’ business needs. To do so, we created the Red Hat Secure Software Management Lifecycle (SSML), a Software Development Life Cycle (SDLC) approach which directly aligns to the NIST Secure Software Development Framework (NIST SSDF SP-800-218).

The NIST framework articulates/aggregates commonly accepted and desired actions to effectively manage software by establishing a core set of secure software development practices. By following these practices, Red Hat reduces the number of vulnerabilities in released software, mitigates the potential impact of undetected or unaddressed vulnerabilities, and addresses the root causes of vulnerabilities to prevent future recurrences.

During the performance of SSML, Red Hat uses manual and automated techniques to scan and test our source code. The process includes threat modeling and static and dynamic code analysis. Selected Red Hat software offerings may go through the Common Criteria and FIPS140 review process, providing additional structured testing of the offering’s core components.

The structured testing includes automated testing, manual testing, testing of vulnerabilities (from internal sources as well as penetration testing done by labs), and regression testing. All issues found during scanning and testing are logged in the appropriate defect tracking system and prioritized by product management with guidance from Red Hat Product Security. As our code is Open Source, customers can access the source code to conduct any validations as Red Hat does not provide testing results.

What processes does Red Hat have to identify potential security issues in upstream codebases?

Red Hat's open and transparent software development approach builds on well-known 'upstream' open source community software projects that reflect the contributions of developers around the world and utilizes widely used and recognized open-source licenses. In contrast to many software vendors or publishers, Red Hat curates upstream community source code for review and inclusion into products that undergo extensive composition, security screening and quality assurance testing prior to release.

We also have a continuous, dedicated process to support our customers through tracking product components, assessing implications of identified vulnerabilities for their impact, and continuously providing security updates, using the process outlined in An Open Approach to Vulnerability Management.

We take a very deliberate approach to adding upstream code to our stable, downstream releases. This approach allows for a more thorough review of potentially inappropriate code or vulnerabilities. It also helps deliver a platform that is stable but not static when it comes to upstream innovation.

Our continuous product security process builds on the National Institute of Standards and Technology (NIST) - Secure Software Development Framework (SSDF). SSDF provides guidelines for scanning and validation, building on global threat landscapes and vulnerability spectrums. Combining these practices and workflows helps deliver a comprehensive look at upstream code integrity, ultimately providing a software foundation with a stronger security posture.

How do our security testing practices fit within Red Hat secure development?

As required by our internal software policies, there are multiple processes that Red Hat uses to ensure software security. These requirements help ensure detection of security issues in our release pipeline. The major steps in the security testing process are discussed below.

Security Architecture Review

Security Architecture Review forms the backbone of the security practices at Red Hat. A Security Architect completes this with assistance from the developer team. The baseline requirements for this review are mapped to NIST SP 800-53r5 secure engineering controls. The primary goals for the review are:

  • Validate Information
  • Validate Mechanisms
  • Review the architecture for unmitigated known security threats
  • Discover potential new weaknesses
  • Build a long-term security roadmap

The security architecture review is performed at the time of change of major versions or annually, whichever occurs first. Generally, threat model changes also trigger architecture reviews. The major activities that take place as a part of the review are:

  • Review the software design to confirm claimed security properties are present and enforced
  • Validate that all weaknesses from the corresponding threat model have been triaged and those above a defined security level have been properly addressed.
  • The development team ensures that the security architect has the resources available to complete the security architecture review.

Threat Modeling

Threat modeling is a core requirement for designing good security evaluations and practices. Following a creative and holistic approach, Red Hat empowers engineering teams to document practical security threats to an offering by using proactive controls framework as guidance and measuring flaws against Product Security and Information Security standards and guidelines. The findings are then presented in an easy-to-understand format to assess business risks based on commercially reasonable decisions.

The threat modeling process starts with identifying the threats from the security architecture diagram. Mitigations are theorized based on the threats identified, followed by validation against the recorded threat model. Some example controls evaluated in a threat model are authentication, authorization, data security, audit and logging, configuration and resilience.

Static Application Security Testing (SAST)

Static Application Security Testing is the process of scanning source code using automated tools for known insecure coding techniques and potential security vulnerabilities in the software.

The precision of a SAST tool is determined by its scope of analysis and the specific techniques used to identify vulnerabilities. Different levels of analysis include:

  • Function level - The sequences of instruction.
  • File or class-level - An extensible program-code-template for object creation.
  • Application level - A program or group of programs that interact.

Red Hat uses all the above-mentioned levels to scan software offered under the Red Hat portfolio. Because the SAST tool is an automated tool, some false positives are generated, which are identified through manual analysis. We also discourage the non-conventional use of memory unsafe languages to enhance performance.

Although Red Hat performs SAST, a good amount of collaborative work remains ahead of us to properly deliver SAST at scale for all Red Hat offerings.

Dynamic Application Security Testing (DAST)

Dynamic Application Security Testing, also known as DAST, is dynamic testing of executable code and aims to identify potential vulnerabilities by employing popular techniques to exploit human software interaction points during runtime. While DAST can be used with any software, most DAST tools are primarily targeted to identify vulnerabilities in Web applications.

DAST begins by probing an application or web service to discover API endpoints to consider as potential attack surfaces. The scanning infrastructure then sends known attack payloads and mutated parameters, observing the application’s behavior and recording its responses. The responses and any exceptional behavior are reported to identify potentially vulnerable interfaces, classified by attack type.

While DAST can run as a part of any release pipeline, Red Hat has taken it further by developing a tool specifically tailored for its offerings. RapiDAST focuses on effective OpenAPI scanning and is regularly used to scan 20+ Red Hat managed services, including console.redhat.com, api.openshift.com, and others.

Fuzzing

Fuzzing or fuzz testing is an automated software testing technique that provides invalid, unexpected, or random data as input to a computer program. The program is then monitored for exceptions such as crashes, failing built-in code assertions, or potential memory leaks.

A fuzzer can be generation-based or mutation-based depending on whether inputs are generated from scratch or by modifying existing inputs in a dumb (unstructured) or smart (structured) depending on whether it is aware of the input structure. It can be white-, grey-, or black-box, depending on whether it is aware of the program structure.

As part of the overall security posture, fuzzing is used by Red Hat to detect memory-related errors, such as buffer overflows and use-after-free, race conditions and deadlocks, undefined behavior, and memory leaks.

Penetration Testing

Penetration testing is a method by which a trusted party attempts to compromise a predetermined target. The goal is to identify weaknesses and vulnerabilities so they can be fixed before a hostile attacker finds the same results. There are three types of penetration testing techniques:

  • Black box testing is performed without prior internal knowledge of the target
  • Grey-box testing is performed with limited internal knowledge of the target
  • White-box testing is performed with complete knowledge of the target

The Red Hat Product Security Research Team performs penetration testing against Red Hat offerings according to the testing framework laid out by Penetration Testing Execution standard (pentest-standard.org). The team spends time based on the past security record and criticality of the software in question. The core activities performed as a part of the testing are:

  • Pre-engagement Interactions
  • Intelligence Gathering
  • Vulnerability Analysis
  • Exploitation
  • Post Exploitation
  • Reporting
  • Post-testing

Conclusion

Software security is a complex problem, requiring various approaches to prevent vulnerabilities entering the product throughout its lifecycle. At Red Hat, we employ all of the techniques discussed in this article to improve the quality of our software and protect our customers. We continually evaluate their efficacy and work across the company to apply the best combination of techniques to build upon and enhance the security of our offerings.

© 2022 Red Hat, Inc.