Introduction

The core values of Red Hat are Freedom, Accountability, Courage, and Commitment . Red Hatters are passionate defenders of these values and it is reflected in the approaches and solutions we derive from tackling novel problems. This approach is not limited to our products and code - it is at the core of our processes and interactions with each other and our customers. In Product Security at Red Hat, we apply this open approach to create a Product Security Partnership to facilitate collaboration on security issues directly related to the software supply chain.

The supply chain background and practices

The software supply chain at any large organization is vast and typically includes several unique components often spread across multiple teams that can potentially span different internal organizations. This creates a supply chain consisting of various technologies, disjointed workflows, and conflicting priorities amongst teams. These challenges can directly impact the response time to security issues and the overall process could be disorganized and confusing to teams.

Throughout the software development and delivery process there are a multitude of technologies and processes. The software delivery process typically begins on the developer’s workstation and ends when the final product is delivered to the consumer. However, when the software producer is providing the software as a service the platforms, networks and infrastructure on which the software operates is also included. To add additional complexity, upstream libraries (for example dependencies and components) that are incorporated into the final product yet are typically created by third parties also fall into the scope of the software development and delivery process.

Breaking it down even further, the systems that must be managed by the supply chain team are not limited to the delivered application and include any service or tool that interacts with the product code. In addition, the infrastructures needed for those services or tools, as well as the infrastructure and networks the code ultimately passes over from beginning to end. This information includes upstream code, repositories, tools (for example, CI/CD tools, build systems, code signers, and scanners), servers, networks, and software delivery platforms. These systems often span multiple teams, perhaps even different organizational structures such as information systems, development, networking, security, and tool owners. Depending on an organization’s size, this can impact only a few to several hundred people.

A supply chain team has several important responsibilities such as evaluating the current security posture of the supply chain, identifying and prioritizing the security gaps that require remediation, and once addressed - validating that security issues have been properly remediated. A supply chain team also creates security policies and implements supportive tooling to aid in enforcing security policies across an organization. But ultimately, the supply chain team continuously evaluates emerging threats and risks to the supply chain and reacts to protect the final delivered product from exploiting those threats.

The attack surface area of the software supply chain is extensive and includes libraries an organization has limited, if any, control over. A compromise at any point in the supply chain process could impact not only the organization producing the software but also every direct, and possibly indirect, consumer of that product. An attack can be performed on included libraries, via the servers supporting the supply chain, the tools the supply chain uses, the network it uses, and the product code itself.

Common constraints on a supply chain team are limited resources and the breadth of specialized knowledge. In order to respond timely and adequately to security issues - security must remain focused on security. Supply chain teams should not be expected to be experts in CI/CD systems or the administration of servers - but should be experts in identifying and prioritizing threats with a risk-based approach to advise system owners on how to best protect their systems. By forming a partnership with subject matter experts (SMEs) of those systems - system owners can focus on being system owners, and security can stay focused on security.

Having a well-defined business and partnership process for supply chain issues can directly impact the efficiency and responsiveness of the supply chain team. Investing the time and effort into implementing such a partnership could reap dividends when the next large scale attack is identified.

Our approach

At Red Hat, we implemented a Product Pipeline Partnership Security Program to increase communication and collaboration while minimizing conflicting priorities. Here is an overview of our Partnership Program.

The Partnership Program

Inventory the supply chain

The first step is to identify and inventory every system or tool that interacts with the product code as it moves through the software build and delivery process. Identify the following:

• Build pipelines (beginning to end paths)
• Services (CI/CD, build systems, repositories, signing services)
• Tools (provenance tools, scanners, QE tools)
• Infrastructures (servers, networks)

For each of the identified pipelines and their services, tools, and infrastructures (referred to as ‘systems’ moving forward) - determine contact points and potential escalation paths. The persons identified for each system should be key participants in the Partnership Program. Persons identified should be empowered to speak for that system and knowledgeable about the underlying technologies. This is key for including decision makers, getting stakeholder buy-in, and ultimately hardening the supply chain systems.

This inventory list could be a few rows or span hundreds or even thousands of identified interaction points. Interacting with teams over time makes it possible to identify services that may be decommissioned as well as overlapping contact points. In our experience, we found that one contact point could cover several teams. This list must be kept current as new systems come online or old ones are decommissioned, with which an onboarding and decommissioning policy will help.

Define and document processes

What would a partnership process be without well defined processes? Well-defined processes are important steps, and designing one should not be rushed. The partnership process is meant to be malleable and responsive to meet the needs of both security and system teams and is not set in stone. To be clear, although this is a partnership, security owns this program and how it is run. Therefore, the processes generated should be defined by the supply chain team with input from system teams. This is the stage to create flowcharts, Gantt charts, roles and responsibilities, and other process documentation. Documentation makes it seamless as people transition into and out of the Partnership Program. The documentation and processes should be specific to an organization. Some processes that could be documented include (but are not limited to):

-Exceptions
-Escalations
-Incident response
-Service onboarding
-Service decommissioning
-Project charter template
-Project sign-off
-Project design template
-Project scheduling
-Project testing
-Defined required meetings (Program meetings, Project specific meetings)
-Defined communication methods (email, chat, ticketing systems)

Identify projects

Armed with a scoped list of systems in the supply chain, the people responsible for those systems, and program documentation - the supply chain can analyze its current security posture and perform a gap analysis. With this in place, a roadmap to reach the ideal state can be obtainable.

The prioritization process is a continual activity for the supply chain team. This prioritization allows the team to be responsive to security issues as they arise. The roadmap created should be periodically re-evaluated for continued relevance.

Achieving alignment

Aiming to attain alignment with security best practices and guidelines to confirm that important security controls are not overlooked. These may include requirements, recommendations, government mandates, security frameworks, and policies. Which ones an organization chooses to pursue depends on several factors. There may be industry specific requirements such as in the healthcare, financial, or government domains. There may be geographical specific requirements such as which country the organization resides in or where it conducts business. Some security guidelines specific to supply chain security that an organization may want to review are The White House Executive Order 14028 [1], NIST 800-161r1 [2], NIST 800-53r5 [3], NTIA's Communications Supply Chain Risk Information Partnership (C-SCRIP) [4], the OpenSSF Mobilization Plan [5], and the Supply chain Levels for Software Artifacts (SLSA) [6] framework.

Perform a gap analysis

To determine which security practices the systems are already in compliance with, you may perform surveys when working with the identified partners in the Partnership. Compare the recommendations and requirements list for the selected best practice with the current state of each identified system. Remediate the items not in adherence through the Partnership Program.

Create and prioritize the roadmap

By knowing the current state of the software supply chain and the desired end state, create a plan to help decrease the gap between the two and facilitate alignment to the chosen recommendation. There may be key security controls that are missing across all of the systems. Some systems may only be missing a few yet critical controls. Review the gap list in its entirety to identify consistently missing controls as well as missing critical and high controls. There may be some less critical controls that can be quickly and easily implemented. Prioritize the list of which items to tackle first and schedule them to go through the partnership program. Consider the execution time, required resources, overall security impact, and the control's criticality. This may be a good time to utilize a risk-based prioritization methodology.

Projects

Now the fun part: based on the roughly scheduled and prioritized list of security controls to implement across the supply chain, work can be scoped and project charters created. Depending on an organization's resources and overall security posture, one or more projects outlined in the Program may be worked on at once. Because there could be several projects at once, there should be an overall program status meeting to review the entire Program. All persons in the Partnership Program should be invited. The current status of all the current and upcoming projects flowing through the Program are presented and any outstanding issues or action items are discussed.

Project charter

A project charter defines the who, what, when, and why. It defines who will be involved in the work, the scope of work that needs to be done, when the work will be performed, and the reason for the work. The detailed ‘how’ will be determined in the design stage; however, if it is known that a certain technology will be used for the project, it is appropriate to call it out in the project charter (i.e., enforce signing using Sigstore).

Each project needs its own charter to prevent confusion and keep projects both manageable and achievable. For the ‘why’, list any business, regulatory, budgetary or other justifications for the project and provide links to references if available. These should be short and clear statements that assist those not involved or new to the project in understanding why the effort is being performed.

For the ‘who’, list all persons or teams impacted by or who need to perform work for the project. Include stakeholders, including project managers, owners, tech leads, and management, for input, review, and sign-off of the entire project charter. Ensure all responsible, accountable, consulted, and informed parties are included, and their roles are specified.

For the ‘what’, the project charter must include the scope of the work. This information is extremely important in order to confirm that the work delivered is what was needed. Include a general statement of work and acceptance criteria, and clarification of any tasks that are out of scope for this particular project. The acceptance criteria should include all of the major accomplishments the project must achieve. At the end of the project, the acceptance criteria should be reviewed to validate that all items have been completed. Listing known out of scope items provides the teams and project managers with clear boundaries and prevents the project from going off track.

For the ‘when’, outline a general project timeline including estimated start and end dates. Specify if the date is immutable so that it is well understood that if priorities change - this project may or may not shift. After the design stage and consulting with the project partners, it is possible that dates may need to be re-evaluated and possibly modified. When a project gets implemented may be dependent on several factors including possible dependencies requiring a specific order of operations, regulatory or compliance deadlines, technologies becoming obsolete, or other internal policies or priorities.

The information included in a project charter will vary from organization to organization; however, the above key elements avoid confusion and make sure all parties are informed about the work to be performed, why it is being performed, and their role in the project.

Project sign-offs

Sign-offs are crucial to the partnership program's success. Obtaining sign-off signifies project buy-in and the signers acknowledgement of their role in the project (ie. informed, consulted, accountable, responsible). Stakeholders must be in agreement that the project is needed, worthy and properly scoped. Managers of teams necessary to perform work must know the project time estimates and the resources required to properly allocate and plan accordingly. If stakeholders or management have concerns with timing, scope, business need, prioritization, required resources, budgetary requirements, or any other concern at this point, open discussions need to occur. Address all concerns prior to sign-off. This sign-off is a hard stopping point in the Partnership Program. It is imperative that the project does not move forward without all required approvals.

If a change is needed to the charter (dates, scope, acceptance criteria, parties involved), all parties must be informed anytime after approvals and sign-offs have occurred. The project charter needs updating, which includes a log of the change, and another sign-off of the project charter must be signed again.

Scoping the project

Communicate with the partners in the Partnership Program and determine if their system is in scope for a particular project. They are out of scope if they are already in compliance or the control does not apply to their system. They are in scope for the project if the control is needed yet missing or poorly implemented and therefore inadequate.

Design

With knowledge of the requirements to meet and the number and type of systems that require a solution, a design to implement the security control can be created. If only a few systems need it, the solution may be simple and homegrown. Similarly, if most systems need a solution and the problem is significant in complexity - commercial solutions may need to be evaluated. One system may have a solution that can easily roll out to other systems with little hassle. Again, how the security issue is mitigated is owned and led by the supply chain team. How to successfully comply with the specified security requirements should be directed by the supply chain team. System teams are consulted and are encouraged to assist; however, the overall direction is controlled by the supply chain team.

Implement security control across the supply chain

Until now, the focus has been on planning and coordinating the project(s). This planning and coordination is a significant amount of upfront work and can seem tedious or irrelevant; however, clear project management is essential given the number of individuals involved in these cross-departmental and cross-functional projects.

At this stage, the supply chain team provides system teams with a complete design on how to implement the security control on their systems. Teams may now take action and implement the solution(s). Each team should have a method to track their work and communicate their progress towards completion using a ticketing system like Jira.

For each project, meetings specific to that project should be created. Only persons involved in that particular project need to be invited. These should include status and or stand-ups between the supply chain and system teams and should occur at least once a week. From this point in the project until teams have completed their work, the supply chain team acts as technical consultants to the system teams and project managers / owners.

Validate the solution

After a system attests that it is now in compliance with the specified security control, the supply chain team must validate that it was correctly implemented. How this is validated is dependent on what the project entailed. For instance, if the goal of the project was to validate certificates on all systems meet specific security requirements , a review of the system certificates is simple enough. If the project was to implement attestations, validate that they exist and are complete. If the project was to confirm that logs meet specific security requirements, the validate logs contain the information needed. Essentially, test the system against the project’s goals and acceptance criteria to confirm the project was completed and then configure monitoring and alerting for the continual validation of the control. If during testing it was determined that the control was not implemented as specified, work with the partner to resolve the issue.

Refine the Partnership Program

In this stage, we learn what works. We learn what doesn't. We solicit honest feedback from everyone involved in the Program for our retrospectives. We take the feedback seriously. We ask questions to determine if something needs to be changed - and, if so, what corrective action should be taken. We make alterations to the Program based on these open conversations. This systematic program analysis paired with strategic improvement to the Program helps quelch budding tensions and prevents the Program from becoming outdated and ineffective.

Conclusion

Given the expansiveness of the software supply chain, there needs to be a baked-in business process to address security issues - proactively and reactively. Knowing key contact points and escalation paths is crucial. Cross-functional business relationships must be fostered. Workstream prioritizations must be clear of ambiguity. With a clear process outlined and key people identified - engagement and awareness increase across the supply chain. This can lead to a more efficient security response to supply chain issues while creating a sense of collaboration and partnership across an organization. By using this collaborative model, we have found that security can focus on increasingly proactive activities in the supply chain while shifting the perception of security from blockers to partners.

References

1. Biden, J. (2021, May 12). Executive order on improving the nation’s cybersecurity. Retrieved August 2022, from https://www.whitehouse.gov/briefing-room/presidential-actions/2021/05/12/executive-order-on-improving-the-nations-cybersecurity/
2. NIST (2022). Special Publication 800-161r1 Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations. Retrieved August 2022, from https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-161r1.pdf
3. NIST (2020). Special Publication 800-53r5 Security and Privacy Controls for Information Systems and Organizations. Retrieved August 2022, from https://csrc.nist.gov/publications/detail/sp/800-53/rev-5/final
4. NTIA's Communications Supply Chain Risk Information Partnership (2022). Retrieved August 2022, from https://www.ntia.doc.gov/cscrip
5. Open Source Security Foundation (2022). The OpenSSF Mobilization Plan, Retrieved August 2022, from https://8112310.fs1.hubspotusercontent-na1.net/hubfs/8112310/OpenSSF/OSS%20Mobilization%20Plan.pdf?hsCtaTracking=3b79d59d-e8d3-4c69-a67b-6b87b325313c%7C7a1a8b01-65ae-4bac-b97c-071dac09a2d8
6. Supply-chain Levels for Software Artifacts. SLSA. (2022). Retrieved August 2022, from https://slsa.dev