AJP File Read/Inclusion in Apache Tomcat (CVE-2020-1938) and Undertow (CVE-2020-1745)

Solution Verified - Updated -

Environment

  • Red Hat JBoss Web Server (JWS)
    • 5.2.0
    • 3.1.7
  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 5.x
    • 6.x
    • 7.x
  • Red Hat Enterprise Linux
    • 5.x ELS
    • 6.x
    • 7.x
    • 8.x (as pki-servlet-container, pki-servlet-engine in pki-deps module)

Issue

  • CVE-2020-1938 is a file read/inclusion using the AJP connector in Apache Tomcat. The AJP protocol is enabled by default, with the AJP connector listening in TCP port 8009 and bond to IP address 0.0.0.0. A remote, unauthenticated/untrusted attacker could exploit this AJP configuration to read web application files from a server exposing the AJP port to untrusted clients. In instances where a poorly configured server allows file uploads, an attacker could upload malicious JavaServer Pages (JSP) code within a variety of file types to gain remote code execution (RCE).

  • CVE-2020-1745 is a file read/inclusion using the AJP connector in Undertow and very similar to CVE-2020-1938.

Resolution

This is a configuration issue with AJP protocol in Tomcat/Undertow. AJP is a highly trusted protocol and should never be exposed to untrusted clients. It is insecure (clear text transmission) and assumes that your network is safe. The preventive measures should be taken by using the configuration that will not allow AJP to be exposed.

In order of preference, one of the following mitigations should be applied:

  • Disable AJP altogether in Tomcat, and instead use HTTP or HTTPS for incoming proxy connections. HTTP and HTTPS do not contain the same trust issues as AJP.
  • Protect the AJP connection with a secret, as well as carefully reviewing network binding and firewall configuration to ensure incoming connections are only allowed from trusted hosts.
  • Use only network binding and firewall configuration to ensure incoming connections are only allowed from trusted hosts.

The first option, disabling AJP, is the most secure and robust recommended solution. Protecting AJP with a secret may be less disruptive, but requires using either mod_jk or a version of httpd that supports the secret parameter. Current versions of httpd in Red Hat Software Collections do not support this parameter, so another mitigation strategy must be employed.

Configuration showing how to disable AJP and how to protect it with a secret is shown below, for various Red Hat products.

JWS (Tomcat)

  • If your site is not using the AJP Connector, disable it by commenting it out from the <TOMCAT_HOME>/conf/server.xml file as:

    <!-- <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" /> -->
    
  • If AJP connector is required and cannot be commented/deactivated, then we recommend to set a secret password for the AJP conduit - Only requests from workers with the same secret keyword will be accepted. At the Tomcat side, edit conf/server.xml:

    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="YOUR_TOMCAT_IP_ADDRESS" requiredSecret="YOUR_AJP_SECRET" />
    

    Note that YOUR_AJP_SECRET must be changed to a value that is highly secure and cannot be easily guessed.

JWS (Tomcat) OpenShift container images

JBoss EAP 5.2 (JBossWeb 2.1.x)

  • If your site is not using the AJP Connector, disable it by commenting it out from the <EAP52_HOME>/server/$PROFILE/deploy/jbossweb.sar/server.xml file as:

    <!-- <Connector protocol="AJP/1.3" port="8009" address="${jboss.bind.address}"
         redirectPort="8443" /> -->
    
  • If AJP connector is required and cannot be commented/deactivated, then we recommend to set a secret password for the AJP conduit - Only requests from workers with the same secret keyword will be accepted. At the EAP 5.2 side, edit <EAP52_HOME>/server/$PROFILE/deploy/jbossweb.sar/server.xml:

    <Connector protocol="AJP/1.3" port="8009" address="${jboss.bind.address}"
       redirectPort="8443" requiredSecret="YOUR_AJP_SECRET"/>
    

    Note that YOUR_AJP_SECRET must be changed to a value that is highly secure and cannot be easily guessed.

JBoss EAP 6.4 (JBossWeb 7.x)

The AJP connector is enabled by default only in standalone-full-ha.xml, standalone-ha.xml and full-ha , ha profiles in domain.xml. AJP connector can be secured as follows:

  • If you do not use AJP, you can disable the AJP port configuration in your standalone-*.xml and/or domain.xml file by setting enabled="false" as shown below or comment out the whole <connector name="AJP" .../> clause:

    <connector name="AJP" protocol="AJP/1.3" scheme="http" socket-binding="ajp" enabled="false"/>
    
  • If AJP connector is a requirement and cannot be commented or deactivated, then, it is recommended to add credential to AJP connector by configuring the following system property. See also this knowledge article about adding system properties:

    <system-properties>
        <property name="org.apache.coyote.ajp.DEFAULT_REQUIRED_SECRET" value="YOUR_AJP_SECRET"/>
    </system-properties>
    

    Note that YOUR_AJP_SECRET must be changed to a value that is highly secure and cannot be easily guessed.

JBoss EAP 7 (Undertow)

The AJP connector is enabled by default only in standalone-full-ha.xml, standalone-ha.xml and full-ha, ha profiles in domain.xml. AJP connector can be secured as follows:

  • In JBoss EAP 7.2 Update 8+ or after applying the One off Patch to EAP 7.2 Update 7 / EAP 7.3 , if using custom AJP and request attributes, see How to allow AJP request attributes after applying the CVE-2020-1745 AJP File Read/Inclusion Vulnerability fix in JBoss EAP 7.2 Update 8+ as they will not be allowed by default after the CVE fix.

  • If you do not use AJP, you can disable the AJP port configuration in your standalone-*.xml and/or domain.xml file by setting enabled="false" as shown below or comment out the whole <ajp-listener .../> clause:

    <ajp-listener name="ajp" socket-binding="ajp" enabled="false"/>
    

    Important: notice that mod_cluster uses AJP by default as a conduit. You need to configure mod_cluster to use HTTP or HTTPS instead of AJP before disabling the AJP connector.

  • If AJP connector is a requirement and cannot be commented or deactivated, configure the following configuration inside undertow subsystem to check secure request attribute on AJP request.

    <subsystem xmlns="urn:jboss:domain:undertow:7.0" ...>
        ...
        <server name="default-server">
            <http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true"/>
            <ajp-listener name="ajp" socket-binding="ajp"/>
            <https-listener name="https" socket-binding="https" security-realm="ApplicationRealm" enable-http2="true"/>
            <host name="default-host" alias="localhost">
                ...
                <!-- add the following with your AJP port (8009) -->
                <filter-ref name="secret-checker" predicate="equals(%p, 8009)"/>
            </host>
        </server>
        ...
        <filters>
            <!-- add the following with your credential (YOUR_AJP_SECRET) -->
            <expression-filter name="secret-checker" expression="not equals(%{r,secret}, 'YOUR_AJP_SECRET') -> response-code(403)"/>
        </filters>
    </subsystem>
    

    The above configuration can be added by the following two command in JBoss-CLI1:

    /subsystem=undertow/configuration=filter/expression-filter=secret-checker:add(expression="not equals(%{r,secret}, 'YOUR_AJP_SECRET') -> response-code(403)")
    /subsystem=undertow/server=default-server/host=default-host/filter-ref=secret-checker:add(predicate="equals(%p,8009)")
    

    Note that YOUR_AJP_SECRET must be changed to a value that is highly secure and cannot be easily guessed.

If the above configuration is not possible, then consider binding AJP port to the loopback interface, or having a firewall that only allows access from trusted hosts would considerably reduce the attack surface to local, or trusted users.


Apache httpd (httpd in JBCS or RHEL)

When the above "secret" setting is configured on Tomcat/JBoss side, the same secret value (YOUR_AJP_SECRET in the above example) will be required to be configured on the front-end proxy (mod_proxy_ajp or mod_jk).

httpd in Red Hat Software Collections (RHSCL) does not support the secret parameter. In this case we recommend using HTTP or HTTPS and disabling AJP in Tomcat as shown above.

mod_proxy (mod_proxy_ajp / mod_proxy_balancer using ajp)

  • The secret property support was added since JBCS httpd 2.4.37 or since RHEL 7 httpd-2.4.6-67.el7.x86_642.
    • For mod_proxy_ajp, the secret property can be added to ProxyPass setting.
    • For mod_proxy_balancer, the secret property can be added to each BalancerMember setting.
  • For example, add secret=YOUR_AJP_SECRET in your configuration (e.g. <HTTPD_HOME>/conf/httpd.conf or <HTTPD_HOME>/conf.d/proxy_ajp.conf) like the following:

    • mod_proxy_ajp:

      ProxyPass /example/ ajp://localhost:8009/example/ secret=YOUR_AJP_SECRET
      
    • mod_proxy_balancer:

      <Proxy balancer://mycluster>
          BalancerMember ajp://node1:8009 route=node1 secret=YOUR_AJP_SECRET
          BalancerMember ajp://node2:8009 route=node2 secret=YOUR_AJP_SECRET
      </Proxy>
      ProxyPass /example/ balancer://mycluster/example/ stickysession=JSESSIONID|jsessionid
      

mod_jk

  • mod_jk - secret can be specified to a worker or a load balancer in workers.properties. If you set a secret on a load balancer, all its members will inherit this secret. This secret property support was added in mod_jk 1.2.12 onwards.
  • For example, add the following in your workers.properties:

    worker.<WORKER_NAME>.secret=YOUR_AJP_SECRET
    

    Ensure that WORKER_NAME must be replaced with the appropriate name.

mod_cluster

  • mod_cluster does not support secret property. 3.
  • If mod_cluster is required, you need to change your configuration to use http or https instead of ajp because you can not connect without a correct secret value when secret is configured on the AJP connector in Tomcat/JBoss side.

Tomcat on RHEL

  • Red Hat Enterprise Linux 8 makes Tomcat available in the pki-deps module. This module is only intended for use to support the Dogtag Certificate System as shipped in the pki-core module, and not for hosting custom or third-party applications. In this configuration, Tomcat is only available locally and therefore remote attacks are not considered possible. As a result, the severity is Moderate.
  • Red Hat Enterprise Linux 7 and 6 are affected. Please follow the CVE page for further updates.
  • Red Hat Enterprise Linux 5 is in the Extended Life Phase of the Support and Maintenance Life Cycle. As this vulnerability is rated Moderate, it is not in support scope for Extended Life Phase, and no updates are planned to be released for Red Hat Enterprise Linux 5. The Mitigation strategies described above should be used.
  • Red Hat Software Collections are not affected. The binary RPMs produced from the rh-java-common-tomcat source RPM do not contain the Apache Coyote code which is affected by this vulnerability unless it is manually recompiled. A future update may fix the code.
  • Red Hat Satellite 6 makes use of Red Hat Enterprise Linux 7's tomcat. Satellite versions 6.5 and older enable AJP. Users of these versions should install RHSA-2020:0855 to get the fixed version of the tomcat component.

Spring Boot with Embedded Tomcat

If we have a Spring boot application with an embedded Tomcat we need to define a bean that handle the embedded application container creation. This bean must make a connector using AJP to connect Apache to Tomcat. We can define this in a @Configuration annotated class as follows:

@Configuration
public class TomcatConfig {


  @Bean
    public TomcatServletWebServerFactory servletContainer() {
        TomcatServletWebServerFactory tomcat = new TomcatServletWebServerFactory();
        Connector ajpConnector = new Connector("org.apache.coyote.ajp.AjpNioProtocol");
        AjpNioProtocol protocol= (AjpNioProtocol)ajpConnector.getProtocolHandler();
        protocol.setSecret("myapjsecret");
        ajpConnector.setPort(9090);
        ajpConnector.setSecure(true);
        tomcat.addAdditionalTomcatConnectors(ajpConnector);
        return tomcat;
    }
}

Restart the app and you should see messages that Tomcat is now listening on both port 8080 and 9090.

[  restartedMain] org.apache.coyote.ajp.AjpNioProtocol     : Initializing ProtocolHandler ["ajp-nio-127.0.0.1-9090"]
[  restartedMain] org.apache.coyote.ajp.AjpNioProtocol     : Starting ProtocolHandler ["ajp-nio-127.0.0.1-9090"]

For further updates please follow up on CVE-2020-1938 and CVE-2020-1745 pages.

Root Cause

This vulnerability leverages a AJP protocol functionality to get access to files at the server side and it is not a code failure.
The AJP protocol, if used, must be properly isolated with proper firewall rules and secret keys to accept only valid content.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

8 Comments

The issue can happen on JBoss EAP 5.x as well.

The AJP setting for JBoss EAP 6.4 (JBossWeb 7.x) did not work for me and resulted in 403 Forbidden messages. The setting is incorrect according to the doc here https://docs.jboss.org/jbossweb/2.1.x/config/ajp.html

Once I set my AJP system property like below everything worked correctly.

Fails - "org.apache.coyote.ajp.DEFAULT_REQUIRED_SECRET" value="YOUR_AJP_SECRET"

Works - "org.apache.coyote.ajp.requiredSecret" value="YOUR_AJP_SECRET"

Thanks Jay - We are evaluating.

The system property org.apache.coyote.ajp.DEFAULT_REQUIRED_SECRET is correct. When you set org.apache.coyote.ajp.DEFAULT_REQUIRED_SECRET, you also need to specify the same value to the "secret" parameter in the front-end proxy (mod_proxy / mod_jk). See "Apache httpd (httpd in JBCS or RHEL)" section for details about the configuration. Otherwise, JBoss rejects requests which do not have the matched secret with "403 Forbidden".

So I did more testing and figured out a couple of things.

I was wrong. The AJP setting for JBoss EAP 6.4 (JBossWeb 7.x) is correct as you state.

Originally I set my jBoss and Apache as this article describes and I could not get Apache to connect to jBoss using mod_jk. I was getting a can't login message not a 403 message. On investigating I found the jBoss AJP config page and changed my config to be as in my earlier comment.

My app, Apache and jBoss were working as expected.

After your email that you were going to Evalutate it more I decide to do more evaluating as well. I used a python script to test AJP on the "fixed" server and found that AJP was still vulnerable.

I went back to the configuration you have on this page and started more testing. In your doc you state "Note that YOUR_AJP_SECRET must be changed to a value that is highly secure and cannot be easily guessed."

So I went with a complex password of Uppercase, Lowercase, Numbers and Specials. After many iteration I've come to the conclusion that my password was to secure.

I am not sure if it is a bug in mod_jk or AJP but if you have a password that has # or % in it the auth fails with a failed login message. A simple example is below.

Password Fails - A#1b2!@%
Password Works - A1b2!@

I did not test all specials. My env setup is below.

Windows server 2016
Apache/2.4.41 (Win64)
    OpenSSL/1.1.1e
    mod_jk/1.2.46
JBoss EAP 6.4.16.GA (AS 7.5.16.Final-redhat-1)

Sorry for jumping to a conclusion.

as per the documentation: "Red Hat Satellite 6 makes use of Red Hat Enterprise Linux 7's tomcat. However, it does not enable AJP, and therefore Satellite 6 is not affected by this issue"

however I can see the following in /etc/tomcat/server.xml:

<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
Thanks

Aurélien

Hello Aurélien, You are correct, although Satellite 6.6 does not use AJP, older versions appears to be using it. The page is going to be corrected. Since Satellite 6 uses RHEL-7's Tomcat, you can update to RHSA-2020:0855 to get the fixed version of the component.

Thank you for the notification!

I was having trouble with other redirects but adding Protocol AJP at the end of the line worked well.

but receive 404 and not the expected 403 :(