OCP3.11: CVE-2021-44228 affecting Elasticsearch (Red Hat OpenShift Logging)

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform 3.11
  • Red Hat Openshift Logging (openshift3/ose-logging-elasticsearch5)

Issue

Red Hat OpenShift Logging mitigation for CVE-2021-44228 in Red Hat OpenShift Container Platform 3.11

Resolution

It is possible to apply ERRATA patch by upgrading Elasticsearch to image tag Elasticsearch 3.11.570-2.gd119820 which contains the fix, or the following workaround is also suitable.

1) Change to project where Logging stack deployed (by default "openshift-logging" project)

$ oc project openshift-logging

2) Find the 'elasticsearch' deploymentConfigs deployed for passing later to oc set env command

$ oc get dc -l component=es

NAME                              REVISION   DESIRED   CURRENT   TRIGGERED BY
logging-es-data-master-kfity61t   9          1         1         
logging-es-data-master-o68rc18y   4          1         1         
logging-es-data-master-u6hh29n4   3          1         1         

3) Set environment variable 'ES_JAVA_OPTS' in 'elasticsearch' for system property log4j2.formatMsgNoLookups to true

$ oc set env -c elasticsearch dc/<elasticsearch_deploymentConfig_name> ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"

Note:
Please check if there are already some custom environment variables set for ES_JAVA_OPTS and append them if needed.

4) Confirm before rolling out the variable is present:

$ oc set env -c elasticsearch dc -l component=es --list | grep ES_JAVA_OPTS

5) Rollout new replicationControllers for 'pods' to start with the new values:. Do this for all deploymentConfigs:

$ oc rollout latest dc/<deploymentConfig_name>

6) Check new ES pod has been spawned automatically after the rollout:

$ oc get pods -l component=es

NAME                                            READY   STATUS    RESTARTS   AGE
elasticsearch-cdm-ba9c6evk-1-796f6cfdbc-4dqc6   2/2     Running   0          27m
elasticsearch-cdm-ba9c6evk-2-7959d4d857-z5km9   2/2     Running   0          2d9h
elasticsearch-cdm-ba9c6evk-3-5f9c5d668c-cr8lj   2/2     Running   0          2d9h

7) Open a shell into the newly-spawned 'ES pods' to check Java command-line arguments passed correctly including "-Dlog4j2.formatMsgNoLookups=true"

$ for es_pod in $(oc get pods -l component=es --no-headers -o jsonpath='{range .items[?(@.status.phase=="Running")]}{.metadata.name}{"\n"}{end}'); \
   do echo "Confirm changes on $es_pod" ;  sleep 1 ; \
   oc rsh -Tc elasticsearch $es_pod ps auxwww | grep log4j2.formatMsgNoLookups ; sleep 3; \
   done

-Dlog4j2.formatMsgNoLookups=true should be visible in above output

8) The pods should also have this variable set:

$ for es_pod in $(oc get pods -l component=es --no-headers -o jsonpath='{range .items[?(@.status.phase=="Running")]}{.metadata.name}{"\n"}{end}'); \
   do echo "Confirm changes on $es_pod" ;  sleep 1 ; \
   oc rsh -Tc elasticsearch $es_pod printenv | grep ES_JAVA_OPTS ; sleep 3; \
   done

You should see something like:

ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"
ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"
ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"

Root Cause

Diagnostic Steps

CVE-2021-44228

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

4 Comments

This command looks at all images in all namespaces to see if the affected OpenShift component images are used somewhere: oc get pod -o custom-columns=NAMESPACE:.metadata.namespace,CONTAINER:.spec.containers[*].name,IMAGE:.spec.containers[*].image -A | grep -e ose-metering-presto -e ose-metering-hive -e ose-logging-elasticsearch5 -e ose-logging-elasticsearch6 -e openshift-elasticsearch6-container

That command can't be used on OCP 3.11, only for OCP4. For OCP 3.11 we would have to use:

$ oc get pods -n openshift-logging -o=jsonpath='{range .items[]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' | sort

Is this workaround still applicable/valid? I see ( at https://logging.apache.org/log4j/2.x/security.html ) that Apache have retracted the initial statement for it. Today the advice is to update to version 2.16 of log4j and enable JNDI explicitly where/when required.

Hi, Marius.

The mitigation is valid.

The article has been updated now with a clarification in this regard.