OCP3.11: CVE-2021-44228 affecting Elasticsearch (Red Hat OpenShift Logging)

Resolution

It is possible to apply ERRATA patch by upgrading Elasticsearch to image tag Elasticsearch 3.11.570-2.gd119820 which contains the fix, or the following workaround is also suitable.

1) Change to project where Logging stack deployed (by default "openshift-logging" project)

$ oc project openshift-logging

2) Find the 'elasticsearch' deploymentConfigs deployed for passing later to oc set env command

$ oc get dc -l component=es

NAME                              REVISION   DESIRED   CURRENT   TRIGGERED BY
logging-es-data-master-kfity61t   9          1         1         
logging-es-data-master-o68rc18y   4          1         1         
logging-es-data-master-u6hh29n4   3          1         1         

3) Set environment variable 'ES_JAVA_OPTS' in 'elasticsearch' for system property log4j2.formatMsgNoLookups to true

$ oc set env -c elasticsearch dc/<elasticsearch_deploymentConfig_name> ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"

Note:
Please check if there are already some custom environment variables set for ES_JAVA_OPTS and append them if needed.

4) Confirm before rolling out the variable is present:

$ oc set env -c elasticsearch dc -l component=es --list | grep ES_JAVA_OPTS

5) Rollout new replicationControllers for 'pods' to start with the new values:. Do this for all deploymentConfigs:

$ oc rollout latest dc/<deploymentConfig_name>

6) Check new ES pod has been spawned automatically after the rollout:

$ oc get pods -l component=es

NAME                                            READY   STATUS    RESTARTS   AGE
elasticsearch-cdm-ba9c6evk-1-796f6cfdbc-4dqc6   2/2     Running   0          27m
elasticsearch-cdm-ba9c6evk-2-7959d4d857-z5km9   2/2     Running   0          2d9h
elasticsearch-cdm-ba9c6evk-3-5f9c5d668c-cr8lj   2/2     Running   0          2d9h

7) Open a shell into the newly-spawned 'ES pods' to check Java command-line arguments passed correctly including "-Dlog4j2.formatMsgNoLookups=true"

$ for es_pod in $(oc get pods -l component=es --no-headers -o jsonpath='{range .items[?(@.status.phase=="Running")]}{.metadata.name}{"\n"}{end}'); \
   do echo "Confirm changes on $es_pod" ;  sleep 1 ; \
   oc rsh -Tc elasticsearch $es_pod ps auxwww | grep log4j2.formatMsgNoLookups ; sleep 3; \
   done

-Dlog4j2.formatMsgNoLookups=true should be visible in above output

8) The pods should also have this variable set:

$ for es_pod in $(oc get pods -l component=es --no-headers -o jsonpath='{range .items[?(@.status.phase=="Running")]}{.metadata.name}{"\n"}{end}'); \
   do echo "Confirm changes on $es_pod" ;  sleep 1 ; \
   oc rsh -Tc elasticsearch $es_pod printenv | grep ES_JAVA_OPTS ; sleep 3; \
   done

You should see something like:

ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"
ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"
ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"