OpenSSL CVE-2014-0160 Heartbleed bug and Red Hat Enterprise Linux

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7 not affected
  • Red Hat Enterprise Linux 7 Release Candidate (RC) not affected
  • Red Hat Enterprise Linux 7 Beta affected
  • Red Hat Enterprise Linux 6 affected
  • Red Hat Enterprise Linux 5 not affected
  • Red Hat Enterprise Linux 4 not affected
  • For other affected products, refer to https://access.redhat.com/site/announcements/781953

Issue

  • Does CVE-2014-0160 affect Red Hat Enterprise Linux?
  • Need fix for openssl heartbleed bug
  • What versions of Red Hat Enterprise Linux are affected by openssl heartbleed vulnerability?
  • Do we have a list of packages/services we ship with RHEL that need a restart after OpenSSL has been updated?

Resolution

Step 1: Determine if RHEL system is vulnerable to flaw described in CVE-2014-0160

  • Red Hat Enterprise Linux 7

    • Red Hat Enterprise Linux 7 include OpenSSL version openssl-1.0.1e-34.el7 which includes a fix backported from openssl-1.0.1g
  • Red Hat Enterprise Linux 7 Release Candidate (RC)

    • Red Hat Enterprise Linux 7 RC include OpenSSL version openssl-1.0.1e-34.el7 which includes a fix backported from openssl-1.0.1g
  • Red Hat Enterprise Linux 7 Beta

    • OpenSSL versions openssl-1.0.1e-33.el7 and earlier include a flawed libssl.so library vulnerable to the issue
    • To determine openssl version, use the command: rpm -q openssl
    • Version openssl-1.0.1e-34.el7 included a fix backported from openssl-1.0.1g
    • See footnote for considerations specific to RHEL 7 Beta1
  • Red Hat Enterprise Linux 6

    • OpenSSL versions openssl-1.0.1e-15 through openssl-1.0.1e-16.el6_5.4 include a flawed libssl.so library vulnerable to the issue
    • The first affected version shipped with RHEL 6.5 (RHEL 6.4 and older shipped with the unaffected openssl-1.0.0 series)
      • Systems which report as RHEL 6.0 - 6.3 could still have been updated to a newer [vulnerable] openssl-1.0.1 series package
    • To determine openssl version, use the command: rpm -q openssl
    • Version openssl-1.0.1e-16.el6_5.7 included a fix backported from openssl-1.0.1g
  • Red Hat Enterprise Linux 5 and Red Hat Enterprise Linux 4

    • Vulnerable OpenSSL 1.0.1 series versions never shipped in RHEL 5 or earlier

Optional Step 2: Look for and/or query processes which are using the vulnerable libssl library


Step 3: Upgrade the openssl package

  • Red Hat Enterprise Linux 7 Beta

    • Update to openssl-1.0.1e-34.el7 (which corrected the flaw) or later
  • Red Hat Enterprise Linux 6

    • Update to openssl-1.0.1e-16.el6_5.7 (which corrected the flaw, as described in RHSA-2014:0376) or later
  • As always, registered systems with internet access (or any RHEL 7 Beta system, or systems connected to Satellites, etc) can be updated via yum, e.g.:

    • yum update openssl
  • Otherwise, use a connected system to download the package or download the package directly from the Customer Portal2

    • After that, transfer the package to the system in question and install it manually with yum, e.g.:

      • yum update <path-to-openssl*rpm>
  • Red Hat Enterprise Linux 6 KVM Guest Image

  • Systems built from the Red Hat Enterprise Linux 6 KVM Guest Image (rhel-guest-image) package can resolve this flaw by updating the openssl package, as noted above. An updated rhel-guest-image package is also available, which includes a version of openssl that is patched for this flaw. For more details, see RHBA-2014:0426.


Step 4: After updating openssl, restart all processes using the flawed libssl.so3

  • The safest and simplest course of action is to perform a full system reboot

  • Alternatively, use the commands from Optional Step 2 to determine which processes need to be restarted and then act accordingly


Optional Step 5: Take additional remediation steps as desired

  • Official statement from Red Hat Security Response Team:

    Red Hat is not aware of any public exploit being used in the wild for this issue prior to the date of disclosure. However, a number of public exploits were published shortly after the issue was disclosed.4 These exploits could lead to the disclosure of information handled by applications using OpenSSL, including private keys, session tokens, and data submitted by users, which could include authentication credentials. It is recommended that you assess the risk this could pose to your systems, and perform additional remediation as you deem appropriate.

  • For more details on additional remediation steps, refer to: How to recover from the Heartbleed OpenSSL vulnerability


  1. Red Hat does not support the use of beta software in production and, therefore, does not normally release errata for betas. Given the great interest in the RHEL 7 Beta and the severity of the Heartbleed issue, Red Hat has made an exception in order to facilitate customer testing -- an updated OpenSSL package for the RHEL 7 Beta was provided. Note however that due to the beta nature of this update, CVE-2014-0160 will not link to any RHEL 7 Beta errata page. 

  2. Package downloads for RHEL 7 Beta are in a different place than normal supported downloads: Navigate to the RHEL 7 Public Beta Download Page and make an appropriate selection. Then click Packages tab, and enter "openssl" in the Filter search field; then select the "openssl" package to download it. 

  3. The most common process which is affected is Apache's httpd when being used in concert with mod_ssl. In contrast, OpenSSH's sshd is not affected. (See Is OpenSSH affected by the OpenSSL Heartbleed bug?

  4. Note that several security researchers have now demonstrated that it is possible to retrieve private keys from an nginx server vulnerable to his flaw. (Note that it has not yet been demonstrated that private keys can be retrieved from Apache httpd servers as shipped in RHEL). If you are using OpenSSL on a web server, for example using Apache httpd & mod_ssl, then remote attackers may have used this flaw to compromise session tokens and plaintext user credentials stored in memory. Tools allowing attackers to automatically harvest session tokens from servers vulnerable to this flaw are now publicly available. 

Root Cause

  • Official statement from Security Advisory RHSA-2014:0376:

    An information disclosure flaw was found in the way OpenSSL handled TLS and
    DTLS Heartbeat Extension packets. A malicious TLS or DTLS client or server
    could send a specially crafted TLS or DTLS Heartbeat packet to disclose a
    limited portion of memory per request from a connected client or server.
    Note that the disclosed portions of memory could potentially include
    sensitive information such as private keys. (CVE-2014-0160)

  • For links to more detail, see the entry for CVE-2014-0160 in Red Hat's CVE Database

Diagnostic Steps

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

53 Comments

Hello,

"This issue does not affect Red Hat Enterprise Linux 6.4 or earlier." Can you tell me why this issue does not affect 6.4? On a 6.4 system, we see openssl-1.0.0-27.el6_4.2.x86_64, which is an earlier version than mentioned openssl-1.0.1e-16.el6_5.4. Please advise.

@Geert:

I've updated the article to be more specific. In reality it is openssl-1.0.1e-15.el6 through openssl-1.0.1e-16.el6_5.4 which are affected by the heartbleed vulnerability.

Hi Geert

This issue was introduced in OpenSSL 1.0.1. Therefore OpenSSL 1.0.0 as shipped with Red Hat Enterprise Linux 6.4 is not affected.

Thanks
David Jorm / Red Hat Security Response Team

When querying the rpm, we see the version string listed above, but when running 'openssl version -a' we still see the old version data. Is this expected behavior?
Regards,
Don

I'm not quite sure what you mean Don because ... well, running openssl version -a will only tell you the upstream OpenSSL version openssl is based on (e.g., OpenSSL 1.0.1e-fips 11 Feb 2013); not the Red Hat release version (e.g., 16.el6_5.7).

In short, you should use rpm -q openssl to confirm you're using the appropriate version (openssl-1.0.1e-16.el6_5.7).

It should be noted that since this vulnerability allows an attacker to obtain the private key [actually, any memory in the process terminating the SSL, not just the private key], and this activity does not show up in the logs, then the key must be treated as compromised, and needs to be recreated.

The following note regarding 'mod_spdy' has been posted by CentOS users:

Anyone who is using mod_spdy from Google is advised that it appears that this module has static copies of the affected openssl code embedded in it and until such time as they release a new version, anyone using mod_spdy on their web server is still vulnerable even if they have openssl-1.0.1e-16.el6_5.7 installed. As far as we are currently aware, the only solution to this is to remove mod_spdy and restart the httpd service.

An update: fixed mod_spdy (v0.9.4.2) is out as seen here.

There is a test tool available at https://gist.github.com/jpicht/10114168 (you'll probably want to use -v 2 to specify TLS 1.1)

Note that deployments where the process doing the SSL termination is the same as that doing the application logic (eg. Apache with mod_ssl and php) will be more at risk, as using this tool (or a modified version which attempts to scan the entire process memory) will tend to leak information such as:

  • credentials that people have typed into the application (to be tested)
  • session cookies
  • credentials that the application might use to access other services (eg. database, LDAP, mail...)
  • encryption keys that might be used to encrypt database contents.
  • in a multi-threaded or multiplexing instance, the exposed data may relate to other clients.

Its a good argument for offloading the SSL termination into a different process, if not off the same machine.

Actually, I'm now not clear on whether this would affect sshd; my previous comment about it being statically linked to openssl is not necessarily correct.

You may find the following command to be useful to give you and indication of what a sshd is using for its SSL implementation; on RHEL 6, you should find that it uses NSS, not OpenSSL

# lsof | grep ^sshd | grep -e ssl -e nss
sshd       1648        root  mem       REG                8,1     65928     132960 /lib64/libnss_files-2.12.so
sshd       1648        root  mem       REG                8,1    182112     299136 /usr/lib64/libnssutil3.so
sshd       1648        root  DEL       REG                8,1               306229 /usr/lib64/libnss3.so
...

Another command of use here is this one, which will tell you what processes are currently using the affected library (but note that the output will be the same pre/post patch). It is useful in helping to determine which processes may need to have the private keys/certificates regenerated, or to figure out which services will need to be restarted (if not rebooting to apply other errata such as a kernel update)

lsof -Pni | grep LISTEN | grep -f <(lsof | grep libssl.so.1.0.1e | awk '{print $1}' | sort -u) | grep -v 127.0.0.1 | awk '{print $1,$9}' | sort -u

This is not correct.

The use of the NSS cryptographic libraries in openssh on RHEL6 is to fulfill other cryptographic dependencies.

A more useful grep reveals the following

# lsof | grep ^sshd | grep -e ssl -e nss -e libcrypto
sshd      1944      root  mem       REG              253,0    44656        364 /lib64/libnss_ldap.so.2
sshd      1944      root  mem       REG              253,0    27424       2003 /lib64/libnss_dns-2.12.so
sshd      1944      root  mem       REG              253,0    65928       2005 /lib64/libnss_files-2.12.so
sshd      1944      root  mem       REG              253,0   179416       5867 /usr/lib64/libnssutil3.so
sshd      1944      root  mem       REG              253,0  1653032       6048 /usr/lib64/libnss3.so
sshd      1944      root  mem       REG              253,0  1950976      20043 /usr/lib64/libcrypto.so.1.0.1e
sshd      1994      root  mem       REG              253,0    65928       2005 /lib64/libnss_files-2.12.so
sshd      1994      root  mem       REG              253,0   179416       5867 /usr/lib64/libnssutil3.so
sshd      1994      root  mem       REG              253,0  1653032       6048 /usr/lib64/libnss3.so
sshd      1994      root  mem       REG              253,0  1950976      20043 /usr/lib64/libcrypto.so.1.0.1e

The "/usr/lib64/libcrypto.so.1.0.1e" being provided by openssl.

As noted in the article

* This issue affects OpenSSL's libssl that implements TLS/SSL and DTLS protocols. OpenSSH does not use those protocols, and hence does not use libssl. It uses OpenSSL's cryptographic algorithms implementation provided via separate libcrypto library.

One small rant about versioning of the package. Probably RH just followed its versioning policy but the Advisory suggesting "Affected users should upgrade to OpenSSL 1.0.1g." (http://www.openssl.org/news/secadv_20140407.txt) and the RPM being named openssl-1.0.1e-16.el6_5.4.x86_64 is a bit misleading. I had to spend too much time explaining that the package released by RH was actually openssl-1.0.1e ...

Ah, but the package is not 1.0.1g, its 1.0.1e with a number of revisions made to it to do things like backport security patches (as has been done here).

What you should be looking at is not the version number, but the CVE's that an erratum addresses.

You may find the yum-plugin-security package to be helpful there, or checking the Changelog for a package, or the Errata for the CVE of interest.

Hello,

There is some information about this (in general) on https://access.redhat.com/site/security/updates/backporting/ that may help if you need to point people to the "why" in future (specifically, the "Explaining Common Release-Numbering Confusion" section).

Thanks Cameron & Murray for your responses.

I realised it was that about 10 minutes after sending the comment but I could not post while commuting. :)

I keep the link for future references.

I do agree with César Alba Pérez opinion!

Hello all,

I wonder if the SSL module (mod_ssl.so), used in the HTTPd Apache webserver, is impacted as well?
We are currently using this version: mod_ssl/2.2.22

Thanks a lot for the answer.

mod_ssl uses the openssl libraries for the SSL procotol.

Thus it is impacted by this issue.

Sources on the web indicate we need to recreate keys. Is this true? Does service httpd restart accomplish this?

This is what we found - and we are -highly interested- in the answer to Fred Magee's quesetion on recreating keys.

Hi,

Since the exploit allowed an attacker access to data which included the private key of the server, one must treat the key as compromised and generate a new one. The key in question is the one used to generate the SSL certificate request (or certificate if self signed).
An example of that command is: openssl req -nodes -newkey rsa:2048 -keyout domain.key -out domain.csr
The httpd key location can be found by looking for the SSLCertificateKeyFile in the httpd conf file.

Hope this helps.

Edit: The reason regenerating the key is necessary is that with the private key an attacker can decrypt encrypted communication even after the hole is plugged.

Hi Fred

I have updated the resolution section of this article to include the following advice:

Red Hat is not aware of any public exploit being used in the wild for this issue prior to the date of disclosure. However, a number of public exploits were published shortly after the issue was disclosed. These exploits could lead to the disclosure of information handled by applications using OpenSSL, including private keys and session tokens. It is recommended that you assess the risk this could pose to your systems, and perform additional remediation as you deem appropriate.

Thanks
David Jorm / Red Hat Security Response Team

Hello David,

I understand that this KB tells openssl from RHEL6.5 is affected.
Have you checked other possibility such as some statically linked
binary from RHEL include affected openssl library?
As you have seen in previous posts, mod_spdy from google is such an example.

Hi Masanari

We have investigated this possibility, and have not located any vulnerable copies of OpenSSL that are embedded in another component shipped with Red Hat Enterprise Linux.

Thanks
David Jorm / Red Hat Security Response Team

Hi,
i have not openssl package to update on my RHEL 65.5 box (yum openssl update --> No Packages marked for Update) but the rpm -qa openssl show me "openssl-1.0.1e-16.el6_5.7.x86_64". So can i update openssl?

Thanks & Regards,
Stefano

Hi Stefano,

This is correct. You have the newest version of the package and it is fixed.

  • The Red Hat Security Advisory (RHSA-2014:0376) reports that if you have version openssl-1.0.1e-16.el6_5.7 then you have a fixed version that is no longer a problem.

  • Currently, the latest (fixed) version is:

    openssl-1.0.1e-16.el6_5.7
                          ^
    

    (The .7 in the .el6_5.7 means that it is the fixed version)

  • If the "rpm -q openssl" command returns that package version (or a newer version) then that is good.

Kind regards,
Michael Kolbs

Michael or other Red Hat Team members,

I have followed the instructions here and upgraded my openssl to the latest (fixed) version (openssl-1.1.0.1e-16.e16_5.7)as well as restarted my server. However, when I try the Red Hat check your site tool that is linked from this Red Hat article it shows my site is vulnerable. Before I go changing the private keys I want to make sure my system is safe. What would you advice?

Thank you,
Constance

Please open a support case Constance.

Something must have went wrong in the initial restart (some services using openssl must not have restarted); because, after I restarted the server a second time and then tried the Red Hat site tool again it now shows me as no longer vulnerable - otherwise I would have opened a ticket.

Thanks!

If el6_0 to el6_5.4 were vulnerable, and the fix is el6_5.7, what of the missing versions el6_5.5 and el6_5.6? Did it take three patches to finally fix this, or was it noticed and fixed earlier? If such a gaping hole were fixed earlier, why do we hear about it later? I can't imagine it takes three patches in as many days to resolve such a straightforward memory leak. Needing to match (and validate) 1e's 6_5.7 to industry jargon of 1g is challenging to say the least.

If el6_0 to el6_5.4 were vulnerable, and the fix is el6_5.7, what of the missing versions el6_5.5 and el6_5.6? Did it take three patches to finally fix this, or was it noticed and fixed earlier?

Duane, you can get part of your answer by running the following command from an internet-connected (and registered) RHEL6 system:

yum --showduplicates list openssl

You will see that no el6_5.5 or el6_5.6 version was ever released. This is typical due to the way that rpm packages are worked on. Any time someone internal has a patch that they trigger a new build for, the rpm gets a new version number. Some of these versions are only dead-end test versions; others solely implement a single patch, whereas the version that gets released might include that plus multiple other patches.

If such a gaping hole were fixed earlier, why do we hear about it later? I can't imagine it takes three patches in as many days to resolve such a straightforward memory leak.

There was less than 24 hours from the time the bugzilla was created and errata was released to the public.

Needing to match (and validate) 1e's 6_5.7 to industry jargon of 1g is challenging to say the least.

You might find info about our backporting policy helpful.

I have updated openssl to 1.0.1e-el6_5.7, rebooted the box and the test shows that my server is still vulnerable - now what?

Please open a support case.

hey Dinko, just to make sure, your version string should look like:

$ rpm -q openssl
openssl-1.0.1e-16.el6_5.7.x86_64

assuming that matches, I'd suggest you open a support case (https://access.redhat.com/support/cases/). You may have, for example, another host proxying or handling SSL ahead of the host you updated.

Assuming openssl has been updated, using:

lsof | awk '/\/usr\/lib64\/libssl.so.1.0.1e/'

detects processes that have file handles open on the deleted library, using $9 missed these because they have no SIZE/OFF column. This will also list deleted prelink files.

Assuming openssl has been updated, using:

grep libssl.so.1.0.1e /proc/*/maps | grep -v "ls -i /usr/lib64/libssl.so.1.0.1e|awk {'print $1'}" | cut -d/ -f3 | sort -u | perl -MFile::stat -lne 'BEGIN{$libssl_ctime=stat(q{/usr/lib64/libssl.so.1.0.1e})->ctime };chomp;print $_ if( stat(qq{/proc/$_})->ctime <= $libssl_ctime )' | xargs -r -- ps uf

will filter out processes that are using the inode of the current libssl, and also filter out processes with start time (/proc/pid ctime) more recent than the ctime on /usr/lib64/libssl.so.1.0.1e. ctime is the inode change time in seconds since the epoch.

@Peter:

Good call on the lsof issue with $9! I was playing around with ways to make it simpler and more succinct last night and I fat-fingered that (should have been $0).

Also, your perl script is awesome. I'm not at all familiar with perl so I can't really audit or critique it, but I would recommend changing "/usr/lib64/" to "/usr/lib(64)?/" in any of your regex (for grep you'll have to turn on extended regex) if you want to catch stuff on 32-bit systems.

We have been using the following command:

lsof -d DEL | grep libssl.so.1.0.1e

If you've updated OpenSSL and don't want to reboot, the above command will show you exactly what processes are still running using the previous vulnerable OpenSSL library.

Indeed.

For the record: the commands already mentioned in this article will show the same thing. (I don't want anyone to worry that they must use your command in addition to what our article shows.)

So what is the best way to force prelink to update. After the daily prelink cron job ran I still have several binaries linked to the deleted openssl library.

I don't understand what you mean. You shouldn't have to be concerned about waiting for the nightly cron job, nor should you have to do anything manually with prelink. If you've got more detailed questions, it might be best to open a support case.

Interesting. Thanks for the reply.

Please reference https://access.redhat.com/site/solutions/795723 for a further explanation of prelinking.

Dear Red Hat supporter,

Could you please confirm that: "Red Hat Enterprise Linux 5 not affected".
RHEL 5 is using openssl-0.9.8e-26 that is lower version currently, so I want to make sure about this.

Thank for your explaination in return.

Yes. I can confirm that every version of openssl shipped in RHEL 5, RHEL 4, and below are unaffected by the issue described in CVE-2014-0160. Excerpt:

The MITRE CVE dictionary describes this issue as:

The (1) TLS and (2) DTLS implementations in OpenSSL 1.0.1 before 1.0.1g do not properly handle Heartbeat Extension packets, which allows remote attackers to obtain sensitive information from process memory via crafted packets that trigger a buffer over-read, as demonstrated by reading private keys, related to d1_both.c and t1_lib.c, aka the Heartbleed bug.

As this article details, only certain 1.0.1-series versions of the openssl package as shipped in RHEL 6.5 and RHEL 7 Beta are affected.

Thanks for your explaination.

I have Red Hat 5 ans openssl 0.9.8e and my server status is: Vulnerable

We'll be much better able to help you through a support case Humberto. Please open one.

EDIT: That said, I've created a new solution to address the situation you're in ...

Heartbleed detector tools report RHEL system vulnerable despite patched or non-vulnerable openssl version

Thank you Ryan!

Another point to that problem - a customer could install some software with static libssl complied in. First example - nginx, where you could easily find it from 'nginx -V' output. This way heartbleed scanner could give false alarms for systems with updated openssl package.

In looking at this issue I'm wondering if one of my systems is impacted as it is not clear what testing was done for Redhat to state that 6.4 is not impacted. In our instance we are using squid as a reverse proxy. The rpm download did not support ssl so we had to compile it with ssl option.

Would this be impacted?

Regards

@Norman: Check step 2 above. It has links to scanners you can use -- if TLS queries against your server report it as vulnerable, you can use the lsof or grep commands (also in step 2) to confirm that it's your custom version of squid (or whatever else).

I have a proxy server where it directs all access to other webservers. It's like A>X1,X2,X3. I have updated A to the current openssl version. but unfortunately after scanning, it is still vilnerable.