RHUI: clients below RHEL 6.4 unable to download content if RHUI server is installed on or updated to RHEL 6.6+ with latest httpd/mod_ssl

Solution Verified - Updated -

Environment

  • Server (RHUA or CDS) runnig Red Hat Enterprise Linux 6.6 or newer: either new RHUI installation on RHEL 6.6+, or existing CDS/RHUA updated to latest httpd/mod_ssl
    • mod_ssl-2.2.15-39.el6.x86_64
    • httpd-2.2.15-39.el6.x86_64
  • Client (client instance or even a CDS syncing from RHUA) running RHEL 6.4 or earlier
    • NSS RPMs below 3.16
  • Using a SSL Certificate private key greater than 2048 bits

Issue

  • Installed RHUI on RHEL 6.6+, getting SSL error / cannot yum update
  • Upgraded CDS/RHUA to RHEL 6.6+ or latest httpd/mod_ssl packages, now RHEL 6.4 or older client instances cannot form a SSL connection using curl/nss to the server, yum cannot to download content
  • Sample curl failure:
curl: (35) SSL connect error
  • Sample logs from RHEL 6.4 server which had httpd/mod_ssl upgraded:
==> /var/log/httpd/ssl_error_log <==
[Wed Nov 05 17:51:00 2014] [info] [client 1.2.3.4] Connection to child 0 established (server 127.0.0.1:443)
[Wed Nov 05 17:51:00 2014] [info] Seeding PRNG with 144 bytes of entropy
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_kernel.c(1845): OpenSSL: Handshake: start
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_kernel.c(1853): OpenSSL: Loop: before/accept initialization
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1924): OpenSSL: read 11/11 bytes from BIO#7f33a29266b0 [mem: 7f33a2949fa0] (BIO dump follows)
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1857): +-------------------------------------------------------------------------+
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1896): | 0000: 16 03 01 00 80 01 00 00-7c 03 01                 ........|..      |
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1902): +-------------------------------------------------------------------------+
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1924): OpenSSL: read 122/122 bytes from BIO#7f33a29266b0 [mem: 7f33a2949fae] (BIO dump follows)
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1857): +-------------------------------------------------------------------------+
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1896): | 0000: 54 5a a9 d4 90 ed 1a 01-d0 6e f7 9a f4 83 a9 12  TZ.......n...... |
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1896): | 0010: 8c 4b ae 17 1f a9 83 ae-59 c8 aa 4f 98 bd 06 91  .K......Y..O.... |
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1896): | 0020: 00 00 20 00 ff 00 39 00-38 00 35 00 33 00 32 00  .. ...9.8.5.3.2. |
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1896): | 0030: 04 00 2f fe ff 00 0a fe-fe 00 09 00 64 00 62 00  ../.........d.b. |
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1896): | 0040: 03 00 06 01 00 00 33 00-00 00 2f 00 2d 00 00 2a  ......3.../.-..* |
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1896): | 0050: 72 68 75 69 32 2d 63 64-73 30 31 2e 65 75 2d 63  rhui2-cds01.eu-c |
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1896): | 0060: 65 6e 74 72 61 6c 2d 31-2e 61 77 73 2e 63 65 2e  entral-1.aws.ce. |
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1896): | 0070: 72 65 64 68 61 74 2e 63-6f 6d                    redhat.com       |
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1902): +-------------------------------------------------------------------------+
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_kernel.c(1972): [client 1.2.3.4] No matching SSL virtual host for servername rhui2-cds01.eu-central-1.aws.ce.redhat.com found (using default/first virtual host)
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_kernel.c(1863): OpenSSL: Write: SSLv3 read client hello C
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_kernel.c(1853): OpenSSL: Loop: SSLv3 read client hello A
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_kernel.c(1853): OpenSSL: Loop: SSLv3 write server hello A
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_kernel.c(1853): OpenSSL: Loop: SSLv3 write certificate A
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_kernel.c(1237): [client 1.2.3.4] handing out built-in DH parameters for 4096-bit authenticated connection
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_kernel.c(1853): OpenSSL: Loop: SSLv3 write key exchange A
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_kernel.c(1853): OpenSSL: Loop: SSLv3 write server done A
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_kernel.c(1853): OpenSSL: Loop: SSLv3 flush data
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_io.c(1935): OpenSSL: I/O error, 5 bytes expected to read on BIO#7f33a29266b0 [mem: 7f33a2949fa3]
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_kernel.c(1882): OpenSSL: Exit: error in SSLv3 read client certificate A
[Wed Nov 05 17:51:00 2014] [debug] ssl_engine_kernel.c(1882): OpenSSL: Exit: error in SSLv3 read client certificate A
[Wed Nov 05 17:51:00 2014] [info] [client 1.2.3.4] (70014)End of file found: SSL handshake interrupted by system [Hint: Stop button pressed in browser?!]
[Wed Nov 05 17:51:00 2014] [info] [client 1.2.3.4] Connection closed to child 0 with abortive shutdown (server 127.0.0.1:443)

Resolution

This is a known issue. Until long-term fix is available, there are 2 equally valid workarounds (customer can choose one, it's not necessary to do both) below:

  • Client-side Only Workaround
    • Upgrade to NSS 3.16 on client system

or:

  • Server-side Only Workaround
    • Use a key of 2048 bits or smaller
    • Steps to append a new key to an existing SSL Certificate
      1) Determine the SSL Certificate httpd is configured with
      View: /etc/httpd/conf.d/ssl.conf
      Look for SSLCACertificateFile /etc/pki/cds/rhua-ssl-ca-cert.crt
      2) Create a new 1024 sized key
      ~~~
      openssl dhparam 1024 -out dhp1024.pem
      ~~~
      3) Append this new key to the existing certificate file
      ~~~
      cat dhp1024.pem >> /etc/pki/cds/rhua-ssl-ca-cert.crt
      ~~~
      4) Restart httpd

Root Cause

  • RHEL 6.6 introduced a change to httpd/mod_ssl which breaks some functionality in older RHEL clients
  • Issue is seen with HTTP SSL keys greater than 2048 bits

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.