Satellite 5 and the HTTPD Whitespace Problem

Updated -

NOTE - if you would like to jump directly to the "how to fix" section, it's outlined in Proposed Corrective Workflow below, with steps to be applied in order.

Introduction

In 2010, a fix to the yum-rhn-plugin was introduced which contained a bug involving the whitespace between headers in the HTTP requests between a Red Hat Satellite or Proxy and its clients. The error was incorrect by the HTTP RFC standards, but (at the time) was harmless due to lenient whitespace processing in httpd.

In 2016, CVE-2016-8743 was made public concerning a potential security exploit involving this lenient processing. The subsequent tightening of httpd’s rules uncovered the latent issue in yum-rhn-plugin during automated testing leading up to the release of Red Hat Enterprise Linux (RHEL) 6.9 in early 2017.

This does not impact Satellite 6 or Red Hat Subscription Management (RHSM).

Implications

Correcting the rules of httpd resulted in communications from clients affected by the yum-rhn-plugin bug being rejected by Satellite and Proxy servers with updated httpd instances.

Providing a fix for the client is insufficient, since once httpd on the server is updated, the client can no longer call home to get the fix.

A configuration-change on the server is needed to “turn off” strict whitespace enforcement. The httpd update provided an option for doing so - however, the option to disable strict whitespace enforcement only exists in newer versions of httpd, and should only be enabled in versions of httpd which support it.

Symptoms and Workarounds

The specific symptom on client systems is a traceback caused by receiving an “HTTP Error 400” response from the Satellite or Proxy server. An example traceback might look like this:

# tail /var/log/up2date
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1685, in _getRepoXML
    self._loadRepoXML(text=self.ui_id)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1676, in _loadRepoXML
    return self._groupLoadRepoXML(text, self._mdpolicy2mdtypes())
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1650, in _groupLoadRepoXML
    if self._commonLoadRepoXML(text):
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1470, in _commonLoadRepoXML
    result = self._getFileRepoXML(local, text)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1248, in _getFileRepoXML
    size=102400) # setting max size as 100K
  File "/usr/share/yum-plugins/rhnplugin.py", line 399, in _getFile
    start, end, copy_local, checkfunc, text, reget, cache, size)
  File "/usr/share/yum-plugins/rhnplugin.py", line 501, in _noExceptionWrappingGet
    raise urlException
<class 'yum.Errors.RepoError'>: failed to retrieve repodata/repomd.xml from rhel-x86_64-server-7
error was [Errno 14] HTTP Error 400 - Bad Request

With the following showing on the server-side:

# tail /var/log/httpd/error_log
[Wed Feb 15 07:37:56 2017] [error] [client 10.10.10.10] request failed: error reading the headers

The initial workaround is to downgrade httpd to the previous version.

A configuration-option workaround is to add the following new httpd config file:

# echo “HTTPProtocolOptions unsafe” >> /etc/httpd/conf.d/123unsafe.conf; service httpd restart

Either of these changes turns off strict-enforcement until such time as all clients can be updated with the fix to yum-rhn-plugin.

Complications

The first priority for the fix is to insure that clients, no matter what their current state, can always call home to get fixes. This leads to the preferred option on the Satellite and/or Proxy being a configuration-file change with the above HTTPProtocolOptions unsafe set. As noted, however, that option is only available for the affected version of httpd, and later. Also, the change should optimally be correct for both Fedora and RHEL, to minimize distance between the upstream and downstream code bases.

The standard way to “protect” an httpd config option from specific versions of httpd, is via the <IfVersion> tag. So

<IfVersion >= 2.4.25>
  HttpProtocolOptions unsafe
</IfVersion>

should do the job for us (where 2.4.25 is the version of httpd released to Fedora with the strict-whitespace changes)

However - RHEL versions stay at the upstream version they were initially pulled from, with only the release field being updated (for RHEL6, for example, httpd is always version httpd-2.2.15). And <IfVersion> doesn’t understand release-differences.

Addressing this required the addition of a directive built into the RHEL6.9 version of httpd, _RH_HAS_HTTPPROTOCOLOPTIONS. With this directive available, we could protect older versions of httpd on Fedora using <IfVersion>, and protect older versions of httpd on RHEL by checking for the existence of the directive with <IfDefine>. The result is a set of configuration options like so:

<IfVersion >= 2.4.25>
  HttpProtocolOptions unsafe
</IfVersion>
<IfDefine _RH_HAS_HTTPPROTOCOLOPTIONS>
  HTTPProtocolOptions unsafe
</IfDefine>

Current State

README: Before applying errata, please thoroughly read and follow Proposed Corrective Workflow section below, since the order of steps matter.

Errata have been released to fix the client-side issue with yum-rhn-plugin for the following versions of RHEL:

Since the RHEL5 clients never received the whitespace patch, there is no need to patch them.

Errata have been released to address the server-side issue by providing a new httpd configuration file which turns off the strict-whitespace-enforcing behavior of httpd, that works for all RHEL6 installs of Red Hat Satellite and Proxy 5:

The whitespace-enforcing version of httpd has been released for RHEL 6 - please read "Proposed Corrective Workflow" section below for any potention action needed before applying this httpd errata.

The client-side fix is available as part of the RHEL 6.9 release.

The server-side fix is available as part of the Red Hat Satellite 5.8 release. However Satellite 5.8 customers still need to apply the client-side fix per steps below.

Installations on Red Hat Enterprise Linux 5 are unaffected by the problem, because the httpd fix was never provided for RHEL5. Also please NOTE that Red Hat Satellite/Proxy 5 running on RHEL 5 have reached End-of-Life: End-of-Life (EOL) notification for Red Hat Satellite 5 running on Red Hat Enterprise Linux 5

Spacewalk, the upstream for Satellite5, has had the backend-config-file released for Fedora24 and Fedora25, and RHEL6. Unfortunately, the version of httpd on RHEL7 has had the fix backported, but is still a lower version than that found in Fedora. Investigation into how to correctly handle the “Spacewalk installed on RHEL7 case” continues.

Proposed Corrective Workflow

In order to be able to use the newly-secured httpd(RHSA-2017:1721), with strict-whitespace-enforcement turned on, while still allowing all clients to be able to communicate with server, the Satellite user needs to execute the following steps in order:

HOW TO KNOW IF ACTION IS NEEDED:

You can confirm whether the version of Satellite you are running is affected.

You can do this via the rpm command

rpm -q spacewalk-backend

If the version of the spacewalk-backend package is lower than the below, please update using the yum command.

If the version of the spacewalk-backend package is greater than or equal to the below, no action is required.

  • Red Hat Satellite 5.7 (spacewalk-backend-2.3.3-48.el6sat)
  • Red Hat Satellite 5.6 for RHEL 6 (spacewalk-backend-2.0.3-44.el6sat)
  • Red Hat Satellite 5.6 for RHEL 5 (spacewalk-backend-2.0.3-44.el5sat)
  • Red Hat Satellite with Embedded Oracle 5.5 for RHEL 6 (spacewalk-backend-1.7.38-56.el6sat)
  • Red Hat Satellite with Embedded Oracle 5.5 for RHEL 5 (spacewalk-backend-1.7.38-56.el5sat)
  • Red Hat Satellite with Embedded Oracle 5.4 for RHEL 6 (spacewalk-backend-1.2.13-84.el6sat)
  • Red Hat Satellite with Embedded Oracle 5.4 for RHEL 5 (spacewalk-backend-1.2.13-84.el5sat)

Applying the Errata.

  1. Apply the Satellite and Proxy errata described above (Current State section) to the Satellite and Proxy server, in order to let all clients continue to connect. Satellite/Proxy 5.8 customers: skip this step and move on to Step 2 (the server-side fix is included in 5.8, however you still need to apply client-side steps below).
  2. Sync the latest content for the RHEL base channels in use on the Satellite. This will pick up the yum-rhn-plugin fix.
  3. Promote that fix (as carried in the client-side errata described above) through the system’s SOE.
  4. Update ALL CLIENTS with the fix.
  5. Update ALL KICKSTARTS to obtain the new yum-rhn-plugin as part of kickstarting
  6. Once ALL clients and kickstarts are updated, comment out the contents of /etc/httpd/conf.d/aa-spacewalk-server.conf and restart httpd on the Satellite and/or Proxy.

ORDER MATTERS - turning strict-enforcement back on, before insuring that all clients have the yum-rhn-plugin fix installed, will results in those clients being unable to call home for updates!

Timeline

  • 2010-01-25 Fix to yum-rhn-plugin working around urlgrabber issue implemented; fix includes subtly-incorrect whitespace issue in Satellite5 client-to-server HTTP communications
  • 2016-12-20 CVE-2016-8743, “httpd: Apache HTTP Request Parsing Whitespace Defects” made public
  • 2016-12-20 httpd 2.4.25 released to Fedora
  • 2017-01-13 Fix backported to RHEL6 for 6.9 release (BZ#1412974)
  • 2017-02-15 automated Satellite5 testing for RHEL6.9 discovers breakage (BZ#1422518)
  • 2017-02-17 bug in httpd patch for RHEL6 discovered (BZ#1423414)
  • 2017-03-01 through 2017-04-12 errata released to fix yum-rhn-plugin for various RHEL6 and RHEL7 variants
  • 2017-04-11 spacewalk-backend config-file fix released for Satellite5 and Proxy5 (5.4, 5.5, 5.6, 5.7)
  • 2017-04-14 Problem with Spacewalk installed on RHEL7 opened (BZ#1442477)
  • 2017-07-11 whitespace-enforcing version of httpd released for RHEL 6 (RHSA-2017:1721), having given Satellite customers time to apply the yum-rhn-plugin and spacewalk-backend errata (see "Proposed Corrective Workflow" section above before updating)

Comments