A fix to tighten up HTTPD's white space enforcement uncovered a bug in the yum-rhn-plugin that would prevent affected clients from being able to call home to Satellite5 or RHN/Hosted. This article describes the problem and its timeline, the fixes, and the current state of affairs.
Table of Contents
NOTE - if you would like to jump directly to the "how to fix" section, it's outlined in Proposed Corrective Workflow below, with steps to be applied in order.
In 2010, a fix to the
yum-rhn-plugin was introduced which contained a bug involving the whitespace between headers in the HTTP requests between a Red Hat Satellite or Proxy and its clients. The error was incorrect by the HTTP RFC standards, but (at the time) was harmless due to lenient whitespace processing in
In 2016, CVE-2016-8743 was made public concerning a potential security exploit involving this lenient processing. The subsequent tightening of
httpd’s rules uncovered the latent issue in
yum-rhn-plugin during automated testing leading up to the release of Red Hat Enterprise Linux (RHEL) 6.9 in early 2017.
This does not impact Satellite 6 or Red Hat Subscription Management (RHSM).
Correcting the rules of
httpd resulted in communications from clients affected by the
yum-rhn-plugin bug being rejected by Satellite and Proxy servers with updated
Providing a fix for the client is insufficient, since once
httpd on the server is updated, the client can no longer call home to get the fix.
A configuration-change on the server is needed to “turn off” strict whitespace enforcement. The
httpd update provided an option for doing so - however, the option to disable strict whitespace enforcement only exists in newer versions of
httpd, and should only be enabled in versions of
httpd which support it.
Symptoms and Workarounds
The specific symptom on client systems is a traceback caused by receiving an “HTTP Error 400” response from the Satellite or Proxy server. An example traceback might look like this:
# tail /var/log/up2date File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1685, in _getRepoXML self._loadRepoXML(text=self.ui_id) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1676, in _loadRepoXML return self._groupLoadRepoXML(text, self._mdpolicy2mdtypes()) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1650, in _groupLoadRepoXML if self._commonLoadRepoXML(text): File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1470, in _commonLoadRepoXML result = self._getFileRepoXML(local, text) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1248, in _getFileRepoXML size=102400) # setting max size as 100K File "/usr/share/yum-plugins/rhnplugin.py", line 399, in _getFile start, end, copy_local, checkfunc, text, reget, cache, size) File "/usr/share/yum-plugins/rhnplugin.py", line 501, in _noExceptionWrappingGet raise urlException <class 'yum.Errors.RepoError'>: failed to retrieve repodata/repomd.xml from rhel-x86_64-server-7 error was [Errno 14] HTTP Error 400 - Bad Request
With the following showing on the server-side:
# tail /var/log/httpd/error_log [Wed Feb 15 07:37:56 2017] [error] [client 10.10.10.10] request failed: error reading the headers
The initial workaround is to downgrade
httpd to the previous version.
A configuration-option workaround is to add the following new
httpd config file:
# echo “HTTPProtocolOptions unsafe” >> /etc/httpd/conf.d/123unsafe.conf; service httpd restart
Either of these changes turns off strict-enforcement until such time as all clients can be updated with the fix to
The first priority for the fix is to insure that clients, no matter what their current state, can always call home to get fixes. This leads to the preferred option on the Satellite and/or Proxy being a configuration-file change with the above
HTTPProtocolOptions unsafe set. As noted, however, that option is only available for the affected version of
httpd, and later. Also, the change should optimally be correct for both Fedora and RHEL, to minimize distance between the upstream and downstream code bases.
The standard way to “protect” an
httpd config option from specific versions of
httpd, is via the
<IfVersion> tag. So
<IfVersion >= 2.4.25> HttpProtocolOptions unsafe </IfVersion>
should do the job for us (where 2.4.25 is the version of
httpd released to Fedora with the strict-whitespace changes)
However - RHEL versions stay at the upstream version they were initially pulled from, with only the release field being updated (for RHEL6, for example,
httpd is always version
<IfVersion> doesn’t understand release-differences.
Addressing this required the addition of a directive built into the RHEL6.9 version of
_RH_HAS_HTTPPROTOCOLOPTIONS. With this directive available, we could protect older versions of
httpd on Fedora using
<IfVersion>, and protect older versions of
httpd on RHEL by checking for the existence of the directive with
<IfDefine>. The result is a set of configuration options like so:
<IfVersion >= 2.4.25> HttpProtocolOptions unsafe </IfVersion> <IfDefine _RH_HAS_HTTPPROTOCOLOPTIONS> HTTPProtocolOptions unsafe </IfDefine>
README: Before applying errata, please thoroughly read and follow Proposed Corrective Workflow section below, since the order of steps matter.
Errata have been released to fix the client-side issue with
yum-rhn-plugin for the following versions of RHEL:
Since the RHEL5 clients never received the whitespace patch, there is no need to patch them.
Errata have been released to address the server-side issue by providing a new
httpd configuration file which turns off the strict-whitespace-enforcing behavior of
httpd, that works for all RHEL6 installs of Red Hat Satellite and Proxy 5:
The whitespace-enforcing version of
httpd has been released for RHEL 6 - please read "Proposed Corrective Workflow" section below for any potention action needed before applying this httpd errata.
The client-side fix is available as part of the RHEL 6.9 release.
The server-side fix is available as part of the Red Hat Satellite 5.8 release. However Satellite 5.8 customers still need to apply the client-side fix per steps below.
Installations on Red Hat Enterprise Linux 5 are unaffected by the problem, because the
httpd fix was never provided for RHEL5. Also please NOTE that Red Hat Satellite/Proxy 5 running on RHEL 5 have reached End-of-Life: End-of-Life (EOL) notification for Red Hat Satellite 5 running on Red Hat Enterprise Linux 5
Spacewalk, the upstream for Satellite5, has had the backend-config-file released for Fedora24 and Fedora25, and RHEL6. Unfortunately, the version of
httpd on RHEL7 has had the fix backported, but is still a lower version than that found in Fedora. Investigation into how to correctly handle the “Spacewalk installed on RHEL7 case” continues.
Proposed Corrective Workflow
In order to be able to use the newly-secured
httpd(RHSA-2017:1721), with strict-whitespace-enforcement turned on, while still allowing all clients to be able to communicate with server, the Satellite user needs to execute the following steps in order:
HOW TO KNOW IF ACTION IS NEEDED:
You can confirm whether the version of Satellite you are running is affected.
You can do this via the
rpm -q spacewalk-backend
If the version of the
spacewalk-backend package is lower than the below, please update using the
If the version of the spacewalk-backend package is greater than or equal to the below, no action is required.
- Red Hat Satellite 5.7 (spacewalk-backend-2.3.3-48.el6sat)
- Red Hat Satellite 5.6 for RHEL 6 (spacewalk-backend-2.0.3-44.el6sat)
- Red Hat Satellite 5.6 for RHEL 5 (spacewalk-backend-2.0.3-44.el5sat)
- Red Hat Satellite with Embedded Oracle 5.5 for RHEL 6 (spacewalk-backend-1.7.38-56.el6sat)
- Red Hat Satellite with Embedded Oracle 5.5 for RHEL 5 (spacewalk-backend-1.7.38-56.el5sat)
- Red Hat Satellite with Embedded Oracle 5.4 for RHEL 6 (spacewalk-backend-1.2.13-84.el6sat)
- Red Hat Satellite with Embedded Oracle 5.4 for RHEL 5 (spacewalk-backend-1.2.13-84.el5sat)
Applying the Errata.
- Apply the Satellite and Proxy errata described above (Current State section) to the Satellite and Proxy server, in order to let all clients continue to connect. Satellite/Proxy 5.8 customers: skip this step and move on to Step 2 (the server-side fix is included in 5.8, however you still need to apply client-side steps below).
- Sync the latest content for the RHEL base channels in use on the Satellite. This will pick up the
- Promote that fix (as carried in the client-side errata described above) through the system’s SOE.
- Update ALL CLIENTS with the fix.
- Update ALL KICKSTARTS to obtain the new
yum-rhn-pluginas part of kickstarting
- Once ALL clients and kickstarts are updated, comment out the contents of
httpdon the Satellite and/or Proxy.
ORDER MATTERS - turning strict-enforcement back on, before insuring that all clients have the
yum-rhn-plugin fix installed, will results in those clients being unable to call home for updates!
- 2010-01-25 Fix to
urlgrabberissue implemented; fix includes subtly-incorrect whitespace issue in Satellite5 client-to-server HTTP communications
- 2016-12-20 CVE-2016-8743, “httpd: Apache HTTP Request Parsing Whitespace Defects” made public
httpd2.4.25 released to Fedora
- 2017-01-13 Fix backported to RHEL6 for 6.9 release (BZ#1412974)
- 2017-02-15 automated Satellite5 testing for RHEL6.9 discovers breakage (BZ#1422518)
- 2017-02-17 bug in
httpdpatch for RHEL6 discovered (BZ#1423414)
- 2017-03-01 through 2017-04-12 errata released to fix
yum-rhn-pluginfor various RHEL6 and RHEL7 variants
spacewalk-backendconfig-file fix released for Satellite5 and Proxy5 (5.4, 5.5, 5.6, 5.7)
- 2017-04-14 Problem with Spacewalk installed on RHEL7 opened (BZ#1442477)
- 2017-07-11 whitespace-enforcing version of
httpdreleased for RHEL 6 (RHSA-2017:1721), having given Satellite customers time to apply the
spacewalk-backenderrata (see "Proposed Corrective Workflow" section above before updating)