Sysadmins: How do you manage errata, and how can we make it easier?

Latest response

Hi all,

I've noticed a few threads mentioning issues with errata management and notifications recently, so I'd like to start up a thread to cover the topic.

I'd like to get an idea of how you're using these systems, and what you'd like to see improve in your experience managing errata.

Any comments on these systems are welcome, but here are some questions to start with:

  • Which system do you use to manage your errata ("Classic" (RHN) management, Red Hat Subscription Management (RHSM) or Satellite?
  • Do you review errata notifications as part of your workflow?
  • Do you receive too many?
  • Would you prefer to receive more targeted errata notifications based on system information you provide, or would you rather review and decide for yourself?
  • What features could we add to make errata management easier for you?

Responses

Hello,

We use satellite 5.6 to manage our errata, we receive a lot of notifications from satellite which it tells you which system is candidate for that errrata however since you could take on demand reports we dont give a lot of importance to those notifications.
Im pretty happy with the way satellite 5.6 allow us to handle the errata.

Pleased to hear Satellite is working for you, Ivan.

Hi,
We have just started use Classic RHN and receive the errata for our RHEL/RHEV environment.
Reviewing the errata is planned to be part of our daily/weekly workflow.However, I find it to be too detailed and too many at a time.It would definitely help to be more targeted and categorized as per our type of systems.I believe partners should assist more in this regard. The subject line on errata alert email should also indicate if it should be applied Urgent/Critical like security updates that may compromise the system.

Great feedback - I appreciate it, Neville.

Thanks for your comment, Neville! Wanted to point out to you that for security advisories via RHN, the notification email does features a rating of "Low," "Moderate," or "Important." This can be seen in the subject line of the notification email, as well as in the Summary section of the email itself.

Would love to also hear more of your thoughts on how these could be more targetted or how you might like them categorized. Could you elaborate on what you mean when you say "type" of system?

Currently using Satellite 5.4, to provide a "gatekeeper" function similar to WSUS in the Windows environment. The two key functions being to be sure we apply the patches to test/dev system before production, and that the patches applied to prod are exactly the same (regardless of what Red Hat may have released in the days between test & prod patching runs).

Errata notices: the big "point upgrades" (e.g. 5.9 to 5.10 or 6.4 to 6.5) are overwhelming, and generally not studied in much detail unless something goes wrong on the test/dev systems. The normal stream of security/bug-fix patches is not a big deal.

I have some concerns about Satellite 6.0 not providing functionality equivalent to RHNSat 5.x (mostly the "Systems" view, with a concise list of systems & their outstanding-errata state/errata count/"channel" membership (i.e., major version & locally-cloned vs. Red Hat default channel membership)). I can find screens that list multiple systems, or show a single system in (sometimes excessive) detail, but not both at the same time.

James,

With regards to reporting on errata within Satellite 6, I've opened the following BZ (https://bugzilla.redhat.com/show_bug.cgi?id=1126570) addressing this concern.

So - I have a few things that come to mind (and I anticipate I would be updating this list over time)
My issues might be my own lack of understanding ;-)

  • I would like if errata, as a general rule, was date-based. For example, if I want to sync a system to a point-in-time that ALL the packages that are XYZ-compliant will align with that date. It seems that now Red Hat packages respect a date-release whereas other vendors do not. (This example is specific to an issue I have with spacewalk-clone-by-date - all of the Red Hat provided packages are fine, but the 3rd-party vendor packages do not seem to follow the same standard and therefore either none get cloned, or ALL get cloned - regardless of date)

  • It woudl be cool if Errata packages had a "reboot" flag - I do not know of an easy way to identify whether a package will require a reboot once applied, or restart a service.

James,

What version of Satellite are you running? Satellite 5.6 '...now scans all published errata for the reboot_required keyword and will display systems which have applied the errata but have not rebooted' [1]. This is true of both Red Hat errata & custom errata. If the errata has the reboot_required set, there is a new page in Satellite 5.6 that shows which systems have applied it, but haven't rebooted.

Also, rememeber that tools such as spacewalk-clone-by-date work on errata, not packages. If you are ingesting 3rd party packages such as via spacewalk-repo-sync, you probably wont have any associated errata with them. You can create your own errata [2], and clone-by-date will honor those dates. (Or you can ask your ISVs to do this, and everyone benefits :) )

[1] - https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/5.6/html-single/Release_Notes/index.html#chap-Red_Hat_Satellite-Release_Notes-Major_Corrections
[2] - https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/5.6/html-single/Getting_Started_Guide/index.html#sect-Getting_Started_Guide-Errata_Management-Creating_and_Editing_Errata

We are, indeed, running 5.6 - for almost a week now ;-) I will look in to the reboot indicator.
I need to educate myself on the difference between packages and errata (in the context of this conversation). The specific example that comes to mind is the oracleasm package which I add to my base channel, but it does not get updated when I "reclone" with a new date. I have not review my EPEL channels closely.

Thanks for the updates.

Satellite Server 5.4.1

I've been asked to produce a report on outstanding patches by server. The easiest way is with running a "spacewalk-report" and then collating the details for each client. The problem is that I've been asked to show the oldest outstanding patch per-client by the date it was issued. I can only find the year of issue, and currently have to then look up the outstanding patch on the RH internet site to confirm the dates.

We're miles out on some servers and they will be addressed, but it's those that are missed off regular patching that need to be highlighted.

Thanks, in advance,
Robin

Have you looked at the OVAL files that Red Hat produce?

I have used these for reporting purposes in the past, and they include the 'issued date' of the advisory.

http://www.redhat.com/security/data/oval/

That OVAL data can be leveraged via OpenSCAP [1][2] to perform errata reporting.

also, you can leverage Satellite's API to grok this information, namely the system.getRelevantErrata call [3]if you are up for a little scripting.

[1] - https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/5.6/html-single/User_Guide/index.html#chap-Red_Hat_Satellite-User_Guide-Maintaining_System_Security_Using_OpenSCAP
[2] - http://www.open-scap.org/page/Documentation#How_to_run_vulnerability_scan_on_Red_Hat_Enterprise_Linux

[3] - https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/5.6/html-single/API_Overview/index.html#sect-system-getRelevantErrata

I really wish Red Hat would revisit this bugzilla (feature request):
https://bugzilla.redhat.com/show_bug.cgi?id=677296

ie. Being able to instruct yum to only download packages up to a certain date.

I know the bug comments state that there are issues because the date the patch is added to the repo isn't captured so yum can't leverage it, but surely something can be worked out / incorporated? perhaps with a plugin that gathers this information from Satellite?

3 years on from this bug being closed and I still frequently see this causing issues in Red Hat environments, with the general workaround to 'clone by date' on the Satellite side, which is a tedious task when maintaining multiple environments.

If someone can suggest a cleaner alternative I would love to hear it, because I have seen this solved the same way everywhere I go.

Scenario: You want to patch your dev/test/prod environments to the same patch level so testing carried out in development validates patches to be applied to test, and then production.

Often, the change process imposes restrictions / delays on patch release through environments so the patches can be tested and regression checked. I accept it isn't the most ideal process, but it is common. I am not interested in suggestions on how to change this business process, but more interested in how the available Red Hat tools can be used to enhance the process.

  1. Week 1 (1st of August). Patch development
  2. Week 2 (8th of August). Patch test
  3. Week 3 (15th of August). Patch production

Scenario 1: Patching to latest available version
Week 1 you patch all outstanding updates to development. Red Hat then release errata on August 5th, so patching to test to the latest available version results in a patch that has not been tested being applied to test and then production. Any patches released between the 2nd and the 14th will go into production without first going through development.

Scenario 2: Clone by date in Satellite
Week 1 you clone the available Red Hat updates to a channel called eg.. '1st of August'. Subscribe development servers to this channel and patch. On the 8th of August subscribe test to the '1st of August' channel and patch. On the 15th of August subscribe the production channel to the '1st of August' channel and patch.

I see this method work, but then you run into issues with cloning future errata to the '1st of August' channel and you end up making a lot of work for yourself to keep the different channel 'snapshots' maintained. One way to solve the errata maintenance issue is to move these servers back to the main OS channels after they have been patched with the cloned channel.

Scenario 3: Yum transaction log suggested in the Bugzilla above
Week 1 you patch the development environment to the latest available patches and store a yum transaction log of this process. On the 8th you run this transaction log in the test environment. On the 15th you run this transaction log in production. No cloning of channels is required in Satellite so no ongoing maintenance of cloned channels. The only concern I have with this approach is if there are any package discrepancies between environments, they may not be patched if the transaction log doesn't capture these packages in development. (eg. someone installed curl in test, but curl doesn't exist in development, so isn't capture in
the transaction log for patching)

There is mention of this patch being supplied to upstream but I haven't followed it up. Can anyone confirm if/when this made its way into the Red Hat shipped yum packages?

Scenario 4: If yum supported '--update-to-date'
Week 1, patch your environment with 'yum --update-to-date 20140801'
Week 2, patch your environment with 'yum --update-to-date 20140801'
Week 3, patch your environment with 'yum --update-to-date 20140801'

This way you ensure that only patches released up to and including the date of deployment into the development environment (1st of August) are deployed to test and production. Satellite channels don't need to be created so additional ongoing management burden of additional channels isn't created. The errata problem doesn't exist because the servers are still subscribed to the primary OS channel receiving the errata updates.

If you have another method to approach this style of staged patching I would be keen to hear about it! If you use one of the above methods, I would appreciate if you dropped a message so I can see what others are doing :)

"If you have another method to approach this style of staged patching I would be keen to hear about it!"

Surely! :) Take a look at the absurdly long titled: 'RHN Satellite Channel Lifecycle Management with spacewalk-clone-by-date' guide [1]. It provides a method for doing staged/phased rollouts of content via Satellite, while providing the rigidity to ensure that errata isn't relased into Test (for example) that hasn't been released in Dev.

Overall, I think there is a somewhat philosophical discussion on whether errata management should happen server side or client side. Each has their pros & cons. I am a big fan on doing it server side because of the inherent economy of scale that you get.

[1] - https://access.redhat.com/articles/469173

Hi Richard,

Thanks for the response.

I have read this document in the past but the solution doesn't fit any environment I am currently working in. From what I understand, this is scenario 2 that I posted above, but instead of moving servers to channels, you have a channel for environments and update the errata sequentially in each of these channels.

The major problem I have with this (please correct me if I am wrong) is that, as these are 'clone' channels, and only have errata up to a certain date, there is no easy way in Satellite to see which errata is outstanding for these channels after the cloned date.

For example, in the examples there is a channel per environment. If I 'clone by date' the development environment to the 1st of August, everything in that channel will then patch to the 1st of August. But after the 1st of August, how do I get visibility of what errata is outstanding in the development channel?

ie. How do I answer the logical business question on August 15th "What errata/advisories are currently outstanding in development?"

This document provides a great example method for patching if you patch on a fixed cycle (eg. monthly or quarterly) because it doesn't need to provide outstanding errata for those channels between patching cycles.

I am interested in seeing an example method that applies to environments that use a trigger of specific outstanding advisories to release patches (eg. start the patching cycle through environments if there is an advisory outstanding with a 'high' risk level).

I agree that managing from the server side is preferred, especially when scaling out.. I just think (especially for smaller environments) having the capability to restrict / filter packages at the client side would also be useful.

I have tried several methods thus far:
* (legacy method) move the clients to a channel based on the environment, and then sync the channel when it's time to update - we ultimately did not like this method
* (current method) make a clone every quarter and then migrate clients to that particular clone. Once I figured out how to easily create the clones, activation keys (with config channels included), update my bootstrap to include all the clones as a selectable option ANNNNNNDDD migrate the hosts between the clones, we like this method ;-)

ie. How do I answer the logical business question on August 15th "What errata/advisories are outstanding in development?"
spacecmd softwarechannel_diff rhel-x86_64-server-6 clone-20140401-rhel-x86_64-server-6 mostly provides that data. However, if your systems is already out-of-date with the clone and the clone is out-of-date with the Red Hat Channel - it seems like a lot of perl scripting to get an answer. I had not looked whether there is a spacecmd that would compare a client's current state to another channel that it is not subscribed to.

Cheers for the info James,

From what I understand of this, it's close to Scenario 2 I posted above, with clone channels for a patch level then hosts are moved between these to update (you just use a different frequency/delay for the move)?

Great answer re: diffing the 2 channels, my problem is this isn't visible in the Satellite interface. If your system is subscribed to the base OS channel you will essentially get 'live updates' of what is outstanding on that system, as soon as you move that system into a clone channel the process of determining outstanding errata becomes far more involved/difficult.

I often feel like I am working against/around Satellite when I want to get work done. I really hope the fresh approach in Satellite 6 addresses some of these issues.

Regarding "But after the 1st of August, how do I get visibility of what errata is outstanding in the development channel? ie. How do I answer the logical business question on August 15th "What errata/advisories are currently outstanding in development?""

Let's talk one small step backwards. With Satellite 5.x, a system's view of 'what it needed' is dependent upon which channel(s) is assigned to. This effectively means that there are two ways to look at errata:

  • Available - What would land on the system if I ran 'yum update'
  • Applicable - What errata apply to this system. (Which may or may not have been cloned yet)

When a system is subscribed to a Red Hat provided channel, its view of Available and Applicable errata are one in the same. When it is moved to a cloned channel, its view of Available and Applicable errata now differ (As the cloned channel is a point in time snapshot). Cloned Channels only affect [the systems view of] Available errata.

There are a couple of ways to answer the question you posed above of 'what errata do I need?':

  • OpenSCAP + OVAL data. Extra cookies if you use it with Satellite 5.6, but you don't have to. The OpenSCAP stuff works standalone too.
  • Comparing the errata of your base channel to your cloned channel. This can be easily done via the API (channel.software.listErrata) and a little set-theory.

Clone channels are great for locking in a specific set of content, but they do make reporting a little more challenging. The clone-by-date guide that I wrote provides both a date-driven and event-driven methodology towards patch management. We'd put the systems on a pre-determined periodic cycle to receive errata. That list of errata can be a liberal as 'everything' or more conservatively (and commonly) 'security_only' errata. This allows our systems to stay relatively current. And if a high priority errata is released before our next patching window, we can add just that specific errata (such as RHSA-2014:0376)

I personally didn't find this method very efficient. I think its strange that you need to resort to SCAP, API or command line in a tool that provides a UI for errata status.

This method works pretty well in environments where you have one set of systems you are moving through dev/test/prod but I haven't seen it scale nicely. If you add different base OS (RHEL 5) you go from 3 clone channels to 6. If you have servers with different configurations,zones or patching cycles (eg. DMZ hosts) you add another 3 channel clones (not to mention the child channels that have to also get copied around), it's such a 'heavy' way to solve the problem.

I wish the system would work more like 'tagging' in version control systems, where you tag a specific point in time and associate a group of systems with the tag. With this approach, no patches need to be replicated/copied or other channels created. Systems are still associated only with the single instance of the base channel so will get errata reports past the 'tagged' date. The repo could then only expose patches up to this 'tagged' date to the client so a yum update won't push past this specified point in time.

Dear Richard Jerrido and PixelDrift.NET,

Thanks for the responses and the very good suggestions. The problem I have with the levering OVAL in an automated way is that this is for Satellite Server 5.6 and we're only 5.4.1, but I suppose it gives me a stick to encourage an upgrade.

On the second point, I too am having fun with multiple clone channels. The problem about changing subscriptions is simply the number of servers gives a high risk of error. Scenario 4 would do away with this all together. An excellent suggestion.

It doesn't get round my auditors What's the oldest outstanding patch? question, but I would hope that this could also be incorporated, or I could at least script up a harness and log yum updates with an --update-to-date set, even syslog report it or write to some central audit file that they can look at any time without the chance of someone just fiddling the results when they ask the question.

Kind regards,
Robin

Robin,

I personally use the OVAL file with OpenSCAP outside of Satellite. I understand this may not be ideal for large numbers of hosts or if you want to keep your environment management in a single tool, but it can provide some excellent information.

http://www.open-scap.org/page/Documentation

The 'Scanning' heading has an OVAL example.

It may be worth trying this out first to confirm it provides the information you need/want and then you can commit to the Satellite upgrade!.

I personally use the OVAL file with OpenSCAP outside of Satellite.

I do a little of both. Reasoning: I was unable to generate a local report using the Satellite portal (it might have been my syntax). And the Audit Reporting seems a bit too summary. However, it's nice to have Satellite manage my files and job scheduling for the scans.
That said - I am fairly inexperienced when it comes to SCAP, in general, and even more so when trying to use Satellite to manage SCAP.
I have a feeling that we will be seeing a lot more from Red Hat in the future regarding SCAP and the coordination with Satellite - and I'm excited about that.

I wish that Satellite could easily display the point release that each server is at. (5.11, 6.5, etc). You can get this indirectly by using space-cmd and then matching the kernel levels to releases. I really wish the Satellite GUI would just display the release level as it would help us quickly determine which boxes are most out of date. (200+ patches for a 5.8 vs 5.9 box kind of appear similar in the current GUI!)

We patch the critical stuff as needed but past that our patching policy is based on point releases and the age of the release, with minimum supported levels. Seeing a number like 200+, 53, or whatever number of outstanding patches per system isn't helpful to us. In a large shop, most systems are going to be out of date and the number just ends up being arbitrary number rather than helpful indicator.

Native CentOS Errata support would be nice too. We have a RHEL site subscription but we have a number of CentOS systems left to migrate. Satellite is a paid product, so I think its fair to ask for the feature. ;) I've done the various hacks to import the CentOS errata, but native support really should be there as I'm sure there are other sites in similar situations.

Thanks for the feedback, Joshua! Would it also be helpful to be able to enable a notification (in-product and/or email) that would fire when a box got a certain number (specified by you) of point releases out of date?

I recently came up with a new approach to solve the problem of "which patches are outstanding for X hosts" which is problematic if you are using cloning methods because they are only cloned to a specific date (I have full details in a post above).

This solution meets the following requirements (which I have found quite common across sites):
1. Standard base installation 'snapshotted' at a specific date
2. Only security related errata is applied to these servers after the chosen snapshot date

Channel configuration is as follows:

-+-Base channel RHEL 6.x SNAPSHOT 201401010
 +---Child channel RHEL 6.x Security updates (updated nightly)

ie. The server is subscribed to two channels on build, the base channel that is a snapshot of the primary OS channel at a specific date and the security updates child channel which has all security updates.

Unfortunately, constructing this configuration with automated processing is a little painful because clone-by-data (which can filter for security only) doesn't appear to be able to target just a child channel, ie. I can't say "clone the security errata from the primary OS channel to the following child channel", and softwarechannel_adderrata is all errata, it can't be filtered for security only.

So the automated process is as follows:
1. Download nightly into primary OS channel
2. Use clone-by-date to clone only security errata into OS-Security (this is where the filtering of errata occurs)
3. Use spacecmd softwarechannel_adderrata to copy security only errata from the OS-Security channel into the child security channel of each snapshot

The most basic channel configuration to use this workflow looks like the following:

-+-Base channel RHEL 6.x OS (updated nightly from RHN)
-+-Base channel RHEL 6.x Security only (updated nightly from OS base channel)
-+-Base channel RHEL 6.x SNAPSHOT 201401010 (updated/created manually)
 +---Child channel RHEL 6.x Security updates (updated nightly from Security base channel)

I would really appreciate if either softwarechanne_adderrata or clone-by-date were updated to allow the cloning of security only errata directly into child channels from base channels for the above workflow... unless someone can suggest an alternative method to get this errata into the right locations?

Hi,

I have RedHat Satellite Ver 5.5 running and now i want to merge all the errata (close to 200) applicable to a single child channel and get it registered to all the clients. Is it possible...?
My understanding is, i need to create a child channel and clone the errata to the child channel created and later subscribed to all the clients. Is my understanding is correct..? Please help me.

Vijay, that is the approach I would take.

  1. Create child channel
  2. Clone Errata into child channel
  3. Modify host channel subscription to include the newly created child channel
  4. Update your activation keys to include the newly created child channel so new hosts are subscribed on creation

Thanks for the comment. Already we have a custom channel and can i use "spacewalk-clone-by-date" to clone the errata's till date to this custom channel..?

Also, how to remove the kernel errata(s) in the custom channel to avoid kernel being update while performing "yum update".

Thanks

So. In Satellite 6 it's not even possible to create your own errata. I've created a RFE for this.. Hopefully this mistake will be corrected ASAP. Using package filters for content views does not scale.

So this was back in March. Heard any more on this?

how do you determine that errata needs a reboot or not?

Mark,

There are a few complexities in answering the question, but there is some good information here:

https://access.redhat.com/solutions/27943

There is also an RFE to add this functionality detailed here:

https://access.redhat.com/solutions/432903

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.