a-PATCH-aclypse Now (or The Heart of Patching Darkness)

Latest response

"The horror....the horror..."
I've got a customer going through a major upgrade/conversion and afterwards hopes to keep a regular cadence for updates to keep their systems shiny. Some companies are compelled to roll out updates frequently for legal/regulatory reasons, others do it just as a matter of good common sense these days, some others...well.....

Patching and vulnerability management are two sides of the same coin that ultimately helps your organization manage their Risk profile. Every business is different and has different tolerances to what downtime they can take or how frequently things can/should change and how all of that gets documented and executed. In this day of always-on connectivity not-patching and maintaining your systems is not a viable option (unless you really want to experience a data breach and lose vital information to "the internets").

As a general rule of thumb, from what I've seen, most big companies have moved to a quarterly patching cycle for their *NIX-based systems, but they all have plans in place to account for that "zero-day" earth-shattering major super-super-critical issue that if they need to they can react more quickly and get vital fixes deployed sooner. Some of the more highly-regulated places are moving towards monthly patching for *NIX to match the already hectic pace required for other Enterprise Operating Systems.

Patching is not an endeavour to be taken lightly; it takes time to review what's being changed and assess the impact of those changes on production-like systems (note: NOT Production. Never ever ever ever never ever ever test a new patch or config straight on your production environment unless you're LOOKING to create a RGE (resume-generating event) ["Hey Bob, it's a funny thing you decided to try out that undocumented change on the online customer system on your last day here at work."]. Patching and remediating vulnerabilities is a very large cost in terms of human capital and person-hours. There are a lot of ways to get your patches out to where they need to go and understand where you need to apply your efforts: Red Hat Satellite, Cobbler, Puppet and a whole battery of FOSS and 3rd-Party products.

For those interested in developing a patching program for their very own we provide some guidance:

What Patching Policy Should I Adopt For My RHEL Servers?

Now ideally 100% of your patching goes exactly as planned, but sometimes it doesn't or someways it won't, so this article is nice to have in your back pocket:

What strategies can be used to rollback updates containing many packages, maybe different spanning different RHEL versions?

or

How can I use yum to downgrade or rollback some package updates?

So let us know how you are patching? Why are you patching? How often? Are you using some great tool or have you developed your own super-scripting mechanism? Do you have any really great results or totally EPIC fails you'd like to share with us? The lessons from yesterday you share can help others in the Community today!

Responses

Thanks for the write up. However, I can't seem to access the last link you provided:
https://access.redhat.com/site/solutions/29617

We are currently going through some changes regarding our patch policy. We are striving to patch every RHEL system every 30 days. I do this all using a Python script that talks to Satellite's API. We run 30 days behind for testing purposes so if a system is patched in October it won't receive errata released after September 30th. Furthermore, I filter all errata down to Security only. We are not applying bug fixes or enhancements. Although, I'm curious how common this is? Do most organizations focus strictly on RHSA Errata? Are there issues that come from only installing RHSA's?

Thanks again!

Chris, looks like the solution you mentioned was unpublished in error. It should be available now.

Hey Chris. Yes, you certainly can only install security updates. The following article tells you how you can do that through yum:
Is it possible to limit yum so that it lists or installs only security updates?

Yes, people do just do this. It's not the majority of folks out there, but there are enough doing it that I wouldn't classify it as "rare". The security piece gets you past your compliance obligations, but what you're missing out on are bugfixes and (and probably less desirable for you) new features/enhancements.

With any kind of change to your environment, every patch should be thoroughly tested, but doing what you're doing (which may be just fine in your particular business circumstance) leaves you a bit exposed as problems get corrected. I'd advise you perhaps quarterly to review all the patches you didn't push out monthly to see if you could benefit from the bugfixes that are released as well.

Thanks for the reply!

Almost all of our servers house some kind of home grown application so minimizing the amount of change/risk was the goal for limiting to only security errata. I imagine we will have a quarterly discussion about which "out of band" patches to apply. Thanks for your help.

Patching: also known as "why I love the ability to do snapshot restores of systems"

Yes. I've only recently discovered this functionality in Satellite so I'm trying to see how we can use it to our advantage. Sounds great so far.