[Moved] a-PATCH-aclypse Now (or The Heart of Patching Darkness)
[Note: The Groups area will soon be archived. This topic has been moved to the new community discussion area, so head over and check it out.]
"The horror....the horror..."
I've got a customer going through a major upgrade/conversion and afterwards hopes to keep a regular cadence for updates to keep their systems shiny. Some companies are compelled to roll out updates frequently for legal/regulatory reasons, others do it just as a matter of good common sense these days, some others...well.....
Patching and vulnerability management are two sides of the same coin that ultimately helps your organization manage their Risk profile. Every business is different and has different tolerances to what downtime they can take or how frequently things can/should change and how all of that gets documented and executed. In this day of always-on connectivity not-patching and maintaining your systems is not a viable option (unless you really want to experience a data breach and lose vital information to "the internets").
As a general rule of thumb, from what I've seen, most big companies have moved to a quarterly patching cycle for their *NIX-based systems, but they all have plans in place to account for that "zero-day" earth-shattering major super-super-critical issue that if they need to they can react more quickly and get vital fixes deployed sooner. Some of the more highly-regulated places are moving towards monthly patching for *NIX to match the already hectic pace required for other Enterprise Operating Systems.
Patching is not an endeavour to be taken lightly; it takes time to review what's being changed and assess the impact of those changes on production-like systems (note: NOT Production. Never ever ever ever never ever ever test a new patch or config straight on your production environment unless you're LOOKING to create a RGE (resume-generating event) ["Hey Bob, it's a funny thing you decided to try out that undocumented change on the online customer system on your last day here at work."]. Patching and remediating vulnerabilities is a very large cost in terms of human capital and person-hours. There are a lot of ways to get your patches out to where they need to go and understand where you need to apply your efforts: Red Hat Satellite, Cobbler, Puppet and a whole battery of FOSS and 3rd-Party products.
For those interested in developing a patching program for their very own we provide some guidance:
What Patching Policy Should I Adopt For My RHEL Servers?
https://access.redhat.com/site/solutions/64094
Now ideally 100% of your patching goes exactly as planned, but sometimes it doesn't or someways it won't, so this article is nice to have in your back pocket:
What strategies can be used to rollback updates containing many packages, maybe different spanning different RHEL versions?
https://access.redhat.com/site/solutions/65856
or
How can I use yum to downgrade or rollback some package updates?
https://access.redhat.com/site/solutions/29617
So let us know how you are patching? Why are you patching? How often? Are you using some great tool or have you developed your own super-scripting mechanism? Do you have any really great results or totally EPIC fails you'd like to share with us? The lessons from yesterday you share can help others in the Community today!
