How often to update?

Latest response

I am a newbie systems admin.

At present I am in a heated discussion with a application vendor about how often I can update my RHEL 5.9 production server.

I dont have a test instance of the server and application so testing is not possible.

My opinion is that once every month is a safe bet? The server is web facing, and mission critical so security is important. I have had cases where some of the vendor apps stopped working after an update even though they state that they fully support RHEL 5.9.

I dont want to wait months as it increases the bulk of updates that has to be checked should an issue occur. The vendor states every 18 months (?? Whoa!) but I must make sure that all security issues is dealt with. So I take responsibility either way....
 

What is best practice and what makes sense on a RHEL server?

Regards

Responses

Hi Barrie,

There are a lot of factors that go into determining a patch schedule. Things like regulatory and compliance issues, maintenance windows and downtime, . A similar discussion was recently started by one of our technical account managers at https://access.redhat.com/node/72915. It's a good read and has some great links to documents that can give you some guidance.

From experience (although I'm trying not to generalise too much here) companies who actually implement patching policies tend to grade critical Errata as requiring production patching within 30 days of release and all other patches to be applied within 90 days of release.

A) It depends whether you agree with Red Hat as to what constitutes "Critical".  I tend to as I don't want to spend the time investigating each Errata released and come up with my own grading system.  IT Security departments may request you do this, but I tend to throw requests like that back over the fence at them.  If they want to re-grade Errata - it can come out of their own manpower and budget (they'll soon give up)

B) The 30-day schedule I've seen as low as 7-days and as high as 90 days.  Same for the 90-days for non-essential Errata in the example above.

On the other hand there are companies who talk a good game and come out with patching policies, but never look to implement them.  Time constraints, business users not happy to accept downtime in Dev, "we'll roll this critical Errata into the quarterly patch cycle next month".  Pick your excuse - I've heard more than a few.

I've even seen places where the gold build of RHEL is never patched.  The risk of patching was seen as greater than the risk of having unpatched systems (although this was born from a mistrust of patches when a patch for an OS from Redmond completely took out key systems for a few days).

There's always a risk with patching, but that's why you patch and test Errata through your Dev, Test, UAT, Staging systems before you go anywhere near Production . . . right?

D

Hi Barrie - I'll echo Duncan and Chris H's comments and add some of my own.  At the end of the day you need to review what your business drivers are and what your risk exposure is to determine what a good patching schedule is for you.  How much downtime can your business afford related to this system?  Some companies require "5 Nines" of system availability, others not.  I mentioned in a recent security talk I gave..."The customer facing ordering system probably deserves more attention than the internal pencil procurement app." 

I'll note right off the bat that you're a bit riskier that I'd advise with the lack of a Dev/Test server.  While you save some money by not having that sandbox to test things in, you open yourself up to total system unavailability if something should happen to that production box (hardware issue, patch problem, application failure/incompatibility).  You'll need to take extra precautions to ensure you are regularly backup up that prod system and validating the backups are good.  One misstep could make a very very long day for you or perhaps a RGE (Resume' Generating Event) if it goes really poorly.

So to cadence/frequency of patching, the customers I've interacted with are fairly varied, but most are performing full system updates on a Quarterly calendar with mechanisms in place to allow for quicker patching if critical patches or bugfixes are released that they must react to.  Sometimes management likes things neatly wrapped up so they'll mandate ALL servers (regardless of OS) get patched on the same schedule (typically monthly...read just after "Patch Tuesday").  I'd debate the necessity of this, but again, that's a business decision.  Just because everything runs on a "server" doesn't mean they intrinsically have the same risk profile and maintenance needs.

Oftentimes companies are compelled to patch more quickly for legal, compliance, or policy reasons.  Things like the PCI-DSS 2.0 are very prescriptive about how frequently you are required to patch.  Consult with your internal security team or your management about any legal or industry requirements you may be governed by. 

At the end of the day your business is the best indicator of how often you can/should patch.  Open a dialogue with those leaders, explain what you HAVE TO do and get their perspective for what they WANT YOU to do (I'll wager they'll initially say it never can be down, but once things are explained to them you'll come to a reasonable compromise).  You state this is web-facing and pretty important, so i'd always error on the side of caution and be as up-to-date as possible.  But back to my comment on your lack of a test environment and your statement you've seen apps stop workign after patching.....Yes, that's why you test patches on a non-prod system first so you understand how things are going to react.

As Duncan pointed out, you may not think a patch that's advertised as "Critical" is so in your environment.  As long as you have a process to review this items and make a risk-based judgement on them, it's OK to do that.  For example, you might not have a GUI on your server nor are you running Flash, so patches related to those items are completly irrelevant to you, /dev/null them and focus in on things that are in your environment.  It's a good idea to have a server baseline of all packages and 3rd-party tools you deploy at hand so you can quickly check if any given update is in scope for you.

Best of luck in keeping this server secured!

-CRob

Thanks for all the great comments! I am studying the docs and will discuss with my managers.

I do a data backup and verification everyday, but as we dont have a test server it is not possible to test the implications of patching. The vendor licence on a test server is outside our ability to afford, so I will be having hardware ready with the OS installed and ready if we have total failure on the production system. We have only recently started experiencing problems when doing RHEL patching. Historically it always went smooth as I presumed that the vendor was keeping up with errata etc. It seems I am getting ahead of them so I will cool down a bit and give them a chance to test the updates. By the way, they will be responsible for restoring their app in a disaster situation.

Regards

Barrie

I dunno that I'd soley blame "Redmond" for wariness about patches. If you've worked with any platform for long enough, you've gotten burnt at least once by a "bad patch" situation. I've had it happen on Solaris, IRIX, AIX and HP/UX (years before "Redmond" was more than a desktop issue). Likely the only reason I haven't had it happen on RHEL (and derivatives) is that places I've worked at that have used RHEL have usually run 60+ days behind on patches (and the RHEL derivatives I've used for personal web-hosting tend to run days to months behind RedHat's release-cycle).

And, while it's fine to say "should have caught it prior to production", it's also (traditionally) been a tad unrealistic. Until virtualization came about, it was frequently simply not practicable to have your pre-production even come close to 100% mimicing production.

Barrie,

have a look into yum yum-plugin-security. It is a plugin to yum that allows you to apply only security-related updates skipping everything else.

You could run 'yum --security update'  daily, and apply all other updates monthly. 

Also it is important that you familiarize yourself with rolling back any updates ;) https://access.redhat.com/site/solutions/65856