Upgrading Satellite 5.4.1 to 5.5
I was looking for feedback from folks that have went through the process.
Is it relatively straight-forward?
Were there any gotcha's?
How long did the process take?
Did you complete the task with the first attempt?
Thanks!
Responses
Guru
6222 points
Hi James,
Plan to do this the coming days with my test Satellite.
Will keep you posted.
Kind regards,
Jan Gerrit Kootstra
Red Hat
Active Contributor
323 points
I did it 2 days ago -- It is pretty straightforward, just like previous major Satellite updates. I would highly reccomend taking a backup of the database beforehand. Make sure you read through the txt documents that are part of the rhn-upgrade package that the web link states below:
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Network_Satellite/5.5/html-single/Installation_Guide/index.html#sect-Installation_Guide-Upgrades-Procedure
These 'comprehensive' instructions in the rhn-upgrade package also have a few additional steps that are not mentioned on the documentation portal. A few that come to mind are the optimizer, gathering stats, etc. Dont forget to use db-control to make sure you have adequate tablespace before getting started (again, see txt documents in the rhn-upgrade package).
As far as how long, it really depends on the hardware and how big your database is. Start to finish it took me about 2 hours including the database backup and at least 45 mins of that was exporting and compressing the database files. This is on 3 or 4 year old hardware using kvm virtual machine with 16GB of RAM and 1 vcpu. My embedded database is pretty small, 10-15 GB. The satellite contains about 30-40 registered systems and approx 75GB of channel content.
EDIT: You may want to monitor this bug before you upgrade. https://bugzilla.redhat.com/show_bug.cgi?id=862051 Its nothing destructive and doesn't affect the upgrade itself, but syncs will take longer than they should. I expect a fix to be out relatively soon -- said patch will be delivered via RHN as always.
Red Hat
25369 points
Great topic, James... thanks for posting. Definitely interested to hear people's experiences with this.
Upgrade to RHN Satellite v 5.5 from satellite v 5.4.1 is simple do not have much complications.
Prerequisites:
- Take backup of current working state of RHN Satellite database.
- Make sure you have certificate for RHN Satellite v5.5 . (Contact Red Hat Technical Support to get it)
- Make sure latest version of rhn-upgrade package is installed (Latest package version till date: rhn-upgrade-5.5.0.17-1.elX )
- Download latest RHN Satellite v 5.5 ISO from Customer Portal
Upgrade process:
- Latest version of rhn-upgrade package will provide you documents required for upgrade.
- Refer /etc/sysconfig/rhn/satellite-upgrade/README file before starting upgrade process.
- Follow stems mentioned in file /etc/sysconfig/rhn/satellite-upgrade/doc/preparations.txt and file based on upgrade scenario (OS + Satellite upgrade : satellite-and-os-upgrade.txt , Only RHN Satellite upgrade without changing OS version : satellite-upgrade.txt)
- Before executing any commands mentioned in these files read it carefully, because if you are upgrading it from RHN Satellite v 5.4.1 then some steps will not be applicable for you.
- Schema upgrade process may take some time , it does not show any output on screen or in the log file. But do not panic wait for the script to complete.
Post upgrade:
- No major issues seen on RHN Satellite v 5.5, satellite-sync command is slow at "Importing package metadata" stage but this issue is on high priority and fix will be released soon.
If you face any issues with upgrade document or while satellite upgrade or after upgrade, feel free to open service request with Red Hat.
Hi James (and everyone else),
I'd highly recommend making sure that the rhn-upgrade package is 5.5.0.17 (this was a post-GA errata), as there is a bug fix for Satellites that contain Source RPMs that could potentially cause the schema upgrade to fail. Also remember to put SELinux into Permissive mode for the upgrade per the documentation.
Ashish and John have covered the other points regarding the upgrade, I personally found my Satellite upgrade to be quicker than expected.
Lastly, I'd recommend that you obtain/request (if you haven't already) a Satellite 5.5 entitlement certificate (like you would have got when you installed or last updated/renewed your Satellite subscription, a couple of days prior to when you expect to upgrade the server.
Guru
6222 points
James,
An update.
Like others state the update goes smooth. On my very small kvm guest it took about an hour to complete, but that is due to the small size of both host and guest.
Just a warning: SAM and Satellite do not go together. SAM needs replaces some apache project stuff that Satellite needs, so your login page will disappear.
Kind regards,
Jan Gerrit Kootstra
Guru
2135 points
Hi folks,
Looking at the 5.4.1 to 5.5.0 upgrade for my Satellite too. One complication I'm considering is rebuilding the entire stack on RHEL6 instead of my current RHEL5 box. I've already got my Proxies running on RHEL6, so would consider upgrading my Satellite to RHEL6 also if the benefits outweighed the negatives.
I'm only considering the extra OS upgrade as we're also upgrading our ESX estate which this all runs on from ESX 3.5 (yes I know) to ESXi5.1. Running RHEL6 on ESXi5.1 will open up the latest vmxnet3 NIC's/drivers to us. It will also standardise the Satellite/Proxy infrastructure on one OS version.
My gut feel is that adding the OS upgrade to the mix is a step too far at present. Any comments?
Thanks
D
Hi Duncan,
My personal opinion is that if you are considering migrating the machine hosting the Satellite from RHEL 5 to RHEL 6 then doing it at the same time as the Satellite is a good opportunity to do it, although it will take a lot longer to perform.
My recommendation if you are interested in making the switch is to read the upgrade documentation from the rhn-upgrade RPM (you can install this without triggering an upgrade automatically) and read the README followed by the documentation in /etc/sysconfig/rhn/satellite-upgrade/, this documentation outlines both scenarios (straight 5.4.1 to 5.5 upgrade, and Satellite + RHEL upgrade).
The main increase in time with the Satellite + RHEL upgrade option is the database export/import time.
The only other recommendation I'd make, is please ensure that you have the Certificate Authority password for the Satellite/Proxy certificate infrastructure, especially if you'd intend to change the Satellite's hostname as part of the upgrade.
Cheers,
Nigel
Red Hat
Active Contributor
307 points
Hello,
Great discussion. The only thing that I would add would be to ensure that you extend the satellite database tablespaces prior to upgrading from 5.4.1 to 5.5. This has been one of the biggest pitfalls of upgrading so far.
Calvin Locklear
I attempted the upgrade from 5.4.1 to 5.5 last week. I ended with a database that lost most of my custom channels, lost all users except satadmin and showed only 25 out of 264 servers. Not a good day to say the least.
I uploaded the database backup (5.4) to Redhat. They ended up with the same result I did. Not sure what happened but the db was definitely pooched. The backup was taken the day before the upgrade and I ran verify against it before starting.
Make sure you have a good server backup prior to starting. It sounds like it works fine 99% of the time but that doesn't help if you're the 1% like me. :-)
Restored the server and will be retrying the upgrade next week.
Hi,
I performed the upgrade from 5.4.1 to 5.5 this past Saturday. Had *plenty* of documentation which I arranged logically (leaving out only non-relative information). FYI: Ours is installed on an RHEL 5 VM with embedded Oracle db.
Hickups/Observations:
- in addition to backing up your database, be *absolutely* sure you have a good backup of the server itself. Somehow, we lost our kickstart files, only to find that our server wasn't *really* being backed up!
- check for yum updates! In our case the RHEL 5 machine was lacking more than two-hundred updates, including Satellite and Cobbler related updates. Obviously, OS updates had to be done before anything else.
- My yum updates included kernel, so a reboot was required. I had a bit of a panic when it took a long time before the machine was back. (Next time, I'll check the mount count of our partitions: the VM took >31 min for FSCK.)
- When backing up the database, follow update instructions *carefully* (especially when to be oracle and when to be root). Be prepared for the backup process to take a while.
- After your successful upgrade to 5.5 you will want to *again* do yum check-update. There were a number of 5.5-related updates, including schema, that had to be applied. (lead to additional Oracle backup too).
Now that we're back online, we've started re-establishing our kickstarts. We were rather surprised to find that iptables somehow lost port 80 during the update. That temporarily broke my engineer's ability to download the kickstart scripts he was rebuilding.
All in all, the process took >6 hours with the OS updates, Oracle db back up and verification, the actual Satellite update, new yum updates, new schema, etc.
Now, I have a question about intra-satellite sync... perhaps in another thread already... why don't kickstarts sync?
Kevin
We recently upgraded to 5.5 also. There was a step missing from the instructions that would have been very helpful. From other comments I see that I was not the only one bitten by this.
After installing the 5.5 packages from the ISO image, and BEFORE upgrading the satellite schema, you should run a yum update. This is because errata have been released since the ISO image was produced, including a schema upgrade. Since we did not do this, our schema was out of date the next time we updated.
I was told that you could apply schema updates while satellite is running, but I would not recommend it and I would never do it without a cold backup of the database.
Guru
6222 points
I still have one issue, if I try to sync the development tools 1.1 channel
satellite-sync -c rhel-x86_64-server-dts-6 --force-all-errata --debug-level=6
the serverprofiles get screwed, the server things it still has new updates to provide, but the clients do not think so.
So I have to remove all channel content and setup the channels and subscriptions again.
Fortunately it is a small test environment, but still annoying.
Red Hat support still has not found the cause.
Red Hat
25369 points
Hi Kevin. Thanks for sharing your experience with this. I'd definitely recommend sharing your kickstart question in a separate discussion!
Unfortunately I too ran into several issues during the upgrade.
I upgraded from RHNSS 5.4.1 (new installation) on RHEL6.
It's been posted before but it's certainly worth repeating: make sure you have full and good backups you know you can restore of both the DB and the server.
First gotcha:
The upgrade command shown in the installation guide assumes the use of the embedded DB. If you - like we - use and external DB check section 4.2.1 "Options for the installer script". This is documented after the upgrade command but I noticed only the "disconnected" which does not apply.
Probably as a result of the fact that we ran the upgrade without the --external-db flag the DB connection strings had to be corrected later on.
Secondly:
The upgrade commands takes a while to run (as indicated by the satellite-upgrade-estimates script). It is advisable to start the command in a "screen" session or something similar to avoid issues when you're connection to the server is lost or terminated (timeout).
Then we really got into problems:
- something went wrong with the schema upgrade (starting from a clean install of 5.4.1)
- The upgrade replaces or updates /etc/rhn/rhn.conf so if you've made any modifications you will need to restore these. In our case the integration with PAM to allow authentication against Active Directory was removed which prevented logons except for the rhnadmin account.
- several of our kickstart profiles became "missing or invalid". The only explanation I can come up with is the fact that we have renamed some kickstart profiles that had been created using the wizard.
To fix this I had to modify the json files in /var/lib/cobbler/config/profiles.d and run cobbler sync manually.
Now the upgrade appears to be complete though I'm still uncertain why we've been told that we don't need to fix the DB packages that failed to compile.
Our support team that performs our installations has reported a delay of up to 30 minutes between the calculation of dependencies and the actual package installation that we never experienced before. I'm unable to reproduce this myself so it's too early to say much about this.
There's one more thing I noticed since the upgrade. A compare deployed config files job is now scheduled daily, I can't remember if this existed in 5.4.1 as well but at least since the upgrade we ran into this known issue: https://access.redhat.com/knowledge/solutions/262833.
Perhaps it's a good idea to upgrade the rhn* packages on all clients after and maybe even before the upgrade.
Another gotcha:
Previously rhnpush used the same PAM configuration as the satellite server itself. Since the upgrade we can no longer push packages with our Active Directory accounts (integrated via LDAP).
/usr/share/rhn/config-defaults/rhn_server.conf contains these lines:
# PAM authentication, turned off by defaultpam_auth_service =But I can't find documentation on this, do I add the same service as should be added to /etc/rhn/rhn.conf for LDAP integration here?
Guru
2135 points
I can confirm that more vCPU's improves performance somewhat with channel sync loads.
Satellite originially had 2x vCPU's and 8Gb RAM. After upgrading to 4x vCPU's and 12Gb RAM the time taken for large errata synching was clearly reduced. The CPU loading graphs used to show nearly 100% usage for several hours after errata synching was started, along with RAM usage climbing above 85%.
Now we see much shorter vCPU peaks which only climb as high as perhaps 85-90% usage. The RAM usage will also climb past the 8Gb that the machine used to have, but still leaves a much more comfortable amount of free RAM. These stats were all grabbed from /prox on a 60 second interval and dumped into an RRD setup I have. I can't vouch for the size of each sync, but I wouldn't call any of them minor.
So if you're saying load scales well from 1 to 2 vCPU's, I'm also saying it scales fairly well from 2 to 4 vCPU's (if somewhat flatter). From what I'm seeing so far, though, further vCPU's might not achieve such good returns.
Cheers
D
Hi,
Just wondering if this something I missed or something the upgrade should have taken care of but didn't.
While checking one of the tickets raised since the upgrade I was recommended to upgrade spacewalk-java because it contains a fix for duplicate results in some API calls.
The update is not available on our satellite server because the yum repo still points to RHNSS 5.4 instead of 5.5.
[mertensb@defrlpsat01 ~]$ sudo yum repolist Loaded plugins: changelog, product-id, rhnplugin, security This system is receiving updates from RHN Classic or RHN Satellite. repo id repo name status redhat-rhn-satellite-5.4-server-x86_64-6 Red Hat Network Satellite (v5.4 for Server v6 AMD64 / Intel64) 524 rhel-x86_64-server-6 Red Hat Enterprise Linux Server (v. 6 for 64-bit x86_64) 10,400 repolist: 10,924
I can't find any description that this should be taken care of manually and didn't check because I assumed the upgrade would take care of this automatically.
Is this a step I missed, can anyone point me to the right documentation?
Regards
Bram
Hi Bram,
Yes it seems you missed the satellite certificate activation step. Depending on whether you did a satellite upgrade or satellite+os upgrade I assume you referred satellite-upgrade.txt or satellite-and-os-upgrade.txt. Both the txt files have the step to activate new satellite certificate. Here once you activate the new Satellite v5.5 certificate, the system would be subscribed to RHN Satellite v5.5 channel.
I would recommend revisiting the installation procedure and completing the remaining steps including activating the new cert.
Paresh
Expert
851 points
I just finished our upgrade from 5.4.1 to 5.5 I hit a few bumps but for most part it was smooth process. I followed the Red Hat solution https://access.redhat.com/site/solutions/11041 which I haven't seen listed here. Made sure I had db backup, read the entire upgrade-satellite doc after installing/updating rhn-upgrade package.
When I first logged in to my Satellite after upgrade, it stated that still required another schema upgrade. According to the latest info in the rhn-upgrade, my schema was right version. I ran the spacewalk-schema-upgrade command again and this moved it from 5.5.0.13 to 5.5.0.17-1. This schema upgrade did resolve some items, such as now I could delete archived items from schedule.
Another note, even though my I had no errors during upgrade taskomatic and rhn_search wouldn't stay running. It would start then die after about a minute. This would cause my kickstarts to fail. There is a knowledge base about the errors I found in /var/log/messages.
Errors I seen
Apr 23 15:13:56 mySat wrapper[29704]: Launching a JVM...
Apr 23 15:13:56 mySat wrapper[29704]: JVM exited while loading the application.
Apr 23 15:13:56 mySat0 wrapper[29704]: There were 5 failed launches in a row, each lasting less than 300 seconds. Giving up.
Apr 23 15:13:56 mySat wrapper[29704]: There may be a configuration problem: please check the logs.
Apr 23 15:13:56 mySat wrapper[29704]: <-- Wrapper Stopped
Here's article
"After upgrading Red Hat Network Satellite to 5.5, the rhn_search and taskomatic services failing with 'Unrecognized option: -Xdump:heap:file=/var/crash/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd' error" https://access.redhat.com/site/solutions/232243
Solution was to make sure ibm java was set. I did a alternatives --config java and my ibm was selected. I changed it and exited then ran again and reset it back to ibm java. After this satellite would start perfectly. I then ran a cobbler-update, satellite-sync, and tested installs and config files and other items in Satellite. Everything was now working.
Make sure you get a new cert before beginning upgrade.
OS was RHEL 6.3
Hi,
Thanks, I see what the problem is now: we encountered errors during the schema upgrade and I stopped the upgrade and started working with support on this. The satellite is functional so we never noticed that the remaining steps had not been completed.
Thanks for the quick response I'll go complete the upgrade 3 months after starting it...
Regards
Bram
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.