Upgrade from 6.1 to 6.2

Latest response

Hello all, hope you are well :) !

Do you know how to check the progress of:
Upgrade Step: migrate_pulp...
2649
is still outgoing after more than 30 hours

Can't find anything in logs:
/var/run/mongodb/mongodb
/var/log/messages
/var/log/foreman-installer/satellite.log

"Found the following type definitions that were not present in the update collection [node, repository]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 24
Migration package pulp_docker.plugins.migrations is up to date at version 2
Migration package pulp_puppet.plugins.migrations is up to date at version 5
Applying pulp_rpm.plugins.migrations version 20"

/dev/mapper/vg_data-lv_pgsql 213G 17G 188G 8% /var/lib/pgsql
/dev/mapper/vg_data-lv_mongodb 1.1T 65G 999G 7% /var/lib/mongodb
/dev/mapper/vg_data-lv_pulp 19T 17T 2.2T 89% /var/lib/pulp

Thanks :)

Responses

Hello TSS UCOMP,

This sounds like the upgrade is hung and not proceeding (seemingly obvious). Satellite is a paid and supported product -- I'd highly recommend submitting a case with Red Hat. It sounds to me like you are upgrading your satellite from 6.1 to 6.2. That last link is a upgrade helper that Red Hat put together.

Please keep these known upgrade issues Red Hat has listed as well.

Please submit that case with Red Hat, be prepared to give them logs.

Regards,

RJ

Hello R.Hinton,

We already create RHEL support case but satellite 6.1 is end of life and support and suggestion from RHEL was to make fresh installation, I have no idea why have upgrade procedure when they advise to make fresh install ...
We use upgrade helper. Only error that was able to find: /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]

Hi TSS UCOMP

UPDATE see next post too

They may not have explained it well, and I can kinda see their point. However, you do have some options.

The link I made in my previous post named "upgrading your satellite from 6.1 to 6.2" leads to a means to incrementally upgrade your satellite from it's unsupported state to something supported. So that is an "option" of sorts. The other overarching option is to just build a new satellite at something like 6.5.1 (see a different thread, if you take this route, use rhel 7.7 to avoid installation issues from a discussion here).

So to review, one option is to incrementally update step by step from your current version to a supported version. The other option is to load a new satellite from scratch.

In my own experience, I was not an early adopter of satellite 6 due to all the issues I read in the forums here. I held on to Satellite 5.8 as long as I could until finally 6.2.x came out. I had three satellite 6.2.11 servers that I have upgraded to 6.3.5.1. But I had to incrementally upgrade them and it was not a simple thing. Each satellite presented some issues that I had to resolve and I used my Red Hat Technical Account manager through the process. In one case, I had to deal with undocumented issues in the install instructions and also undocumented issues within the "known issues" link I cited in my previous post. So that was quite a "rabbit hole" (meaning a load of not-fun diagnosis, and resolution steps, digging into logs, research). My current satellites are at a version that is supported by Red Hat. I'm about to upgrade them all to 6.4.x. That being said, I'm installing Satellite 6.5.1 in our test/development area because it's the latest, and there's features I really want, not just FIPS.

So that is perhaps why Red Hat recommended building a new satellite. The website I linked to in my previous post does cite a means (in theory) to upgrade from 6.1 to a current supported version, but is it worth the potential hassle? In my own experience, you might just want to install a fresh new shiny satellite 6.5.1 and then migrate all your clients over from the old satellite to the new one that is running 6.5.1. Maybe.

You could ask them - put in your case something about a desire to upgrade it to a current supported version. If you have a Technical Account Manager with Red Hat, they ought to help you with this process. I can tell you from my own experience, it was not as simplistic as the instructions at the link I provided. I had to do some real digging. Red Hat figured out and resolved some of the issues, I figured out some of the other issues, and got it up and running.

Examine the output of "foreman-tail" command and examine the output for issues. I wouldn't also be surprised if some hung task is getting in your way. In my own case, I was willing to take the risk of reloading my satellite and I ran foreman rake commands that would remove a lot of the clutter successfully ran tasks, however, that's a risk. That might show what tasks are remaining. You could examine the web interface tasks area and use filtering to see if there are any hung tasks. However, there's a myriad of things that could have gone wrong with this.

Wish you well with this,

Regards

RJ

TSS COMP,

I remember something that occurred with one of my upgrades and what I did to fix it that might relate to what you originally posted.

NOTE: I make no guarantee, and if possible, run this against Red Hat

First Examine /var/log/messages shows the pulp server failed to start due to the following reasons:

- the database has not been migrated to the current version.  "run pulp-manage-db" and restart the application.  

If yes, consider this https://access.redhat.com/solutions/3138921 (contact Red Hat and run by them too if possible)

 # for i in pulp_celerybeat pulp_resource_manager pulp_workers pulp_streamer; do service $i stop; done

   # sudo -u apache pulp-manage-db
   # katello-service restart

Perform this below command before and after the above:

hammer ping

Regards,

RJ

Hello TSS UCOMP,

The rebuild is suggested, because to get to a supported release you need to update at least to 6.3, via 6.3 and starting with 6.1 it is a time consuming and like RJ wrote above a non failure free way to go. If you need to go to 6.5.2 (current release) you need to update to 6.2, 6.3, 6.4 to 6.5.2. Might take a week.

If you go for a rebuild, I suggest 6.5.2 it is since a few weeks the latest release which solved some minor issues with 6.5.2.

Advantage of 6.5.x is the support of RHEL 8 provisioning. Syncing the content may take a day or two, maybe three. Less chance of a failure then the multiple update path.

Regards,

Jan Gerrit

Hi ! :)

I agree with what Jan Gerrit and RJ mentioned and (between the lines) recommended to you, "TSS UCOMP". My 2 cents :
This is a good example for why it is a good idea to always upgrade to the latest stable release as soon as possible, once it is
being made available. Leaving out minor point releases, or sticking too long to older versions may lead to a lot of trouble ...

Regards,
Christian

My esteemed colleagues Christian Labisch and Jan Gerrit distilled this nicely. If you can, build a fresh new satellite. If you can wait until satellite 6.5.2, I'd recommend that too. I posted all the stuff I did yesterday, because sometimes you don't get a choice from your customer/employer. If you do have a choice, or can influence one, I'd do a fresh install. All I wrote is the long way of saying that the effort to incrementally upgrade to a supported version in your case is probably not worth that effort.

Regards

RJ

Thank you, RJ ! :) ... "(probably) not worth that effort." ... Exactly that ! Sometimes less words "do the trick" ... :)

Regards,
Christian