FFU upgrades fails in 12 to 13 step

Issue

• We are following instructions from the FFU documentation.

• In the step 12 to 13 after we run command openstack undercloud upgrade, we get an error:

2021-06-29 12:01:42,776 INFO: ---> Package rhosp-director-images-ipa-x86_64.noarch 0:13.0-20210511.1.el7ost will be installed
2021-06-29 12:01:42,776 INFO: ---> Package rhosp-release.noarch 0:13.0.16-1.el7ost will be installed
2021-06-29 12:01:43,328 INFO: --> Finished Dependency Resolution
2021-06-29 12:01:43,379 INFO: Error: Package: python2-qpid-proton-0.24.0-2.el7.x86_64 (installed)
2021-06-29 12:01:43,380 INFO:            Requires: qpid-proton-c(x86-64) = 0.24.0-2.el7
2021-06-29 12:01:43,380 INFO:            Removing: qpid-proton-c-0.24.0-2.el7.x86_64 (installed)
2021-06-29 12:01:43,380 INFO:                qpid-proton-c(x86-64) = 0.24.0-2.el7
2021-06-29 12:01:43,381 INFO:            Updated By: qpid-proton-c-0.33.0-4.el7_9.x86_64 (rhel-7-server-openstack-13-rpms)
2021-06-29 12:01:43,381 INFO:                qpid-proton-c(x86-64) = 0.33.0-4.el7_9
2021-06-29 12:01:43,381 INFO:            Available: qpid-proton-c-0.26.0-3.el7.x86_64 (rhel-7-server-openstack-13-rpms)
2021-06-29 12:01:43,381 INFO:                qpid-proton-c(x86-64) = 0.26.0-3.el7
2021-06-29 12:01:43,381 INFO:            Available: qpid-proton-c-0.27.0-3.el7.x86_64 (rhel-7-server-openstack-13-rpms)
2021-06-29 12:01:43,382 INFO:                qpid-proton-c(x86-64) = 0.27.0-3.el7
2021-06-29 12:01:43,382 INFO: **********************************************************************
2021-06-29 12:01:43,382 INFO: yum can be configured to try to resolve such errors by temporarily enabling
2021-06-29 12:01:43,382 INFO: disabled repos and searching for missing dependencies.
2021-06-29 12:01:43,383 INFO: To enable this functionality please set 'notify_only=0' in /etc/yum/pluginconf.d/search-disabled-repos.conf
2021-06-29 12:01:43,383 INFO: **********************************************************************
2021-06-29 12:01:43,383 INFO:
2021-06-29 12:01:43,383 INFO: --> Running transaction check
2021-06-29 12:01:43,383 INFO: ---> Package kernel.x86_64 0:3.10.0-957.21.3.el7 will be erased
2021-06-29 12:01:44,315 INFO: ---> Package qpid-proton-c.x86_64 0:0.24.0-2.el7 will be updated
2021-06-29 12:01:44,316 INFO: --> Processing Dependency: qpid-proton-c(x86-64) = 0.24.0-2.el7 for package: python2-qpid-proton-0.24.0-2.el7.x86_64
2021-06-29 12:01:44,320 INFO: ---> Package rhosp-director-images.noarch 0:10.0-20180628.2.el7ost will be erased
2021-06-29 12:01:44,321 INFO: ---> Package rhosp-director-images-ipa.noarch 0:10.0-20180628.2.el7ost will be erased
2021-06-29 12:01:44,398 INFO: --> Finished Dependency Resolution
2021-06-29 12:01:44,399 INFO: Error: Package: python2-qpid-proton-0.24.0-2.el7.x86_64 (installed)
2021-06-29 12:01:44,399 INFO:            Requires: qpid-proton-c(x86-64) = 0.24.0-2.el7
2021-06-29 12:01:44,400 INFO:            Removing: qpid-proton-c-0.24.0-2.el7.x86_64 (installed)
2021-06-29 12:01:44,400 INFO:                qpid-proton-c(x86-64) = 0.24.0-2.el7
2021-06-29 12:01:44,400 INFO:            Updated By: qpid-proton-c-0.33.0-4.el7_9.x86_64 (rhel-7-server-openstack-13-rpms)
2021-06-29 12:01:44,400 INFO:                qpid-proton-c(x86-64) = 0.33.0-4.el7_9
2021-06-29 12:01:44,401 INFO:            Available: qpid-proton-c-0.26.0-3.el7.x86_64 (rhel-7-server-openstack-13-rpms)
2021-06-29 12:01:44,401 INFO:                qpid-proton-c(x86-64) = 0.26.0-3.el7
2021-06-29 12:01:44,401 INFO:            Available: qpid-proton-c-0.27.0-3.el7.x86_64 (rhel-7-server-openstack-13-rpms)
2021-06-29 12:01:44,401 INFO:                qpid-proton-c(x86-64) = 0.27.0-3.el7
2021-06-29 12:01:44,401 INFO:  You could try using --skip-broken to work around the problem
2021-06-29 12:01:45,004 INFO:  You could try running: rpm -Va --nofiles --nodigest
2021-06-29 12:01:45,069 INFO: Uploading Enabled Repositories Report
2021-06-29 12:01:45,086 INFO: Loaded plugins: product-id, subscription-manager

and it seems that there is some problems with repository configuration:

• yum provides python2-qpid-proton-0.24.0-2.el7.x86_64 returns an error:

Loaded plugins: enabled_repos_upload, package_upload, product-id, search-disabled-repos, subscription-manager
rhel-7-server-extras-rpms/x86_64/filelists                                                                                                                                              | 759 kB  00:00:00
https://satellite/pulp/repos/DOMAIN/content/dist/rhel/server/7/7.9/x86_64/optional/os/repodata/e5117d90d0e794378f6422b72822762837c2cf90-filelists.xml.gz: [Errno
14] HTTPS Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article

https://access.redhat.com/articles/1320623

If above article doesn't help to resolve this issue please open a ticket with Red Hat Support.

 One of the configured repositories failed (Red Hat Enterprise Linux 7 Server - Optional (RPMs)),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo=rhel-7-server-optional-rpms ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable rhel-7-server-optional-rpms
        or
            subscription-manager repos --disable=rhel-7-server-optional-rpms

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=rhel-7-server-optional-rpms.skip_if_unavailable=true

failure: repodata/e5117d90d0e794378f6422b72822762837c2cf90-filelists.xml.gz from rhel-7-server-optional-rpms: [Errno 256] No more mirrors to try.https://satellite/pulp/repos/DOMAIN/content/dist/rhel/server/7/7.9/x86_64/optional/os/repodata/e5117d90d0e794378f6422b72822762837c2cf90-filelists.xml.gz: [Errno
14] HTTPS Error 404 - Not Found
Uploading Enabled Repositories Report
Loaded plugins: product-id, subscription-manager

• The following repositories are seen in yum repolist:

Loaded plugins: enabled_repos_upload, package_upload, product-id, search-disabled-repos, subscription-manager
rhel-7-server-extras-rpms                                                                                                                                                               | 2.0 kB  00:00:00
rhel-7-server-openstack-13-rpms                                                                                                                                                         | 2.3 kB  00:00:00
rhel-7-server-optional-rpms                                                                                                                                                             | 1.8 kB  00:00:00
Not using downloaded rhel-7-server-optional-rpms/repomd.xml because it is older than what we have:
  Current   : Wed Jun 23 11:21:39 2021
  Downloaded: Wed Jun 23 11:20:05 2021
rhel-7-server-rh-common-rpms                                                                                                                                                            | 2.1 kB  00:00:00
Not using downloaded rhel-7-server-rh-common-rpms/repomd.xml because it is older than what we have:
  Current   : Wed Jun 23 11:26:15 2021
  Downloaded: Wed Jun 23 11:26:13 2021
rhel-7-server-rhceph-2-mon-rpms                                                                                                                                                         | 2.1 kB  00:00:00     Not using downloaded rhel-7-server-rhceph-2-mon-rpms/repomd.xml because it is older than what we have:
  Current   : Wed Jun 23 11:26:10 2021
  Downloaded: Wed Jun 23 11:26:01 2021
rhel-7-server-rhceph-2-tools-rpms                                                                                                                                                       | 2.1 kB  00:00:00
Not using downloaded rhel-7-server-rhceph-2-tools-rpms/repomd.xml because it is older than what we have:
  Current   : Wed Jun 23 11:26:14 2021
  Downloaded: Wed Jun 23 11:26:11 2021
rhel-7-server-rpms                                                                                                                                                                      | 2.0 kB  00:00:00
rhel-ha-for-rhel-7-server-rpms                                                                                                                                                          | 2.0 kB  00:00:00
Not using downloaded rhel-ha-for-rhel-7-server-rpms/repomd.xml because it is older than what we have:
  Current   : Wed Jun 23 11:26:20 2021
  Downloaded: Wed Jun 23 11:26:15 2021
repo id                                                                             repo name                                                                                                            status
rhel-7-server-extras-rpms/x86_64                                                    Red Hat Enterprise Linux 7 Server - Extras (RPMs)                                                                     1,404
rhel-7-server-openstack-13-rpms/x86_64                                              Red Hat OpenStack Platform 13 for RHEL 7 (RPMs)                                                                       3,266
rhel-7-server-optional-rpms/x86_64                                                  Red Hat Enterprise Linux 7 Server - Optional (RPMs)                                                                  19,766
rhel-7-server-rh-common-rpms/x86_64                                                 Red Hat Enterprise Linux 7 Server - RH Common (RPMs)                                                                    242
rhel-7-server-rhceph-2-mon-rpms/x86_64                                              Red Hat Ceph Storage MON 2 for Red Hat Enterprise Linux 7 Server (RPMs)                                                 297
rhel-7-server-rhceph-2-tools-rpms/x86_64                                            Red Hat Ceph Storage Tools 2 for Red Hat Enterprise Linux 7 Server (RPMs)                                               366
rhel-7-server-rpms/x86_64                                                           Red Hat Enterprise Linux 7 Server (RPMs)                                                                             31,958
rhel-ha-for-rhel-7-server-rpms/x86_64                                               Red Hat Enterprise Linux High Availability (for RHEL 7 Server) (RPMs)                                                   717
repolist: 58,016
Uploading Enabled Repositories Report
Loaded plugins: product-id, subscription-manager