OpenStack 17.1 upgrade fails with "Check for valid ceph version during FFU... Target ceph version cannot be for FFU."

Issue

• When upgrading RHOSP from 16.2 to 17.1 with Ceph deployed through Director also being upgraded from 4 to 5, following the step 3 from chapter 6.2 of the official RHOSP 17.1 FFU documentation, and using Satellite as the RPMs and container images provider, the following error message is shown:

  025-01-03 13:20:57,324 p=544247 u=stack n=ansible | 2025-01-03 13:20:57.322539 | 005056a3-2c4c-2864-af0c-0000000000b3 |      FATAL | Check for valid ceph version during FFU | undercloud | error={"changed": false, "msg": "Target ceph version cannot be  for FFU."}
  2025-01-03 13:20:57,364 p=544247 u=stack n=ansible | PLAY RECAP *********************************************************************
  2025-01-03 13:20:57,366 p=544247 u=stack n=ansible | ceph1                      : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2  
  2025-01-03 13:20:57,368 p=544247 u=stack n=ansible | ceph2                      : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2  
  2025-01-03 13:20:57,370 p=544247 u=stack n=ansible | ceph3                      : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2  
  2025-01-03 13:20:57,372 p=544247 u=stack n=ansible | compute1                   : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2  
  2025-01-03 13:20:57,373 p=544247 u=stack n=ansible | controller1                : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2  
  2025-01-03 13:20:57,375 p=544247 u=stack n=ansible | controller2                : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2  
  2025-01-03 13:20:57,376 p=544247 u=stack n=ansible | controller3                : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2  
  2025-01-03 13:20:57,378 p=544247 u=stack n=ansible | localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0  
  2025-01-03 13:20:57,379 p=544247 u=stack n=ansible | undercloud                 : ok=4    changed=1    unreachable=0    failed=1    skipped=0    rescued=0    ignored=2  
  

• Although Controllers are apparently OK, the Undercloud node is not:

            "ok: [controller1 -> controller1] => {\"changed\": false, \"cmd\": [\"podman\", \"run\", \"--rm\", \"--net=host\", \"-v\", \"/etc/ceph:/etc/ceph:z\", \"-v\", \"/var/lib/ceph:/var/lib/ceph:z\", \"-v\", \"/var/run/ceph:/var/run/ceph:z\", \"--entrypoint=ceph\", \"satellite.example.com/container-images-osp-17_1-rhceph-5-rhel8:latest\", \"--cluster\", \"ceph\", \"--version\"], \"delta\": \"0:00:01.202827\", \"end\": \"2025-01-03 11:42:27.549883\", \"rc\": 0, \"start\": \"2025-01-03 11:42:26.347056\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"ceph version 16.2.10-266.el8cp (07823b29a11c047cffc11d81c3c975986573a225) pacific (stable)\", \"stdout_lines\": [\"ceph version 16.2.10-266.el8cp (07823b29a11c047cffc11d81c3c975986573a225) pacific (stable)\"]}",
         "TASK [ceph-common : get ceph version] ******************************************",
         "ok: [controller1 -> controller1] => {\"changed\": false, \"cmd\": [\"podman\", \"run\", \"--rm\", \"--net=host\", \"-v\", \"/etc/ceph:/etc/ceph:z\", \"-v\", \"/var/lib/ceph:/var/lib/ceph:z\", \"-v\", \"/var/run/ceph:/var/run/ceph:z\", \"--entrypoint=ceph\", \"satellite.example.com/container-images-osp-17_1-rhceph-5-rhel8:latest\", \"--cluster\", \"ceph\", \"--version\"], \"delta\": \"0:00:01.438645\", \"end\": \"2025-01-03 11:56:21.588101\", \"rc\": 0, \"start\": \"2025-01-03 11:56:20.149456\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"ceph version 16.2.10-266.el8cp (07823b29a11c047cffc11d81c3c975986573a225) pacific (stable)\", \"stdout_lines\": [\"ceph version 16.2.10-266.el8cp (07823b29a11c047cffc11d81c3c975986573a225) pacific (stable)\"]}",
         "TASK [ceph-common : get ceph version] ******************************************",
  2025-01-03 13:20:55,682 p=544247 u=stack n=ansible | 2025-01-03 13:20:55.681916 | 005056a3-2c4c-2864-af0c-0000000000b2 |       TASK | Get Ceph version
  2025-01-03 13:20:57,182 p=544247 u=stack n=ansible | 2025-01-03 13:20:57.178632 | 005056a3-2c4c-2864-af0c-0000000000b2 |    CHANGED | Get Ceph version | undercloud
  2025-01-03 13:20:57,226 p=544247 u=stack n=ansible | 2025-01-03 13:20:57.224788 | 005056a3-2c4c-2864-af0c-0000000000b3 |       TASK | Check for valid ceph version during FFU
  2025-01-03 13:20:57,324 p=544247 u=stack n=ansible | 2025-01-03 13:20:57.322539 | 005056a3-2c4c-2864-af0c-0000000000b3 |      FATAL | Check for valid ceph version during FFU | undercloud | error={"changed": false, "msg": "Target ceph version cannot be  for FFU."}
  2025-01-03 13:20:57,389 p=544247 u=stack n=ansible | 2025-01-03 13:20:57.388104 | 005056a3-2c4c-2864-af0c-0000000000b2 |    SUMMARY | undercloud | ceph : Get Ceph version | 1.50s
  2025-01-03 13:20:57,401 p=544247 u=stack n=ansible | 2025-01-03 13:20:57.400883 | 005056a3-2c4c-2864-af0c-0000000000b3 |    SUMMARY | undercloud | Check for valid ceph version during FFU | 0.10s
  2025-01-03 17:07:15,418 p=119900 u=stack n=ansible | 2025-01-03 17:07:15.417204 | 005056a3-2c4c-e102-498e-0000000000b2 |       TASK | Get Ceph version
  2025-01-03 17:07:16,828 p=119900 u=stack n=ansible | 2025-01-03 17:07:16.826486 | 005056a3-2c4c-e102-498e-0000000000b2 |    CHANGED | Get Ceph version | undercloud
  2025-01-03 17:07:16,858 p=119900 u=stack n=ansible | 2025-01-03 17:07:16.857324 | 005056a3-2c4c-e102-498e-0000000000b3 |       TASK | Check for valid ceph version during FFU
  2025-01-03 17:07:16,944 p=119900 u=stack n=ansible | 2025-01-03 17:07:16.941936 | 005056a3-2c4c-e102-498e-0000000000b3 |      FATAL | Check for valid ceph version during FFU | undercloud | error={"changed": false, "msg": "Target ceph version cannot be  for FFU."}
  2025-01-03 17:07:17,021 p=119900 u=stack n=ansible | 2025-01-03 17:07:17.020324 | 005056a3-2c4c-e102-498e-0000000000b2 |    SUMMARY | undercloud | ceph : Get Ceph version | 1.41s
  2025-01-03 17:07:17,050 p=119900 u=stack n=ansible | 2025-01-03 17:07:17.049217 | 005056a3-2c4c-e102-498e-0000000000b3 |    SUMMARY | undercloud | Check for valid ceph version during FFU | 0.09s
  

• Checking the container images available in the Controller nodes, they all are in the correct version, which is pacific, but even like that the upgrade procedure fails:

  {
     "mon": {
         "ceph version 16.2.10-266.el8cp (07823b29a11c047cffc11d81c3c975986573a225) pacific (stable)": 3
     },
     "mgr": {
         "ceph version 16.2.10-266.el8cp (07823b29a11c047cffc11d81c3c975986573a225) pacific (stable)": 3
     },
     "osd": {
         "ceph version 16.2.10-266.el8cp (07823b29a11c047cffc11d81c3c975986573a225) pacific (stable)": 3
     },
     "mds": {
         "ceph version 16.2.10-266.el8cp (07823b29a11c047cffc11d81c3c975986573a225) pacific (stable)": 3
     },
     "rgw": {
         "ceph version 16.2.10-266.el8cp (07823b29a11c047cffc11d81c3c975986573a225) pacific (stable)": 3
     },
     "rgw-nfs": {
         "ceph version 16.2.10-266.el8cp (07823b29a11c047cffc11d81c3c975986573a225) pacific (stable)": 5
     },
     "overall": {
         "ceph version 16.2.10-266.el8cp (07823b29a11c047cffc11d81c3c975986573a225) pacific (stable)": 20
     }
  }