Performing online database updates failed

Issue

• We are updating a RHOSP 16.2.0 to 16.2.3, step by step as per this documentation and the command openstack overcloud external-update run --tags online_upgrade fails with the following error:

PLAY [External update step 1] **************************************************
2022-07-19 14:02:39.985256 | 566fa72f-0000-63f5-1f0d-000000000037 |       TASK | Online data migration for Cinder
2022-07-19 14:02:43.890812 | 566fa72f-0000-63f5-1f0d-000000000037 |      FATAL | Online data migration for Cinder | undercloud -> 10.10.10.10 | error={"changed": true, "cmd": ["podman", "exec", "cinder_api", "cinder-manage", "db", "online_data_migrations"], "delta": "0:00:02.717831", "end": "2022-07-19 14:02:43.809488", "msg": "non-zero return code", "rc": 2, "start": "2022-07-19 14:02:41.091657", "stderr": "", "stderr_lines": [], "stdout": "Running batches of 50 until complete.\nError attempting to run untyped_volumes_online_data_migration\nError attempting to run untyped_snapshots_online_data_migration\n+-----------------------------------------+--------------+-----------+\n|                Migration                | Total Needed | Completed |\n+-----------------------------------------+--------------+-----------+\n| untyped_snapshots_online_data_migration |      0       |     0     |\n|  untyped_volumes_online_data_migration  |      0       |     0     |\n+-----------------------------------------+--------------+-----------+\nSome migrations failed unexpectedly. Check log for details.", "stdout_lines": ["Running batches of 50 until complete.", "Error attempting to run untyped_volumes_online_data_migration", "Error attempting to run untyped_snapshots_online_data_migration", "+-----------------------------------------+--------------+-----------+", "|                Migration                | Total Needed | Completed |", "+-----------------------------------------+--------------+-----------+", "| untyped_snapshots_online_data_migration |      0       |     0     |", "|  untyped_volumes_online_data_migration  |      0       |     0     |", "+-----------------------------------------+--------------+-----------+", "Some migrations failed unexpectedly. Check log for details."]}

[WARNING]: ('undercloud -> 10.10.10.10',
'566fa72f-0000-63f5-1f0d-000000000037') missing from stats

• We can see the following error on that controller in /var/log/containers/cinder/cinder-manage.log:

2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage [req-f0d620be-c723-4abd-9f76-460535c03af8 - - - - -] Error attempting to run untyped_snapshots_online_data_migration: cinder.exception.VolumeTypeNotFoundByName: Volume type with name __DEFAULT__ could not be found.
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage Traceback (most recent call last):
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/cmd/manage.py", line 331, in _run_migration
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     found, done = migration_meth(ctxt, count)
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/db/api.py", line 437, in untyped_snapshots_online_data_migration
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     return IMPL.untyped_snapshots_online_data_migration(context, max_count)
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1015, in wrapper
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     return fn(*args, **kwargs)
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 601, in untyped_snapshots_online_data_migration
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     '__DEFAULT__')
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/volume/volume_types.py", line 186, in get_volume_type_by_name
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     return db.volume_type_get_by_name(context, name)
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/db/api.py", line 656, in volume_type_get_by_name
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     return IMPL.volume_type_get_by_name(context, name)
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     return f(*args, **kwargs)
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 3990, in volume_type_get_by_name
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     return _volume_type_get_by_name(context, name)
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     return f(*args, **kwargs)
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 3968, in _volume_type_get_by_name
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     raise exception.VolumeTypeNotFoundByName(volume_type_name=name)
2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage cinder.exception.VolumeTypeNotFoundByName: Volume type with name __DEFAULT__ could not be found.