Performing online database updates failed

Solution In Progress - Updated -

Issue

  • A fast forward upgrade from RHOSP 16.2 to 17.1 fails, or
  • An update of RHOSP 16 (for example from 16.2.0 to 16.2.3) fails during the external-update task with the following error:

    PLAY [External update step 1] **************************************************
    2022-07-19 14:02:39.985256 | 566fa72f-0000-63f5-1f0d-000000000037 |       TASK | Online data migration for Cinder
    2022-07-19 14:02:43.890812 | 566fa72f-0000-63f5-1f0d-000000000037 |      FATAL | Online data migration for Cinder | undercloud -> 10.10.10.10 | error={"changed": true, "cmd": ["podman", "exec", "cinder_api", "cinder-manage", "db", "online_data_migrations"], "delta": "0:00:02.717831", "end": "2022-07-19 14:02:43.809488", "msg": "non-zero return code", "rc": 2, "start": "2022-07-19 14:02:41.091657", "stderr": "", "stderr_lines": [], "stdout": "Running batches of 50 until complete.\nError attempting to run untyped_volumes_online_data_migration\nError attempting to run untyped_snapshots_online_data_migration\n+-----------------------------------------+--------------+-----------+\n|                Migration                | Total Needed | Completed |\n+-----------------------------------------+--------------+-----------+\n| untyped_snapshots_online_data_migration |      0       |     0     |\n|  untyped_volumes_online_data_migration  |      0       |     0     |\n+-----------------------------------------+--------------+-----------+\nSome migrations failed unexpectedly. Check log for details.", "stdout_lines": ["Running batches of 50 until complete.", "Error attempting to run untyped_volumes_online_data_migration", "Error attempting to run untyped_snapshots_online_data_migration", "+-----------------------------------------+--------------+-----------+", "|                Migration                | Total Needed | Completed |", "+-----------------------------------------+--------------+-----------+", "| untyped_snapshots_online_data_migration |      0       |     0     |", "|  untyped_volumes_online_data_migration  |      0       |     0     |", "+-----------------------------------------+--------------+-----------+", "Some migrations failed unexpectedly. Check log for details."]}
    
    [WARNING]: ('undercloud -> 10.10.10.10',
    '566fa72f-0000-63f5-1f0d-000000000037') missing from stats
    
  • We can see the following error on that controller in /var/log/containers/cinder/cinder-manage.log:

    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage [req-f0d620be-c723-4abd-9f76-460535c03af8 - - - - -] Error attempting to run untyped_snapshots_online_data_migration: cinder.exception.VolumeTypeNotFoundByName: Volume type with name __DEFAULT__ could not be found.
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage Traceback (most recent call last):
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/cmd/manage.py", line 331, in _run_migration
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     found, done = migration_meth(ctxt, count)
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/db/api.py", line 437, in untyped_snapshots_online_data_migration
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     return IMPL.untyped_snapshots_online_data_migration(context, max_count)
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1015, in wrapper
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     return fn(*args, **kwargs)
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 601, in untyped_snapshots_online_data_migration
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     '__DEFAULT__')
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/volume/volume_types.py", line 186, in get_volume_type_by_name
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     return db.volume_type_get_by_name(context, name)
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/db/api.py", line 656, in volume_type_get_by_name
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     return IMPL.volume_type_get_by_name(context, name)
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     return f(*args, **kwargs)
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 3990, in volume_type_get_by_name
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     return _volume_type_get_by_name(context, name)
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     return f(*args, **kwargs)
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 3968, in _volume_type_get_by_name
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage     raise exception.VolumeTypeNotFoundByName(volume_type_name=name)
    2022-07-19 14:39:29.858 1882 ERROR cinder.cmd.manage cinder.exception.VolumeTypeNotFoundByName: Volume type with name __DEFAULT__ could not be found.
    

Environment

  • Red Hat OpenStack Platform (RHOSP) 17.1
  • Red Hat OpenStack Platform (RHOSP) 16.2

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content