Recoverable error: This service is older (v15) than the minimum (v30) version

Solution Verified - Updated -

Environment

Red Hat OpenStack Platform 13.0

Issue

  • Horizon GUI Unable to Retrieve instance list after performing FFU of RHOSP10 environment.
  • After updating to RHOSP13, we get the following error in log horizon logs:

    WARNING horizon.exceptions Recoverable error: This service is older (v15) than the minimum (v30) version of the rest of the deployment. Unable to continue. (HTTP 400) (Request-ID: req-669c408d-2487-4622-8179-cb9930716034)
    

Resolution

  • First, take DB backup as a manual workaround is required to resolve the issue. Run the below command on the controller node.

    # docker ps | grep galera
    # docker exec -it container_id bash
    
  • Run the below command from inside the galera-container.
    Note: Make sure you connect with the support team before making any direct changes at DB level.

    # mysql -u root -e "select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';" -s -N | xargs mysqldump -u root --single-transaction --databases > /var/lib/mysql/openstack_database.sql
    
  • Then backup all the grants.

    # mysql -u root -e "SELECT CONCAT('\"SHOW GRANTS FOR ''',user,'''@''',host,''';\"') FROM mysql.user where length(user) > 0"  -s -N | xargs -n1 mysql -u root -s -N -e | sed 's/$/;/' > /var/lib/mysql/grants.sql
    
  • Logout from the container and move the backup to /root of the controller.

    # mv /var/lib/mysql/openstack_database.sql /root/
    # mv /var/lib/mysql/grants.sql /root/
    
  • Perform the manual workaround by updating the DB. Delete the entries of version 15. Run these commands inside the galera-container.

    # mysql nova 
    # select id, host, version, uuid, created_at, updated_at from services where version = 15;
    # delete from services where version = 15;
    

Root Cause

  • The error occurs when we have db entries for both versions 15 and 30 for nova service.
  • The issue was tracked in the BZ and fixed in the version openstack-nova-17.0.10-10.el7ost of RHOSP13.

Diagnostic Steps

  • Verify in nova DB the version of services.

    MariaDB [nova]> select id, host, version, uuid, created_at, updated_at from services ;
    +-----+------------------------------------+---------+--------------------------------------+---------------------+---------------------+
    | id  | host                               | version | uuid                                 | created_at          | updated_at          |
    +-----+------------------------------------+---------+--------------------------------------+---------------------+---------------------+
    |   2 | 172.16.3.11                        |      15 | 8fe60bb4-dd04-42a2-b901-ff4690ff395c | 2018-02-12 13:32:38 | 2020-07-01 07:23:28 |
    |  26 | 172.16.3.11                        |      30 | 7489cda2-470a-4ed0-a5c0-353b89605bba | 2018-02-12 13:32:38 | 2020-07-01 08:37:04 |
    |  65 | 172.16.3.12                        |      15 | 35da0840-cbdc-412f-9ac9-4833c400f202 | 2018-02-12 13:36:29 | 2020-07-01 07:23:28 |
    |  98 | 172.16.3.12                        |      30 | 4301fdf1-545a-4565-96ce-9a494222caf1 | 2018-02-12 13:36:29 | 2020-07-01 08:37:00 |
    +-----+------------------------------------+---------+--------------------------------------+---------------------+---------------------+
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.