Red Hat Training

A Red Hat training course is available for Red Hat CloudForms

Chapter 3. Migrating from CloudForms 4.6 (CFME 5.9) to CloudForms 4.7 (CFME 5.10)

3.1. Overview

This procedure describes the process of migrating Red Hat CloudForms 4.6 (CFME 5.9) to Red Hat CloudForms 4.7 (CFME 5.10). This procedure does not necessarily include migration of all possible customer modifications, so it is recommended that you fully test any modifications before migrating your environment.

Important
  • Read through all of the steps in this procedure before beginning the migration process.
  • CloudForms 4.7 appliances require 12 GB memory which is the same as the previous version. However, before migrating your appliances, adjust resources in your environment to avoid performance issues. See Migration Considerations in the Release Notes for more information.
  • CloudForms 4.6 high availability environments require additional configuration. See Additional Configuration in High Availability Environments.

You can classify the migration into three groups of appliances:

  • VMDB appliance - An appliance with workers running, which also hosts its own database that other appliances can also connect to.
  • Non-VMDB appliance - An appliance with workers running which does not host a database. The appliance is connected to an external database.
  • Dedicated database appliance - A CloudForms appliance or non-CloudForms virtual machine with no workers running on it: the appliance contains only a database for other appliances to connect to.

Migration Workflow Summary

In summary, the migration process from CFME 5.9 to CFME 5.10 follows this workflow:

Note

Appliances must be offline during migration; ensure you plan for downtime when migrating.

  1. Back up appliances (optional but recommended).
  2. Prepare appliances:

    1. Disable older CloudForms repositories and enable new repositories.
    2. Resize the disk space on the virtual machines hosting the appliances.
    3. Shut down evmserver on the master or global appliance.
  3. Migrate appliances:

    1. Update CFME packages on all appliances.
    2. Load the new version of the pglogical library on the VMDB and dedicated database appliances.
    3. Migrate the non-VMDB and VMDB appliance databases and update the Automate Model.
    4. Restart PostgreSQL on the VMDB and dedicated database appliances.
    5. Restart evmserver on the VMDB and non-VMDB appliances.
  4. Configure replication after the migration process is complete and appliances are running once again.

3.2. Backing Up Current Appliances

These steps will not affect the operations of your CloudForms infrastructure. However, they will help ensure that you are able to roll back if required and replicate the network settings.

  1. Back up the databases of your CFME 5.9 appliances. Take a snapshot if possible.
  2. Back up the following files for disaster recovery, noting which appliance each comes from:

    • /var/www/miq/vmdb/GUID
    • /var/www/miq/vmdb/REGION
  3. During the upgrade, the iptables configuration file (/etc/sysconfig/iptables) is removed. If you have changed the iptables configuration from the default (run iptables --list -n to see the current configuration), use the following command to back up the iptables configuration:

    # iptables-save > /etc/iptables.conf

    You can restore your iptables configuration file with the following command:

    # iptables-restore < /etc/iptables.conf

    Alternatively, add this command to /etc/rc.local to reload the rules at every reboot.

Note

For 5.9 appliances with the User Interface server role: Before migration, ensure that the Web Services role is enabled (it is enabled by default in CFME 5.9). If the Web Services role is disabled, it will not be turned on during the migration process. This role is required in CFME 5.10 to log in to the user interface.

3.3. Preparing the Appliances for Migration

On all appliances:

  1. Disable the CloudForms 4.6 (CFME 5.9) repositories:

    # subscription-manager repos --disable=cf-me-5.9-for-rhel-7-rpms
    Note

    See Enabling Supplementary and Optional Repositories in Using and Configuring Red Hat Subscription Manager for more information.

  2. Enable the CloudForms 4.7 (CFME 5.10) repositories:

    # subscription-manager repos --enable=rhel-7-server-rpms \
    --enable=cf-me-5.10-for-rhel-7-rpms \
    --enable=rhel-7-server-extras-rpms \
    --enable=rhel-7-server-ansible-2.7-rpms \
    --enable=rhel-7-server-rh-common-rpms \
    --enable=rhel-server-rhscl-7-rpms

3.4. Resizing the Disk Space

Note

This section only applies for customers upgrading from CFME 5.9.0.17 versions. Customers upgrading from the latest CFME 5.9 versions already have the partition changes and need not follow this procedure.

CloudForms 4.6 and newer require more disk space than previous CloudForms versions because of the addition of built-in Ansible features. Before migrating your appliances to CloudForms 4.7, resize the virtual machine partition hosting the appliances to ensure sufficient space is available for the appliance.

Complete the following steps to resize the disk space, replacing filenames as needed:

  1. Install the xfsdump tool for backing up filesystems:

    # yum -y install xfsdump
  2. Back up the partition’s existing filesystem, /repo, to a temporary repository, /tmp/repo:

    # xfsdump -F -f /tmp/repo /repo
  3. Unmount the existing filesystem:

    # umount /repo
  4. Remove the logical volume:

    # lvremove -f /dev/VG-CFME/lv_repo
  5. Create a new 1GB logical volume in the existing volume group lv_repo:

    # lvcreate --yes -L 1GB -n lv_repo VG-CFME
  6. Construct the volume path:

    # mkfs.xfs /dev/VG-CFME/lv_repo
  7. Mount the volume to /repo:

    # mount /dev/VG-CFME/lv_repo /repo
  8. Restore the /tmp/repo filesystem data to the old filesystem:

    # xfsrestore -f /tmp/repo /repo
  9. Resize the volume to allow sufficient space for the CloudForms 4.7 appliance:

    # lvextend --resizefs --size +9GB /dev/VG-CFME/lv_var

3.5. Migrating from CFME 5.9 to 5.10

Perform the following steps on your CloudForms VMDB, non-VMDB and dedicated database appliances to migrate to CFME 5.10. Migrate all remote regions before migrating the global region. See Chapter 1, Ordering Migrations for more information.

Note

Some steps are run on certain appliances. Ensure you wait for each command to finish before going to the next step.

  1. Connect to the appliance using SSH.
  2. On the VMDB and non-VMDB appliances, stop the evmserver process:

    [root@VMDB]# systemctl stop evmserverd
    [root@non-VMDB]# systemctl stop evmserverd
  3. Update packages on all appliances:

    [root@VMDB]# yum update
    [root@non-VMDB]# yum update
    [root@dedicatedDB]# yum update
  4. On the VMDB and dedicated database appliances, restart the server to load the new version of the pglogical library:

    [root@VMDB]# systemctl restart $APPLIANCE_PG_SERVICE
    [root@dedicatedDB]# systemctl restart $APPLIANCE_PG_SERVICE
  5. On the VMDB and non-VMDB appliances, log out of the appliance, then log back in to fully reload Ruby.

    Important

    Failure to log out of the old shell environment and back into a new one results in an error reporting a Ruby gem is missing.

    If you did not log out and log back in and received the error message about the missing gem, you can re-login and continue with the next step. But if you ran bundle install after the error, you must run yum reinstall cfme-gemset to fix the broken environment in order to continue.

  6. On the VMDB and non-VMDB appliances, change to the vmdb directory:

    [root@VMDB]# cd /var/www/miq/vmdb/
    [root@non-VMDB]# cd /var/www/miq/vmdb/
  7. On the VMDB and non-VMDB appliances, run the below command appropriate to your environment to migrate everything in the database to work with the latest 5.10 configuration:

    1. In a single (standalone database) or replicated environment, run this command on each VMDB appliance:

      [root@VMDB]# rake db:migrate
    2. In a dedicated database or highly available environment, run this command on a single non-VMDB appliance pointed at that environment:

      [root@non-VMDB]# rake db:migrate
  8. On the VMDB and non-VMDB appliances, update the Automate Model to the latest version. This resets the ManageIQ and Red Hat domains (base domains) to a new and upgraded version. Run the command appropriate to your environment to update the Automate Model:

    1. In a single (standalone database) or replicated environment, run this command on each VMDB appliance:

      [root@VMDB]# rake evm:automate:reset
    2. In a dedicated database or highly available environment, run this command on a single non-VMDB appliance pointed at that environment:

      [root@non-VMDB]# rake evm:automate:reset
  9. On the VMDB and dedicated database appliances, restart PostgreSQL:

    [root@VMDB]# systemctl restart rh-postgresql95-postgresql
    [root@dedicatedDB]# systemctl restart rh-postgresql95-postgresql
  10. On the VMDB and non-VMDB appliances, start the evmserver process:

    1. In a single (standalone database) or replicated environment, run this command on each VMDB appliance:

      [root@VMDB]# systemctl start evmserverd
    2. In a dedicated database or highly available environment, run this command on each non-VMDB appliance:

      [root@non-VMDB]# systemctl start evmserverd

3.6. Additional Configuration for High Availability Environments

Red Hat CloudForms 4.6 high availability environments require additional configuration after upgrading to CloudForms 4.7.

Run the following commands on all VMDB and dedicated database appliances:

  1. Update packages:

    # yum update
  2. Reload the systemd manager configuration:

    # systemctl daemon-reload
  3. Stop the repmgrd daemon. This prevents unwanted failovers and ensures repmgrd will start properly after the upgrade:

    # systemctl stop rh-postgresql95-repmgr
  4. Restart postgresql:

    # systemctl restart $APPLIANCE_PG_SERVICE
  5. Verify that shared_preload_libraries lists repmgr:

    # psql -d vmdb_production -c 'show shared_preload_libraries'
    Note

    If repmgr_funcs shows in shared_preload_libraries manually change the value in the postgresql configuration file.

On the primary database-only appliance(node):

  1. Edit /etc/repmgr.conf to:

    node_id=<id>
    node_name=<hostname>
    conninfo='host=<hostname> user=root dbname=vmdb_production'
    use_replication_slots=1
    pg_basebackup_options='--xlog-method=stream'
    failover=automatic
    promote_command='repmgr standby promote -f /etc/repmgr.conf --log-to-file'
    follow_command='repmgr standby follow -f /etc/repmgr.conf --log-to-file --upstream-node-id=%n'
    log_file=/var/log/repmgr/repmgrd.log
    service_start_command='sudo systemctl start rh-postgresql95-postgresql'
    service_stop_command='sudo systemctl stop rh-postgresql95-postgresql'
    service_restart_command='sudo systemctl restart rh-postgresql95-postgresql'
    service_reload_command='sudo systemctl reload rh-postgresql95-postgresql'
    data_directory='/var/opt/rh/rh-postgresql95/lib/pgsql/data'
  2. Execute the command string 'CREATE EXTENSION repmgr FROM unpackaged':

    # psql -d vmdb_production -c 'CREATE EXTENSION repmgr FROM unpackaged'
  3. Initialize repmgr installation and register the primary node:

    # su -c 'repmgr primary register -f /etc/repmgr.conf --force' - postgres

On the standby database nodes:

  1. Edit /etc/repmgr.conf to:

    node_id=<id>
    node_name=<hostname>
    conninfo='host=<hostname> user=root dbname=vmdb_production'
    use_replication_slots=1
    pg_basebackup_options='--xlog-method=stream'
    failover=automatic
    promote_command='repmgr standby promote -f /etc/repmgr.conf --log-to-file'
    follow_command='repmgr standby follow -f /etc/repmgr.conf --log-to-file --upstream-node-id=%n'
    log_file=/var/log/repmgr/repmgrd.log
    service_start_command='sudo systemctl start rh-postgresql95-postgresql'
    service_stop_command='sudo systemctl stop rh-postgresql95-postgresql'
    service_restart_command='sudo systemctl restart rh-postgresql95-postgresql'
    service_reload_command='sudo systemctl reload rh-postgresql95-postgresql'
    data_directory='/var/opt/rh/rh-postgresql95/lib/pgsql/data'
  2. Initialize repmgr installation and register the standby node:

    # su -c 'repmgr standby register -f /etc/repmgr.conf --force' - postgres
  3. Start repmgrd:

    # systemctl start rh-postgresql95-repmgr

On all CloudForms appliances:

  1. Restart the evm-failover-monitor:

    # systemctl restart evm-failover-monitor