Chapter 6. Migrating Embedded etcd to External etcd

6.1. Overview

Until OpenShift Container Platform 3.6, it was possible to deploy a cluster with an embedded etcd instance. This embedded etcd instance was deployed on your OpenShift Container Platform instance. Starting in OpenShift Container Platform version 3.7, this is no longer possible.

Important

Embedded etcd was not supported in high availability clusters. If you use a high availability cluster, do not migrate etcd.

This migration process performs the following steps:

  1. Stop the master service.
  2. Perform an etcd backup of embedded etcd.
  3. Deploy external etcd (on the master or new host).
  4. Perform a backup of the original etcd master certificates.
  5. Generate new etcd certificates for the master.
  6. Transfer the embedded etcd backup to the external etcd host.
  7. Start the external etcd from the transfered etcd backup.
  8. Re-configure master to use the external etcd.
  9. Start master.

Additionally, the etcd API version since OpenShift Container Platform 3.6 defaults to v3. Also, since OpenShift Container Platform 3.7, v3 is the only version allowed. Therefore, older deployments with embedded etcd with the etcd API version v2 need to migrate to the external etcd first, followed by data migration, before they can be upgraded to OpenShift Container Platform 3.7.

6.2. Running the Automated Migration Playbook

Migration to external RPM etcd or external containerized etcd is currently supported.

A migration playbook is provided to automate all aspects of the process; this is the preferred method for performing the migration. You must have access to your existing inventory file with both the master and external etcd host defined in their separate groups.

In order to perform the migration on Red Hat Enterprise Linux Atomic Host, you must be running Atomic Host 7.4 or later.

  1. Add etcd under the [OSEv3:children] section if it does not already exist: 

    [OSEv3:children]
masters
nodes
etcd

  2. Your inventory file is expected to have exactly one host in an [etcd] host group. In most scenarios, it is best to use your existing master, as there is no need for a separate host.

    Add an [etcd] host group to your inventory file if it does not already exist, and list the host to migrate your etcd to: 

    [etcd]
master1.example.com
    Important

    If you find etcd in the [OSEv3:children] section, and the [etcd] host group already contains host names, you do not need to migrate etcd. Do not follow the remaining steps.

  3. Pull the latest subscription data from Red Hat Subscription Manager (RHSM): 

    # subscription-manager refresh

  4. To get the latest playbooks, manually disable the OpenShift Container Platform 3.6 channel and enable the 3.7 channel on the host you are running the migration from: 

    # subscription-manager repos --disable="rhel-7-server-ose-3.6-rpms" \
    --enable="rhel-7-server-ose-3.7-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-fast-datapath-rpms"
# yum clean all

  5. Run the embedded2external.yml playbook using your inventory file: 

    # ansible-playbook [-i /path/to/inventory] \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-etcd/embedded2external.yml

    Successful completion of the playbook will show the following: 

    INSTALLER STATUS **************************************
Initialization             : Complete
etcd Install               : Complete

  6. To verify that the migration from embedded to external etcd was successful, run the following on the etcd host and check for an etcd process: 

    # ps -aux | grep etcd
etcd      22384  2.1  3.9 5872848 306072 ?      Ssl  10:36   0:02 /usr/bin/etcd --name=master1.example.com --data-dir=/var/lib/etcd/ --listen-client-urls=https://192.168.122.197:2379

6.3. Running the Manual Migration

Currently, manual migration is not recommended, as it requires a deployment of the new etcd cluster and re-deployment of etcd master certificates.