Red Hat Training

A Red Hat training course is available for Red Hat Enterprise Linux

Chapter 21. Xen live migration

The Xen hypervisor supports Virtualization Migration for para-virtualized guests and fully virtualized guests. Migration is only supported on Red Hat Enterprise Linux 5.1 and newer systems. Migration can be performed offline or live.
  • Offline migration suspends the guest on the original host, transfers it to the destination host and then resumes it once the guest is fully transferred. Offline migration uses the virsh migrate command. 
    # virsh migrate GuestName libvirtURI
  • A live migration keeps the guest running on the source host and begins moving the memory without stopping the guest. All modified memory pages are monitored for changes and sent to the destination while the image is sent. The memory is updated with the changed pages. The process continues until the amount of pause time allowed for the guest equals the predicted time for the final few pages to be transfer. The Xen hypervisor estimates the time remaining and attempts to transfer the maximum amount of page files from the source to the destination until Xen predicts the amount of remaining pages can be transferred during a very brief time while the guest is paused. The registers are loaded on the new host and the guest is then resumed on the destination host. If the guest cannot be merged (which happens when guests are under extreme loads) the guest is paused and then an offline migration is started instead.
    Live migration uses the --live option for the virsh migrate command. 
    # virsh migrate--live GuestName libvirtURI

Important

Migration is presently unsupported on the Itanium® architecture.
To enable migration with Xen a few changes must be made to the /etc/xen/xend-config.sxp configuration file. By default, migration is disabled as migration can be a potential security hazard if incorrectly configured. Opening the migration port can allow an unauthorized host to initiate a migration or connect to the migration ports. Authentication and authorization are not configured for migration requests and the only control mechanism is based on hostnames and IP addresses. Special care should be taken to ensure the migration port is not accessible to unauthorized hosts.

Important

IP address and hostname filters only offer minimal security. Both of these attributes can be forged if the attacker knows the address or hostname of the migration client. The best method for securing migration is to isolate the network from external and unauthorized internal connections.
Enabling migration

Modify the following entries in /etc/xen/xend-config.sxp to enable migration. Modify the values, when necessary, and remove the comments (the # symbol) preceding the following parameters:

(xend-relocation-server yes)
The default value, which disables migration, is no. Change the value of xend-relocation-server to yes to enable migration.
(xend-relocation-port 8002)
The parameter, (xend-relocation-port), specifies the port xend should use for the relocation interface, if xend-relocation-server is set to yes
The default value of this variable should work for most installations. If you change the value make sure you are using an unused port on the relocation server.
The port set by the xend-relocation-port parameter must be open on both systems.
(xend-relocation-address '')
(xend-relocation-address) is the address the xend listens for migration commands on the relocation-socket connection if xend-relocation-server is set.
The default is to listen on all active interfaces. The (xend-relocation-address) parameter restricts the migration server to only listen to a specific interface. The default value in /etc/xen/xend-config.sxp is an empty string(''). This value should be replaced with a single, valid IP address. For example: 
(xend-relocation-address '10.0.0.1')
(xend-relocation-hosts-allow '')
The (xend-relocation-hosts-allow 'hosts') parameter controls which hostnames can communicate on the relocation port.
Unless you are using SSH or TLS, the guest's virtual memory is transferred in raw form without encryption of the communication. Modify the xend-relocation-hosts-allow option to restrict access to the migration server.
If the value is empty, as denoted in the example above by an empty string surrounded by single quotes, then all connections are allowed. This assumes the connection arrives on a port and interface which the relocation server listens on, see also xend-relocation-port and xend-relocation-address.
Otherwise, the (xend-relocation-hosts-allow) parameter should be a sequence of regular expressions separated by spaces. Any host with a fully-qualified domain name or an IP address which matches one of these regular expressions will be accepted.
An example of a (xend-relocation-hosts-allow) attribute: 
(xend-relocation-hosts-allow '^localhost$ ^localhost\\.localdomain$')
After you have configured the parameters in your configuration file, restart the Xen service. 
# service xend restart

21.1. A live migration example

Below is an example of how to setup a simple environment for live migration. This configuration is using NFS for the shared storage. NFS is suitable for demonstration environments but for a production environment a more robust shared storage configuration using Fibre Channel or iSCSI and GFS is recommended.
The configuration below consists of two servers (et-virt07 and et-virt08), both of them are using eth1 as their default network interface hence they are using xenbr1 as their Xen networking bridge. We are using a locally attached SCSI disk (/dev/sdb) on et-virt07 for shared storage using NFS.
Setup for live migration

Create and mount the directory used for the migration: 

# mkdir /var/lib/libvirt/images
# mount /dev/sdb /var/lib/libvirt/images

Warning

Ensure the directory is exported with the correct options. If you are exporting the default directory /var/lib/libvirt/images/ make sure you only export /var/lib/libvirt/images/ and not/var/lib/xen/ as this directory is used by the xend daemon and other tools. Sharing /var/lib/xen/ will cause unpredictable behavior. 
# cat /etc/exports
/var/lib/libvirt/images  *(rw,async,no_root_squash)
Verify it is exported via NFS: 
# showmount -e et-virt07
Export list for et-virt07:
/var/lib/libvirt/images *
Install the guest

The install command in the example used for installing the guest: 

# virt-install -p -f /var/lib/libvirt/images/testvm1.dsk -s 5 -n\
testvm1 --vnc -r 1024 -l http://example.com/RHEL5-tree\
Server/x86-64/os/ -b xenbr1
For step by step installation instructions, see Chapter 8, Guest operating system installation procedures.
Verify environment for migration

Make sure the virtualized network bridges are configured correctly and have the same name on both hosts: 

[et-virt08 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr1          8000.feffffffffff       no              peth1
vif0.1

[et-virt07 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr1          8000.feffffffffff       no              peth1
vif0.1
Verify the relocation parameters are configured on both hosts: 
[et-virt07 ~]# grep xend-relocation /etc/xen/xend-config.sxp |grep -v '#'
(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-address '')
(xend-relocation-hosts-allow '')

[et-virt08 ~]# grep xend-relocation /etc/xen/xend-config.sxp |grep -v '#'
(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-address '')
(xend-relocation-hosts-allow '')
Make sure the relocation server has started and is listening on the dedicated port for Xen migrations (8002): 
[et-virt07 ~]# lsof -i :8002
COMMAND  PID  USER   FD   TYPE  DEVICE SIZE NODE NAME
python 3445 root 14u IPv4 10223 TCP *:teradataordbms (LISTEN)

[et-virt08 ~]# lsof -i :8002
COMMAND  PID USER   FD   TYPE DEVICE SIZE NODE NAME
python 3252 root 14u IPv4 10901 TCP *:teradataordbms (LISTEN)
That the default /var/lib/libvirt/images directory is available and mounted with networked storage on both hosts. Shared, networked storage is required for migrations. 
[et-virt08 ~]# df /var/lib/libvirt/images
Filesystem           1K-blocks      Used Available Use% Mounted on
et-virt07:/var/lib/libvirt/images    70562400   2379712  64598336   4% /var/lib/libvirt/images

[et-virt08 ~]# file /var/lib/libvirt/images/testvm1.dsk 
/var/lib/libvirt/images/testvm1.dsk: x86 boot sector; partition 1: ID=0x83,
active, starthead 1, startsector 63, 208782 sectors; partition 2: ID=0x8e, 
starthead 0, startsector 208845, 10265535 sectors, code offset 0x48

[et-virt08 ~]# touch /var/lib/libvirt/images/foo
[et-virt08 ~]# rm -f /var/lib/libvirt/images/foo
Verify saving and restoring the guest

Start the virtual machine (if the virtual machine is not on): 

[et-virt07 ~]# virsh list
 Id Name                 State
----------------------------------
Domain-0                 running

[et-virt07 ~]# virsh start testvm1
Domain testvm1 started
Verify the virtual machine is running: 
[et-virt07 ~]# virsh list
 Id Name                 State
----------------------------------
Domain-0                 running
testvm1        blocked
Save the virtual machine on the local host: 
[et-virt07 images]# time virsh save testvm1 testvm1.sav
real    0m15.744s
user    0m0.188s
sys     0m0.044s

[et-virt07 images]# ls -lrt testvm1.sav
-rwxr-xr-x 1 root root 1075657716 Jan 12 06:46 testvm1.sav

[et-virt07 images]# virsh list
 Id Name                 State
----------------------------------
Domain-0                 running
Restore the virtual machine on the local host: 
[et-virt07 images]# virsh restore testvm1.sav

[et-virt07 images]# virsh list
 Id Name                 State
----------------------------------
Domain-0                 running
testvm1        blocked
Start the live migration of domain-id from et-virt08 to et-virt07. The hostname you are migrating to and <domain-id> must be replaced with valid values. This example uses the et-virt08 host which must have SSH access to et-virt07 
[et-virt08 ~]# xm migrate --live testvm1 et-virt07
Verify the virtual machine is no longer present on et-virt08 
[et-virt08 ~]# virsh list
 Id Name                 State
----------------------------------
Domain-0                 running
Verify the virtual machine has been migrated to et-virt07: 
[et-virt07 ~]# virsh list
 Id Name                 State
----------------------------------
Domain-0                 running
testvm1        running
Testing the progress and initiating the live migration

Create the following script inside the virtual machine to log date and hostname during the migration. This script performs I/O tasks on the guest's file system. 

#!/bin/bash

while true
do
touch /var/tmp/$$.log
echo `hostname` >>  /var/tmp/$$.log
	echo `date`     >>  /var/tmp/$$.log
	cat  /var/tmp/$$.log
	df /var/tmp
	ls -l  /var/tmp/$$.log
	sleep 3
	done
Remember, that script is only for testing purposes and unnecessary for a live migration in a production environment.
Verify the virtual machine is running on et-virt08 before we try to migrate it to et-virt07: 
[et-virt08 ~]# virsh list
 Id Name                 State
----------------------------------
Domain-0                 running
testvm1        blocked
Initiate a live migration to et-virt07. You can add the time command to see how long the migration takes: 
[et-virt08 ~]# xm migrate --live testvm1 et-virt07
run the script inside the guest: 
# ./doit
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 62 Jan 12 02:26 /var/tmp/2279.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:30 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 124 Jan 12 02:26 /var/tmp/2279.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:30 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:33 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 186 Jan 12 02:26 /var/tmp/2279.log
Fri Jan 12 02:26:45 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:48 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:51 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:54:57 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:55:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:55:03 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 744 Jan 12 06:55 /var/tmp/2279.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007
Verify the virtual machine has been shut down on et-virt08: 
[et-virt08 ~]# virsh list
 Id Name                 State
----------------------------------
Domain-0                 running
Verify the virtual machine has started up on et-virt07: 
[et-virt07 images]# virsh list
 Id Name                 State
----------------------------------
Domain-0                 running
testvm1        blocked
Run through another cycle migrating from et-virt07 to et-virt08. Initiate a migration from et-virt07 to et-virt08: 
[et-virt07 images]# xm migrate --live testvm1 et-virt08
Verify the virtual machine has been shut down: 
[et-virt07 images]# virsh list
 Id Name                 State
----------------------------------
Domain-0                 running
Before initiating the migration start the simple script in the guest and note the change in time when migrating the guest: 
# ./doit
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 62 Jan 12 06:57 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 124 Jan 12 06:57 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 186 Jan 12 06:57 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:00 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 248 Jan 12 02:30 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:03 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 310 Jan 12 02:30 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:03 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:06 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 372 Jan 12 02:30 /var/tmp/2418.log
After the migration command completes on et-virt07 verify on et-virt08 that the virtual machine has started: 
[et-virt08 ~]# virsh list
 Id Name                 State
----------------------------------
Domain-0                 running
testvm1        blocked
and run another cycle: 
[et-virt08 ~]# time virsh migrate --live testvm1 et-virt07
real    0m10.378s
user    0m0.068s
sys     0m0.052s
At this point you have successfully performed an offline and a live migration test.