What is Relax and Recover(ReaR) and how can I use it for disaster recovery?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6
  • Package: rear

Issue

  • Red Hat Enterprise Linux now contains Relax and Recover(ReaR) which can be used for image based backups. What is it? Any example for using this backup utility?

Resolution

What's Relax and Recover

Relax-and-Recover(ReaR) is a recovery and system migration utility. The utility produces a bootable image and restores from backup using this image. It also allows to restore to different hardware and can therefore be used as a migration utility as well.

ReaR is available for Red Hat Enterprise Linux (RHEL):

Support Note: Support of ReaR with RHEL is intended for the image based recovery capabilities. While the package includes all of the upstream capabilities, please note that there are many options that are not well tested.

How to set up ReaR to backup with NFS method

Before setting up ReaR, you need to make a backup-restore plan according to your environment. In this tutorial, we will:

  • Set up an NFS server for holding the backup files.
  • Back up the files to the NFS server and store the disaster recovery system to an ISO image.
  • Boot the restore server with disaster recovery system and finish the recovery.

Detailed steps are as follows:

  1. Create an export on the NFS server(in this example, it has the IP address 192.168.56.1). We use /storage directory for storing the back up files later.

    # yum install nfs-utils
    
    # mkdir /storage
    
    # cat /etc/exports
    /storage        *(fsid=0,rw,sync,no_root_squash,no_subtree_check,crossmnt)
    
    # service nfs start
    
  2. Install packages which are required for creating ISO image on the backup server.

    # yum install rear genisoimage syslinux
    
  3. Modify the configuration file /etc/rear/local.conf on the backup server with these settings.

    # cat /etc/rear/local.conf
    OUTPUT=ISO
    OUTPUT_URL=nfs://192.168.56.1/storage
    BACKUP=NETFS
    BACKUP_URL=nfs://192.168.56.1/storage
    BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')
    NETFS_KEEP_OLD_BACKUP_COPY=
    

    Be careful when changing the BACKUP_PROG_EXCLUDE variable. Always preserve the default excludes to avoid issue described in solution RHEL 7 restored from ReaR doesn't boot if custom backup excludes were used for backup.

  4. Create the disaster recovery system and generate backup files.

    # rear -d -v mkbackup
    

    Rear will examine the current backup system and gather necessary information, like the disk layout, files to be excluded and the boot loader, etc. Then a bootable ISO image with disaster recovery system will be created under /var/lib/rear/output by default. At last the files to be backed up as well as the bootable ISO image will be transfered to the NFS server.

  5. Recovery test.
    5.1 Burn the bootable ISO image to a CD or DVD.
    5.2 Boot the test server from the recovery medium.
    5.3 Select Recover <hostname>. Here, the hostname is the backup server's name.
    5.4 Login as root without password.
    5.5 Recover the backup to this test server:

    # rear -d -v recover
    

    5.6 After restoring the backup, you may choose to stay in the shell of the disaster recovery system to check the recreation on the test server. If everything is okay, reboot the test server and starts from local harddisk.
    TIPS:You can also choose the Automatic Recover <hostname> to automate the recovery process, as long as no error occurs when you manually recover the test server.

  6. In case a disaster has occurred, you can proceed recovery steps as described in Recovery test section on a server which has been tested and has the same hardwares as the backup server.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

24 Comments

  1. How do you have rear include the hostname of the server you are creating the iso for ? If you have 100 servers to generate backup iso for recover and send to the same nfs location. How do you have rear include the hostname on each iso it generates from each server ?
  2. Can you set rear to automate iso creation for backup so you have new copies available for your system incase it crashes?

Thanks

Hello Arrey, you can define the "ISO_PREFIX" variable in /etc/rear/local.conf, for example:

ISO_PREFIX="$(hostname 2>/dev/null)"

For details, feel free to open a case with us. Have a nice day!

Thank you.

According to the man-page, the default is to use a naming-convention specifying the hostname; though, I'm not sure if this was also the case at the time of your question.

OUTPUT=ISO                                                                                                      │
           (Default) Create a bootable ISO9660 image on disk as rear-$(hostname).iso

Much appreciated.

Could someone, please, "examinate" this post for grammar errors and syntactical issues?

In the meantime, this seems like an exciting new tool to facilitate backup/recovery procedures. I hope to see some more extensive documentation soon.

Using this line I got it to work under RHEL7.2 using NETFS and a GPT based system BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/crash')

Excluding /var/tmp gave this error situation on restore

WARNING!! initramfs creation for Kernel 3.10.0-327.13.1.el7.x86_64 failed. Please check /var/log/rear/rear-hostname.log to see the error messages in detail and decide your self, wether the system will boot or not. ... Installing GRUB2 boot loader Finished recovering your system.

And the restore system cant boot because of the bad initramfs

The /etc/rear/local.conf that worked for us TMPDIR=/mnt/rear_tmp OUTPUT=ISO BACKUP=NETFS NETFS_URL=file:///mnt/rear_tmp

BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash' '/mnt')

BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/crash' '/mnt')

Hello Joseph, could you please open a case with us? We can help to look into your issue further. Thanks!

I opened a case and closed it once I figured out our workaround.

As far as I could track it down We have this setup in $HOSTNAME/etc/fstab

/tmp /var/tmp none bind 0 0

This is local site policy for CIS compliance.

Removing the /var/tmp from exclusion in /etc/rear/local.conf allowed the ReaR utility to recreate the initramfs when using the DVD+USB recovery method.

Thanks!

Is it possible to restore the system through network (pxe boot). How the support for this product from redhat? Need to have separate subscription? Which all systems (RHEL Versions) supported by this product?

Hello all, Would like to consult step 5.1, "5.1 Burn the bootable ISO image to a CD or DVD."... Can we support boot ISO from network, in some automated way ?

Thank you for your answers ! Regards, Stefan

In steps 2 & 3, the "backup server" seems to be the system that needs to be backed up, not the NFS server where the backup images are stored. On this "backup server", the NFS share must be mounted, but the step is missing.

Please also see : https://access.redhat.com/solutions/3213391 as you may also need to install grub2-efi-x64-modules via "yum -y install grub2-efi-x64-modules" as part of rear install.

Can we use this method to take backup of Physical server? Does it will restore application also which is installed in system?

is it possible to configure this so the backup data is wrriten to nfs as above, but the boot image is written to a usb key left plugged in?

Hello, This should be possible. OUTPUT controls where ISO is stored, BACKUP controls where backup will:

OUTPUT=USB
USB_DEVICE=/dev/disk/by-path/pci-0000:00:04.7-usb-0:4:1.0-scsi-0:0:0:0-part1
BACKUP=NETFS
BACKUP_URL=nfs://<nfsserver>/<path>

Here I formatted a USB disk with 1 partition and EXT4 on it (it cannot be VFAT).

Hello Team, Can I use this tools into my hypervisor hosts which is in cluster. If yes then, please tell me the procedure. If I use tool and install new rpm then will those RPM do any effect on my servers. My servers are RHEV-M & RHEV hypervisor.

So, on the rescue system I dont need to install RHEL or OS. it should be just blank bare metal ? and rear will do the rhel install + device/partition layout same as the original system ?

Indeed. You boot the rescue ISO created by ReaR and it will recover your partitions then restore the backup of the system.

ReaR is a great tool for baremetal backup on linux. We've been using it successfully for quite some time. We created a article that describes the step-by-step procedure to use this tool http://carroll.net/blog/red-hat-bare-metal-backup/

Their bash scripts may be a marvel of coding to behold, but it is the outcome that matters. In your own words, recovering with ReaR will fail several times and you have to chase dependecies. I ran into an issue that seems to not be addressed and the case is simply closed with no resolution in sight. The project might as well be dead if they are meeting issues of users that way, closing threads on github without resolution. Their code for example attempts to mount without specifying the filesystem type. If mounter doesn't like that (and many times it seems from github posts it doesn't) the whole recovery process fails. Issue is posted, discussion ensued and then the climax was case was closed. Resolution? None.

Gjorgi, I work in Red Hat support, and I happened to notice your comment left here. I have relayed this internally to Red Hat product management in charge of ReaR with regards to RHEL. Just wanted to let you know that we have noticed the feedback, and we appreciate it.

in May, I discovered that Relax and Recover(ReaR) does not work with Bacula Enterprise Edition client because it installs things to different paths that the Community Edition does. The bacula-client in both the Red Hat and CentOS main repositories installs things to the same locations as does the Community Edition.

I opened an issue with the maintainers of rear and I think I can fix it.

How long would it take for the fix to propagate into the Red Hat repositories ?

Hi I'm very much new to ReaR, I need to recover a RHEL 6.6 server, but my generated ISO image seems to be very small(95 MB). Does ReaR backup my entire root volume with all LVs to ISO?(much like AIX mksysb of rootvg) Or am I missing something in my ReaR config

/etc/rear/local.conf OUTPUT=ISO

OUTPUT_URL=file:///vbs_archive/rear_z/

OUTPUT_URL=file:///sapnim_nfs/rear_backups/

BACKUP=NETFS OUTPUT_URL=null BACKUP_URL="iso:///vbs_archive/rear_z/" ISO_DIR="///sapnim_nfs/rear_backups/" ONLY_INCLUDE_VG=( "rootvg" )

[root@HOSTNAME_CHANGED rear_backups]# vgs VG #PV #LV #SN Attr VSize VFree crashvg 1 1 0 wz--n- 80.13g 0 dbvg 1 1 0 wz--n- 16.02g 24.00m mirrorvg 1 1 0 wz--n- 16.02g 24.00m onlinevg 1 1 0 wz--n- 256.00g 0 rootvg 2 9 0 wz--n- 237.99g 260.00m stream1vg 1 1 0 wz--n- 384.00g 0 stream2vg 1 1 0 wz--n- 384.00g 0 stream3vg 1 1 0 wz--n- 384.00g 0 stream4vg 1 1 0 wz--n- 384.00g 0 stream5vg 1 1 0 wz--n- 384.00g 0 stream6vg 1 1 0 wz--n- 384.00g 0 stream7vg 1 1 0 wz--n- 384.00g 0 stream8vg 1 1 0 wz--n- 384.00g 0 sybaseoldvg 3 3 0 wz--n- 48.07g 18.07g sybasevg 1 6 0 wz--n- 256.00g 50.00g vbsarchvg 2 1 0 wz--n- 1.00t 12.84g vbsvg 1 1 0 wz--n- 512.00g 107.00g You have mail in /var/spool/mail/root [root@HOSTNAME_CHANGED rear_backups]#