Server Recovery method
AIX has mksysb
HP has make_recovery
Solaris has Flash archive (ok it isn't very good but still)
Redhat has ... freeware applications that attempt to cover the above requirements. Baremetal, etc.
Why not provide a fully supported product, one that comes with the OS. Save to ISO image and then let the user handle it from there.
Just seems like a big hole that shouldn't be that hard to fill.
Responses
Guru
3262 points
You state in your note "freeware applications that attempt to cover the above requirements." What are the "above requirements"?
All of the conventional UNIX bare-metal backup and restore tools exist and are useable with RHEL. tar, cpio, dump, rsync, dd, etc.
You can place a system backup on an ISO image by using cpio and mkisofs.
Community Member
98 points
Dissimilar hardware is where the issue really is. If you have a machine truly die, you will not get exact hardware. When life cycling hardware you will also have issues with drivers. Similar issues with P2V or V2P conversions.
We have about 3.000 server:
200 linux (130 RHEL)
50 AIX
10 SOLARIS
rest WINDOWS
In the past we had also HP-UX servers.
I'm root of linux/aix/solaris servers.
On linux we need the equivalent of mksysb for AIX or make_recovery for HP-UX.
On AIX we schedule weekly mksysb onto a NetAPP share which is replicated on the DR site with no down time. I case of disaster we can recovery our aix systems in the dr site using new hw restoring the mksysb (before we did this also with make_recovery on HP-UX).
With our linux system we are in trouble. Because of this lack we are forced to install most of our RHEL on VMs but when we need a lot of RAM, CPU, disk this is not optimal.
This is an issue we're confronting right now; we're a sizeable shop with about 600+ AIX servers, and a growing (~200) list of Linux servers. One hole in our supportability of Linux is exactly this; the ability to recover from a backup image of a server, such as in the case of a patching gone wrong, fat-fingered sys admin, etc.
AIX does have mksysb, but IBM also offers a 'wrapper' that used to be called sysback, now 'blue washed' and called Tivoli somethingorother. Sysback allows for restore onto dissimilar metal, adding or removing LVM mirrors and/or backup/restore of whole non-root volume groups, resizing of physical extents, etc. Very nice, smooth, easy to use and Rock Solid. We even use sysback over the network to 'clone' AIX servers, it allows for post-recovery removal of all networking configs.
We're looking into the BMR product integrated into NetBackup, but it's a bear. It seems all planets have to align, incense burned, and a chant to get it to work.
AIX also has a couple of other handy commands for making "alt root" volume groups or even bootable filesystems within the current rootvg.
True, Linux has lots of low-level tools like tar and cpio and mkisofs, but then you need to figure out how to put those tools together and in effect everyone ends up reinventing the wheel to accomplish the same end result.
Would be so much better if RH could roll a tool like sysback/mksysb that everyone could just use without having to make their own.
Community Member
98 points
We use Storix www.storix.com for our bare metal image backup/recovery on Linux. It is a very solid product. That said we have also tried Mondo Rescue www.mondorescue.org as a free alternative without much success. I so wish that Red Hat would just purchase Storix and make it available as way of recovery. It is a great tool and handles P2V, V2P and disimilar hardware easily. It also supports several other operating systems, but we have free tools that come with the OS to handle those.
Red Hat's answer of reimaginge the server by reinstalling the OS and then applications and configuration files with Satellite does not satify our need to be able to quickly recover to our exact configuration before an OS upgrade or allow us to move around images easily.
We have 1000 linux systems today, and it is expected to keep growing - our biggest issue today, is the ability to restore the OS to the last good backup, to recover the exact configuration of the system; in a simple and efficient manner. Which can be standardized and documented to a fairly experienced linux admin can complete the process.
99% of restores would be the to original hardware; but some will be DR restores, so different hardware - and on process for both would be wonderful.
We have 1000 linux systems today, and it is expected to keep growing - our biggest issue today, is the ability to restore the OS to the last good backup, to recover the exact configuration of the system; in a simple and efficient manner. Which can be standardized and documented to a fairly experienced linux admin can complete the process.
99% of restores would be the to original hardware; but some will be DR restores, so different hardware - and on process for both would be wonderful.
Which obviously overlap a lot, but imho the goal with imaging is to be able to, in a very consistent and easy way, restore your OS from bare metal to a state from where you can launch your traditional restore.
Most backup solutions tend to assume you'll reinstall your OS, the backup agent(s), and configure it all, before starting the restore. And backups generally won't tell you how your hardware was set up, it's just a filesystem-level copy. That's the gap that mksysb fills. Apps and app data can be left to traditional backups.
Personally I don't care about booting from tape or iso, a network-based system via e.g. PXE + HTTP much like kickstart would have the least limitations and be the easiest imho.
Simply capturing basic system info (networking, partition layout, etc) and being able to boot up a glorified RHEL rescue mode without .iso's (from PXE) that could fetch and set up this basic config from e.g. the kickstart server, and make it easier for me to customize the rescue mode (install backup agents) and it'd be a good start, and good enough for me.
Active Contributor
136 points
Hi
we had the same Problem in early 2006 and found a Company in Germany that implement a DR Tool for us.
It's now also public available as Product
http://www.atix.de/en/produkte/com.oonics/modul-d
Before on Tru64 we had something similar:
http://www.unix-wissen.de/Tru64/cloning/
With the Enterprise Copy you are able to Clone your Installation. We use it only for the OS. We add another Grub entry and you are able boot into the Clone direct from the Bootmenu. Also if you have a SAN Installation you are able to map your Disk to a new Server and can also boot it.
Also this Tool is integrated with DR / Live DVD Tools, so you can create your own DR DVD.
We use that very often, would be great if Red Hat could implement such a Tool in RHEL7
Mike
When we started our Linux service finding a replacement for tools like flash archive were a key requirement. We ended up settling on System Imager . Its served us extremely well for RHEL 4 and 5. We use it on any system that has DR requirements to create an image offsite. I even submitted an RFE to Red Hat to have this evaluated/included in future RH releases. Unfortunately, there isn't much activity from the maintainers so I don't think RH is willing to adopt it.
Having said that something is needed and I have to wonder if any of the virt tools could/would be capable of filling this gap (P2V + V2P = P2P).
I have try Relax and Recover (abbreviated ReaR) http://rear.sourceforge.net/ which is also in epel repo and its work great. It's also included in fedora from version 12, so I think they will put in rhel soon. By the way, its also officialy included in suse.
Hi,
I thought the OS install is not hard to install, but one needs to track the customised data eg. passwd, groups, ip etc. I'd assume that in case of disaster, one gets some new h/w, install the O/S, load the customised data (passwd, group files), then import the data volume groups. I think AIX has an advantage that it uses IBM h/w and the import of volume group always work. I use mksysb a lot on AIX and it works really well.
In RHEL, I'd assume the way to organise data is to have a small OS with minimum customised data and huge volume groups on other disks/storage to the OS. So if the h/w and/or O/S goes wrong, the data volume groups can be imported into another working h/w and O/S. If this is the case, then the trick seems to be securing the data on the volume groups and not the O/S. There are utilities like tar, cpio which can track these data. (But not the size of the filesystem on the volumr groups, I suppose).
The problem which I can't see a way of tracking easily are the chkconfig on/off, installed rpm's etc. Assuming the O/S is really minimal, then maybe a Desktop/Server install plus a few rpm's will suffice. Or arguably use a kickstart install file. This will still leave the chkconfig on/off etc to deal with. I maybe missing something on this front though, my linux knowledge is patchy.
Lina
I'm a platform engineer at a giant pharma and we have thousands of UNIX servers, mostly Solaris. Our base of RHEL systems has shot past 1,000 this year and everything new seems to be moving to RHEL.
Every enterprise datacenter needs to be able to recover every system to exactly the way it was before it crashed and burned. Not by starting over with a fresh installation and reinstalling everything; we need to restore an exact copy of what was on the system drive including all of the changes and tweaks, including every ownership and permission. Since RH has no solution, I wound up using freeware called Mondo Rescue, which did the job with a lot of manual intervention, guesswork and trial and error. I finally had to dig into the source code to fix and enhance it to the point where it's reliable and supportable. Now all of our RHEL systems save an archive image on a NAS share once a week. We can restore images across many different physical and virtual platforms. We do it every time we have a DR exercise and SunGard gives us whatever they happen to have available. Works like a charm.
The point is that this is a basic requirement of anything that calls itself an enterprise operating system. Every UNIX species has some form of this functionality available; Red Hat has nothing.
Although our home-grown solution works, we should not have had to expend the effort to create it. We would much prefer that it be a supported part of the OS distribution.
Red Hat seems to have a lingering hobbyist / SOHO mentaility that really holds RHEL back from being a true top tier enterprise operating system.
The approach needs to be to look at what Solaris, AIX and HP-UX can do and what they include and then make sure that RHEL can match them feature for feature.
Guru
2135 points
Isn't this the kind of problem that Satellite & System Engine are aimed at?
None of my config is stored solely on a server. Config management is done using Puppet for us. This keeps a system's config in line with the defined standards. I prefer doing my network config direct from DHCP so that when a server crash does occur, I know that I can re-provision it from scratch quicker than I can try to diagnose the problem.
I know that exactly the same software will be installed on the rebuild. It will be patched to exactly the same level as it was before crash. The config on the system will also be deployed to the latest released config for the system.
Any static data is either stored on alternative drives, network, SAN or backed up for restoring after crash.
This way I know that I can rebuild any machine within minutes. Faster than I've managed on any kind of ISO restore.
Duncan
I totally agree with Duncan here. You should not need to be in need of this type of tooling. Simply redeploy. It forces you (and your collegues) to not do one-off adjustments on servers, because they will get lost on redeploy. Any change on a server is discussed, auto-documented and neatly deployed this way. Backup should only be needed for data, not systems.
and is there any sign of this in rhel7 ...
Interested to know how/when you plan to use this in your environment?
This is something I was looking for several years back when installing on bare metal was common (and the AIX admins used to gloat about mksysb), but to be honest I haven't installed Linux on anything but a hypervisor for at least 18 months now, so this hardly seems relevant anymore... and with puppet configuration from a base image.. I am having trouble thinking of a use for it.
I understand different people have different requirements, so interested to know if you are using it to clone systems, migrate to different hardware? or just for disaster recovery etc.?
A lot of enterprise level backup tools now support bare metal restore of Linux (something that wasn't supported when I was looking) and may be worth investigating if you've already got them licensed at your site. eg. Commvault 1 touch restore / Acronis.
All that being said.. it would be nice to run a single command and have full RHEL backup!
We have automated rebuilds, snapshots for virtual and lvm but its not gaureented across our environment its just too diverse at this time, applications are intergrated to deeply with the O/S. I would agree virtual and even more so containers would alleviate this need, were just not there yet. Our use case is one of system maintenace updates and gaureenting ability to rollback a system to the pre update state within an alloted maintenance window e.g. change risk controls. I guess some comes down to poor application design but never the less for an O/S that likes to put the word "enterprise" in its title you think this would be included.
I agree that using a configuration management system like Puppet, Ansible, Chef, etc. is a great way to be able to provision servers no matter the underlying hardware (combined with kickstart installs, ideally). I used mondorescue for backup and migration purposed and it worked ok, but now rear is available as of RHEL 6.8. I may give it a try eventually (hopefully before the next time I need it).
Yes, I'd vote for Rear. It is serving the purpose as expected. I can see rear packages included in the standard image in RHEL6.8 on-wards.
Hello,
Red Hat has official documentation for ReaR:
Test your backups before relying on them!
Feedback on the above documentation is welcome.
I'd like to second (third, fourth, etc..) the endorsement of ReaR. It's an opensource project that has really matured in the past few years and is now an excellant option for RHEL server recovery. We've collected our notes together and put together an online article that documents our most common configuration options: http://carroll.net/blog/red-hat-bare-metal-backup/
Note: We are no connected to the ReaR project -- just satisfied users.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.