Introduction

Proper planning and system installation can make the job of recovery fairly painless by separating user data from the root file system, and by installing software systems independent of the vendor's packaging system, and implementing a policy where non-vendor software is installed outside the root file system.

1. Partition the disk.

2. Separate non-vendor data from root file system.

3. Build non-vendor software outside the root file system.

4. Create /backroot file system, mirroring root system.

5. Recovery procedures.

Installation and partitioning

The first step is to partition the disk(s) on the system to provide space for a spare root file system, and a /home file system where all user data and non-vendor software will be installed. Ideally one would put the operating system on one hard disk, and all data and non-vendor programs on one or more separate drives.

The following discussion is based on SuSE 9.2 Professional, but applies to any Linux system, and other *nix system including FreeBSD, SCO OpenServer, etc. although the terms may change slightly.

1. Create a root file system (mount point ``/''). We tend to install all the development software and libraries which requires at least 6GB for SuSE 9.2 Professional. I suggest using the ext3 file system rather than the default reiser file systems as (a) it provides extended attributes (e.g. ``immutable''), and (b) we have seen problems with the reiser file system on occassion.

2. Create a file system named /backroot, of the same size as the root file system. This will not be mounted by default, and will be initially used for a backup of the root file system, and may be used for updates and emergency installs if necessary.

3. Create a swap partition. A good rule of thumb is to make this twice as large as the physical RAM installed (you may want to make it large to provide for future RAM additions.

4. Create a /home partition using the remaining space on the hard drive. This can be either reiser or xfs file systems. Each has its own advantages. The xfs file system may be more robust as it has been in use for years on the SGI systems.

NOTE: If using multiple hard drives, I would create the swap partition first, then split the remainder in half for the / and /backroot partitions.

Create /backroot file system.

When the system is fully installed and stable, create the initial /backroot file system.

mount /backroot
cd /
find . -xdev -print | cpio -pdumv /backroot
vi /backroot/etc/fstab # change the partition numbers for ``/`` and ``/backroot''
umount /backroot
# copy linux entry to linux-back changing the partition numbers and
# root file system to the backroot partition.
vi /boot/grub/menu.lst

Separate non-vendor data and programs from the root file system.

The root file system is normally the first thing to go in a system crash, and is not to be trusted in the event of a cracking event.

Separating data and programs from the vendor's by putting it outside the root file system allows one to reinstall the root system without losing any data or programs that aren't in the vendor's installation.

We have been using the OpenPKG portable packaging system for this since switching from Caldera Linux to SuSe several years ago. This has simplified my life quite a bit, and is what allowed me to recently restore a fairly large ISP mail system in about an hour after a hardware failure on the root file system.

The first thing to do is to copy /usr/local to /home/local, remove /usr/local, and replace it with a symbolic link to /home/local. This will the keep all non-vendor software that is installed under /usr/local on the /home file system where it doesn't have any issues during reinstalls and updates. The FreeBSD ports system, and most Open Source software builds to install by default under /usr/local keeping the vendor's software under /bin, /usr/bin.

Building non-vendor software on Linux

NOTE: Unfortunately much (most) Linux software installs under /usr/bin, and in the root file system which makes it difficult to keep this separate. For this reason, I don't recommend installing binary RPMs or other packages, but recommend starting with the source packages (SRPMS), modifying the .spec files to install under /usr/local, or better under /home/local. This is usually fairly easy. The first thing one should do is to copy the vendor's RPM package directory (/usr/src/packages on SuSE systems) to some appropriate directory, say /home/rpms, and symlink that directory back to the vendor's directory. This

1. Get the appropriate SRPMS package, and install it using the command ``rpm -i $package.src.rpm''. This will put the package in the appropriate subdirectories of the /home/rpms directory you installed above.

2. Edit the package .spec file which will be under the /home/rpms/SPECS directory (one of the major advantages of OpenPKG is that it puts all files associated with a package under a single directory instead of spread it out over several). Change the ``Release:'' of the package. I normally use YYYYMMDD for this rather than using the many inconsistent Release schemes found in different packages. Look for all ``prefix='' lines in the .spec file, changing them to read ``prefix=/home/local''.

3. Rebuild the package with the command ``rpmbuild -ba $package.spec''. This should build a package that installs under /home/local if the package uses the normal GNU, ./configure, make, make install'' installation procedures. If the package doesn't use this, you may have to dig further into the .spec file to see how the author does things. This can be very messy depending on the skill and experience of the .spec file author.

4. Install the package using the command ``rpm -U $binarypackage''.

Recovery Example

This discussion assumes that the system has been installed as described above.

If one is using the OpenPKG package management system, all programs and data are normally installed under a single directory in a directory structure that mimics the standard *nix system.

As an example, we did a recovery on a SuSE 9.2 Linux system in which the root file system became corrupted, probably as the result of not having properly cleaned out a root kit. The disk partitioning on this system before the recovery was:

We should have had a mirror of the root file system in /backroot, bot for one reason or another didn.t.

The first step was to reinstall SuSE 9.2 Professional. During this procedure the disk partitioner was told to install everything in /dev/hda2, the old /backroot file system, and not to touch the other partitions other than to use the same swap partition. After the install, the partition table looked like this:

The first step was to mount the /data1 file system which contained the old root file system.

The next step was to copy old /etc/fstab file to the current file then edit it to change the devices so it looked like this:

Next we unmounted /data1, moved the /data1 directory to /backroot, mounted the file systems, and copied the critical files over to the new system.

umount /data1
mv /data1 /backroot
mount /backroot
mount /home
cd /backroot/etc
cp -a passwd group shadow /etc/
# the name of our OpenPKG file system is ``csrel22''

# add the OpenPKG entries to crontab
grep csrel22 crontab >> /etc/crontab
# copy the OpenPKG startup scripts to /etc
find init.d | grep csrel | cpio -pdumv /etc
# Make the symbolic links for OpenPKG
ln -fs /home/csoft/csrel22 /csrel22
ln -fs /home/csoft/csrel22 /csoft
# Replace the SuSE version of postfix with an RPM placeholder
/bin/rpm -U /csoft/etc/postfix/openpkg-postfix*.rpm
# kill off SuSE's version of sshd as OpenPKG's knows about its
# executable directories in the default PATH
cd /etc/init.d
sh sshd stop
# remove sshd from startup
insserv -r sshd
# start the OpenPKG system
/csoft/etc/rc all start

After doing this, the system was up an running.