Creating and Using a Recovery CD ROM

This page describes the creation of a backup / system recovery CD-R disk for a hardened OpenBSD server. (See Cheap Backup Solutions for Linux and NT to CD-ROM backups.)

Besides being used to quickly restore a specific system configuration in the event of a failed server, the resulting recovery CD may also be used to migrate a configuration to other hardware of the same architecture. It could also be used to build multiple similar systems. The created recovery CD ROM contains both the install components necessary to perform a minimal operating system installation and a backup of the configured system less hardware specific components.

The CD ROM can also be a physical key that allows selected functions with significant security implications to be turned on or off at will. As discussed in the preceding page a significant part of hardening a system is removing unnecessary, infrequently used or potentially dangerous files. The suggested hardening procedure attempts to get the best of both worlds by relocating to a CD ROM, programs that would otherwise simply be removed from the system. Further, this approach allows programs to be removed that otherwise could not be removed because they are necessary to the operation of the system. Programs with significant security implications are replaced with symbolic links that point to the location of the program when the CD is mounted at a standard mount point. Thus these security sensitive programs can only be used when the CD is in the CD ROM drive.

Except for convenience, the four components of the CD-R Do not need to be located on a single CD. The four components are 1) minimal install files, 2) full backup less hardware specific components, 3) tar of the files deleted from hardened server and 4) extracted and executable copies of the deleted files. Because OpenBSD is compact and all four components will likely fit on a single CD there is no reason not to put them all on one CD. If there is room, having all executables in an executable format on the CD allows further experimentation with removing more files without creating new CD ROMs.

If the full system tar is zipped and only the deleted files are included as expanded files on the CD, a single CD has room for a fairly substantial web site. As a system grows, the components could be split across multiple CDs in which case there would be no reason to build a new install CD. The limit to this approach is reached when either a gzipped system tar or the subsequent daily differential backups no longer fit on a CD.

Building the Restore CD

Previously I thought there was a significant advantage to having the deleted files in the nearly full system tar. As long as the deleted.tar is preserved, available and in sync with the remove script and not normally accessible from the system from which files have been removed, there is no need to have a second copy of these files in the full system tar. Executing the remove script and creating the subsequent deleted.tar can be done at any time as long as they are kept synchronized.

The likelihood of correctly identifying every file that you want removed without including any that need to be kept and getting this into a several hundred line script with no errors on the first attempt is nil. Thus you will run the script, move files, fix the errors, put things back and repeat the process.

Each time the remove script is run, a tar of all the deleted files is made as deleted.tar from within the /deleted directory. Use "tar -cvf /var/local/backup/deleted.tar ."; change the output tar directory to the one you normally use for online backups. This goes to the system with the CD-R drive. It's smart to transfer a copy each time one is made.

If more changes are made to the remove script, the deleted files must be restored and the /deleted directory removed before the remove script can be run again. From / use "tar -xpvf /var/local/backup/deleted.tar ." Include the 'p' flag with tar when restoring from deleted.tar to preserve the SUID and GUID bits on the deleted files that have them. If both the restore and delete are not done each time, the remove script will generate many error messages and there will be no way of knowing if it would have worked properly.

If your move and link script follows the example provided here, it should run with zero error messages (no output). Otherwise there is an error in the script or something was not restored to its original state before running the script.

It's a good idea to make the almost full system tar a few times during the hardening process, after significant steps have been completed. Touch /etc/installed and then from the root directory, "tar -cvf /tmp/reinstall.tar bin bsd dev etc home root sbin usr var", gets all needed files and leaves behind the primary hardware specific file. /boot must not be included as it contains hard disk geometry information and will prevent the system from booting if its wrong. /boot will be properly created for the specific hard disk by the basic install. After the hardened system configuration is stable, a final reinstall.tar will be made and used to create the CD ROM.

Other files that relate to hardware such as the network card configuration files can be adjusted after the system boots. I've been including /dev; I don't know if it's needed. Not included were /.cshrc, /.profile, /altroot, /mnt, /stand, /sys and /tmp. These are created in a basic install and not modified or have contents that should not be saved for subsequent installs (/mnt and /tmp).

When you're ready to make the CD, both reinstall.tar and deleted tar must be on the system with the CD-R drive if they are not already. Both tars are placed in the same working directory, gzipped if necessary. The deleted.tar must be extracted into the working directory so that bin, sbin and usr are subdirectories of the working directory.

Depending on space, as much or little of the reinstall.tar as you wish, may also be extracted into the same working directory. Having all the executables in an executable format, allows additional files to be deleted but available without creating a new CD. If reinstall.tar was made after the last time that the remove script was run and thus contains links to the deleted files, extract from deleted.tar after reinstall.tar. This will ensure that executable versions of and not links to the deleted files are on the CD.

If space permits, you can avoid swapping CD-ROMs during a reinstall by placing the core OpenBSD install files on the CD-R you're building; copy them to a subdirectory of the working directory. Only the five files that are actually installed, base28.tgz, bsd, comp28.tgz, etc28.tgz and man28.tgz are needed.

With my small web site, both unzipped tars, a fully expanded reinstall.tar and the required install files fit on a CD ROM with over eighty megabytes to spare. When these pieces are in place, the full contents of the working directory is burned onto a CD-R disk. The contents of the working directory become the contents of the CD-R root directory.

A more complex but space saving alternative to an almost full system tar would be to use a script and find to do a differential backup of all files created and modified as part of and since the basic install. You might use the directories in /etc (afs, amd, KerberosIV, mail, photuris, ppp, skel, sliphome, ssl, uucp) as a guide to just when the system was installed. You could also determine the date the install CD was made and use a date slightly after that.

Find's "-newer" option can be used to locate files in the directories /, /bin, /dev, /etc, /home, /root, /sbin, /usr, /var created or modified as part of the install or since the install. This avoids duplicating files that are provided by the basic install. Because of the additional complexity, I would not use this approach unless it was the only way to fit the necessary files on a CD-R.

Once the CD ROM with both the reinstall and deleted tars is verified as good and that the links to the expanded files work, deleted.tar should be removed from the original system. Reinstall.tar should also be removed if it contains deleted files. Failure to do so defeats the purpose of creating the CD with sensitive programs, as an intruder who has gained root privileges can extract the deleted programs from the tars. Also the /deleted directory must be deleted.

Restoring a System

Subsequently, only a boot floppy, this CD that's just been made and a differential tar is needed to install the hardened server on another machine or reinstall on the original machine if there was a hard disk failure. Up to the network configuration step, restoring to the original machine or migrating to a new machine is like a new install as described previously.

In the networking portion, a host and domain name are required to prevent boot errors. No network card or other IP information needs to be entered. Type done at the configure the network card interface prompt and pass the rest of the network prompts without supplying any information. Don't edit the hosts table or escape to the shell. Any installation information you enter here will be overwritten by the restored tar. A simple root password is OK.

The software install is similar to the minimal software install described previously. If you created a reduced install set, only the files present will be displayed by the install program for selection. Add comp28.tgz or you may type 'all' instead of individual file names if you've left the unnecessary games, misc and Xwindow install sets off the CD.

As soon as the basic install completes, reboot the machine and login as root with the temporary password. So that a CD and a floppy can be mounted at the same time I prefer an /mnt/cd directory as a mount point for the CD. Create this directory or whatever directory the remove script assumed as the CD ROM mount point. Use "mount /dev/cd0a /mnt/cd" or change as necessary to mount the cd.

From the root directory, restore the full system with "tar -xpvf /mnt/cd/reinstall.tar" This will overlay the standard install. Without the 'p' option in the tar flags, the SUID and GUID files will not be restored with these bits set. As soon as /etc is restored, the users and passwords will be from the system being rebuilt.

Assuming that you have been creating differential backups since the reinstall.tar was made, restore from the latest backup available to bring the just built system up to date.

Differential Backups

Here differential means every file created or modified after a specific date and time, i.e. when reinstall.tar was created. A file changed a few minutes after the cutoff date and time should be in every backup. Differential backups get bigger every day and are cumulative whereas incremental backups only include new or changed files since the most recent backup. To properly restore incremental backups, every backup since the last full backup must be restored in order; they are not practical over more than a few days. Differentials require only the full backup and one differential restore; they are limited only by backup media size limitations.

A sample shell script for creating differential backups follows:

# Cd to the location for online backups.
cd /var/local/backup

# The subsequent find instruction will 
# append each file it finds to a growing 
# tar file.  The tar file must be created
# before find.  Any small file, known
# to exist, will do.
tar -cvf backup /etc/shells

# Use find to identify all normal files
# modified more recently than the empty
# marker file /etc/installed.  Skip files
# that match one of the backup file names.  
# Invoke tar, passing it each matching name
# and append to the growing tar file.
# (Update the date on /etc/installed with
# touch before creating new reinstall.tars.)
find / -type f -newer /etc/installed ! \
   \( -name "*.tar" -or -name "backup" -or \
   -name "*.gz" -or -name "*.tgz" \) \
   -exec tar -rvf backup {} \;
   
# Create a table of contents to assist
# locating specific files for subsequent
# restores.
tar -tvf backup > backup.log

# Change the owners, permissions and
# file names to relace "backup" with
# today's date (yymmdd).
chgrp wheel backup*
chmod 640 backup*
mv backup `date +%y%m%d`.tar
gzip `date +%y%m%d`.tar
mv backup.log `date +%y%m%d`.log

# Send the backup file and log to another
# system.
. /usr/local/bin/backup_to_other

Several specific filenames including "backup" are excluded. Since the archive is being built in a directory that will be backed up, this prevents a partly built archive from being added to itself. It also avoids the need to explicitly specify every directory root to be backed up. Files containing other backups and install files readily available elsewhere are skipped; depending on acceptable differential backup sizes and how components added to a system are tracked, it may be preferable to keep these in the differential backups.

This script is not efficient as tar is invoked separately for every file to be backed up. I wouldn't use it on a large rapidly changing server. For servers hosting modest web sites with limited capacity backup media, it works well.

Post Restore Choices

What's next depends on why you rebuilt the server. If you're restoring a crashed server or migrating a specific configuration to new hardware, as soon as the restore is complete, the system can be rebooted and it will be, for all practical purposes, the machine that the CD-R was built from. Whether the deleted files are present will depend on the order and times that you created the tars and ran the remove script.

If the cutoff time used for differential backups is before the last time the remove script was run, the deleted files will not be present. The differential backups will pick up the symbolic links as newer than the cutoff time and the links will replace files from the basic install or reinstall extract. Likewise, if you did an almost full or reinstall.tar after the deletion, the deleted files will not be present.

If the reinstall.tar was created before the removal and the time used for your differential cutoff is after the remove script was run, then the deleted files will be present. Looking in /usr/bin is the simplest and surest way determine if executables or the links from the remove script are present; if the directory has numerous links to the CD ROM then the system has links, otherwise executables. If the system has executables, the remove script can be run and the resulting /deleted directory tree erased.

Differential backups can resume where they left off on the previous server. If you touched /etc/installed just before creating the reinstall.tar, it will be restored with the same date and time as the original system.

If the server is being migrated to a new machine, this is a good time to make configuration adjustments, change deleted files and create a new recovery CD. If a test or development machine is being created or clones are being made, the network files will need to be adjusted before rebooting. You don't want the just restored server to reboot with conflicting IP and host name information if the original server is still online. Alternatively the new machine(s) could be kept off-line until the networking adjustments are complete.

The host name is kept in /etc/myname and the default gateway IP address in /etc/mygate. If you change a host name then you should also delete the four /etc/ssh_host* key files so they will be regenerated the next time the system boots. IP address information is in /etc/hostname.interfacename, one file for each NIC, where interfacename matches the NIC device name. If NICs are not the same model as the source system, the hostname. interfacename files may need to be renamed. Look in /var/run/dmesg.boot to see what NIC or NICs were detected.

A CD-R created as described here allows a complete system configuration to be installed on another machine in about 20 minutes if you have minimal install, fast CPU and small hard disks. The time consuming steps are disk partitioning, creating filesystems and extracting from reinstall.tar. If you have to figure out partitioning, the install may take much longer. If you know exactly how you will partition the disk or can use existing partitions, this only takes a couple of minutes. Creating filesystems can be quite time consuming with large disks. A slow CD ROM drive or a large reinstall.tar will add significantly to the install time.

If you're using a CD-R like this to build multiple similar systems, you will need to manually adjust host names, IP addresses and the hosts table. I have successfully rebuilt systems with these CD-Rs on systems with different motherboards, CPUs, RAM and hard disks. The architecture must remain the same and if the system includes a custom kernel, that will need to support any hardware on the new system(s).

Changing Deleted Files

It's likely that over time you will want to change the deleted files. If sufficient space is available on the CD ROM, having all the executable files in an executable format on CD ROM is more convenient and flexible for experimenting with additional file removal. After the CD is made, when you decide a program should not routinely be kept on the system, you can easily replace it with a symbolic link.

Once it's determined that the CD is the right location for a program, the removal script should updated to reflect this. If you're experimenting with what can be removed, care needs to be exercised that any files removed manually are restored manually, before a subsequent version of the remove script is tested and matching deleted files tar is created.

When a once deleted file is put back, I prefer to keep the file in the remove script but comment it out and add a comment why it was put back.

Backup Cycle

After a near full system tar is made, differential backups will initially be small in size. On any system that's used they will grow some every day. How fast they grow will depend on how the systems are used. When the differential backup size becomes cumbersome, a new full backup / install disk can be built. What's cumbersome will vary. Some sites may burn a CD per night per machine (wasting most of the space shortly after a new full backup has been made); other's may want to get multiple weeks of multiple machines on a single CD.

This creates a backup, install, system recovery process that requires only occasional full backups and relatively small daily backups for sites with systems that change at a slow to modest rate. This approach should work, with some variation, as long as a zipped reinstall.tar fits on a single CD ROM. This is including any web document trees or other application data. The full backups should not include log files or online backups which should be archived before making the reinstall.tar.

The best feature of this approach, besides providing for very low cost backups, is that the system can be restored quickly to almost any roughly similar hardware. The main drawbacks are that it won't scale to large systems and the complexity creating the recovery CD ROM.

Other Systems

Everything discussed here has been in reference to OpenBSD. There is little reason that a similar approach should not work with any operating system that meets two criteria. A basic install should be small enough that a gzipped tar of the full system fits on a CD ROM. The hardware specific components need to be localized in a few identifiable and preferably documented files. This probably applies to all the open source, UNIX like systems as well as to some of the more compact commercial UNIXs.

This approach obviously cannot be applied to Microsoft Windows systems. The confused mixture of hardware and software, operating system and applications, programs and data represented by the registry and the "system" directory prevent backups that can be reliably restored to any but essentially identical hardware as the original system.

Top of Page - Site Map

Copyright © 2000 - 2014 by George Shaffer. This material may be distributed only subject to the terms and conditions set forth in https://geodsoft.com/terms.htm (or https://geodsoft.com/cgi-bin/terms.pl). These terms are subject to change. Distribution is subject to the current terms, or at the choice of the distributor, those in an earlier, digitally signed electronic copy of https://geodsoft.com/terms.htm (or cgi-bin/terms.pl) from the time of the distribution. Distribution of substantively modified versions of GeodSoft content is prohibited without the explicit written permission of George Shaffer. Distribution of the work or derivatives of the work, in whole or in part, for commercial purposes is prohibited unless prior written permission is obtained from George Shaffer. Distribution in accordance with these terms, for unrestricted and uncompensated public access, non profit, or internal company use is allowed.