Moving a Red Hat 6.2 Linux Web Server to Red Hat 7.2 - 11/29/2001
Less Than Two Minutes Downtime!
Migrating vs. Upgrading
This page covers issues relating to moving or migrating a web
server from Red Hat 6.2 to Red Hat 7.2 Linux. This is NOT
an upgrade. Red Hat 7.2 is installed freshly on a replacement
machine. Desired components including the web site and system
administration scripts are restored from backups of the 6.2
system. The system configuration is manually updated on the new
system to keep the settings necessary for the web site and
related applications. This might be more work than an upgrade
but the benefits more than outweigh the costs. It eliminates
nearly all downtime and virtually guarntees a successful
transition. The result is a clean secure system including all
relevant new Red Hat 7.2 features that performs all the
application and other custom functions of the original Linux web
The Red Hat 6.2 Linux "server" was the last server I installed
before getting a full time Internet connection (mid 2000). It
was a custom install that included both desktop and server
components with every additional package I thought I might want.
The new 7.2 server is stripped of everything not supporting
needed functions. Issues dealt with include: moving to a "huge"
disk much larger than needed by the system and partitioning
choices; removing unnecessary packages; replacing inetd and TCP
Wrappers with xinetd; replacing telnet and telnetd with SSH;
trying to replace FTP with SSH; installing NTP as a Red Hat
package rather than compiling from source; enabling shadow
passwords; Apache configuration changes and enabling SSL;
installing components from previous system; migrating log files;
planing physical changeover and IP configuration changes.
The traditional approach to upgrading operating systems on
business servers is almost obsolete. It may still be viable on
desktop systems in both home and work environments, but even here
it is increasing likely to provide an unsatisfactory outcome.
The complexity of operating systems has increased by orders of
magnitudes since the early days of DOS where upgrading meant
replacing a few key system files and utilities in a few
directories. This process was nearly always successful.
Today, by the time that an operating system needs to be upgraded,
two things have typically occured. Hardware has advanced to the
point that the current systems may have some difficulty with the
older hardware (depending on both how widely supported the
individual components are and whether the system to be upgraded
was state of the art when it was new or was a mid range to low
end system already containing dated components). Over the life
of the system, software is likely to have been added and
configurations changed, resulting in an increasingly less
standard system with variations that the upgrade program must
cope with. The more customized and dated the system to be
upgraded, the lower is the likelihood of success.
With desktop systems, assuming good backups are made before the
upgrade process begins, it should generally be possible to
restore the system to it's original state in the event the
upgrade is not successful. With luck, the upgrade will be
successful, but will include components that should have been
discarded. With less luck, the upgrade will fail and the system
will be restored as it was prior to the upgrade attempt. All
that is lost is time. In the worst case, a previously working
system may be left inoperable. At home an unsuccessful upgrade is
not likely to be critical but will be frustrating; at work it may
result in some significant loss of productivity, depending on how
dependent the employee is on a specific desktop system.
For business servers the situation changes dramatically. All the
issues relevant to desktop systems are likely to apply, but an
additional factor becomes even more important. This is the
availability or lack of availability of the server during the
upgrade. For the installation of patches and service packs,
depending on the operating system there may no perceptible
downtime to a few minutes if all goes well. For full operating
system upgrades, downtime cannot be reliably predicted.
In ideal circumstances downtime may be measured in minutes
but if anything goes wrong, it could easily be hours and in
difficult situations requiring a reinstall and restore of the old
system, could be a day or two.
In the old days, that is before every business was connected to
the Internet, small to mid size shops scheduled their upgrades
for weekends - often Saturday morning. If things went well,
you'd be home for lunch but if not, there were two days to be
ready for business on Monday morning. Larger businesses with
redundant systems had and still do have much more latitude
regarding these issues. Regardless of the actual
schedule, nearly all small to medium size businesses had off
hours and system maintenance, including operating system upgrades
could be scheduled so as to have minimum impact on business
Today, things are different. Even if the employees are at home,
nearly all business have public web sites that are accessed by the
public seven days a week, 24 hours a day. When these sites are
not available, it reflects poorly on the company. Unlike
internal systems, you can't post messages or send e-mails saying
some system may not be available for some of its normal operating
period. Public web servers must be available continuously or
their absence will be noticed by those you hope to attract to
your business. Any backend server that provides real time
services to the public web servers is subject to similar
considerations. Until the business is large enough to have
redundant systems and able to phase in upgrades without the
changes being visible to the web public, a business must
carefully plan how to upgrade their web servers and any systems
providing services to the web servers.
Unless you don't care about your public image, the traditional
operating system upgrade is simply not a viable method for
upgrading a public web server or any system providing services to
a public web server. Even under ideal circumstances it will be
visible to some of your web users. Everyone who's been in the
computer business for any length of time knows that uneventful
(perfect) major system changes are very much more the exception
than the rule.
The traditional operating system upgrade should now be replaced
by a system migration. Essentially this is installing a new
operating system, applications and data on a replacement machine
and planning a machine substitution that entails a minimum
disruption. The replacement machine may be functionally similar
to the machine being replaced or it may be an upgrade if the old
system was nearing any of its performance limits. Where advance
planning is an option, the same techniques may be used to replace
any major components including both hardware and software that
may need replacing or upgrading.
Depending on your planning and testing, and the physical
environment in which the change is being made, it should be
possible to make a switch that holds the disruptions in services
to 15 seconds. Most web browsers actively using the site, should
be unaware of any change or service disruption. If they request a
page at just the wrong time, hopefully they will stop the
retrieval, and try again before the original request times out.
(Web applications that require state maintenace, will unavoidably
To effect a change this smooth, several conditions must be met.
There needs to be sufficient cable connectivity (power and UPS,
network, KVM) that both machines are fully connected when the
switch is made. Shutdown and boot times must be known for both
machines to within a few seconds. The idea is to boot the
replacement so that its networking functions are activated
approximately 10 seconds after the networking functions on the
machine being replaced are shutdown. The network parameters on
the replacement machine should be identical to the replaced
machine, which means they must be reconfigured and tested on a
physically separate network, which may be as few as two machines.
The replaced system should be left off and not changed until it
is clear the replacement is working. The replaced system cannot
be re-connected to the live network until its networking
parameters have been changed.
If physical constraints don't allow for both systems to be
connected simultaneously, or it's desired that the replacement
system occupy the current location and connections of the machine
being replaced, which may be a requirement in the case of
firewalls, the disruption will necessarily be somewhat longer. It
will be the sum of the time necessary to shutdown the machine to
be replaced, move the machines and switch the cables to the
replacement machine and boot the new system. With systems that
halt and boot quickly, and a convenient work area, two minutes
should be a reasonable goal.
The primary advantages of a migration compared to an upgrade are
that the system being migrated to can be fully tested prior to
being placed in service. Even in the event of some entirely
unexpected failure on the new server, the original should be
available and unaltered so it may simply be returned to its original
location to continue performing its normal functions.
The main disadvantages to a migration versus an upgrade is that first
it requires an available and suitable machine. It will also be
substantially more work than an upgrade that goes smoothly but likely
less work than an upgrade that is not successful. Theoretically,
until it goes wrong, an upgrade is a largely automated process.
A migration requires significant detailed planning and testing.
The Old Red Hat 6.2 System and Hardened Servers
When I installed the Red Had 6.2 Linux web server it included
every software component I thought I might be useful. This was
the last machine I ever installed as completely general purpose
computer, able to perform any function I might desire on a
computer. I still had the Windows', maximum-functionality, it's-
easier-to-install-it-all-at-once, mentality that still dominates
much of the computer industry. Fortunately, I soon understood
the security implications of this approach. Before I connected
this machine to the Internet (behind a custom firewall on a
hardened OpenBSD system), I'd disabled all unneeded services and
placed additional restrictions via TCP Wrappers on those few that
were left. The system was hardly a well configured Internet
server but with the outer firewall and good tradition host
security, it was reasonably secure. The GUI desktop did allow me
to view my web sites, which where still in transitional designs,
via multiple browsers and platforms.
Over much of the next year, I focused more on OpenBSD than Linux
and my OpenBSD firewalls and web servers went through an
iterative hardening process more extreme than that described in
any security books or literature I've read. This led to my
Hardening OpenBSD Internet Servers web
section that's been revised and expanded multiple times. Meanwhile
the Linux web server reliably served web pages without a reboot for
over 11 months, until I accidentally pressed the reset button (when I
intended to reset an NT server on an identical box sitting next to the
After the release of Red Hat 7.1, I spent significant time with
Linux, working with Red Hat as well as the secure Linux, Trustix
and EnGarde, distributions. Though both of these secure distributions
have much to recommend, each has issues that suggest it may not be the
best platform for a web server in my environment at this time.
Following the release of Red Hat 7.2, I decided it was time to
replace the Linux web server. The primary requirement was that all
functionality related to the web site on the existing server be
moved to the new server. This included Swish-e full text
searching, a virtual management site with web statistics provided
by Analog and other CGI scripts; I hoped to get SSL to work
(which was still under RSA patent restrictions when I set up the
previous server). All existing custom administrative functions
including backup, various custom logging and intrusion detection
had to work on the new machine.
The new Red Hat 7.2 server would be hardened but within the
framework of conventional hardening techniques avoiding the more
extreme techniques I've used on my OpenBSD systems.
Hard Disk Partitioning Issues
The 6.2 web server had a 13GB hard disk of which about 2.5GB was
used. The only disk management I've performed is about every
4 months, I've archived the online backups. So far, there is
probably enough space that all of these could have been left online.
The replacement machine has a 40GB disk drive which is far more
than needed for normal operations. The question was how best to
use (or not use) this excess disk capacity.
I've never used the auto partitioning features of Red Hat. I
have used Disk Druid but find it reorders the partitions in ways
I don't want. All recent and future installs have and will be
done with fdisk which gives full control over what partitions are
placed where. Partitioning is about trading security and
flexibility. A single partition occupying the entire hard disk
with a single filesystem is most flexible because all available
space will be available to any new file regardless of what
directory it goes in. The drawback is that one large partition
is most susceptible to damage such as when a power outage crashes
your machine, especially if it's writing to the disk at the time.
Separate partitions significantly increase recovery prospects.
If you use multiple partitions on small disks you need to
accurately predict where your file growth will occur. If all
your disk is partitioned and you fill one partition, even if
other partitions have plenty of space, using that space will
become a significant management issue. As disks get larger you
have more options how to use your disk space. You can leave
unpartitioned space and create new partitions as needed, mounting
them inside of partitions that are becoming full. Generally your
are limited to a total of 16 partitions per disk. Some may not
be useable so you could have only 12 - 14 useable partitions.
Instead of keeping unpartitioned space you can allocate all
space, estimating where it will be needed most. After a system
crash, the system checks the integrity of the filesystems and
normally fixes any problems automatically. If a hard disk is
much larger than needed, the checking will take much longer than
on a smaller hard disk or the same hard disk with either
unpartitioned space, or partitions that are not normally mounted.
Partioning and mounting all available space on huge hard disks
will greatly slow the boot process after a crash when you will
want the system back quickly.
There are many opinions on which directories should be mount
points for separate filesystems. There must be a system root
directory, /, and filesystem. It's not clear to me if /boot
needs to be a separate filesystem in Linux. It is clear
that /bin, /dev, /etc, /mnt, /proc, /root and /sbin must be part
of the / filesystem so the devices, configuration files, and key
utilities will be on the first mounted partition. For example,
without /bin/mount, none of the other filesystems can be mounted.
Several top level directories are often mount points for
filesystems. These are /home, /opt, /tmp, /usr, and /var.
Sometimes /usr/local is a mount point for a separate filesystem.
None of these need to be separate filesystems provided / has
enough space to hold the contents of the respective directories.
The reason these often are mount points for filesystems is that
they do not have to be part of / and one or more of these will
likely consume large amounts of disk space, depending on how the
system is used. /usr contains the large majority of system
installed programs on most UNIX systems. After the initial
install /usr typically does not grow by large amounts unless a
significant number of products are added. /var doesn't start
with much but is often the fastest growing directory. This is
where most system logs as well as mail and print queues go. On a
high volume firewall, web server, or intrusion detection system,
logs may grow at an enormous rate. Local custom logs or online
backups may be kept under /var contributing to rapid growth.
/var may be the largest filesystem on some systems.
/tmp is for temporary files but how temporary is temporary? Some
systems clear /tmp on reboot and others automatically remove old
files in /tmp and any subdirectories. Nothing that you care
about or need to keep more than briefly should go in /tmp. It
may only contain editor work files and print files in progress in
which case it might be part of /. On the other hand, with large
hard disks /tmp can be a good working directory for system
backups and projects that briefly require very large files. If
disk size permits, I like /tmp as a separate filesystem large
enough to do a full system backup and compress it as a separate
step. If a backup goes in any /var subdirectory, then it's
necessary to program around that file location so the growing
backup file doesn't get recursively included in itself. Creating
a backup in /tmp is an easy way around this. All top level
directories that are wanted can easily be listed and backed up to
a file in /tmp, which should never have contents that need to be
backed up. Following the backup, the backup file might be moved
to a /var or other location.
/home is for user files. Each user gets a subdirectory matching their
username and this contains their initialization files. Depending on
the system and the number of users, user files my be negligible or
What to do with applications and their data that are not standard
parts of a system install is a key disk management issue. Such
programs and their data are likely to be placed in one of three
locations. /usr/local contains bin, etc, and sbin subdirectories
as well as several other directories that mimic the system root.
The "local" designation suggests this is a place to add system
specific applications and many applications install by default
into this area. Some place the executables in bin or sbin.
Others create their own subdirectory tree directly under
/usr/local. If it's expected there will be many such applications
/usr/local is likely to be a separate filesystem. Typically the
applications that go in /usr/local do not have large quantities
Large data sets are most likely to be placed in one of two
locations. Perhaps the more conventional location is /home.
Typically a directory will be created to match the application
name or use. Web document trees often go in /home and the
directory names might match the domain or virtual site names.
Databases might also go in home. Databases can be from less than
a megabyte to multiple terabytes in size. Larger ones are often
a machine's sole reason for existing.
The other location for data and applications is /opt for
"optional" which is like "local" in /usr/local. /opt may have
applications and related data together. Personally, I've always
preferred to keep data separate from programs, but this does
require a straight forward mechanism for relating programs and
data. On a traditional multi user host with many active users,
/opt makes good sense to separate application data from personal
user data. On a dedicated server with a handful of
administrative users, the choice of /home or /opt for data is
little more than personal preference or habit. Every business or
organization should establish standards, preferably formal and
documented, for what types of files are stored in which
directories and subdirectories.
In Red Hat 6.2 the default document root was /home/httpd/html
with cgi-bin and icons as subdirectories also in httpd. Somebody
stopped thinking when Red Hat set the default locations for 7.2.
The default web document root and cgi-bin directories were moved
to /var/www. I know *BSD does this also. I wonder what or if
anyone was thinking when this choice was made. Programs don't
belong in /var; data doesn't belong in /var. Web document trees
are data and programs; they don't belong in var. The web server
logs do belong in /var but not web documents. People edit web
documents and update databases from which web pages are created.
Nobody but intruders should ever be editing logs. Nothing in
/var should be edited or changed by legitimate users. Depending
on how the rest of the system is set up, web sites belong in /home
Actual Partitioning Decisions
So how did I partition a 40GB drive when the old system used about
2.5GB and the new minimum install created a much smaller install of
only about 750MB? First I decided to partition just over half of
the total disk and of the approximately 20GB partitioned, I
As I said above, I used fdisk for full control. I used three of
the four primary partitions. The first was a small 50MB partition
(/dev/hda1) for /boot. By default fdisk starts at cylinder 1 and
lets you pick partitions 1 - 4 as primary or extended. It gives
you the first available cylinder and lets you specify ending
cylinder or size in KB or MB. If you've made an extended
partition you can create logical partitions inside the extended
partition. /boot is only 4% used and won't grow so it should
have been about 10 - 15MB regardless of disk size.
The second primary partition was a 250MB swap partition
(/dev/hda2). A common recommendation is to make swap partitions
twice the physical RAM size. With 128MB RAM and Linux, I'd do
this but with 256MB, that seems a waste. Linux is typically
using just over half the physical RAM and has yet to touch the
swap space. By comparison OpenBSD only uses 20 some megabytes
leaving the rest free and does not go to swap. Windows (at least
NT), even with 384MB leaves much memory free but makes needless
heavy use of swap space.
Though I have no need for all four partitions, on a future
install, I want to try a larger primary for / and locate /boot in
that and make the second partition an extended partition and see
if the swap partition will go in that. The third partition was
an extended partition and I went back and forth over using the
entire disk or not. Since I was sure the machine would never be
a dual boot, I chose to make the extended partition the rest of
the disk (/dev/hda3).
In the extended partition, I made /dev/hda5 250MB for /. /dev/hda4
is reserved for the fourth possible main partition. / is about 35%
full and not likely to grow much. If however the filesystem for /tmp
is not mounted this leaves enough room for normal system activities
so seems about right. I'd only make it smaller if space were an
For /usr, I made /dev/hda6 3GB. After the install this is only about
15% full and not likely to grow significantly. 1GB would have been
comfortable. With the large number of packages I removed 500MB might have
been sufficient but would have allowed almost no room for growth. A
basic Red Hat 7.2 server install without the X Window system is
For /home, I made /dev/hda7 2GB. With the current files this is
only 7% full. 500MB would have been comfortable. With a largely
static site there would be almost no possibility of ever needing
more space. If however, dynamic applications are added that
create significant amounts of data, or multiple dynamic
applications are added, 500MB might become crowded. 1GB would
allow sufficient room for growth for almost any plausible web
site development that the site would almost certainly be upgraded
or migrated again before space became an issue. This assumes no
non web related applications are added to /home. Unexpected
changes make any disk use projections unreliable.
For /var, I made /dev/hda8 6GB. This is the only filesystem on
the new server that's smaller than the previous system which had
an 8GB /var. The old system /var was 20% full. More space in var
would be needed only if I wanted to accumulate all differential
backups online for well over a year. All foreseeable web and
system logs should easily be accommodated. I keep all web logs
from the very beginning of the site and migrate old logs to the
new server. Even if somehow the site were producing a gigabyte
of logs per day, I'd find a way to preserve them all. Other than
the actual web content, nothing is more important for a web site
than a complete collection of all raw logs. With these, any
future analysis tool can perform its analysis from the begining
and produce valid results. If only reports, abstracts, or totals
are saved, there is no way to switch or add to the analysis tools
and get any valid results except going forward. I would
definitely preserve web logs in preference to online backups,
assuming of course that the backups have been migrated to some
I created an 8GB /dev/hda9 for /tmp. This is more than large
enough for any foreseeable backups or other potential uses. One
of the first things that I did, was to untar the last full backup
of the existing server into /tmp/rh62. I thought this would give
me more opportunity than directly restoring from a tar file to
view, evaluate, and selectively restore files from the old system
to the new. In addition, as work configuring the new system
progressed, I placed full system backups in /tmp.
A review of the filesystems and sizes, starting at cylinder 1
follows. The /wrk partition is explained later.
I used fdisk and set this physical arrangement hoping to group
the most used partitions together and put the rarely used partitions
outside the frequently used ones. After the system boots, there
is no reason to return to the /boot partition. With more RAM
than the system needs, swap should rarely if ever be used, so
after it boots, the disk heads should rarely go to the outer
cylinders. / contains a number of frequently used utilities such
as ls, grep, less and others. Hopefully these will get cached
and rarely be accessed from disk. This leaves three adjacent
partitions, where programs from /usr work with data from /home
and write logs to /var. After the program is read in from /usr,
hopefully there will be little need for the disk heads to return
there. Some programs will use temporary files in /tmp. That at
least is the theory and why I partitioned in the order that I
did. I don't know how to actually test the results. /wrk is not
The importance of planning disk partitions has been based on
limited disk space and the need to reasonably anticipate system
growth. Large disks reduce the importance of properly
anticipating future disk use needs. In addition, Linux
now includes Logical Volume Management (LVM) capabilites that
greatly increase the flexibility of how physical disk space is
allocated and used. Another new program, parted, allows ext2,
FAT and FAT32 partitions to be created, destroyed, resized, moved
and copied. These capabilities have not existed (for Linux
systems ) in the past and mitigate the need to anticipate future
The "Hidden" Partition
One of the automated process I run on my systems is to create
md5sum logs of all /etc files and all executable files.
Consecutive days are automatically compared and any changes
displayed. This lets me verify that system configuration files
and executables only change when I expect them to. The saved log
file was dramatically altered by the /tmp/rh62 contents which
were essentially a replica of another full system.
To deal with this I decided to put the remaining unpartitioned disk
space to use. fdisk from the command line as root quickly turned
the available space in the extended partition into an 18GB ext2
partition. mkfs did not recognize the new partition and I had to
reboot the system for it to do so. After rebooting, within a few
minutes I had a new 18GB disk work area that I could mount and use
when I wanted. I created a /wrk for a mount point and after
mounting /dev/hda10 on it, moved all /tmp contents to /wrk. If
I don't need this stuff, I simply "#umount /wrk" and it disappears.
This prevents my checksum scripts from including this, but any
old files I might still need to check are easily available. As
I do more full backups, they will also be to this filesystem that is
not normally mounted. Since it's not normally mounted, if the
system crashes, it won't spend time checking this filesystem
when it reboots. Even if /wrk happened to be mounted during a
crash, it wouldn't be checked until the next time it was mounted.
The Red Hat 7.2 network install appears to be a step backwards.
It no longer defaults most of the TCP/IP fields and those that
are defaulted, are wrong. I don't know what happened the first
time I did this install. I entered all the IP parameters but when
I booted after the install no networking was set up. There were
some other problems so I started over and this time all the
settings entered were preserved as expected.
I chose not to set up any firewall during the install. I've not been
satisfied with the firewall settings resulting from the Red
Hat 7.1 and 7.2 install process. I'll deal with these manually
later. This machine is behind a strong outer firewall that
shields it from everything but requests on port 80. The only
other services running are sshd and in.ftpd; both are protected
by /etc/hosts.allow /etc/hosts.deny and limited to local traffic
so another layer of firewall protection is not a pressing need.
With the limited and unpredictable results of firewalls setup by
the Red Hat 7 series install routines, the resulting firewall is
more likely to cause problems than benefits. Red Hat needs to
deal with IPTables in its install routines and needs the install
process to clearly indicate what will and will not be allowed
depending on which options are chosen.
Following the network install the root password and a non
privileged user and password were set up. These passwords
contained mixed case letters, multiple digits and multiple
symbols and or punctuation. Using my
Password Evaluator they
had strength ratings of 8 or 9 and should take a fast desktop
PC between 10 and 20 million years to crack, i.e., even
NSA should not be able to crack them with today's technology.
Package Installation Choices
For the new Linux web server, I started with Red Hat's server
install option and deleted the X Window System and other
unnecessary packages. I selected web server and the select
individual packages option. I deselected about 150 packages but
dependency requirements put about 20 of these back. The packages
that I removed were: 4Suite, LPRNG, PyXML, VFLIB2 & devel,
XFree86-libs, XFree86-xfs, a2ps, alchemist, apmd, arts, ash,
asp2php, audiofile & devel, aumix, authconfig, autoconf, autofs,
awesfs, bc, bcm5820, bzip2-devel, cdda2wav, edlahelgen, cdp,
cdparanoia, cdrecord & devel, cipe, cracklib & dicts, cuvl &
devel, dhcpd, efax, emacs, esound & devel, expat & devel,
fetchmail, finger & server, foomatic, freetype, gcc-g77, gd &
devel, gdb, gettext, ghostscript & fonts, gsm & devel, hotplug,
indent, isdn4k-utils, kbdconfig, krb5-libs & devel, libogg &
devel, libpng & devel, libtiff & devel, libvorbis & devel, links,
lokkit, lrzsz, micq, mikmod, minicom, mod_dav, mpage, mpg321, mt-
st, nfs-utils, newt, nkf, nmap, nmh, nscd, nss_ldap, openjade,
openldap*, pam_krb5, php*, pidentd, ppp, printconf, procmail,
psutils, python*, radvd, raidtools, rdate, reiserfsutils,
rhn_register, rp-pppoe, rpm-python, rsh & server, rsync, rusers &
server, rwall & server, rwho, , sndconfig, sox, statserial, swig,
syslinux, talk & server, tamang, tcl, tcpdump, tcsh, telnet &
server, ttfonts-ja, udc-snmpt, up2date, urw-fonts, vorbis,
wantanbek-vf, whois, wvdail, xdelta, yp*.
The packages kept due to dependencies were: autoconf, cracklib,
crack-dict, expat, freetype, gd, krb5-libs, libjpeg, libpng,
libcap, newt, openjade, openldp, popt, postgresql-libs,
postgresql-tcl, procmail, syslinux, tcl, tcsh, XFree86-libs.
I added bash-doc and ntp and following the install, added wu-
ftpd. The result was 264 packages for 501MB. In looking over many
of the packages that I removed, some are completely harmless and
others such as the various sound related functions, simply
irrelevant to a machine without a sound card or speakers. These
are consumer trash that should never be selected in the first
place for a machine designated as a server. From the packages
included by Red Hat in a "server" install one might draw two
conclusions. First, Red Hat is still caught in a Windows type
mentality, trying to make a single computer serve a variety of
unrelated functions. Second, Red Hat thinks its core audience is
home hobbyists. Otherwise, why would all the packages needed to
play audio CDs be included on a server. Does Red Hat think a
professional system administrator would be listening to CDs on the
same machine that they are configuring as a web or other server?
Several programming languages were removed; any language that I
don't expect to use on a specific machine is a hazard. If it's
not used to support the machine's intended functions, it has no
value. In the hands of an intruder, any programming language can
become a powerful tool. All the insecure old protocols related
to rsh and yp are removed; they simply have no place on a machine
intended for use as a public Internet server.
Some of the programs are downright dangerous including nmap and
tcpdump. Nmap is a cracker tool. While it certainly has legitimate
security uses for probing one's own LAN or possibly those of
customers in need of security assistance, it has no business
sitting on a public web server. If the machine is compromised,
tcpdump is just the program the intruder needs to analyze traffic
on the LAN in preparation for compromising nearby machines. Both
telnet and telnetd are removed; these have been completely
replaced by SSH. I'll leave a telnet client on my workdstation
for access to clients' systems who have not yet migrated to SSH
but it has no business on a server where, if the server is
compromised, it becomes a tool to attack other systems.
For the time being I have ftp, ncftp, and wu-ftpd. I don't want
any of these but I have yet to get scp (the secure copy part of
SSH) to work between Windows and UNIX machines. It works fine
between UNIX systems. Until this problem is solved, I need file
transfer both directions and initiated from both sides between
Windows and UNIX systems. Wu-ftpd is restricted to accepting
only connections from my workstation. I'll look into options to
hinder the use of ftp and or ncftp and likely remove one. I may
set my outer firewall to block FTP requests from the web server.
In some ways, a very dangerous program is the text browser,
links. Nearly every firewall is set to allow outbound web
browsing sessions. It's trivial for an intruder to set up their own
web server (possibly limited to accepting connections from
compromised machines). With a web browser on a compromised machine,
an intruder can use their own web server as a transfer agent to
pass any cracking tools desired to the compromised system. These
might be used to cover tracks on the compromised system or to
load tools needed to use the compromised machine as a base of
operations for attacks on other machines.
Ideally the server would lack all client programs, including both
ftp and links mentioned above, capable of initiating connections
to arbitrary destination computers. The server should have only
passive servers capable of receiving connections from other
computers. The outer firewall would block all traffic except the
intended public services between the server and the Internet.
The non public services, ideally only sshd, would be configured
only to accept connections from designated management workstations.
With a server set up like this, even if an application bug and or
misconfiguration allowed an intruder to gain root access, they
would be trapped on that server, unable to use the server to
attack any other targets. To make use of the compromised server,
the intruder would also have to independently crack, either the
outer firewall protecting the server or a management workstation
that is set up to communicate with and control the server.
Following the install there were a number of modifications to
make. I turned off sendmail, xfs and ipchains; when I set up a
firewall it will be with IPTables. Even though I added the ntp
package, I had to turn it on.
To review the services the system will be starting or not, run
"chkconfig --list|less". Primarily what matters is the setting
for run level 3 (4 and 5 are normally the same) which is normal
multi user mode. Service names are listed on the left; a "3:on"
indicates the service will be started by the system at boot time
and a "3:off" that it will not be started. To change whether a
service is started or not use "#chkconfig servicename on"
or "#chkconfig servicename off" as appropriate.
SSH Replaces Telnet
The SSH daemon, sshd, is started by default on Red Hat 7.2 Linux
systems. Configuration files and server keys are in /etc/ssh.
The file sshd_config is the configuration file for sshd. There
is only one important change that should be made in this file.
The line "PermitRootLogin yes" should be changed to
"PermitRootLogin no". This will prevent root from logging in
directly from a remote location and forces a remote user to know
two passwords to get remote root access.
If access to su is limited to a few non privileged users who also
have strong passwords this would effectively eliminate
unauthorized remote root logins, even if network configuration
did not restrict access to sshd and the shadow password file fell
into the wrong hands. I also changed "X11Forwarding yes" to
"X11Forwarding no"; this is not significant and if you insist on
running X on servers you will probably want this left yes.
With these settings you can use ssh to log into the system using
a local username and password as you would with telnet. The
difference is that the passwords and connection are passed back
and forth with strong encryption so someone on your LAN or a
network segment between cannot eavesdrop on your session by
packet sniffing. Also the root password is protected when a user
using ssh, su's to root. It is not necessary to set up public and
private keys though many will feel this is a more secure way to
use SSH. My
beginners SSH tutorial covers key setup
if you wish to use this approach. See the discussion of
/etc/hosts.allow and /etc/hosts.deny below for further
restrictions related to ssh connections.
Xinetd Replaces Inetd and TCP Wrappers
An important change between Red Hat 6.2 Linux and Red Hat 7.2 is that
inetd has been replaced by xinetd. Inetd was a daemon that
listened for a variety of connection types, typically including
telnet, ftp, and a variety of protocols that do not normally
remain as memory resident daemons. When a connection request comes
in for one of these, inetd and now xinetd, starts the appropriate
service. Inetd had no security and another program known as
TCP Wrappers was added that allowed control of requests by the IP
address or hostname from which the requests were received. Xinetd
adds TCP Wrappers capabilities, plus additional controls to inetd.
Part of the access control is implemented in two files
/etc/hosts.allow and /etc/hosts.deny. See "man hosts.allow" for
format details and options. Lines that start with a # are
comment lines. I strongly recommend adding a single active line
to hosts.deny that consists solely of "ALL: ALL". This blocks
all connections controlled by the hosts.deny and hosts.allow
files and forces all connections that are to be allowed to be
explicitly authorized in hosts.allow. I tend to make my
restrictions extremely limited, often allowing connections
only from a single management workstation.
I installed wu-ftpd as my ftpd server and ftp clients access it
through xinetd. If I wanted to allow ftp connections from
anywhere, the line "in.ftpd: ALL" would allow this. If the local
LAN network address was 192.168.123.0 and all machines on the LAN
were to be allowed to connect, either "in.ftpd: 192.168.123." or
"in.ftpd: 192.168.123.0/255.255.255.0" would work. Obviously the
first format is much simpler, but if the LAN did not have a
standard netmask, the second format provides the necessary
flexibility to accomodate subnets. A single host connection
could be enabled with "in.ftpd: 192.168.123.22".
There is much more to xinetd. Inetd had a simple /etc/inetd.conf
configuration file with one line per service. Xinetd is
controlled by an /etc/xinetd.conf file which can, and by default
on Red Hat Linux, does specify an /etc/xinetd.d directory. See
"man xinetd.conf" for the format which is much more complicated
than was the configuration for inetd. Besides options for
specific services there are default settings that may apply to
all services started by xinetd. Options can be single values or
lists matched to the option name. For lists, values may be added
or removed or the whole list replaced. Defaults set in
xinetd.conf may be overridden by specific files in /etc/xinetd.d.
When the wu-ftpd package was installed, the necessary
/etc/xinetd.d/wu-ftpd file was created but it was set to
disabled. The file had to be manually edited and xinetd
restarted to make ftp connections. Among the parameters that can
be controlled are "only_from" and "no_access" which provide
access controls similar to the access restrictions
provided by hosts.allow and hosts.deny.
The settings in xinetd.conf and in xinetd.d are read when xinetd
starts. If options in these files are changed, xinetd must be
restarted to put the changes into effect. In contrast, the
settings in hosts.allow and hosts.deny are checked every time a
connection attempt is made, making them much easier to change and
test. The hosts.allow and hosts.deny settings override similar
settings made in the configuration files. The surest way to
restart xinetd is to use the command "#/etc/rc.d/init.d/xinetd
restart". If it's successful, you'll see "Stopping xinetd:" with
an "[ OK ]" followed by
"Starting xinetd:" and "[ OK ]". The OKs will be in green and
to the right of the screen just the way they are when the system
starts or stops. If there was a problem, you would see a red [
Though sshd runs continuously as a daemon and does not use xinetd
to control connections, it does use the hosts.allow and hosts.deny
file in the same manner that xinetd does. If hosts.deny has an
"ALL: ALL" line then ssh connections must be authorized in hosts.allow
or none will ever succeed regardless of any other ssh setup. The
line "sshd: ALL" will allow connections from anywhere. To be more
selective set up IP addresses or ranges as suggested above or see
"man hosts.allow" for the various supported formats.
No Shadow Passwords
One of the things that really surprised me on the new Red Hat
Linux install, was that shadow passwords were not enabled. It's
my recollection that shadow password files were used as long as
I've been using Red Hat (since 6.1). I don't recall ever having
to enable them on any other Red Hat system I've installed or
used. Except for Corel Linux, I don't recall every working on or
installing any UNIX or UNIX like system that didn't use shadow
password files. A recent, default install of Red Hat 7.2 as a
desktop (workstation) system clearly has shadow passwords enabled
and I did not turn them on. This server install though, had no
/etc/shadow or /etc/gshadow files. It was very simple to turn
them on. The two commands "#pwconv" and "#grpconv" did it.
Still, I'm perplexed by seeing variations from one Red Hat
7.2 install to another, that have nothing to do with options
selected during the install, and no rational explanation why there
is a variation.
What to Move?
Perhaps the largest task was identifying what to move from the
Red Hat 6.2 system to the new 7.2 system. Having a fully
expanded copy of the old system online made this easier. Over
time, any system accumulates some junk, and this should be left
behind when moving to a new system. Obviously non of the old Red
Hat 6.2 executables would be wanted. I try to be systematic
about where new things are added to a system. Besides /home and
/root, I use /usr/local and also create a /var/local.
/usr/local/bin gets homegrown scripts and most individual
utilities. A few products have directories in /usr/local and I
created a /usr/local/tools directory for most products that I
build locally on the system; some of these executables are
installed in /usr/local/bin or sbin.
Having just about everything in /usr/local makes it pretty easy to
select what to keep. A locally built copy of nmap as well as
some other hacker tools were discarded in keeping with the stripped
and secure nature of the new system.
Red Hat's Strange NTP Setup
Built copies of ntpd and related executables were removed from
/usr/local/bin in deference to the Red Hat supplied package. Two
related scripts in /usr/local/bin were kept but modified. These
were ntp to start ntpd and ntp.stop to stop it. They were
modified so that their contents simply became
"/etc/rc.d/init.d/ntpd start" and "/etc/rc.d/init.d/ntpd stop" to
keep manual stops and starts of ntpd consistent with the bootup
starts and halt or shutdown stops. I did not like the Red Hat
default parameters passed to ntpd in /etc/sysconfig/ntpd which
included an option not documented in the NTP man pages. I
replaced the Red Hat parameters with ones I was comfortable with.
Nothing about the Red Hat setup of NTP makes a bit of sense to me.
NTP is something I've spent considerable time with; see my
NTP Tutorial. The default Red Hat
ntp.conf file is an empty shell with comments that are not likely
to be of use to anyone. It's meaningless without consulting the
official documentation at
Review the ntpd man page and the configuration file man page.
I set up ntpd to use a strictly defined, hierarchical client
server setup. I have top level servers that each use three outside,
independent Stratum 2 time sources, and peer with each other.
These machines in turn act as servers for all other machines on
the LAN which also peer with each other so my machines are
clearly defined to be running at Strata 3 and 4. Obviously the
new Linux system will use the carefully build ntp.conf file from
the system it will be replacing. Also, at least for my purposes,
the Red Hat default location for the ntp.drift file in /etc is
not acceptable. This file is regularly updated by ntpd. As I
have automated alerts for any files that are changed, added to or
removed from /etc on all my systems, I placed ntp.drift in
/var/local/conf/ where it won't trigger change alerts every day.
The default Red Hat 7.2 ntpd startup code resets the system time
each time the system boots. This makes no sense. Unless the
computer has been powered down for an extended period, any
computer running ntpd will have close to the correct time. If it
is off by more than a 1000 seconds, ntpd will exit and the system
time will need to be set manually. Otherwise, ntpd will set the
correct time - that's what ntpd is designed to do. Red Hat
provides no instructions on using the /etc/ntp/step-tickers file
which is not a standard part of NTP but is part of Red Hat's
startup code. With my ntp.conf file, which has worked on
all my production systems in the past year and a half, the
standard Red Hat startup code hangs the boot process indefinitely.
This code also uses the deprecated ntpdate program which will be
phased out. Red Hat does not seem to keep up with developments of
the supplementary products it includes in its installs, but then
with 7 CDs of software in its high end commercial version, how
could it? Whoever set un NTP at Red Hat either knows a lot more
about NTP than I do, but neglects to tell us how to use it, or
they know a lot less. In the end, I kept the standard System V
style initialization used by Linux, but replaced all the NTP
specifics provided by Red Hat.
Moved Applications, Files, and Logs
Two applications which were brought forward were the Swish-e text
searching programs to provide text searching for the web site and
Analog, the web traffic analysis program. I could have used this
as an opportunity to upgrade these; both are performing adequately
and rather than complicate the migration, these look like tasks
better left as separate projects. Both ran as they had on the old
system after being restored. Each has CGI and cron components.
The entire web site was copied from the /tmp/rh62 directories and
updated with the latest differential backup to bring it into sync
with the live sites. Likewise the current crontab was brought
forward intact from the old system. This runs mostly scripts
from /usr/local/bin. Most ran as expected. Red Hat 6.2 included
/usr/bin/ksh but not 7.2. A few of the scripts used ksh as their
interpreter. As bash is upwardly compatible with ksh, adding a
ksh symbolic link in /usr/bin to /bin/bash fixed these problems
without editing the scripts that started with "#!/usr/bin/ksh".
I create a "local" subdirectory in /var to accomodate a
significant number of custom logs my scripts create as well as
online backups. /var/local normally contains backup, conf,
download, and logs subdirectories. I never move old backups
forward to a new system. If I need some specific old file, it
should be on the appropriately dated CD-R drive. conf only has
the ntp.drift file which should not be moved from one system to
another as it is system specific; it's a measure of how accurate
and in which direction (fast or slow) the system's hardware clock
is. This information is what allows ntpd to keep accurate time
on a computer if it is up but becomes disconnected from its time
The /var/local/download directory normally has .tar and .tgz
files from which additional products were added to the local
system. If the product works out, these files are normally
moved to an installed subdirectory.
/var/local/logs contains both individual logs and additional
subdirectories for some logs that proliferate quickly. Among the
logs I keep are those related to
Homegrown Intrusion Detection.
This includes md5sums of system configuration and executable
files as well as frequent snapshots of all processes running on
the system. As both change dramatically between system releases,
there is no point in moving these to a new system. It is
important to get a checksum file as soon after the new system is
installed as possible, i.e. shortly after the install is complete
and before other setup changes are made. This should be moved to
another system as well as a CD-R drive and serves as a baseline
comparison to identify any unauthorized changes to system
executables that might be made in the future.
I always use a script when I edit crontab. This captures an
image of crontab before any changes are made. These do not move
to a new system. If you make a mistake in crontab, having the
last good copy available will greatly speed the correction and
reduce tension as you know its merely a matter of a few minutes
until any problem is corrected. These also establish a record of
how the system's automated processes develop over time.
Another log I keep on every system I manage is the output of
uptime every 15 minutes for the life of the system. Once a day, the
date and time are inserted into this log. A record of how long the
system has been up can be handy as well as the number of users
at different times of the day and on various days of the week.
On a busy system these can track major trends. To me the most
important part and reasons for creating this log are the load averages
that are included. Over the life of a system it can be invaluable
to know when and how high the peak loads are and how they compare to
a month, six months or a year before. If users complain of system
performance, it becomes simple to determine if there is a true
long term trend, a temporary anomaly that needs to be addressed
or simply a user's poor memory. Load average is simply the grossest
measure of system use, but it's an excellent indicator when more
refined measures need to be taken.
For disk use I keep two sets of logs, each produced once a week.
Saving the output from a simple df in a file named by date will
show long term disk use trends. This I normally preserve across
a system migration or when disks or partitions are added. A
second set of du outputs (created at the same times as the df
outputs) for the entire system, help find where disk space is
being consumed. If a filesystem runs short of disk space, a
simple grep of the df files can identify anytime there was a
significant change in disk use. The matching du files can then
identify the directories in which the change occured and normally
lead to specific files that account for large parts of the
change. An apparent disk management problem may be nothing other
than a forgotten online backup or other large work file created
and forgotten weeks or months before. While the df logs are
useful accross major system changes, the du logs typically are
On a web server, it's most important to verify that all web
content has been transferred and is up-to-date, that any automated
updating mechanisms are working and that any CGI or web applications
are properly working.
GeodSoft sites are pretty simple and all the functions came
accross without problems. The only issue was getting the wu-ftpd
access permissions correct to allow the automated transfer of
updated content. The biggest web related change between Red Hat 6.2 and
7.2 is that the RSA patents on which SSL depends, expired.
New Apache web servers are coming with SSL components ready for use.
The default site immediately following the basic Red Hat install
had SSL operational, verifying that everything was properly set
My httpd.conf file is customized to account for virtual sites,
non standard document root and cgi-bin locations, odd port
numbers, and multiple IP address. These had to be preserved to
access my existing sites. To get the SSL functionality I wanted
required a careful comparison of the new and old httpd.conf
files. Section by section, all the new capabilites that did not
conflict with custom settings in the old file were moved from the
new to the old file. For example the greatly expanded language
section completely replaced the old, still default language
section. The big SSL section at the bottom of the new httpd.conf
file was added to the old.
I could manually start the server with "#/usr/sbin/httpd" and the
old sites worked but not SSL. If I rebooted to use the system
startup scripts, httpd would not initialize. Running
"#/etc/rc.d/init.d/httpd start" proved the most practical way to
see what was going on with the standard initialization scripts.
They were starting httpd with additional options that were using
sections of httpd.conf that failed. Careful review showed I'd
missed loadmodule and addmodule statements for the needed SSL
library module among several other modules I did not want
enabled. A port conflict caused by a Listen statement I'd added
long ago and which was repeated in the new SSL section also
caused httpd to fail. Two additional minor edits in httpd.conf
and the GeodSoft site ran with SSL enabled.
The Hardware Switch
After configuring the new Linux web server as described above,
all the functionality of the old server worked on the new server,
except on different IP addresses. The final step was to make the
physical hardware switch. As the updated server was to
physically replace the current server, a couple minutes of
downtime was expected. Prior to moving the new server, all the IP
settings in /etc/sysconfig/network and /etc/sysconfig/network-
scripts and matching settings in httpd.conf on the Red Hat 7.2
server were set to match the old 6.2 server. The new server was
then halted and disconnected from the network and connected to a
single test workstation via a crossover cable. After rebooting
the new Linux web server was tested from the workstation to
verify the IP address changes had all been made correctly and the
web server was functioning correctly.
The server cases were located side by side for a speedy cable
swap. A web browser on a machine near the servers being swapped
was used to display the current site. The old server was halted
with the -p (power down) option and all cables reconnected to the
new machine as quickly as possible. As soon as the cables were
connected the new server was powered up. Once the boot sequence
completed, the web browser machine was used to continue fetching
web pages from what was now the new server at the old server's
network address. The total time from when Enter was pressed on
the halt command on the old server until new pages were
successfully fetched on the browser machine was 2 minutes and 2
seconds. Thus the total down time was clearly under 2 minutes.
(There was a several second delay from the initiation of the halt
command until network and web services were stopped and several
more seconds switching the keyboard, video, mouse (KVM) switch
box from the new server following the boot to the browser machine
and actually retrieving multiple pages to be sure the new server
After the new server was checked out both from local computers
and from outside testing services that retrieved pages, and it
was clear that it was functioning properly, the web logs from the
two servers were merged. The date and time of the last
access_log entry already restored to the new machine via backup and the
date and time of the first access_log entry as the new live
server were found. All access_log and error_log entries between
these dates and times on the old server were extracted and merged
into the new server's logs, so web traffic records would be
As a final step, full backups of both the old and new systems
were made and burned onto a CD-R. The old server will be kept
intact for a few days to be sure no surprise problems prevent the
continued use of the new server. After that, I'll wipe the disk
clean and use the machine for some new project.
Regardless of how thoroughly I think a machine was backed up
while it was in regular use, or how briefly a test or development
machine was in use, I try to remember to do a full system backup
before taking it out of service and reusing it for something
else. If I don't do these backups, then it seems when it's too
late, I remember some "minor" changes I made on the old system
that I now have use for. Sometimes it's simply handy to be able
check how an old system was set up. The few minutes it takes to
start a backup, transfer it to another system, and burn it to a
CD-R, plus twenty some cents for a blank CD-R, seem cheap
compared to the effort of trying to figure out what is on a
system that might matter in the future.
Top of Page -
Copyright © 2000 - 2014 by George Shaffer. This material may be
distributed only subject to the terms and conditions set forth in
These terms are subject to change. Distribution is subject to
the current terms, or at the choice of the distributor, those
in an earlier, digitally signed electronic copy of
http://GeodSoft.com/terms.htm (or cgi-bin/terms.pl) from the
time of the distribution. Distribution of substantively modified
versions of GeodSoft content is prohibited without the explicit written
permission of George Shaffer. Distribution of the work or derivatives
of the work, in whole or in part, for commercial purposes is prohibited
unless prior written permission is obtained from George Shaffer.
Distribution in accordance with these terms, for unrestricted and
uncompensated public access, non profit, or internal company use is