Moving a Red Hat 6.2 Linux Web Server to Red Hat 7.2 - 11/29/2001

Less Than Two Minutes Downtime!

• Migrating vs. Upgrading
• The Old Red Hat 6.2 System and Hardened Servers
• Hard Disk Partitioning Issues
• Actual Partitioning Decisions
• The "Hidden" Partition
• Network Install
• Passwords
• Package Installation Choices
• Services
• SSH Replaces Telnet
• Xinetd Replaces Inetd and TCP Wrappers
• No Shadow Passwords
• What to Move?
• Red Hat's Strange NTP Setup
• Moved Applications, Files, and Logs
• Httpd.conf Updates
• The Hardware Switch
• Backups

Migrating vs. Upgrading

This page covers issues relating to moving or migrating a web server from Red Hat 6.2 to Red Hat 7.2 Linux. This is NOT an upgrade. Red Hat 7.2 is installed freshly on a replacement machine. Desired components including the web site and system administration scripts are restored from backups of the 6.2 system. The system configuration is manually updated on the new system to keep the settings necessary for the web site and related applications. This might be more work than an upgrade but the benefits more than outweigh the costs. It eliminates nearly all downtime and virtually guarntees a successful transition. The result is a clean secure system including all relevant new Red Hat 7.2 features that performs all the application and other custom functions of the original Linux web server.

The Red Hat 6.2 Linux "server" was the last server I installed before getting a full time Internet connection (mid 2000). It was a custom install that included both desktop and server components with every additional package I thought I might want. The new 7.2 server is stripped of everything not supporting needed functions. Issues dealt with include: moving to a "huge" disk much larger than needed by the system and partitioning choices; removing unnecessary packages; replacing inetd and TCP Wrappers with xinetd; replacing telnet and telnetd with SSH; trying to replace FTP with SSH; installing NTP as a Red Hat package rather than compiling from source; enabling shadow passwords; Apache configuration changes and enabling SSL; installing components from previous system; migrating log files; planing physical changeover and IP configuration changes.

The traditional approach to upgrading operating systems on business servers is almost obsolete. It may still be viable on desktop systems in both home and work environments, but even here it is increasing likely to provide an unsatisfactory outcome. The complexity of operating systems has increased by orders of magnitudes since the early days of DOS where upgrading meant replacing a few key system files and utilities in a few directories. This process was nearly always successful.

Today, by the time that an operating system needs to be upgraded, two things have typically occured. Hardware has advanced to the point that the current systems may have some difficulty with the older hardware (depending on both how widely supported the individual components are and whether the system to be upgraded was state of the art when it was new or was a mid range to low end system already containing dated components). Over the life of the system, software is likely to have been added and configurations changed, resulting in an increasingly less standard system with variations that the upgrade program must cope with. The more customized and dated the system to be upgraded, the lower is the likelihood of success.

With desktop systems, assuming good backups are made before the upgrade process begins, it should generally be possible to restore the system to it's original state in the event the upgrade is not successful. With luck, the upgrade will be successful, but will include components that should have been discarded. With less luck, the upgrade will fail and the system will be restored as it was prior to the upgrade attempt. All that is lost is time. In the worst case, a previously working system may be left inoperable. At home an unsuccessful upgrade is not likely to be critical but will be frustrating; at work it may result in some significant loss of productivity, depending on how dependent the employee is on a specific desktop system.

For business servers the situation changes dramatically. All the issues relevant to desktop systems are likely to apply, but an additional factor becomes even more important. This is the availability or lack of availability of the server during the upgrade. For the installation of patches and service packs, depending on the operating system there may no perceptible downtime to a few minutes if all goes well. For full operating system upgrades, downtime cannot be reliably predicted. In ideal circumstances downtime may be measured in minutes but if anything goes wrong, it could easily be hours and in difficult situations requiring a reinstall and restore of the old system, could be a day or two.

In the old days, that is before every business was connected to the Internet, small to mid size shops scheduled their upgrades for weekends - often Saturday morning. If things went well, you'd be home for lunch but if not, there were two days to be ready for business on Monday morning. Larger businesses with redundant systems had and still do have much more latitude regarding these issues. Regardless of the actual schedule, nearly all small to medium size businesses had off hours and system maintenance, including operating system upgrades could be scheduled so as to have minimum impact on business operations.

Today, things are different. Even if the employees are at home, nearly all business have public web sites that are accessed by the public seven days a week, 24 hours a day. When these sites are not available, it reflects poorly on the company. Unlike internal systems, you can't post messages or send e-mails saying some system may not be available for some of its normal operating period. Public web servers must be available continuously or their absence will be noticed by those you hope to attract to your business. Any backend server that provides real time services to the public web servers is subject to similar considerations. Until the business is large enough to have redundant systems and able to phase in upgrades without the changes being visible to the web public, a business must carefully plan how to upgrade their web servers and any systems providing services to the web servers.

Unless you don't care about your public image, the traditional operating system upgrade is simply not a viable method for upgrading a public web server or any system providing services to a public web server. Even under ideal circumstances it will be visible to some of your web users. Everyone who's been in the computer business for any length of time knows that uneventful (perfect) major system changes are very much more the exception than the rule.

The traditional operating system upgrade should now be replaced by a system migration. Essentially this is installing a new operating system, applications and data on a replacement machine and planning a machine substitution that entails a minimum disruption. The replacement machine may be functionally similar to the machine being replaced or it may be an upgrade if the old system was nearing any of its performance limits. Where advance planning is an option, the same techniques may be used to replace any major components including both hardware and software that may need replacing or upgrading.

Depending on your planning and testing, and the physical environment in which the change is being made, it should be possible to make a switch that holds the disruptions in services to 15 seconds. Most web browsers actively using the site, should be unaware of any change or service disruption. If they request a page at just the wrong time, hopefully they will stop the retrieval, and try again before the original request times out. (Web applications that require state maintenace, will unavoidably be interputed.)

To effect a change this smooth, several conditions must be met. There needs to be sufficient cable connectivity (power and UPS, network, KVM) that both machines are fully connected when the switch is made. Shutdown and boot times must be known for both machines to within a few seconds. The idea is to boot the replacement so that its networking functions are activated approximately 10 seconds after the networking functions on the machine being replaced are shutdown. The network parameters on the replacement machine should be identical to the replaced machine, which means they must be reconfigured and tested on a physically separate network, which may be as few as two machines. The replaced system should be left off and not changed until it is clear the replacement is working. The replaced system cannot be re-connected to the live network until its networking parameters have been changed.

If physical constraints don't allow for both systems to be connected simultaneously, or it's desired that the replacement system occupy the current location and connections of the machine being replaced, which may be a requirement in the case of firewalls, the disruption will necessarily be somewhat longer. It will be the sum of the time necessary to shutdown the machine to be replaced, move the machines and switch the cables to the replacement machine and boot the new system. With systems that halt and boot quickly, and a convenient work area, two minutes should be a reasonable goal.

The primary advantages of a migration compared to an upgrade are that the system being migrated to can be fully tested prior to being placed in service. Even in the event of some entirely unexpected failure on the new server, the original should be available and unaltered so it may simply be returned to its original location to continue performing its normal functions.

The main disadvantages to a migration versus an upgrade is that first it requires an available and suitable machine. It will also be substantially more work than an upgrade that goes smoothly but likely less work than an upgrade that is not successful. Theoretically, until it goes wrong, an upgrade is a largely automated process. A migration requires significant detailed planning and testing.

The Old Red Hat 6.2 System and Hardened Servers

When I installed the Red Had 6.2 Linux web server it included every software component I thought I might be useful. This was the last machine I ever installed as completely general purpose computer, able to perform any function I might desire on a computer. I still had the Windows', maximum-functionality, it's- easier-to-install-it-all-at-once, mentality that still dominates much of the computer industry. Fortunately, I soon understood the security implications of this approach. Before I connected this machine to the Internet (behind a custom firewall on a hardened OpenBSD system), I'd disabled all unneeded services and placed additional restrictions via TCP Wrappers on those few that were left. The system was hardly a well configured Internet server but with the outer firewall and good tradition host security, it was reasonably secure. The GUI desktop did allow me to view my web sites, which where still in transitional designs, via multiple browsers and platforms.

Over much of the next year, I focused more on OpenBSD than Linux and my OpenBSD firewalls and web servers went through an iterative hardening process more extreme than that described in any security books or literature I've read. This led to my Hardening OpenBSD Internet Servers web section that's been revised and expanded multiple times. Meanwhile the Linux web server reliably served web pages without a reboot for over 11 months, until I accidentally pressed the reset button (when I intended to reset an NT server on an identical box sitting next to the Linux server).

After the release of Red Hat 7.1, I spent significant time with Linux, working with Red Hat as well as the secure Linux, Trustix and EnGarde, distributions. Though both of these secure distributions have much to recommend, each has issues that suggest it may not be the best platform for a web server in my environment at this time.

Following the release of Red Hat 7.2, I decided it was time to replace the Linux web server. The primary requirement was that all functionality related to the web site on the existing server be moved to the new server. This included Swish-e full text searching, a virtual management site with web statistics provided by Analog and other CGI scripts; I hoped to get SSL to work (which was still under RSA patent restrictions when I set up the previous server). All existing custom administrative functions including backup, various custom logging and intrusion detection had to work on the new machine.

The new Red Hat 7.2 server would be hardened but within the framework of conventional hardening techniques avoiding the more extreme techniques I've used on my OpenBSD systems.

Hard Disk Partitioning Issues

The 6.2 web server had a 13GB hard disk of which about 2.5GB was used. The only disk management I've performed is about every 4 months, I've archived the online backups. So far, there is probably enough space that all of these could have been left online. The replacement machine has a 40GB disk drive which is far more than needed for normal operations. The question was how best to use (or not use) this excess disk capacity.

I've never used the auto partitioning features of Red Hat. I have used Disk Druid but find it reorders the partitions in ways I don't want. All recent and future installs have and will be done with fdisk which gives full control over what partitions are placed where. Partitioning is about trading security and flexibility. A single partition occupying the entire hard disk with a single filesystem is most flexible because all available space will be available to any new file regardless of what directory it goes in. The drawback is that one large partition is most susceptible to damage such as when a power outage crashes your machine, especially if it's writing to the disk at the time. Separate partitions significantly increase recovery prospects.

If you use multiple partitions on small disks you need to accurately predict where your file growth will occur. If all your disk is partitioned and you fill one partition, even if other partitions have plenty of space, using that space will become a significant management issue. As disks get larger you have more options how to use your disk space. You can leave unpartitioned space and create new partitions as needed, mounting them inside of partitions that are becoming full. Generally your are limited to a total of 16 partitions per disk. Some may not be useable so you could have only 12 - 14 useable partitions.

Instead of keeping unpartitioned space you can allocate all space, estimating where it will be needed most. After a system crash, the system checks the integrity of the filesystems and normally fixes any problems automatically. If a hard disk is much larger than needed, the checking will take much longer than on a smaller hard disk or the same hard disk with either unpartitioned space, or partitions that are not normally mounted. Partioning and mounting all available space on huge hard disks will greatly slow the boot process after a crash when you will want the system back quickly.

There are many opinions on which directories should be mount points for separate filesystems. There must be a system root directory, /, and filesystem. It's not clear to me if /boot needs to be a separate filesystem in Linux. It is clear that /bin, /dev, /etc, /mnt, /proc, /root and /sbin must be part of the / filesystem so the devices, configuration files, and key utilities will be on the first mounted partition. For example, without /bin/mount, none of the other filesystems can be mounted.

Several top level directories are often mount points for filesystems. These are /home, /opt, /tmp, /usr, and /var. Sometimes /usr/local is a mount point for a separate filesystem. None of these need to be separate filesystems provided / has enough space to hold the contents of the respective directories. The reason these often are mount points for filesystems is that they do not have to be part of / and one or more of these will likely consume large amounts of disk space, depending on how the system is used. /usr contains the large majority of system installed programs on most UNIX systems. After the initial install /usr typically does not grow by large amounts unless a significant number of products are added. /var doesn't start with much but is often the fastest growing directory. This is where most system logs as well as mail and print queues go. On a high volume firewall, web server, or intrusion detection system, logs may grow at an enormous rate. Local custom logs or online backups may be kept under /var contributing to rapid growth. /var may be the largest filesystem on some systems.

/tmp is for temporary files but how temporary is temporary? Some systems clear /tmp on reboot and others automatically remove old files in /tmp and any subdirectories. Nothing that you care about or need to keep more than briefly should go in /tmp. It may only contain editor work files and print files in progress in which case it might be part of /. On the other hand, with large hard disks /tmp can be a good working directory for system backups and projects that briefly require very large files. If disk size permits, I like /tmp as a separate filesystem large enough to do a full system backup and compress it as a separate step. If a backup goes in any /var subdirectory, then it's necessary to program around that file location so the growing backup file doesn't get recursively included in itself. Creating a backup in /tmp is an easy way around this. All top level directories that are wanted can easily be listed and backed up to a file in /tmp, which should never have contents that need to be backed up. Following the backup, the backup file might be moved to a /var or other location.

/home is for user files. Each user gets a subdirectory matching their username and this contains their initialization files. Depending on the system and the number of users, user files my be negligible or quite large.

What to do with applications and their data that are not standard parts of a system install is a key disk management issue. Such programs and their data are likely to be placed in one of three locations. /usr/local contains bin, etc, and sbin subdirectories as well as several other directories that mimic the system root. The "local" designation suggests this is a place to add system specific applications and many applications install by default into this area. Some place the executables in bin or sbin. Others create their own subdirectory tree directly under /usr/local. If it's expected there will be many such applications /usr/local is likely to be a separate filesystem. Typically the applications that go in /usr/local do not have large quantities of data.

Large data sets are most likely to be placed in one of two locations. Perhaps the more conventional location is /home. Typically a directory will be created to match the application name or use. Web document trees often go in /home and the directory names might match the domain or virtual site names. Databases might also go in home. Databases can be from less than a megabyte to multiple terabytes in size. Larger ones are often a machine's sole reason for existing.

The other location for data and applications is /opt for "optional" which is like "local" in /usr/local. /opt may have applications and related data together. Personally, I've always preferred to keep data separate from programs, but this does require a straight forward mechanism for relating programs and data. On a traditional multi user host with many active users, /opt makes good sense to separate application data from personal user data. On a dedicated server with a handful of administrative users, the choice of /home or /opt for data is little more than personal preference or habit. Every business or organization should establish standards, preferably formal and documented, for what types of files are stored in which directories and subdirectories.

In Red Hat 6.2 the default document root was /home/httpd/html with cgi-bin and icons as subdirectories also in httpd. Somebody stopped thinking when Red Hat set the default locations for 7.2. The default web document root and cgi-bin directories were moved to /var/www. I know *BSD does this also. I wonder what or if anyone was thinking when this choice was made. Programs don't belong in /var; data doesn't belong in /var. Web document trees are data and programs; they don't belong in var. The web server logs do belong in /var but not web documents. People edit web documents and update databases from which web pages are created. Nobody but intruders should ever be editing logs. Nothing in /var should be edited or changed by legitimate users. Depending on how the rest of the system is set up, web sites belong in /home or /opt.

Actual Partitioning Decisions

So how did I partition a 40GB drive when the old system used about 2.5GB and the new minimum install created a much smaller install of only about 750MB? First I decided to partition just over half of the total disk and of the approximately 20GB partitioned, I oversized everything.

As I said above, I used fdisk for full control. I used three of the four primary partitions. The first was a small 50MB partition (/dev/hda1) for /boot. By default fdisk starts at cylinder 1 and lets you pick partitions 1 - 4 as primary or extended. It gives you the first available cylinder and lets you specify ending cylinder or size in KB or MB. If you've made an extended partition you can create logical partitions inside the extended partition. /boot is only 4% used and won't grow so it should have been about 10 - 15MB regardless of disk size.

The second primary partition was a 250MB swap partition (/dev/hda2). A common recommendation is to make swap partitions twice the physical RAM size. With 128MB RAM and Linux, I'd do this but with 256MB, that seems a waste. Linux is typically using just over half the physical RAM and has yet to touch the swap space. By comparison OpenBSD only uses 20 some megabytes leaving the rest free and does not go to swap. Windows (at least NT), even with 384MB leaves much memory free but makes needless heavy use of swap space.

Though I have no need for all four partitions, on a future install, I want to try a larger primary for / and locate /boot in that and make the second partition an extended partition and see if the swap partition will go in that. The third partition was an extended partition and I went back and forth over using the entire disk or not. Since I was sure the machine would never be a dual boot, I chose to make the extended partition the rest of the disk (/dev/hda3).

In the extended partition, I made /dev/hda5 250MB for /. /dev/hda4 is reserved for the fourth possible main partition. / is about 35% full and not likely to grow much. If however the filesystem for /tmp is not mounted this leaves enough room for normal system activities so seems about right. I'd only make it smaller if space were an issue.

For /usr, I made /dev/hda6 3GB. After the install this is only about 15% full and not likely to grow significantly. 1GB would have been comfortable. With the large number of packages I removed 500MB might have been sufficient but would have allowed almost no room for growth. A basic Red Hat 7.2 server install without the X Window system is 687MB.

For /home, I made /dev/hda7 2GB. With the current files this is only 7% full. 500MB would have been comfortable. With a largely static site there would be almost no possibility of ever needing more space. If however, dynamic applications are added that create significant amounts of data, or multiple dynamic applications are added, 500MB might become crowded. 1GB would allow sufficient room for growth for almost any plausible web site development that the site would almost certainly be upgraded or migrated again before space became an issue. This assumes no non web related applications are added to /home. Unexpected changes make any disk use projections unreliable.

For /var, I made /dev/hda8 6GB. This is the only filesystem on the new server that's smaller than the previous system which had an 8GB /var. The old system /var was 20% full. More space in var would be needed only if I wanted to accumulate all differential backups online for well over a year. All foreseeable web and system logs should easily be accommodated. I keep all web logs from the very beginning of the site and migrate old logs to the new server. Even if somehow the site were producing a gigabyte of logs per day, I'd find a way to preserve them all. Other than the actual web content, nothing is more important for a web site than a complete collection of all raw logs. With these, any future analysis tool can perform its analysis from the begining and produce valid results. If only reports, abstracts, or totals are saved, there is no way to switch or add to the analysis tools and get any valid results except going forward. I would definitely preserve web logs in preference to online backups, assuming of course that the backups have been migrated to some removable media.

I created an 8GB /dev/hda9 for /tmp. This is more than large enough for any foreseeable backups or other potential uses. One of the first things that I did, was to untar the last full backup of the existing server into /tmp/rh62. I thought this would give me more opportunity than directly restoring from a tar file to view, evaluate, and selectively restore files from the old system to the new. In addition, as work configuring the new system progressed, I placed full system backups in /tmp.

A review of the filesystems and sizes, starting at cylinder 1 follows. The /wrk partition is explained later.

/boot	50MB
<swap>	250MB
/	250MB
/usr	3GB
/home	2GB
/var	6GB
/tmp	8GB
/wrk	18GB

I used fdisk and set this physical arrangement hoping to group the most used partitions together and put the rarely used partitions outside the frequently used ones. After the system boots, there is no reason to return to the /boot partition. With more RAM than the system needs, swap should rarely if ever be used, so after it boots, the disk heads should rarely go to the outer cylinders. / contains a number of frequently used utilities such as ls, grep, less and others. Hopefully these will get cached and rarely be accessed from disk. This leaves three adjacent partitions, where programs from /usr work with data from /home and write logs to /var. After the program is read in from /usr, hopefully there will be little need for the disk heads to return there. Some programs will use temporary files in /tmp. That at least is the theory and why I partitioned in the order that I did. I don't know how to actually test the results. /wrk is not normally mounted.

The importance of planning disk partitions has been based on limited disk space and the need to reasonably anticipate system growth. Large disks reduce the importance of properly anticipating future disk use needs. In addition, Linux now includes Logical Volume Management (LVM) capabilites that greatly increase the flexibility of how physical disk space is allocated and used. Another new program, parted, allows ext2, FAT and FAT32 partitions to be created, destroyed, resized, moved and copied. These capabilities have not existed (for Linux systems ) in the past and mitigate the need to anticipate future disk needs.

The "Hidden" Partition

One of the automated process I run on my systems is to create md5sum logs of all /etc files and all executable files. Consecutive days are automatically compared and any changes displayed. This lets me verify that system configuration files and executables only change when I expect them to. The saved log file was dramatically altered by the /tmp/rh62 contents which were essentially a replica of another full system.

To deal with this I decided to put the remaining unpartitioned disk space to use. fdisk from the command line as root quickly turned the available space in the extended partition into an 18GB ext2 partition. mkfs did not recognize the new partition and I had to reboot the system for it to do so. After rebooting, within a few minutes I had a new 18GB disk work area that I could mount and use when I wanted. I created a /wrk for a mount point and after mounting /dev/hda10 on it, moved all /tmp contents to /wrk. If I don't need this stuff, I simply "#umount /wrk" and it disappears. This prevents my checksum scripts from including this, but any old files I might still need to check are easily available. As I do more full backups, they will also be to this filesystem that is not normally mounted. Since it's not normally mounted, if the system crashes, it won't spend time checking this filesystem when it reboots. Even if /wrk happened to be mounted during a crash, it wouldn't be checked until the next time it was mounted.

Network Install

The Red Hat 7.2 network install appears to be a step backwards. It no longer defaults most of the TCP/IP fields and those that are defaulted, are wrong. I don't know what happened the first time I did this install. I entered all the IP parameters but when I booted after the install no networking was set up. There were some other problems so I started over and this time all the settings entered were preserved as expected.

I chose not to set up any firewall during the install. I've not been satisfied with the firewall settings resulting from the Red Hat 7.1 and 7.2 install process. I'll deal with these manually later. This machine is behind a strong outer firewall that shields it from everything but requests on port 80. The only other services running are sshd and in.ftpd; both are protected by /etc/hosts.allow /etc/hosts.deny and limited to local traffic so another layer of firewall protection is not a pressing need. With the limited and unpredictable results of firewalls setup by the Red Hat 7 series install routines, the resulting firewall is more likely to cause problems than benefits. Red Hat needs to deal with IPTables in its install routines and needs the install process to clearly indicate what will and will not be allowed depending on which options are chosen.

Passwords

Following the network install the root password and a non privileged user and password were set up. These passwords contained mixed case letters, multiple digits and multiple symbols and or punctuation. Using my Password Evaluator they had strength ratings of 8 or 9 and should take a fast desktop PC between 10 and 20 million years to crack, i.e., even NSA should not be able to crack them with today's technology.

Package Installation Choices

For the new Linux web server, I started with Red Hat's server install option and deleted the X Window System and other unnecessary packages. I selected web server and the select individual packages option. I deselected about 150 packages but dependency requirements put about 20 of these back. The packages that I removed were: 4Suite, LPRNG, PyXML, VFLIB2 & devel, XFree86-libs, XFree86-xfs, a2ps, alchemist, apmd, arts, ash, asp2php, audiofile & devel, aumix, authconfig, autoconf, autofs, awesfs, bc, bcm5820, bzip2-devel, cdda2wav, edlahelgen, cdp, cdparanoia, cdrecord & devel, cipe, cracklib & dicts, cuvl & devel, dhcpd, efax, emacs, esound & devel, expat & devel, fetchmail, finger & server, foomatic, freetype, gcc-g77, gd & devel, gdb, gettext, ghostscript & fonts, gsm & devel, hotplug, indent, isdn4k-utils, kbdconfig, krb5-libs & devel, libogg & devel, libpng & devel, libtiff & devel, libvorbis & devel, links, lokkit, lrzsz, micq, mikmod, minicom, mod_dav, mpage, mpg321, mt- st, nfs-utils, newt, nkf, nmap, nmh, nscd, nss_ldap, openjade, openldap*, pam_krb5, php*, pidentd, ppp, printconf, procmail, psutils, python*, radvd, raidtools, rdate, reiserfsutils, rhn_register, rp-pppoe, rpm-python, rsh & server, rsync, rusers & server, rwall & server, rwho, , sndconfig, sox, statserial, swig, syslinux, talk & server, tamang, tcl, tcpdump, tcsh, telnet & server, ttfonts-ja, udc-snmpt, up2date, urw-fonts, vorbis, wantanbek-vf, whois, wvdail, xdelta, yp*.

The packages kept due to dependencies were: autoconf, cracklib, crack-dict, expat, freetype, gd, krb5-libs, libjpeg, libpng, libcap, newt, openjade, openldp, popt, postgresql-libs, postgresql-tcl, procmail, syslinux, tcl, tcsh, XFree86-libs.

I added bash-doc and ntp and following the install, added wu- ftpd. The result was 264 packages for 501MB. In looking over many of the packages that I removed, some are completely harmless and others such as the various sound related functions, simply irrelevant to a machine without a sound card or speakers. These are consumer trash that should never be selected in the first place for a machine designated as a server. From the packages included by Red Hat in a "server" install one might draw two conclusions. First, Red Hat is still caught in a Windows type mentality, trying to make a single computer serve a variety of unrelated functions. Second, Red Hat thinks its core audience is home hobbyists. Otherwise, why would all the packages needed to play audio CDs be included on a server. Does Red Hat think a professional system administrator would be listening to CDs on the same machine that they are configuring as a web or other server?

Several programming languages were removed; any language that I don't expect to use on a specific machine is a hazard. If it's not used to support the machine's intended functions, it has no value. In the hands of an intruder, any programming language can become a powerful tool. All the insecure old protocols related to rsh and yp are removed; they simply have no place on a machine intended for use as a public Internet server.

Some of the programs are downright dangerous including nmap and tcpdump. Nmap is a cracker tool. While it certainly has legitimate security uses for probing one's own LAN or possibly those of customers in need of security assistance, it has no business sitting on a public web server. If the machine is compromised, tcpdump is just the program the intruder needs to analyze traffic on the LAN in preparation for compromising nearby machines. Both telnet and telnetd are removed; these have been completely replaced by SSH. I'll leave a telnet client on my workdstation for access to clients' systems who have not yet migrated to SSH but it has no business on a server where, if the server is compromised, it becomes a tool to attack other systems.

For the time being I have ftp, ncftp, and wu-ftpd. I don't want any of these but I have yet to get scp (the secure copy part of SSH) to work between Windows and UNIX machines. It works fine between UNIX systems. Until this problem is solved, I need file transfer both directions and initiated from both sides between Windows and UNIX systems. Wu-ftpd is restricted to accepting only connections from my workstation. I'll look into options to hinder the use of ftp and or ncftp and likely remove one. I may set my outer firewall to block FTP requests from the web server.

In some ways, a very dangerous program is the text browser, links. Nearly every firewall is set to allow outbound web browsing sessions. It's trivial for an intruder to set up their own web server (possibly limited to accepting connections from compromised machines). With a web browser on a compromised machine, an intruder can use their own web server as a transfer agent to pass any cracking tools desired to the compromised system. These might be used to cover tracks on the compromised system or to load tools needed to use the compromised machine as a base of operations for attacks on other machines.

Ideally the server would lack all client programs, including both ftp and links mentioned above, capable of initiating connections to arbitrary destination computers. The server should have only passive servers capable of receiving connections from other computers. The outer firewall would block all traffic except the intended public services between the server and the Internet. The non public services, ideally only sshd, would be configured only to accept connections from designated management workstations. With a server set up like this, even if an application bug and or misconfiguration allowed an intruder to gain root access, they would be trapped on that server, unable to use the server to attack any other targets. To make use of the compromised server, the intruder would also have to independently crack, either the outer firewall protecting the server or a management workstation that is set up to communicate with and control the server.

Services

Following the install there were a number of modifications to make. I turned off sendmail, xfs and ipchains; when I set up a firewall it will be with IPTables. Even though I added the ntp package, I had to turn it on.

To review the services the system will be starting or not, run "chkconfig --list|less". Primarily what matters is the setting for run level 3 (4 and 5 are normally the same) which is normal multi user mode. Service names are listed on the left; a "3:on" indicates the service will be started by the system at boot time and a "3:off" that it will not be started. To change whether a service is started or not use "#chkconfig servicename on" or "#chkconfig servicename off" as appropriate.

SSH Replaces Telnet

The SSH daemon, sshd, is started by default on Red Hat 7.2 Linux systems. Configuration files and server keys are in /etc/ssh. The file sshd_config is the configuration file for sshd. There is only one important change that should be made in this file. The line "PermitRootLogin yes" should be changed to "PermitRootLogin no". This will prevent root from logging in directly from a remote location and forces a remote user to know two passwords to get remote root access.

If access to su is limited to a few non privileged users who also have strong passwords this would effectively eliminate unauthorized remote root logins, even if network configuration did not restrict access to sshd and the shadow password file fell into the wrong hands. I also changed "X11Forwarding yes" to "X11Forwarding no"; this is not significant and if you insist on running X on servers you will probably want this left yes.

With these settings you can use ssh to log into the system using a local username and password as you would with telnet. The difference is that the passwords and connection are passed back and forth with strong encryption so someone on your LAN or a network segment between cannot eavesdrop on your session by packet sniffing. Also the root password is protected when a user using ssh, su's to root. It is not necessary to set up public and private keys though many will feel this is a more secure way to use SSH. My beginners SSH tutorial covers key setup if you wish to use this approach. See the discussion of /etc/hosts.allow and /etc/hosts.deny below for further restrictions related to ssh connections.

Xinetd Replaces Inetd and TCP Wrappers

An important change between Red Hat 6.2 Linux and Red Hat 7.2 is that inetd has been replaced by xinetd. Inetd was a daemon that listened for a variety of connection types, typically including telnet, ftp, and a variety of protocols that do not normally remain as memory resident daemons. When a connection request comes in for one of these, inetd and now xinetd, starts the appropriate service. Inetd had no security and another program known as TCP Wrappers was added that allowed control of requests by the IP address or hostname from which the requests were received. Xinetd adds TCP Wrappers capabilities, plus additional controls to inetd.

Part of the access control is implemented in two files /etc/hosts.allow and /etc/hosts.deny. See "man hosts.allow" for format details and options. Lines that start with a # are comment lines. I strongly recommend adding a single active line to hosts.deny that consists solely of "ALL: ALL". This blocks all connections controlled by the hosts.deny and hosts.allow files and forces all connections that are to be allowed to be explicitly authorized in hosts.allow. I tend to make my restrictions extremely limited, often allowing connections only from a single management workstation.

I installed wu-ftpd as my ftpd server and ftp clients access it through xinetd. If I wanted to allow ftp connections from anywhere, the line "in.ftpd: ALL" would allow this. If the local LAN network address was 192.168.123.0 and all machines on the LAN were to be allowed to connect, either "in.ftpd: 192.168.123." or "in.ftpd: 192.168.123.0/255.255.255.0" would work. Obviously the first format is much simpler, but if the LAN did not have a standard netmask, the second format provides the necessary flexibility to accomodate subnets. A single host connection could be enabled with "in.ftpd: 192.168.123.22".

There is much more to xinetd. Inetd had a simple /etc/inetd.conf configuration file with one line per service. Xinetd is controlled by an /etc/xinetd.conf file which can, and by default on Red Hat Linux, does specify an /etc/xinetd.d directory. See "man xinetd.conf" for the format which is much more complicated than was the configuration for inetd. Besides options for specific services there are default settings that may apply to all services started by xinetd. Options can be single values or lists matched to the option name. For lists, values may be added or removed or the whole list replaced. Defaults set in xinetd.conf may be overridden by specific files in /etc/xinetd.d. When the wu-ftpd package was installed, the necessary /etc/xinetd.d/wu-ftpd file was created but it was set to disabled. The file had to be manually edited and xinetd restarted to make ftp connections. Among the parameters that can be controlled are "only_from" and "no_access" which provide access controls similar to the access restrictions provided by hosts.allow and hosts.deny.

The settings in xinetd.conf and in xinetd.d are read when xinetd starts. If options in these files are changed, xinetd must be restarted to put the changes into effect. In contrast, the settings in hosts.allow and hosts.deny are checked every time a connection attempt is made, making them much easier to change and test. The hosts.allow and hosts.deny settings override similar settings made in the configuration files. The surest way to restart xinetd is to use the command "#/etc/rc.d/init.d/xinetd restart". If it's successful, you'll see "Stopping xinetd:" with an "[  OK  ]" followed by "Starting xinetd:" and "[  OK  ]". The OKs will be in green and to the right of the screen just the way they are when the system starts or stops. If there was a problem, you would see a red [ FAILED].

Though sshd runs continuously as a daemon and does not use xinetd to control connections, it does use the hosts.allow and hosts.deny file in the same manner that xinetd does. If hosts.deny has an "ALL: ALL" line then ssh connections must be authorized in hosts.allow or none will ever succeed regardless of any other ssh setup. The line "sshd: ALL" will allow connections from anywhere. To be more selective set up IP addresses or ranges as suggested above or see "man hosts.allow" for the various supported formats.

No Shadow Passwords

One of the things that really surprised me on the new Red Hat Linux install, was that shadow passwords were not enabled. It's my recollection that shadow password files were used as long as I've been using Red Hat (since 6.1). I don't recall ever having to enable them on any other Red Hat system I've installed or used. Except for Corel Linux, I don't recall every working on or installing any UNIX or UNIX like system that didn't use shadow password files. A recent, default install of Red Hat 7.2 as a desktop (workstation) system clearly has shadow passwords enabled and I did not turn them on. This server install though, had no /etc/shadow or /etc/gshadow files. It was very simple to turn them on. The two commands "#pwconv" and "#grpconv" did it. Still, I'm perplexed by seeing variations from one Red Hat 7.2 install to another, that have nothing to do with options selected during the install, and no rational explanation why there is a variation.

What to Move?

Perhaps the largest task was identifying what to move from the Red Hat 6.2 system to the new 7.2 system. Having a fully expanded copy of the old system online made this easier. Over time, any system accumulates some junk, and this should be left behind when moving to a new system. Obviously non of the old Red Hat 6.2 executables would be wanted. I try to be systematic about where new things are added to a system. Besides /home and /root, I use /usr/local and also create a /var/local. /usr/local/bin gets homegrown scripts and most individual utilities. A few products have directories in /usr/local and I created a /usr/local/tools directory for most products that I build locally on the system; some of these executables are installed in /usr/local/bin or sbin.

Having just about everything in /usr/local makes it pretty easy to select what to keep. A locally built copy of nmap as well as some other hacker tools were discarded in keeping with the stripped and secure nature of the new system.

Red Hat's Strange NTP Setup

Built copies of ntpd and related executables were removed from /usr/local/bin in deference to the Red Hat supplied package. Two related scripts in /usr/local/bin were kept but modified. These were ntp to start ntpd and ntp.stop to stop it. They were modified so that their contents simply became "/etc/rc.d/init.d/ntpd start" and "/etc/rc.d/init.d/ntpd stop" to keep manual stops and starts of ntpd consistent with the bootup starts and halt or shutdown stops. I did not like the Red Hat default parameters passed to ntpd in /etc/sysconfig/ntpd which included an option not documented in the NTP man pages. I replaced the Red Hat parameters with ones I was comfortable with.

Nothing about the Red Hat setup of NTP makes a bit of sense to me. NTP is something I've spent considerable time with; see my NTP Tutorial. The default Red Hat ntp.conf file is an empty shell with comments that are not likely to be of use to anyone. It's meaningless without consulting the official documentation at http://www.ntp.org/documentation.html. Review the ntpd man page and the configuration file man page.

I set up ntpd to use a strictly defined, hierarchical client server setup. I have top level servers that each use three outside, independent Stratum 2 time sources, and peer with each other. These machines in turn act as servers for all other machines on the LAN which also peer with each other so my machines are clearly defined to be running at Strata 3 and 4. Obviously the new Linux system will use the carefully build ntp.conf file from the system it will be replacing. Also, at least for my purposes, the Red Hat default location for the ntp.drift file in /etc is not acceptable. This file is regularly updated by ntpd. As I have automated alerts for any files that are changed, added to or removed from /etc on all my systems, I placed ntp.drift in /var/local/conf/ where it won't trigger change alerts every day.

The default Red Hat 7.2 ntpd startup code resets the system time each time the system boots. This makes no sense. Unless the computer has been powered down for an extended period, any computer running ntpd will have close to the correct time. If it is off by more than a 1000 seconds, ntpd will exit and the system time will need to be set manually. Otherwise, ntpd will set the correct time - that's what ntpd is designed to do. Red Hat provides no instructions on using the /etc/ntp/step-tickers file which is not a standard part of NTP but is part of Red Hat's startup code. With my ntp.conf file, which has worked on all my production systems in the past year and a half, the standard Red Hat startup code hangs the boot process indefinitely. This code also uses the deprecated ntpdate program which will be phased out. Red Hat does not seem to keep up with developments of the supplementary products it includes in its installs, but then with 7 CDs of software in its high end commercial version, how could it? Whoever set un NTP at Red Hat either knows a lot more about NTP than I do, but neglects to tell us how to use it, or they know a lot less. In the end, I kept the standard System V style initialization used by Linux, but replaced all the NTP specifics provided by Red Hat.

Moved Applications, Files, and Logs

Two applications which were brought forward were the Swish-e text searching programs to provide text searching for the web site and Analog, the web traffic analysis program. I could have used this as an opportunity to upgrade these; both are performing adequately and rather than complicate the migration, these look like tasks better left as separate projects. Both ran as they had on the old system after being restored. Each has CGI and cron components.

The entire web site was copied from the /tmp/rh62 directories and updated with the latest differential backup to bring it into sync with the live sites. Likewise the current crontab was brought forward intact from the old system. This runs mostly scripts from /usr/local/bin. Most ran as expected. Red Hat 6.2 included /usr/bin/ksh but not 7.2. A few of the scripts used ksh as their interpreter. As bash is upwardly compatible with ksh, adding a ksh symbolic link in /usr/bin to /bin/bash fixed these problems without editing the scripts that started with "#!/usr/bin/ksh".

I create a "local" subdirectory in /var to accomodate a significant number of custom logs my scripts create as well as online backups. /var/local normally contains backup, conf, download, and logs subdirectories. I never move old backups forward to a new system. If I need some specific old file, it should be on the appropriately dated CD-R drive. conf only has the ntp.drift file which should not be moved from one system to another as it is system specific; it's a measure of how accurate and in which direction (fast or slow) the system's hardware clock is. This information is what allows ntpd to keep accurate time on a computer if it is up but becomes disconnected from its time servers.

The /var/local/download directory normally has .tar and .tgz files from which additional products were added to the local system. If the product works out, these files are normally moved to an installed subdirectory.

/var/local/logs contains both individual logs and additional subdirectories for some logs that proliferate quickly. Among the logs I keep are those related to Homegrown Intrusion Detection. This includes md5sums of system configuration and executable files as well as frequent snapshots of all processes running on the system. As both change dramatically between system releases, there is no point in moving these to a new system. It is important to get a checksum file as soon after the new system is installed as possible, i.e. shortly after the install is complete and before other setup changes are made. This should be moved to another system as well as a CD-R drive and serves as a baseline comparison to identify any unauthorized changes to system executables that might be made in the future.

I always use a script when I edit crontab. This captures an image of crontab before any changes are made. These do not move to a new system. If you make a mistake in crontab, having the last good copy available will greatly speed the correction and reduce tension as you know its merely a matter of a few minutes until any problem is corrected. These also establish a record of how the system's automated processes develop over time.

Another log I keep on every system I manage is the output of uptime every 15 minutes for the life of the system. Once a day, the date and time are inserted into this log. A record of how long the system has been up can be handy as well as the number of users at different times of the day and on various days of the week. On a busy system these can track major trends. To me the most important part and reasons for creating this log are the load averages that are included. Over the life of a system it can be invaluable to know when and how high the peak loads are and how they compare to a month, six months or a year before. If users complain of system performance, it becomes simple to determine if there is a true long term trend, a temporary anomaly that needs to be addressed or simply a user's poor memory. Load average is simply the grossest measure of system use, but it's an excellent indicator when more refined measures need to be taken.

For disk use I keep two sets of logs, each produced once a week. Saving the output from a simple df in a file named by date will show long term disk use trends. This I normally preserve across a system migration or when disks or partitions are added. A second set of du outputs (created at the same times as the df outputs) for the entire system, help find where disk space is being consumed. If a filesystem runs short of disk space, a simple grep of the df files can identify anytime there was a significant change in disk use. The matching du files can then identify the directories in which the change occured and normally lead to specific files that account for large parts of the change. An apparent disk management problem may be nothing other than a forgotten online backup or other large work file created and forgotten weeks or months before. While the df logs are useful accross major system changes, the du logs typically are not.

Httpd.conf Updates

On a web server, it's most important to verify that all web content has been transferred and is up-to-date, that any automated updating mechanisms are working and that any CGI or web applications are properly working.

GeodSoft sites are pretty simple and all the functions came accross without problems. The only issue was getting the wu-ftpd access permissions correct to allow the automated transfer of updated content. The biggest web related change between Red Hat 6.2 and 7.2 is that the RSA patents on which SSL depends, expired. New Apache web servers are coming with SSL components ready for use. The default site immediately following the basic Red Hat install had SSL operational, verifying that everything was properly set up.

My httpd.conf file is customized to account for virtual sites, non standard document root and cgi-bin locations, odd port numbers, and multiple IP address. These had to be preserved to access my existing sites. To get the SSL functionality I wanted required a careful comparison of the new and old httpd.conf files. Section by section, all the new capabilites that did not conflict with custom settings in the old file were moved from the new to the old file. For example the greatly expanded language section completely replaced the old, still default language section. The big SSL section at the bottom of the new httpd.conf file was added to the old.

I could manually start the server with "#/usr/sbin/httpd" and the old sites worked but not SSL. If I rebooted to use the system startup scripts, httpd would not initialize. Running "#/etc/rc.d/init.d/httpd start" proved the most practical way to see what was going on with the standard initialization scripts. They were starting httpd with additional options that were using sections of httpd.conf that failed. Careful review showed I'd missed loadmodule and addmodule statements for the needed SSL library module among several other modules I did not want enabled. A port conflict caused by a Listen statement I'd added long ago and which was repeated in the new SSL section also caused httpd to fail. Two additional minor edits in httpd.conf and the GeodSoft site ran with SSL enabled.

The Hardware Switch

After configuring the new Linux web server as described above, all the functionality of the old server worked on the new server, except on different IP addresses. The final step was to make the physical hardware switch. As the updated server was to physically replace the current server, a couple minutes of downtime was expected. Prior to moving the new server, all the IP settings in /etc/sysconfig/network and /etc/sysconfig/network- scripts and matching settings in httpd.conf on the Red Hat 7.2 server were set to match the old 6.2 server. The new server was then halted and disconnected from the network and connected to a single test workstation via a crossover cable. After rebooting the new Linux web server was tested from the workstation to verify the IP address changes had all been made correctly and the web server was functioning correctly.

The server cases were located side by side for a speedy cable swap. A web browser on a machine near the servers being swapped was used to display the current site. The old server was halted with the -p (power down) option and all cables reconnected to the new machine as quickly as possible. As soon as the cables were connected the new server was powered up. Once the boot sequence completed, the web browser machine was used to continue fetching web pages from what was now the new server at the old server's network address. The total time from when Enter was pressed on the halt command on the old server until new pages were successfully fetched on the browser machine was 2 minutes and 2 seconds. Thus the total down time was clearly under 2 minutes. (There was a several second delay from the initiation of the halt command until network and web services were stopped and several more seconds switching the keyboard, video, mouse (KVM) switch box from the new server following the boot to the browser machine and actually retrieving multiple pages to be sure the new server was working.)

After the new server was checked out both from local computers and from outside testing services that retrieved pages, and it was clear that it was functioning properly, the web logs from the two servers were merged. The date and time of the last access_log entry already restored to the new machine via backup and the date and time of the first access_log entry as the new live server were found. All access_log and error_log entries between these dates and times on the old server were extracted and merged into the new server's logs, so web traffic records would be complete.

Backups

As a final step, full backups of both the old and new systems were made and burned onto a CD-R. The old server will be kept intact for a few days to be sure no surprise problems prevent the continued use of the new server. After that, I'll wipe the disk clean and use the machine for some new project.

Regardless of how thoroughly I think a machine was backed up while it was in regular use, or how briefly a test or development machine was in use, I try to remember to do a full system backup before taking it out of service and reusing it for something else. If I don't do these backups, then it seems when it's too late, I remember some "minor" changes I made on the old system that I now have use for. Sometimes it's simply handy to be able check how an old system was set up. The few minutes it takes to start a backup, transfer it to another system, and burn it to a CD-R, plus twenty some cents for a blank CD-R, seem cheap compared to the effort of trying to figure out what is on a system that might matter in the future.

Top of Page - Site Map

Copyright © 2000 - 2014 by George Shaffer. This material may be distributed only subject to the terms and conditions set forth in http://GeodSoft.com/terms.htm (or http://GeodSoft.com/cgi-bin/terms.pl). These terms are subject to change. Distribution is subject to the current terms, or at the choice of the distributor, those in an earlier, digitally signed electronic copy of http://GeodSoft.com/terms.htm (or cgi-bin/terms.pl) from the time of the distribution. Distribution of substantively modified versions of GeodSoft content is prohibited without the explicit written permission of George Shaffer. Distribution of the work or derivatives of the work, in whole or in part, for commercial purposes is prohibited unless prior written permission is obtained from George Shaffer. Distribution in accordance with these terms, for unrestricted and uncompensated public access, non profit, or internal company use is allowed.