Linux, OpenBSD, Windows Server Comparison: Logging and Monitoring

• Windows System Logging
• UNIX System Logging
• Limited Windows Monitoring
• Tlist Is Not PS

A fundamental characteristic of a server is the tendency to create a variety of logs recording system events, and the need for administrators to review or analyze these logs for significant events, that require either corrective action or further investigation.

Windows System Logging

Windows NT provides almost no useful logging capabilities, not because the important events aren't logged, but because almost no tools are provided for working with the logs. Like nearly everything else, Windows stores its logs in a proprietary binary format, and provides a limited GUI front end to view and analyze the logs. There are three primary log files on an NT system: System, Security and Application. Some servers and applications write their own logs, but except for the Application log provided by the system, there is no other application logging assistance provided by Windows NT. Each log file has a set size, and one of three options for "Event Log Wrapping": overwrite events when the size limit is reached overwriting the oldest, overwrite after a fixed number of days, or not overwrite at all, which requires a periodic manual clearing of the logs. The only clearing option is to "Clear all events". It's not clear if this will clear all three logs or just the currently selected one as I've never tried. The only option that's ever made any sense to me, was to set a large log file size, and overwrite as needed; on some very active servers I've allowed several hundred megabytes for log files.

The Event Viewer allows you to select one of the three logs and view all entries, set a filter, or use a find option. The filter and find are almost the same, except find shows the next matching entry in the context of other entries, while filter just shows all matching entries. For both find and filter, there are five check boxes for error, warning, information, success, and failure, any or all of which may be checked. There is a source dropdown, which is the generating application in the application log, and a process or service in the system log. There is a category dropdown but there are no available choices. There are three common fields in the filter and find selections: event ID, computer, and user. These accept exact single value entries, with no ranges or wild cards of any kind. Find includes an additional description field.

The foregoing is an essentially complete listing of the logging capabilities included with Windows NT. There is no rotation capability, no selective purging, no reporting and analysis, no selective extraction, and no sending logs to other computers. You can't separate application logs except interactively while viewing them with the filter so there is no practical way to look at two applications at the same time or match any application events to system or security events.

Naturally the format is a proprietary binary format. I've wondered if the format serves any purpose other to ensure that no existing programs can make any sense of the contents. Perhaps this is intended to create a market opportunity for third party log viewers. If such a product were successful, Microsoft would probably buy it. The format is bizarre. Every character that's part of a string is separated from adjacent characters by an ASCII null. Three consecutive ASCII nulls appears to be the equivalent of a space. I cannot figure out what the string or record terminators are.

UNIX System Logging

Contrast these extreme limitations with UNIX, where every program that creates logs, has a choice of writing directly to it's own log file or logging through the syslogd service. Either way, automated rotation options are available and administrators can keep as much or as little on line as they want and break up the what's not part of the active log by almost any method that's desired, such as by hour, day, week, month, or other period, compressing or not compressing the saved results, or automatically discarding results beyond a specified age.

When the syslogd service is used, output from various programs can be merged and split in various ways and can be saved on the local machine or sent to a central log server. Types of logs, such as security or mail related logs, can be separated with different programs contributing to the security or authorization log. The administrator can control how any type or seriousness of message is handled. The same log entry can be routed to more than one destination or printed to a local or remote printer. Different actions can be taken based on the origin and seriousness of a log entry which may range from discarding the entry to simply saving it to disk in the appropriate log. More serious log entries can be sent to the system console, to all logged in users, to a printer, to e-mail, to web pages, or to a pager or any combination of these.

Besides all these more or less built in capabilities (they do require configuration and options like pagers and web pages may require the addition of open source products not included with the original operating system) UNIX log files are almost always text files. The few that are not text files have utilities to convert them to text files. As text files, all of UNIX's scripting and text manipulation utilities can be used to examine and analyze the contents of the log files. Old timers may still use sed and awk in conjunction with shell scripts and various UNIX utilities. Those who haven't been around UNIX as long, are more likely to use Perl, as it wraps more text manipulation functions in a single complete programming language than any other product, and is available on nearly every general purpose operating system.

The ability to examine, manipulate, select, and or summarize information, in a structured text file such as a log file is limited only by what information is actually present, the administrator's knowledge of that information, and his or her imagination and programming skills in isolating and otherwise manipulating the available information. Because UNIX logs can be routed to virtually any destination, the analysis and response can be performed anywhere.

I'm going to use examples from my own systems. One is the daily analysis of my firewall logs. Visually scanning large log files is a boring waste of time rarely likely to reveal much of significance. If the essential information is suitably packaged, a few seconds review is likely to indicate if there is something worth being concerned about and warranting additional investigation.

Firewall logs are like nearly all logs, in that they typically contain large amounts of information, that is not worth the time it takes to look at it. The more detailed the logs are, the more uninteresting information they contain. If however, only interesting events are logged, then it can be almost impossible to see the context in which these occurred. Some events are interesting only when they become a part of a pattern, which can only be visible if many uninteresting details are included in the logs. Thus ideally, logs should be very detailed, but administrators should only need to look at interesting events in the logs. This can only occur if detailed logs are filtered and analyzed before being presented to an administrator.

My total traffic levels are still low enough, that my firewall can log all packets including passed packets. Some will wonder why take up disk space for logs of legitimate traffic. The answer is that only by knowing the complete traffic pattern can you have a complete picture of what is going on. For example if some packets to your web site with the ACK or ACK and FIN flags set are blocked, only if you know that there was or was not a preceding legitimate web session from the sending site, can you know whether these are a port probe or legitimate packets that simply arrived too late to fall within the allowed state window. If you see all traffic, you learn that longer web sessions nearly always have some late packets that get blocked.

If you see a scan from a source following a web session from the same source and further, use web logs to determine that source host was focusing on your security related pages, you can be almost certain that someone has taken a specific interest in cracking your site, which is likely to be much more serious than a random scan. If you see what looks like a port scan and can determine that there has been no other packets from the source site that day or in the preceding few months, your site is more likely to have been picked at random which is not likely to be significant.

Currently my firewall rules allowing outbound traffic are fairly liberal. As a result, if my workstation were to be infected with a Trojan delivered via e-mail, that Trojan could most likely initiate a session with a controlling master outside my firewall. My firewall filter script removes all routine allowed traffic, such high port traffic from my workstation to any remote port 80 (web), POP3 traffic from my workstation to my ISP's POP3 server, etc. Any Trojan traffic would be highlighted and investigated. Were I to spot any odd port traffic to a host that did not reverse DNS to a familiar host, I'd know I had a serious problem that had to be dealt with.

The Perl filter lets me collect large firewall logs that are far too large to deal with manually, and selectively filter anything that looks interesting. Raw log entries are converted to counts of packets blocked and passed between hosts and the time frame over which the activity occurred. When I see an entry that looks like it's worth investigating further, I can search the raw logs by remote IP address and see all the activity in context. Over time the filter becomes more refined and fewer types of events are shown for preliminary analysis. Each day, shortly after midnight, windows are opened on my workstation showing all the logs, filtered and raw as well as tools necessary to do further investigation (nslookup and a web browser on http://www.arin.net/whois/). Each Saturday morning, all online logs are reviewed for historical patterns that might not show in a single day's activity. Currently there are less than 180 lines of Perl scripts and batch files to do all this.

Another example is extensively documented in my Homegrown Intrusion Detection Section. Each UNIX system creates frequent "#w" and "#ps ax" lists which are sent to another system. These are analyzed against normal processes, logged in users and the locations from which they are logged in. Any process or process state, user or login location that's not recognized causes a warning to be displayed on my workstation.

Log files can be automatically analyzed for anything that may be of concern. Other examples could include excessive failed login attempts, odd hour logins, excessive CPU, disk or file transfer use by any users, unusual system load averages, etc. Much that happens on UNIX systems is already logged. Anything of interest that is not logged, can be. Any information in a UNIX text log file can be analyzed in almost any manner by a capable system administrator using only tools that come with UNIX.

A much simpler example relates to backup. Most of my systems do fully automated, daily differential backups that are passed to a system with a CD-R drive which receives the backup files. Periodically, when the differential backups from a system get too large, I do a full manual backup. Each day a three line batch file opens a window on my workstation, showing how much free space is left on the CD-R drive and how much was available several preceding mornings. This alerts me when I need to change disks, the one step that must be manual. Typically this ranges from five to twelve days depending on how recently one or more full backups have been done.

For those paying attention, you'll have noticed the references to "batch" files. Others may have read other parts of this site, and know that the scripts currently execute on an NT workstation. While I firmly believe both Linux and OpenBSD make a better server solutions in most cases, Linux advantages as a workstation are not so compelling and I don't see OpenBSD as being ready for use as a general purpose business desktop system. If I were setting up a new primary workstation today, there is no question, it would (or will) be Linux. Currently however, my NT machine is adequate as a workstation. In addition, I have at least three Windows applications for which I have not found satisfactory substitutes on Linux. These are Adobe Photoshop and Macromedia's Fireworks and Dreamweaver. So for the foreseeable future, I'll either keep the current NT workstation, in addition to a Linux Workstation, or install a retail version of NT Workstation running under VMware on Linux and transfer the necessary applications.

The fact that the Perl scripts being discussed, happen to run on an NT workstation is not significant. The scripts can easily be transferred to a UNIX workstation. Some will require no changes at all except hardcoded locations of log files. Others will require a few more lines to be changed.

The following factors are significant. Except for an occasional tiny batch file that will easily be replaced by a shell script, all the scripts under discussion are written in Perl, and when they need an external utility, use UNIX tools provided via the Cgywin toolkit. The techniques used are all natural parts of standard UNIX administration, and are generally alien to Windows system administration. All analysis is of text output log files created on UNIX like systems, and automatically transferred to a remote system where the analysis is performed. This insures that results are placed in front of me, where they cannot be missed, with no need to move from system to system collecting results. Though the analysis is performed on a Windows system, I cannot perform corresponding analysis of any Windows system, including either the Windows NT Server which directly corresponds to most of the systems analyzed or the NT Workstation itself on which the analysis is performed, because Windows does not provide the tools to obtain or save in an analyzable format the necessary information.

Am I suggesting that most UNIX system administration should be done through a series of small custom scripts? No. Except for the smallest sites, most UNIX system administration will be done like much Windows system administration, through third party tools. There are two important differences. First, nearly all management tasks in the UNIX world have good open source products that can do the necessary jobs. If the open source solutions aren't sufficient, commercial products are available. Second, UNIX provides all the tools necessary to fully automate any management task that existing open source and commercial products don't do or don't do in a manner that the local administrator is comfortable with. This is simply not true on Windows systems. On UNIX, many things can be automated with less than a dozen lines of shell script or Perl code and a cron entry.

Limited Windows Monitoring

In contrast, little is typically logged on Windows systems. This does not need to be so as nearly every type of resource access, attempt, success or failure, as well as any policy or logging setup changes, can be logged. Actually system performance can be seriously affected by turning on sufficient auditing events. But, these massive auditing results are dumped into logs with which almost nothing can be done. If you use Windows auditing you'll be buried in detail that you can't analyze, at least not without buying third party products.

If you want to examine what's occurring on a Windows system, the built in options are very limited. If you want to know who ran what programs with what options, there are no tools to do so, in part because in Microsoft's graphic view of the world, command lines are of little consequence. Because of its single user view of the world, only system services and interactive process started from the local console should be running. Services can run as a user rather than the system; the system should track this. Actually it does but I've yet to find a utility that associates a process to the user who started it, or as Microsoft might say, "the security context" in which it's running. Microsoft has provided basic telnet servers and other remote command line options in the Resource kit but no tools to track who is using them. There are third party telnet and SSH servers for Windows but if these tools don't provide an appropriate management interface, I know of no way of knowing who is "logged in".

Windows NT does include several GUI program monitoring tools but these are focused almost exclusively on performance related issues. The purely interactive Task Manager can show tasks as they come and go but has no mechanism for writing logs. Performance Monitor can create logs and reports besides displaying interactive graphs. Neither shows any user related information. All displayable information is related to performance and resource consumption. Performance monitor cannot even track process that come and go; it can only be set to monitor a process that is currently executing. Once such a process is identified and included in Performance Monitor's watch list, its counts can drop to zero, i.e., it can go inactive and later be restarted and thus monitored. A second, or third or fourth instance of the same program is a different program and needs to be set up separately. Performance monitor can't however track a process, until it can be actively seen executing; thus it's essentially impossible to track any processes that start and run for less than a few seconds.

Tlist Is Not PS

The closest that Windows NT provides to being able to dynamically track processes in a manner that could be logged comes with the Resource Kit program, Tlist. Tlist will list the process ID of all running processes. A different invocation of tlist with a known process id will provide most of what Windows knows or tracks about a process. A custom written Perl script that runs in a loop can use tlist to get the process IDs of all executing process and then feed them as command line arguments to tlist again to get the process details. This approach will still have difficulty catching processes that live for only very short time periods. Either they may have already completed by the time tlist is run in the second loop or they may not have been running when tlist collected active process Ids.

Running tlist with the process ID will give current working directory, the command line that started the program, several memory statistics, thread count, IDs, addresses and states and the DLLs used by the process. No user related information is provided. As information about each process is spread over many lines, a script with several levels of loops is required to analyze this information and put it in a useful format. Doing this requires multiple add-in products including open source UNIX derived scripting tools, Perl, and UNIX like administration skills.

Not having worked with Windows 2000, I can't say what changes Microsoft has made that would change this situation. As Microsoft's orientation is almost entirely towards GUI products, of generally increasing complexity with a near total disregard for command line and text based tools there is no reason to think that Windows 2000 would make any meaningful changes in these regards. Microsoft does not build small limited function utilities that administrators can tie together with scripts to serve their own purposes. Microsoft is placing a much greater emphasis on Microsoft Management Console in Windows 2000 so it's reasonable to expect that many of what have been independent GUI programs are now modules in MMC. As the interface conventions between the older separate GUI management programs were already consistent, it's hard to see any significant consequences except that the location of any specific management option may change and be perhaps one level higher or two to three levels lower than it was previously, depending on where in the user interface you start from and how well organized the MMC modules are.

In short, Windows 2000 will have some additional features. There may be some new reports not previously available but it's highly unlikely any general purpose reporting functions that allow a high degree of user control reporting on any information, selected by any criteria from the system logs will be available. The relocation of management functions will require administrators to relearn basic mechanics of using Windows 2000 that are different than Windows NT but the tools will give administrators no more insight into how the system works nor aid automating routine tasks any more than any previous versions of Windows have. I will be glad to update this review with any corrections provided by readers who have significant work experience with both Windows NT and 2000.

Thus, though Microsoft Windows NT and 2000 systems are almost certainly easier for administrators with limited experience to learn to use initially, they are much harder for experienced system administrators to use and maintain on an ongoing basis, than any UNIX system. This comes from Microsoft's history of developing mass market desktop systems. They are unable to see the important fundamental differences between server and desktop systems. They fail to see the differences between the jobs performed by desktop users and computer system administrators. It's generally advantageous to aid the desktop user in using the computer as they do other office machines such as the telephone, photocopier or fax machines to accomplish tasks without understanding the principles on which the computer works. However, because computers are in fact complex multifunction machines, any user who needs to go beyond the basis, will benefit greatly by actually understanding the concepts of how computers work.

Top of Page - Site Map

Copyright © 2000 - 2014 by George Shaffer. This material may be distributed only subject to the terms and conditions set forth in http://GeodSoft.com/terms.htm (or http://GeodSoft.com/cgi-bin/terms.pl). These terms are subject to change. Distribution is subject to the current terms, or at the choice of the distributor, those in an earlier, digitally signed electronic copy of http://GeodSoft.com/terms.htm (or cgi-bin/terms.pl) from the time of the distribution. Distribution of substantively modified versions of GeodSoft content is prohibited without the explicit written permission of George Shaffer. Distribution of the work or derivatives of the work, in whole or in part, for commercial purposes is prohibited unless prior written permission is obtained from George Shaffer. Distribution in accordance with these terms, for unrestricted and uncompensated public access, non profit, or internal company use is allowed.