If you have ever done any system administration, chances are you have used tail [1] to watch the changes to the end of a file. MultiTail [2] is a supercharged version of Tail that uses ncurses to create multiple windows in a single console so you can view more the one file at a time. It includes options for starting a command with a new window or searching through its windows, as well as extensive options for customizing and manipulating the windows. The result is a command that is not only vastly more powerful than Tail, but easier to read and quicker to use as well.

MultiTail is packaged in many distributions and is also available as source code from the project website. It includes an extensive configuration file, /etc/multitail.conf for the system or ~/.multitailrc in the current account's home directory. How to edit either file would be an article in itself [3], but many of the choices are separate color schemes to indicate the various applications you are likely to use with MultiTail, ranging from the kernel or Apache to Squid and Sendmail.

The basic command is the name, followed by a space-separated list of files to open (Figure 1). Strictly speaking, the list of files should be prefaced by -i, but the option is only necessary if a command begins with a hyphen (-). Notice that each MultiTail window has a number that is used to manipulate it, as well as the file name on the title bar at its bottom.

Figure 1: MultiTail showing mounts and partitions from the /proc directory.

The only problem you are likely to run into with this basic command is if you run it in a virtual terminal that is too small to create windows of the default size (Figure 2). In that case, simply close down the tab and maximize the desktop window before running MultiTail again.

Figure 2: A desktop window that is too small can prevent MultiTail from creating the default MultiTail windows.

This is about as far as the Tail command goes. However, at this point, MultiTail is only beginning [4].

Extending the Basics

To start, running multitail with -l FILE options opens each file in the same window as the previous file listed in the command, merging it with any existing content. You can also use the option -iw "FILE" INTERVAL to set the number of seconds between updates.

The interval between updates of the display can also be set using -r INTERVAL, in which the interval is the number of seconds. If you want to append the updated information to the existing information, use -R INTERVAL instead. The suitable interval depends on what is being viewed. At one extreme, /proc/mounts, which displays mounted filesystems, would rarely require an interval of less than 60 seconds (and then only for external drives), whereas klogd, the kernel log daemon, could require a regular interval of five seconds because new activity is being entered constantly.

However, the most interesting option for starting MultiTail is -l COMMAND, which opens the output of a command in a window or, to be more specific, -L COMMAND, which runs a command in the last window opened. When either option is used, if you add options to the command, you need to place both the command and its options in quotation marks, so you would enter -l "free -m" rather than -l free -m. Either -l or -L can be specified along with a file, so the command multitail -l less /proc/partitions -l less /proc/mount would be the same as piping the command through less.

MultiTail's Keyboard Commands

Once MultiTail is displaying files, you can use keystrokes to manipulate the MultiTail windows. To view a help file that lists the available keyboard shortcuts, press the F1 key (Figure 3). You can close both the help file and any key command in progress by pressing Ctrl+G (for good-bye, apparently).

Figure 3: MultiTail's help file appears in a pop-up menu in the middle of the screen.

Using the keyboard, you can press a to add a new file or command to the MultiTail window, or d to delete a file or command. If deletion seems too drastic to you, z hides and unhides windows, creating a less cluttered display. Similarly, you can press p or P to stop updating temporarily. Yet another alternative is to type o to wipe a MultiTail window's contents while keeping it open, or O to wipe all contents.

Each of these keyboard commands open simple wizards that are part of the help system (Figure 4). Some of the questions, such as which color scheme to use, may not interest you, in which case, you can choose one of the available choices at random. You can add files in new MultiTail windows or in part of the space currently used by existing windows. You may also press s to swap window positions to organize the display.

Figure 4: MultiTail includes several keyboard commands that open simple wizards to help guide you through an operation.

When you use MultiTail, you are usually looking for specific information. Typing a forward slash assists your search by opening a search wizard with an option for case insensitivity, whereas typing b lets you scroll back in a window in a separate buffer, and typing e signals that you about to use regular expressions.

If you want to see what was happening in different files at the same time, B lets you scroll back through all open windows at the same time – which can be confusing but also handy for troubleshooting if you take your time.

If a MultiTail window gets too long, press T to truncate it. R can restart it to ensure you have the very latest information. Still another key command to aid a search is to add a bookmark using the numbers 0-9. If, in the middle of an analysis, the system of windows momentarily confuses you, so that you forget which files or commands are sharing the same buffer, pressing t gives you a statistical overview of MultiTail as a whole (Figure 5).

Figure 5: Pressing the t key gives you an overview of MultiTail's current state.

Advanced Options

The multitail command also takes several dozen other options. For example, as with tail, -n NUMBER specifies the number of lines to display in each MultiTail window. You can also specify vertical columns, rather than the default horizontal columns with -s NUMBER (Figure 6).

Figure 6: You can specify vertical columns for viewing.

If you really want to be particular, you can also add -sw PIXEL-WIDTH, in which the pixel position is a comma-separated list. If you add a comma but do not specify a pixel position (e.g., -sw 50,50,,50), then that column's is width is whatever is left after the width of the other columns is allotted.

Although the configuration file sets up elaborate color schemes, you can opt to forego all of them with the -w option. Less sweepingly, you can use -c FILE to suppress color when displaying a particular file, or C to suppress color in a list of files. You can also do the reverse: Entering -cs SCHEME or -Cs SCHEME specifies a color scheme to use from the configuration file.

Still other options detail how to merge files and commands into one window or to write to file rather than standard output. Inconveniently, MultiTail's man page lists all options alphabetically, instead of by function, but related options are usually close enough in structure that you can quickly learn the ones that interest you.