Keeping Watch

Linux lets multiple users work in parallel. With multiple users on the system, the system administrator needs a means for monitoring user activity to watch for unusual behavior and to keep tabs on how the system is being used.

In this article, I describe some simple command-line tools that help the admin quickly identify strange behavior and ensure a stable operations.

Users on Your System

The /etc/passwd file contains user account settings (Listing 1). The lines start with the login name, followed by the user and group IDs, the user's real name, the user's home directory, and the program that automatically launches after successful authentication (typically the /bin/sh or the /bin/bash login shell.

You can create a list of user accounts using the universal cut tool to extract the first field from each line of the file. The option d: (delimiter) sets the separator to the colon sign; -f 1 references the first text field in each row. Listing 2 shows the results of the command line.

The first account is always the administrative user root, ID 0. The list includes entries for system services without a genuine login account, such as exim, sshd, avahi, or saned. Other entries only appear in the list under certain circumstances. For example, www-data only appears if the system is running a web server such as Apache or Nginx.

Accounts for system services have a low user ID and login shell entries of the form /bin/false or /usr/sbin/nologin. If you log in as a user with this kind of account, you can start the program; if the entry points to /bin/false, you will not have a shell.

The awk command in Listing 3 will help you filter out all of the possible users by their user ID from the /etc/passwd file. Note that a regular user ID will always have a value greater than 1000. The -F: option sets the colon as the delimiter, and the '$3>999{print $3,$1}' parameter evaluates the content of the third test field and checks whether it is a value greater than 999. If so, the contents of the third and first field (user ID and login) are output to stdout. The simple quotation marks in the call stops the shell from evaluating the option itself.

The | (pipe) operator directs Awk's output to the sort command, which then sorts the individual lines in ascending order. Because the user IDs are in the first column of the output as numerical values, sort uses the -n flag to order the numerical data in the correct way (e.g., 10 after 9).

You could use this list to check the home directories, for example. Occasionally, users who no longer exist but who have not been deleted will be identified. Home directories of users who don't have access should also be suspect, especially if the home directory should be located elsewhere according to /etc/passwd.

Currently Active Users

The list resulting from the execution of Listing  3 will help you in assessing whether the updated list of users currently logged in to your system is plausible. The following three standard tools are useful: users, w, and who. The users tool shows a list of usernames (Listing  4). If a name is listed more than once, that user, according to the system, currently has many active login sessions. These could be open terminals, for example.

Listings 5 and 6 show the outputs of w and who and are considerably more detailed. The w output begins with a header, which contains the running time of the system, as well as the system time and uptime. Next to that are the number of users and their average load on the system. Listed underneath are the users and their current system usage information. In the second and third columns are the terminals (TTY) being used and the originating IP address (FROM).

The mosh- value indicates that an encrypted connection is established with the Mosh program [1] [2]. You also see the login time, any idle time, and CPU usage for the past one, five, and 15 minutes. The last column tells you what command the user is running, which can give you a clue as to the kind of activity taking place.

The who command shows the information seen in Listing 6, which is similar to w, although in a more compact form. You can see the login names, the (pseudo-)terminals users are using, the time of login, and finally the IP address the user is connecting from or the program making the connection. The value in the square brackets indicates the corresponding process number.

The who tool can take the -a option (shows a more verbose output) and the -u option (only shows information for the users that are logged in). In addition to terminals (tty), you can also see if the user is accessing the X Window System. Listing 7 shows the output of all activities on a desktop system. Line 2 shows the time the computer was booted. Line  4 shows the change to runlevel 2, and line 6 shows that an X Window session is running (on display :0). Lines 7 to 12 show real terminals (activated by using Ctrl+Alt+F1 to F6) waiting for activity. Lines 13 through 23 contain pseudo-terminals, two of which have the error codes 0 (lines 20 and 21). Pseudo-terminals appear, for example, when you open a terminal window on an X Window interface in which you activated the program xterm. (See the "Using Zsh" box for additional information.)

rechner% watch=(all)
rechner%
test has logged off pts/7 from localhost.
test has logged on pts/7 from localhost.
rechner%

Colorful Output

If you want to see something more exciting than the monochrome output of the tools mentioned above, you can use the whowatch [3] tool. The program shows users and their processes output graphically, in color, and, more importantly, updated in real time. Figure 1 shows the output of a system with 11 users (7 local, 1 connected via SSH, and 3 others) bustling about.

Figure 1: Process analysis with whowatch.

In the illustration, the indented processes were started within the processes above them. So, you can see that the user dd connected over SSH activated a terminal, which runs a Bash shell and is currently working with Midnight Commander (mc). The whowatch command works with the keyboard and has a menu you can start by pressing F9. The menu offers features such as the possibility of displaying details of a selected process and ending the process if necessary.