Trackers

The Top utility has served as a general-purpose system monitor on Linux systems for many years. Top originally appeared as part of the procps package [1]. Debian, Fedora, and openSUSE forked the original package and released procps-ng version 3.3.0  [2] at the end of 2011. The new Top impresses with many new fields, simpler operation, and better performance. Resourceful developers have also devised alternatives to the Top utility, including Htop [3], Atop [4], and most recently the Python program Glances [5] (see Table  1 for a comparison of features).

Htop delights users with a rollable process list, simple operation via function keys, and ASCII bar graphs for CPU utilization. Atop records the CPU, memory, disk, and network utilization, and colors highlight resources that are working at full capacity. Atop also records performance data and supports analysis with reporting functions or even interactive post-processing.

The newcomer Glances displays as much information as possible on a terminal with 80 characters and 24 rows. Glances also offers the ability to monitor remote systems by running in server mode over the network. Color highlights illuminate resource bottlenecks.

All tools divide the output into two areas: One area displays general system information (CPU, memory, storage, network); the other is a process list. Glances additionally logs the last resource bottlenecks in a small area.

You can configure the list of fields in Top and Htop. However, Atop and Glances only let you hide or restrict certain information. In small windows, both automatically omit some columns. To see all the information, you need to enlarge the window until additional information stops appearing.

Top

The unpretentious but powerful Top is preinstalled on all Linux systems. The fork offers some additional comfort: For example, you can use the arrow keys to scroll through the process list. When you do so, the program also updates the general area; this means that the display can be a little fidgety during scrolling.

Top's header area shows the uptime, the number of users, the load average, the number of processes, and various details relating to the CPU (see the "Key Metrics" box). If you want to see the values for the processor cores separately, press 1. The process list shows the process ID (PID), the user (USER), priority (PR and NI), memory usage (VIRT, RES, and SHR), process state (S, Table 2), percent CPU and memory usage (%CPU and %MEM), the CPU time the process has used so far, and the process name. (See the "Generating Stress" box for more information.)

You can set which fields to display with the f key, which opens the Fields Management view (Figure 1). Top displays the fields in bold and marked with an asterisk in the process list. Pressing the spacebar adds a field to the display or removes it. You can select a field to move with the right arrow, and move it with the up or down arrows (Figure 1). The detailed man page for Top explains each field under FIELDS/Columns.

Figure 1: Flexible: The next generation Top makes easy work of setting the fields to display.

Top does not record the amount of data a process writes to disk or reads from disk. The WCHAN field is ideal for professionals who want to know in which kernel function a process is currently sleeping: It allows conclusions to be drawn on what a process is waiting for. Recent versions even offer fields for namespaces, as used for Linux containers.

The sort field is set by pressing s in the Fields Management view and is the first row in the list (see the "Setting Fields" box). If you select %MEM here, Top sorts the processes by their physical memory consumption. This can be done in the main view by pressing M, whereas C switches back to sorting by CPU usage, and T tells Top to sort by CPU time usage.

Pressing r ("renice") changes the "nice" value of a process, which determines its priority. A lower nice value means a high priority. Possible values range from -20 to 19. To do this, you must specify the process ID (PID) and the signal number. Because the process order constantly changes to reflect the CPU utilization in this view, it is difficult to read the PID of a process from the table given a default refresh interval of three seconds. Pressing d or s sets a different interval, for example, 10 seconds. You can also send a signal to the process with k ("kill," see Table 3). Without further instructions, Top sends the signal SIGTERM, which instructs the process to exit.

For other interesting options, press h or see the man page. For example, when you press H, Top shows you all user threads individually. L tells Top to select a search term in the process or user list. The program also supports powerful filters: Pressing u lets you filter by users, and o, by arbitrary criteria. For example, pressing o then entering COMMAND = mysql filters for all processes with MySQL in their names. Entering %MEM>3.0 would list all processes that consume more than three percent RAM. Note that the commands are case sensitive. Additionally, Top always uses character-based comparisons. An exclamation mark negates a condition.

Top can also use colors, as evidenced when you press z. The color scheme can be determined by pressing Z. A tree view of the processes is shown when you press V. To see each processor individually instead of the single-line CPU overview, press 1. Pressing x highlights the column with the sort field. A tells Top to show even more process lists ("windows" in Top-speak) at the same time. You can try the following key sequence to admire Top in all its colorful splendor: A followed three times by z, x, a, followed by z, x (Figure 2). For more details, see COMMANDS for Windows in the man page.

Figure 2: Top can be very colorful: A stress command fully loads a CPU, and another occupies 1GB of physical memory, as the memory view with the blue title bar shows.

If you want to see just one of the predefined field groups, press g to toggle between the different groups. Alternatively, you can press "-" to close the views that you don't need and keep only those for CPU and memory usage, for example.

To avoid the need to customize Top every time you launch it, press W to save the current configuration in the ~/.toprc file. If you want to have a little fun, on top of all of these options, take a look at the section called STUPID TRICKS Sampler in the Top man page.

Htop

The Htop program is largely similar to Top. However, you can use the arrow keys to scroll the view vertically and horizontally. Otherwise, Htop essentially relies on the function keys for control in typical Midnight Commander style. The bottom row shows a reference.

In the two-column header, Htop shows an ASCII bar (Figure 3) on the left with the CPU utilization per processor core, followed by the main memory usage and the swap allocation. The help feature, which you can open by pressing h, shows you the color mapping: For example, in CPU load, blue stands for reduced priority via nice value, green for "normal" user processes, and red for kernel threads. In the memory bar, green shows memory in use, blue is the block device buffer, and yellow is cache memory. The number of processes, the average load, and uptime are shown on the right side. Htop shows information similar to Top in the process list.

Figure 3: Htop clearly shows the extent to which Kdepim from KDE SC 4.13.2 and the current developer version generates load on the system when retrieving a couple of email messages from POP3 accounts.

To send a signal to a process, press F9. Unlike Top, you do not need to specify the PID here: Htop targets the process currently selected and color highlighted in the list. F7 and F8 reduce and increase the nice value of the process. However, Htop changes the currently selected process when you update the list – unless you press F to follow the process in the view. Alternatively, you can select the process with the spacebar to work around this behavior. In contrast to Top, Htop can work with multiple processes after you select all the target processes. U tells Htop to clear the selection.

F3 searches for processes. F4 permanently filters the list for processes with a specific name, and u filters for users. Htop also displays threads by default. For example, a MySQL server appears with all its threads in the list. To tell Htop to toggle between the sorted view and a tree view, press F5. To sort on another field, press F6; if Htop is currently in the tree view, F6 unfolds or folds a subtree. Htop also supports C, M, and T for sorting by CPU load, memory usage, and CPU time.

Permanent changes to settings – Htop writes them to the ~/.config/htop/htoprc file – are set by pressing F2 (Figure 4). You also choose here which metrics (Meters) Htop shows in the header of the view. For example, in Display options, you can choose Hide userland threads to hide user threads or disable Hide kernel threads to display the kernel threads as well. You can also use the K and H shortcuts for this. Below Colors, you'll find various color schemes from which to choose, such as Light Terminal, which is suitable for light backgrounds. The fields in the process list are set up below Columns, as are, for example, fields for the I/O throughput caused by a process.

Figure 4: Htop also offers convenient configuration of the fields to be displayed in the process list.

For deeper analysis of processes, Htop has a few nice extras on board: For example, pressing s tells the program to tack the strace command onto the current process. This shows you what system functions the process calls. Pressing F8 enables automatic downward scrolling to follow the trace. L tells Htop to use the ltrace command for calls to library functions, and l returns a list of currently open files. The two aforementioned commands rely on the corresponding packages being installed.

Atop

The "AT Computing's System & Process Monitor" consists of three parts: the Top-style program, Atop, which can write to a logfile in addition to monitoring in real time; the Atopsar program, which creates reports from logs; and the Netatop service and kernel module, which lets you break down the network traffic by process.

Popular distributions include Atop; however, Debian only gives you the older 1.26 version. On SUSE, Red Hat, and CentOS the Monitoring repository provides a package for version 1.27 [6]. Choose the appropriate directory for your version of SUSE and add the URL as the package source by typing:

zypper addrepo <URL> server-monitoring

(See the "Building Atop and Netatop Yourself" box for more information.)

<C>tar-xf atop-2.0.2.tar.gz<C>

Atop shows information about processes, the processor load, memory plus swap, mass storage devices, and the network in the header area (Figure 5). It color-highlights resources working at high load; thus, you can see at a glance where bottlenecks are occurring. The configurable limits in the ~/.atoprc file are explained by the man page in the COLORS section. Atop also leaves out uninteresting information about unused resources or sleeping processes. Pressing a reveals all processes, however.

Figure 5: Atop monitoring the Kdepim build process. You can see some load on the CPU and the mSATA SSD. A MySQL instance belonging to Akonadi, which supplies information to Kdepim, also needs a fair amount of CPU performance.

Atop provides numerous additional details compared with Top. For example, the Paging Frequency (PAG) line provides information on the extent to which the kernel is searching for free pages or swapping out. It also provides some information on disk I/O and network traffic. Additionally, Atop takes into account the pro rata resource usage by processes that terminate during the sample interval, relying on the process accounting kernel function to do so. The kernel writes to an accounting file. You should thus never terminate Atop with SIGKILL, because it cannot otherwise stop the accounting process. Atop tags processes that terminate within the sampling with angle brackets.

The generic view (accessible by pressing g) of the process list includes information such as the PID, system and user CPU utilization, physical and virtual address space, and data read and written. VGROW and RGROW state how much additional address space the process has requested. This information is often more interesting than the absolute values.

Atop also supports other views: m shows main memory usage with minor and major page faults, the absolute physical and virtual address space, and percent memory usage. In this view, you can see the volume of data that is read and written, and the data to be written that the kernel didn't write because the process deleted it prior to this action (in WRDSK_CANCEL). In conjunction with Netatop, Atop shows a network usage list  [6] when you press n.

Depending on the view, Atop sorts the process list by CPU, main memory, mass storage, or network utilization as a percentage. Keyboard shortcuts control the sort order regardless of the view. Note that you can sort by the most used resource by pressing A. Atop shows at a glance to what extent processes utilize the most heavily used resource – the bottleneck.