Ins and Outs

If you have kernel 2.6.20 or newer, with the CONFIG_TASK_IO_ACCOUNTING option built in, you can also use pidstat to monitor the I/O load for individual processes or whole groups. On a system with the right support, typing

pidstat -d 2 3

will generate output as shown in Figure 3. Pidstat will list the processes generating I/O and even tell you the type (read or write), along with the data throughput (in Kbps).

Figure 3: Linux administrators can now use pidstat to monitor I/O load for individual processes.

To allow this to happen, you need a recent kernel, the packages from the Sysstat website, and a matching cron configuration.

Listing 1 shows the /etc/cron.d/sysstat file on Ubuntu. In the simplest of all cases, the following two lines in /etc/crontab are all you need:

*/10 * * * * root /usr/lib/sa/sa1 1 1
53 23 * * * root /usr/lib/sa/sa2 -A

The first entry starts sadc (the System Activity Data Collector), the server daemon that collects data every second and stores its findings under /var/log/sa. The logfile format is saXX, where XX is the date. The second part with sar -A just rotates the logfiles.

Iostat and Mpstat

Besides sar and pidstat, the Sysstat package also includes iostat and mpstat. Mpstat monitors the processor you pass in to it; iostat is popular with file, database, and mail server administrators as a tool for measuring system I/O load by drive, disk, partition, and filesystem.

For example,

iostat -m -n 10 2

monitors an NFS mount point and outputs the volume of data in megabytes (Listing 2). For a single partition, you can do the same by typing iostat -m -x sda2 10 2, as shown in Listing 3.

Absolutely Essential

The latest version of the Sysstat package includes some new features that many administrators have been waiting for – above all, the ability to monitor I/O load per process. Linux had some catching up to do in this regard. It is a pity you can't use this feature on older servers or on enterprise distributions. It is questionable whether distributions will respond with matching kernels and packages in future releases.

If you can do without detailed I/O statistics, a combination of sar and kSar will give you a flexible and comprehensive interface that lets you monitor multiple servers at the same time.

Even if you don't have a full-fledged monitoring system, you will still be able to identify recurring bottlenecks and take steps to fix any performance issues before a system crash.