Performance analysis with iostat, sar, Ksar, collectd, and serverstats
Performance at a Glance

© Lead Image © Martin Blech, Fotolia.com
We describe five tools you can use to monitor and troubleshoot your system's performance.
Sooner or later, most server administrators realize they need to do some performance analysis. Whether you hope to detect bottlenecks or plan resources, reliable performance data is essential for any well-managed network. Also, historical records showing performance over an extended period makes it easier to forecast and adapt to changes.
The most important part of measuring performance values is that you don't start measuring right away. To begin, you need to consider carefully what information you want to obtain. Here are five suggestions for what is important during a performance analysis:
- Document your observations and results. You must allocate presumptions and conclusions based on your data.
- Set up a baseline of performance for your systems. This baseline is a defined initial state that you can use to qualify measurements.
- Measure the state of the system during the analysis before making a change. All data you collect for a defined system will help.
- Locate the bottlenecks in your system. Which resources limit performance?
- Make realistic assessments regarding which measures improve performance, and restrict yourself to these parts when optimizing.
In this article, I describe some tools you can use to monitor and troubleshoot system performance.
Iostat
Data for input and output operations are important in so far as the I/O system affects overall system performance in the case of bottlenecks. The parameter iowait is used as an example here; it is the time the CPU waits for I/O requests to be processed. Large I/O latencies can be a cause of high load on the system; but, they do not always need to be.
Iostat is part of the sysstat package that appears in the Ubuntu repositories as "System Performance Tools for Linux." Other programs included in this package are sar – which is described later – pidstat, mpstat, nfsiostat, and cifsiostat. The sysstat programs are consistent front ends to the /proc
filesystem of the Linux kernel. You cannot therefore provide any more statistics than the kernel offers for /proc
.
Iostat also provides the option -d
for querying device statistics. The tool delivers the CPU utilization report using the -c
option. Keep in mind that the first output by default contains the values since the system start. All other reports refer to the interval between individual outputs. If desired, you can skip the first report using the option -y
and receive only the current values.
Performance Indicators
Advanced output using -x
gives a parameter which, at first glance, is a factor for the utilization of devices for many users: %util (Listing 1).
Listing 1
iostat
A look at the man page, however, corrects the common mistaken belief that %util indicates the actual device usage: %util is the percentage of CPU time in which I/O requests were issued to the device. It may be true for individual traditional hard drives that a higher %util value is an indicator of good device utilization. However, RAID devices or SSDs also process several requests in parallel; the CPU time spent for the device is therefore put into perspective.
Better indicators of correct I/O utilization are %iowait in the CPU report and avgqu-sz and await in the disk report. %iowait is the percentage of CPU idle time during which I/O requests were still on the system. However, another pitfall is waiting here: %iowait is a special form of idle time and thus a sub-parameter of the actual CPU idle time. If a task that lets %iowait rise and a CPU-intensive task are running on a CPU core, iowait drops automatically. This situation arises because the second task consumes CPU time and there is thus no longer any idle time.
Listing 2 shows a simple experiment. The corresponding output from Top and iostat bring the idle times to light (Listing 3). If a computationally intensive task is running simultaneously with an I/O task, %iowait is zero, but %util has risen (Listing 4). As previously mentioned, the CPU-related idle times, as well as the values avgqu-sz and await proximal to the device are significant.
Listing 2
Task Set Experiment
Listing 3
Idle Processes
Listing 4
CPU-Intensive Task and I/O Task
Iostat is a tool that provides statistics on the block layer level. Requests are often applied in this context. They are structures that the Linux I/O scheduler works with. It passes them on to the device driver dispatch queue, which presents the last step in the Linux I/O chain.
The value avgqu-sz denotes the average length of the queue that was passed to the device. From time_in_queue and the iostat interval time_in_queue, it calculates the time in which requests had to wait for the device. If several in_flight requests have to wait, time_in_queue increases by the product of time_in_queue and the number of in_flight requests [1]. Finally, avgqu-sz can be calculated from the equation [2]:
(delta[time_in_queue]/interval)/1000.0
A simple example using five in_flight requests in 20ms with an iostat interval of five seconds would result in an avgqu-sz of 0.02. The following therefore applies for a device: The larger avgqu-sz is with a constant await time, the more requests it was able to handle.
The await value is in turn the average time spent waiting for an I/O request:
delta(read_ticks+write_ticks)/delta(read_IOs+write_IOs)
The product rule for several in_flight requests as discussed previously also applies for the ticks used here. It is possible to identify from the derivation of the two parameters that they are suitable as indicators for device utilization. Large await times with small avgu-sz are an indicator that it is necessary to wait for the requests to be processed by the device.
Using plotiostat
You can create graphs from the iostat output using detours. To this end, note the iostat calls in a logfile over a longer period:
iostat -xym 1 > iostat.log
It is best to run this command on a remote server in a screen session or using nohup
. Then, the command will not abort even if the SSH connection ends. A simple Python script called plotiostat.py3
[3] then generates a graphic from the logfile:
python plotiostat.py -f iostat.log -d sda -c rMB/s -c wMB/s
Finally, note that iostat provides figures at the block layer level. Performing an analysis at the application level is best done using tools such as iotop or atop, which display who is currently generating an I/O load at the process level in the style of the well-known program Top.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Direct Download
Read full article as PDF:
Price $2.95
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Find SysAdmin Jobs
News
-
Escuelas Linux 8.0 Now Available
Just in time for its 25th anniversary, the developers of Escuelas Linux have released the latest version.
-
LibreOffice 7.5 Loaded with New Features and Improvements
The favorite office suite of the Linux community has a new release that includes some visual refreshing and new features across all modules.
-
The Next Major Release of Elementary OS Has Arrived
It's been over a year since the developers of elementary OS released version 6.1 (Jólnir) but they've finally made their latest release (Horus) available with a renewed focus on the user.
-
KDE Plasma 5.27 Beta Is Ready for Testing
The latest beta iteration of the KDE Plasma desktop is now available and includes some important additions and fixes.
-
Netrunner OS 23 Is Now Available
The latest version of this Linux distribution is now based on Debian Bullseye and is ready for installation and finally hits the KDE 5.20 branch of the desktop.
-
New Linux Distribution Built for Gamers
With a Gnome desktop that offers different layouts and a custom kernel, PikaOS is a great option for gamers of all types.
-
System76 Beefs Up Popular Pangolin Laptop
The darling of open-source-powered laptops and desktops will soon drop a new AMD Ryzen 7-powered version of their popular Pangolin laptop.
-
Nobara Project Is a Modified Version of Fedora with User-Friendly Fixes
If you're looking for a version of Fedora that includes third-party and proprietary packages, look no further than the Nobara Project.
-
Gnome 44 Now Has a Release Date
Gnome 44 will be officially released on March 22, 2023.
-
Nitrux 2.6 Available with Kernel 6.1 and a Major Change
The developers of Nitrux have officially released version 2.6 of their Linux distribution with plenty of new features to excite users.