When the performance analysis book by Brendan Gregg [1], which I had long been waiting for, appeared recently, I devoured it greedily. I even had access to a prerelease so that in January I was able to give the readers of Linux Magazine some tips on the kernel probes for throughput measurement by means of the Berkeley Packet Filters [2]. Performance guru Gregg also explains at the beginning of the book how he often uses classic command-line tools to find out what is causing a struggling server to suffer.

10 Commands

He starts off with a list of 10 command-line utilities (Listing 1) [3], which every Unix system understands, checks everyday things – such as how long the system has been running without rebooting (uptime) or whether there is anything noticeable in the system log (dmesg). The vmstat command looks for processes waiting for their turn on a CPU. It also checks if all RAM is occupied and the system is wildly swapping. Similarly, free shows the available memory.

# uptime
# dmesg | tail
# vmstat 1
# mpstat -P ALL 1
# pidstat 1
# iostat -xz 1
# free -m
# sar -n DEV 1
# sar -n TCP,ETCP 1
# top

Entering pidstat visualizes how processes are distributed on the system's CPUs; iostat determines whether the data exchange with the hard disk is the bottleneck. The utilization of individual CPUs is illustrated by mpstat, which also shows whether a single process is permanently blocking an entire CPU. The sar commands collect statistics about the network activity on a regular basis, including the throughput data. The best known of the tools, top, gives an overview of RAM usage and running processes, but according to Gregg, should be called last.

As the Go language offers both excellent interaction with external processes and can conjure up fast and attractive terminal UIs, my idea was to fire off all the analysis commands more or less at the same time and then to display the neatly structured results in separate panes of the same terminal window (Figure 1).

Figure 1: The Greggalizer fires off all of Listing 1's performance analysis commands and displays the results in separate window panes.

Listing 2 shows the runit() function, the central part of the newly created command-line utility Greggalizer – the name I gave the analysis program in honor of Gregg. The function receives a command in the form of an array of strings, executes it, and returns the contents of its standard output to the caller. This comes courtesy of the exec package's Command() function in the Go standard library, which accepts a command with parameters. The Command() call executes the requested program with the specified arguments and returns an object, on which the code then calls the Output() method to retrieve its output as a character string.

01 package main
02
03 import (
04   "fmt"
05   "os/exec"
06   "regexp"
07 )
08
09 func runit(argv []string) string {
10   out, err := exec.Command(
11     argv[0], argv[1:]...).Output()
12
13   if err != nil {
14     return fmt.Sprintf("%v\n", err)
15   }
16
17   r := regexp.MustCompile("\\t")
18   return r.ReplaceAllString(
19     string(out), " ")
20 }

Since various Unix command-line tools structure their text output using tabs, but the widgets in the terminal UI cannot cope with these special characters, line 17 defines a regular expression that matches and later replaces tabs with spaces. Since regexes expect the tab character to be \t and the expression is double quoted, Listing 2 needs to double the backslash (\\t) to preserve the single backslash within double quotes.

Go has to compile regular expressions. However, nothing can go wrong with a single tab, so line 17 uses MustCompile(), which does not return any error code, but would blow up in your face if a regex failed to compile. ReplaceAllString() then replaces all tabs in the out byte array with spaces, and runit() returns the result as a string to the caller.

Drawing the Test Grid

Listing 3 shows the main() program, which starts the Greggalizer. In the initial import statements, the code pulls in the Termdash project from the GitHub server, to draw and manage the terminal UI. The array of string slices in the commands variable starting in line 16 define the various commands that the program will run, along with their parameters.

01 package main
02
03 import (
04   "context"
05   "fmt"
06   "github.com/mum4k/termdash"
07   "github.com/mum4k/termdash/cell"
08   "github.com/mum4k/termdash/container"
09   "github.com/mum4k/termdash/linestyle"
10   "github.com/mum4k/termdash/terminal/termbox"
11   "github.com/mum4k/termdash/terminal/terminalapi"
12   "github.com/mum4k/termdash/widgets/text"
13 )
14
15 func main() {
16   commands := [][]string{
17     {"/usr/bin/uptime"},
18     {"/bin/bash", "-c",
19       "dmesg | tail -10"},
20     {"/usr/bin/vmstat", "1", "1"},
21     {"/usr/bin/mpstat", "-P", "ALL"},
22     {"/usr/bin/pidstat", "1", "1"},
23     {"/usr/bin/iostat", "-xz", "1", "1"},
24     {"/usr/bin/free", "-m"},
25     {"/usr/bin/sar",
26       "-n", "DEV", "1", "1"},
27     {"/usr/bin/sar",
28       "-n", "TCP,ETCP", "1", "1"},
29   }
30
31   t, err := termbox.New()
32   if err != nil {
33     panic(err)
34   }
35   defer t.Close()
36
37   ctx, cancel := context.WithCancel(
38     context.Background())
39
40   widgets := []container.Option{
41     container.ID("top"),
42     container.Border(linestyle.Light),
43     container.BorderTitle(
44       " Greggalizer ")}
45
46   panes := []*text.Text{}
47
48   for _, command := range commands {
49     pane, err := text.New(
50       text.RollContent(),
51       text.WrapAtWords())
52     if err != nil {
53       panic(err)
54     }
55
56     red := text.WriteCellOpts(
57       cell.FgColor(cell.ColorRed))
58     pane.Write(
59       fmt.Sprintf("%v\n", command), red)
60     pane.Write(runit(command))
61
62     panes = append(panes, pane)
63   }
64
65   rows := panesSplit(panes)
66
67   widgets = append(widgets, rows)
68
69   c, err := container.New(t, widgets...)
70   if err != nil {
71     panic(err)
72   }
73
74   quit := func(k *terminalapi.Keyboard) {
75     if k.Key == 'q' || k.Key == 'Q' {
76       cancel()
77     }
78   }
79
80   err = termdash.Run(ctx, t, c,
81     termdash.KeyboardSubscriber(quit))
82   if err != nil {
83     panic(err)
84   }
85 }

Some commands, such as pidstat, accept both an update interval and – optionally – the maximum number of iterations to perform their task. For example, pidstat 1 prints out tasks currently being processed by the kernel, as an infinite loop in one second intervals. A second parameter specifies the maximum number of calls; pidstat 1 1 terminates after the first result, just like the Greggalizer wants.

You might have noticed that the top command is missing from the list; this is because it uses terminal escape sequences for its display, similar to the Greggalizer's terminal UI. Its output would have needed to be preprocessed and has for that reason been excluded. Also, because exec can only execute simple executable programs with arguments, it cannot process commands like dmesg | tail -10 directly. Two commands linked by a pipe can only be understood by the shell. Therefore, line 18 simply uses bash -c to pass the whole command to a bash shell as a string for execution.

In line 31, termbox.New() defines a new virtual terminal, which the defer call in line 35 neatly collapses when the program ends. The widgets slice in line 40 defines the widget panes in the window and populates the UI with the main "top" widget, which draws a frame and writes the program title at the top.

The panes slice in line 46 defines pointers to the various stacked text widgets in the top window. The for loop from line 48 creates a text widget with scrolling content for each of the commands. This means that the widgets can handle longer output from chatty commands without going haywire or losing content. The user can scroll up the output with the mouse wheel if the content size exceeds the dimensions of the widget.

Stacking Building Blocks

Into each of these text widgets, line 58 first writes the name of the scheduled command in red and then passes the command to runit() to have it executed. It then intercepts the output and feeds it to the text widget. All the widgets end up in the panes slice, each of them appended at its end thanks to the append command (line 62).

Line 69 then uses the ... operator to individually pass the elements in the slice to the container.New() function courtesy of the termdash library. The Run() function in line 80 builds and manages the UI until the user presses Q to terminate the endless event loop. The keyboard handler intercepts this event starting at line 74 and calls the cancel() function of the background context previously defined in line 37, which pulls the rug out from under the terminal UI's processing loop.

But how does the graphical user interface stack the individual widgets on top of each other, while giving each one the same amount of space, no matter how many commands the user defines? A trick is needed here, because as a layout method for vertically stacking two widgets, termdash only supports the SplitHorizontal function. It accepts two widgets and a percentage value that determines how much space the upper widget gets in relation to the lower one.

Figure 2 shows how any number of widgets can be stacked in steps of two: At the top of each partial stack is the conglomerate of all previously stacked widgets, and at the bottom the new widget that the algorithm attaches. The percentage value, which determines the ratio of the upper widget height to the lower one, needs to change dynamically, depending on the number of widgets already in the group, so that in the end all individual widgets really appear to be the same size.

Figure 2: Building up the panel structure in steps of two.

If there is only one widget at the top, it gets exactly 50 percent of the space (orange box), just like the one at the bottom. But if there are already three widgets on top and one is added at the bottom, the widget group on top gets 75 percent of the space and the new widget 25 percent (blue box). Accordingly, the function panesSplit() from Listing 4 takes a slice of text widgets and initializes the resulting group widget rows by adding the first text widget.

01 package main
02
03 import (
04   "github.com/mum4k/termdash/container"
05   "github.com/mum4k/termdash/widgets/text"
06   "github.com/mum4k/termdash/linestyle"
07 )
08
09 func panesSplit(
10   panes []*text.Text) container.Option {
11   var rows container.Option
12
13   if len(panes) > 0 {
14     rows =
15       container.PlaceWidget(panes[0])
16     panes = panes[1:]
17   }
18
19   for idx, pane := range panes {
20     itemsPacked := idx + 2
21
22     rows = container.SplitHorizontal(
23       container.Top(rows),
24       container.Bottom(
25         container.PlaceWidget(pane),
26         container.Border(
27           linestyle.Light),
28       ),
29       container.SplitPercent(
30         100*(itemsPacked-1)/itemsPacked),
31     )
32   }
33
34   return rows
35 }

The for loop then iterates over the remaining widgets to be packed starting in line 19, and in itemsPacked keeps track of how many widgets have already been grouped in the upper part. Each call to SplitHorizontal() in line 22 now receives the widget group at the top (rows) as container.Top() and the newly added widget with a thin border to cordon it off as container.Bottom().

The space distribution is determined by SplitPercent() in line 29 based on the formula 100*(n-1)/n, where n stands for the number of widgets grouped at the top. Where n=2, this means 50 percent, while n=3 gives you 66 percent, and n=4 is 75 percent – just like the doctor ordered.