On Fire

perf can also even create pretty graphics. Thanks to flame graphs, admins and developers can determine the most frequently used software paths. The developer tools integrated into browsers can also generate graphics (Figure 3).

Figure 3: Browser-based developer tools also offer performance analysis features.

The x-axis shows the load of the individual processes; the y-axis the resolution of each function call. The wider the blocks, the more resources their processes require, which helps you see which areas are problematic are most in need of optimizing.

Admins can also generate flame graphs from perf data. You do not have to write the script yourself – look for the script on GitHub [5]:

git clone https://github.com/brendangregg/FlameGraph

The commands from Listing 5 record the performance data and create a flame graph with perf script (Figure 4). The flame graph shows that, although the browser is playing a 720p video, the kernel is pretty much in sleep mode. In the browser, individual sections of the flame graph can be zoomed in and out.

01 sudo perf record -F 99 -a -g -- sleep 10
02 sudo perf script > out.perf
03 ./stackcollapse-perf.pl out.perf > out.folded
04 ./flamegraph.pl out.folded > out.svg
05 firefox out.svg

Figure 4: A script-generated flame graph of the perf data. If desired, you can zoom in or out at the push of a button.

Dynamic Tracing

Using the perf probe subcommand and the --add switch, admins can define dynamic trace points. Listing 6 shows the procedure using the Linux tcp_sendmsg() function as an example.

01 $ sudo perf probe --addd='tcp_sendmsg'
02 Added new event:
03 probe:tcp_sendmsg (on tcp_sendmsg)
04 [...]
05 $ sudo perf record -e probe:tcp_sendmsg -aR sleep 5
06 [ perf record: Woken up 1 times to write data ]
07 $ sudo perf report --stdio
08 [...]

Dynamic Duo

The Linux perf subsystem and the perf tool collection offer many possibilities for analyzing a system. Admins can easily collect and evaluate data with existing tools and scripts to achieve an overview. Correct interpretation of the data requires some knowledge of the perf subsystem. Because the Linux kernel delivers perf directly, you won't need to worry about installing external modules.