Measuring performance with the perf kernel tool

On Fire

perf can also even create pretty graphics. Thanks to flame graphs, admins and developers can determine the most frequently used software paths. The developer tools integrated into browsers can also generate graphics (Figure 3).

Figure 3: Browser-based developer tools also offer performance analysis features.

The x-axis shows the load of the individual processes; the y-axis the resolution of each function call. The wider the blocks, the more resources their processes require, which helps you see which areas are problematic are most in need of optimizing.

Admins can also generate flame graphs from perf data. You do not have to write the script yourself – look for the script on GitHub [5]:

git clone

The commands from Listing 5 record the performance data and create a flame graph with perf script (Figure 4). The flame graph shows that, although the browser is playing a 720p video, the kernel is pretty much in sleep mode. In the browser, individual sections of the flame graph can be zoomed in and out.

Listing 5

Commands for Creating a Flame Graph

01 sudo perf record -F 99 -a -g -- sleep 10
02 sudo perf script > out.perf
03 ./ out.perf > out.folded
04 ./ out.folded > out.svg
05 firefox out.svg
Figure 4: A script-generated flame graph of the perf data. If desired, you can zoom in or out at the push of a button.

Dynamic Tracing

Using the perf probe subcommand and the --add switch, admins can define dynamic trace points. Listing 6 shows the procedure using the Linux tcp_sendmsg() function as an example.

Listing 6

tcp_sendmsg as Dynamic Trace Point

01 $ sudo perf probe --addd='tcp_sendmsg'
02 Added new event:
03 probe:tcp_sendmsg (on tcp_sendmsg)
04 [...]
05 $ sudo perf record -e probe:tcp_sendmsg -aR sleep 5
06 [ perf record: Woken up 1 times to write data ]
07 $ sudo perf report --stdio
08 [...]

Dynamic Duo

The Linux perf subsystem and the perf tool collection offer many possibilities for analyzing a system. Admins can easily collect and evaluate data with existing tools and scripts to achieve an overview. Correct interpretation of the data requires some knowledge of the perf subsystem. Because the Linux kernel delivers perf directly, you won't need to worry about installing external modules.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Core Technologies – Profiling

    We all want our programs to run fast. With profiling, you can locate the code that is slowing you down and fix it.

  • Open Heart Surgery

    Who is constantly creating the new processes that are paralyzing the system? Which process opens the most files and how many bytes is it reading or writing? Mike Schilli pokes inside the kernel to answer these questions with bpftrace and its code probes.

  • Kernel News

    Chronicler Zack Brown reports on the latest news, views, dilemmas, and developments within the Linux kernel community.

  • PowerTOP 2.0

    The PowerTOP tool developed by Intel helps you save power. In version 2.0, it can also handle GPUs and offers advanced tracking features.

  • Kernel News

    Chronicler Zack Brown reports on the latest news, views, dilemmas, and developments within the Linux kernel community.

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More