Bakers and burners

Life's too short for manual testing, says the website for openQA [1] (Figure 1), and most software developers will agree. Many developers feel like Charlie Chaplin in Modern Times, working in a factory, repeating the very same steps over and over again.

Figure 1: The website for openQA gives precise data on current tests.

Most available server software, especially when it is offering APIs, can be tested by specially written clients that perform the very same tasks over and over again, counting success or failure. But what about desktop software, and what about distributions?

Some years ago, the openSUSE project suffered from a severe lack of testing and human testers. Nobody wanted to do the tedious clicking-watching-comparing job. But somebody had to do it, and they did it fairly well, it seems, at least if you look at the success that SUSE and openSUSE have had in recent years. However, the developers saw much room and need for improvement, and, as QA experts often warn, "human QA does not scale."

Even worse, "Distributions got complicated," says Bernhard Wiedemann of SUSE, who invented openQA in 2010. In his talks at conferences, Wiedemann loves to show an urban power pole covered with a huge mess of cables. "Finding the root cause of an error," he explains, "is like trying to find the right cable to your house." So, he started the openQA project.

According to the openSUSE project, "openQA works by booting up an ISO image (which is automatically generated daily from the Open Build Service) in a virtual machine. It installs the image, boots, starts some applications and does a few simple actions. It takes screen shots during that process and then compares the screenshots it just took in the VM with what was expected."

Automate it!

Wiedemann's major aim was to implement "frequent in-depth testing of operating systems and functions, (…) by a tool that works entirely hands-off." He wanted to "enable developers to catch and fix issues faster, roll back to yesterday's versions, and find out who changed what. And, he wanted the whole suite to be integrated into (open)SUSE's toolchain, which is why openQA had some of its roots in SUSE's old-time deployment tool AutoYast.

The shortest way to explain what openQA does is the quote from SUSE people themselves: "openQA is an automated testing system that runs daily tests on the openSUSE Development tree named Factory." Recently, the rolling release Tumbleweed took over Factory, so openQA has focused on testing the Tumbleweed release (Figure 1).

What For?

Today, openQA is helping the openSUSE community for a variety of purposes. Developers used openQA to determine if a special openSUSE Factory/Tumbleweed build works, to give Factory users an idea about the current quality, to find serious bugs as early as possible, and to track the source of bugs.

The software does fully automated installation and upgrade testing, daily regression testing, application and network testing, and benchmarking for new ISO images, providing a strikingly clear overview of the test results, along with screenshots and videos.

These image collections are used by other teams as well: The documentation team and speakers use these up-to-date images, and sometimes the marketing team uses videos to help promote new releases. If you ever need a screenshot of a Linux system booting, just go to the openQA website [1] and pick one.

How Does It Work?

Digging into the details, you'll find several Unix/Linux dogmas applied at their best, like KISS (Keep it simple…): If a new ISO image pops up in a directory, openQA will simply fire up a KVM/Qemu virtual machine with this ISO as boot media (and some specific environment variables).