Exploring openSUSE's automated testing tool for ISO images
Bakers and burners

© Lead Image © Markus Feilner, CC-BY-SA 4.0
Quality Assurance is "like a baker testing his recipe for a cake by trying it once it is out of the oven," says Bernhard Wiedemann, inventor of openQA. OpenSUSE's openQA project is a powerful tool for testing Linux distributions – and even Android images.
Life's too short for manual testing, says the website for openQA [1] (Figure 1), and most software developers will agree. Many developers feel like Charlie Chaplin in Modern Times, working in a factory, repeating the very same steps over and over again.
Most available server software, especially when it is offering APIs, can be tested by specially written clients that perform the very same tasks over and over again, counting success or failure. But what about desktop software, and what about distributions?
Some years ago, the openSUSE project suffered from a severe lack of testing and human testers. Nobody wanted to do the tedious clicking-watching-comparing job. But somebody had to do it, and they did it fairly well, it seems, at least if you look at the success that SUSE and openSUSE have had in recent years. However, the developers saw much room and need for improvement, and, as QA experts often warn, "human QA does not scale."
Even worse, "Distributions got complicated," says Bernhard Wiedemann of SUSE, who invented openQA in 2010. In his talks at conferences, Wiedemann loves to show an urban power pole covered with a huge mess of cables. "Finding the root cause of an error," he explains, "is like trying to find the right cable to your house." So, he started the openQA project.
According to the openSUSE project, "openQA works by booting up an ISO image (which is automatically generated daily from the Open Build Service) in a virtual machine. It installs the image, boots, starts some applications and does a few simple actions. It takes screen shots during that process and then compares the screenshots it just took in the VM with what was expected."
Automate it!
Wiedemann's major aim was to implement "frequent in-depth testing of operating systems and functions, (…) by a tool that works entirely hands-off." He wanted to "enable developers to catch and fix issues faster, roll back to yesterday's versions, and find out who changed what. And, he wanted the whole suite to be integrated into (open)SUSE's toolchain, which is why openQA had some of its roots in SUSE's old-time deployment tool AutoYast.
The shortest way to explain what openQA does is the quote from SUSE people themselves: "openQA is an automated testing system that runs daily tests on the openSUSE Development tree named Factory." Recently, the rolling release Tumbleweed took over Factory, so openQA has focused on testing the Tumbleweed release (Figure 1).
What For?
Today, openQA is helping the openSUSE community for a variety of purposes. Developers used openQA to determine if a special openSUSE Factory/Tumbleweed build works, to give Factory users an idea about the current quality, to find serious bugs as early as possible, and to track the source of bugs.
The software does fully automated installation and upgrade testing, daily regression testing, application and network testing, and benchmarking for new ISO images, providing a strikingly clear overview of the test results, along with screenshots and videos.
These image collections are used by other teams as well: The documentation team and speakers use these up-to-date images, and sometimes the marketing team uses videos to help promote new releases. If you ever need a screenshot of a Linux system booting, just go to the openQA website [1] and pick one.
How Does It Work?
Digging into the details, you'll find several Unix/Linux dogmas applied at their best, like KISS (Keep it simple…): If a new ISO image pops up in a directory, openQA will simply fire up a KVM/Qemu virtual machine with this ISO as boot media (and some specific environment variables).
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Direct Download
Read full article as PDF:
Price $2.95
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Find SysAdmin Jobs
News
-
MNT Seeks Financial Backing for New Seven-Inch Linux Laptop
MNT Pocket Reform is a tiny laptop that is modular, upgradable, recyclable, reusable, and ships with Debian Linux.
-
Ubuntu Flatpak Remix Adds Flatpak Support Preinstalled
If you're looking for a version of Ubuntu that includes Flatpak support out of the box, there's one clear option.
-
Gnome 44 Release Candidate Now Available
The Gnome 44 release candidate has officially arrived and adds a few changes into the mix.
-
Flathub Vying to Become the Standard Linux App Store
If the Flathub team has any say in the matter, their product will become the default tool for installing Linux apps in 2023.
-
Debian 12 to Ship with KDE Plasma 5.27
The Debian development team has shifted to the latest version of KDE for their testing branch.
-
Planet Computers Launches ARM-based Linux Desktop PCs
The firm that originally released a line of mobile keyboards has taken a different direction and has developed a new line of out-of-the-box mini Linux desktop computers.
-
Ubuntu No Longer Shipping with Flatpak
In a move that probably won’t come as a shock to many, Ubuntu and all of its official spins will no longer ship with Flatpak installed.
-
openSUSE Leap 15.5 Beta Now Available
The final version of the Leap 15 series of openSUSE is available for beta testing and offers only new software versions.
-
Linux Kernel 6.2 Released with New Hardware Support
Find out what's new in the most recent release from Linus Torvalds and the Linux kernel team.
-
Kubuntu Focus Team Releases New Mini Desktop
The team behind Kubuntu Focus has released a new NX GEN 2 mini desktop PC powered by Linux.