Exploring openSUSE's automated testing tool for ISO images
Bakers and burners
Quality Assurance is "like a baker testing his recipe for a cake by trying it once it is out of the oven," says Bernhard Wiedemann, inventor of openQA. OpenSUSE's openQA project is a powerful tool for testing Linux distributions – and even Android images.
Life's too short for manual testing, says the website for openQA [1] (Figure 1), and most software developers will agree. Many developers feel like Charlie Chaplin in Modern Times, working in a factory, repeating the very same steps over and over again.
Most available server software, especially when it is offering APIs, can be tested by specially written clients that perform the very same tasks over and over again, counting success or failure. But what about desktop software, and what about distributions?
Some years ago, the openSUSE project suffered from a severe lack of testing and human testers. Nobody wanted to do the tedious clicking-watching-comparing job. But somebody had to do it, and they did it fairly well, it seems, at least if you look at the success that SUSE and openSUSE have had in recent years. However, the developers saw much room and need for improvement, and, as QA experts often warn, "human QA does not scale."
Even worse, "Distributions got complicated," says Bernhard Wiedemann of SUSE, who invented openQA in 2010. In his talks at conferences, Wiedemann loves to show an urban power pole covered with a huge mess of cables. "Finding the root cause of an error," he explains, "is like trying to find the right cable to your house." So, he started the openQA project.
According to the openSUSE project, "openQA works by booting up an ISO image (which is automatically generated daily from the Open Build Service) in a virtual machine. It installs the image, boots, starts some applications and does a few simple actions. It takes screen shots during that process and then compares the screenshots it just took in the VM with what was expected."
Automate it!
Wiedemann's major aim was to implement "frequent in-depth testing of operating systems and functions, (…) by a tool that works entirely hands-off." He wanted to "enable developers to catch and fix issues faster, roll back to yesterday's versions, and find out who changed what. And, he wanted the whole suite to be integrated into (open)SUSE's toolchain, which is why openQA had some of its roots in SUSE's old-time deployment tool AutoYast.
The shortest way to explain what openQA does is the quote from SUSE people themselves: "openQA is an automated testing system that runs daily tests on the openSUSE Development tree named Factory." Recently, the rolling release Tumbleweed took over Factory, so openQA has focused on testing the Tumbleweed release (Figure 1).
What For?
Today, openQA is helping the openSUSE community for a variety of purposes. Developers used openQA to determine if a special openSUSE Factory/Tumbleweed build works, to give Factory users an idea about the current quality, to find serious bugs as early as possible, and to track the source of bugs.
The software does fully automated installation and upgrade testing, daily regression testing, application and network testing, and benchmarking for new ISO images, providing a strikingly clear overview of the test results, along with screenshots and videos.
These image collections are used by other teams as well: The documentation team and speakers use these up-to-date images, and sometimes the marketing team uses videos to help promote new releases. If you ever need a screenshot of a Linux system booting, just go to the openQA website [1] and pick one.
How Does It Work?
Digging into the details, you'll find several Unix/Linux dogmas applied at their best, like KISS (Keep it simple…): If a new ISO image pops up in a directory, openQA will simply fire up a KVM/Qemu virtual machine with this ISO as boot media (and some specific environment variables).
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.
-
New Pentesting Distribution to Compete with Kali Linux
SnoopGod is now available for your testing needs
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.
-
ZorinOS 17.1 Released, Includes Improved Windows App Support
If you need or desire to run Windows applications on Linux, there's one distribution intent on making that easier for you and its new release further improves that feature.
-
Linux Market Share Surpasses 4% for the First Time
Look out Windows and macOS, Linux is on the rise and has even topped ChromeOS to become the fourth most widely used OS around the globe.
-
KDE’s Plasma 6 Officially Available
KDE’s Plasma 6.0 "Megarelease" has happened, and it's brimming with new features, polish, and performance.
-
Latest Version of Tails Unleashed
Tails 6.0 is based on Debian 12 and includes GNOME 43.
-
KDE Announces New Slimbook V with Plenty of Power and KDE’s Plasma 6
If you're a fan of KDE Plasma, you'll be thrilled to hear they've announced a new Slimbook with an AMD CPU and the latest version of KDE Plasma desktop.
-
Monthly Sponsorship Includes Early Access to elementary OS 8
If you want to get a glimpse of what's in the pipeline for elementary OS 8, just set up a monthly sponsorship to help fund its continued existence.