Don't rush the testing process
Rigorous Review

"maddog" recounts a tale illustrating the importance of thorough product testing.
The newest Free and Open Source exploit is called Shellshock, and I hope it will finally illustrate that GNU/Linux is not free of bugs, any more (or less) than closed source code is free of bugs.
However, I've seen several articles in which columnists continue to write as if every piece of open source code has many eyes looking at it (which is not always true) or as if every piece of closed source code goes though rigorous design review and testing by legions of engineers (equally untrue). The answer, as always, lies somewhere in the middle.
Yes, some companies have good design and code reviews that proceed to a field test or beta release that allows end users to test and submit bug reports before the product is unleashed on the general public. Even in the largest companies, however, some projects receive little engineering consideration. The code is written once by engineering, then patched and expanded over time according to bug fix requests and feature requests from end users. Over time, fewer engineers work on each part, with fewer "eyes" on the code.
When I worked at Digital Equipment Corporation (DEC), we were in the process of going to field test with a brand new operating system and a brand new piece of hardware (including a new tape drive called the TK50), when I found out that the field-test software had never actually been tested on the new hardware. I raised the flag in the field test meeting, but people did not want to hold up the start of testing because it would look bad to upper management. Finally, the Senior Vice President of that workstation line called me and asked what was going on. I told him that the rush to bring the product to market was making people take shortcuts that I did not approve of, including the fact that the field test documentation was incomplete and not reviewed. He listened carefully and said he would attend the next meeting and that – no matter what was said in the meeting – we would not be going to field test that week.
The VP came to the meeting and listened while everyone gave their reports, then he asked each person, "Are we ready to go to field test?" Each speaker said "yes." Finally, he came to me. I gave my report and said that I was not happy that each report had some caveats, but I could not second-guess the rest of engineering and I would go along with what other people said.
The VP then stood up and said, "I have to get back to headquarters, but I want to tell you that we are not going to field test tomorrow. You have all been doing a good job, but I feel it is much too rushed, and I want you to take another week."
Some of the engineering managers started to object, but the VP said, "There is no use arguing this, it is my decision. Ken Olsen (at that time the President of DEC) and I both feel you have made your commitment to schedule, but we are going to give it a week to mature." Immediately after that, the documentation manager left the room to tell his people to continue developing the documentation.
That meeting was on a Friday. On Monday, I went to the manufacturing site where the 15 field test units were waiting to be shipped. I tried to install the field-test software on the hardware, planning to wipe the hard disk clean after testing. Thirteen of the 15 systems would not even install, and the two that did install had other hardware problems.
I called over the QA manager of the field test and asked how these systems had passed final diagnostic testing. "They did not pass," he said, "but the main engineering group said the diagnostics were wrong and just to ignore the results of the diagnostics."
Fuming, I went back to central engineering and asked what was going on. "Oh no!" declared the diagnostic engineer, "ONE of the diagnostics was bad, and I told them to ignore the result of that ONE diagnostic. The other 368 diagnostics were fine, and he should have been flagging any of those that failed." The engineering team then did an evaluation of why every single one of the field test systems had failed manufacturing and fixed them before they left DEC.
If we had gone to field test according to schedule, we would have lost time and money fixing these issues out in the field, as well as credibility in the eyes of the customer.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Direct Download
Read full article as PDF:
Price $2.95
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Find SysAdmin Jobs
News
-
OpenMandriva Lx 23.03 Rolling Release is Now Available
OpenMandriva "ROME" is the latest point update for the rolling release Linux distribution and offers the latest updates for a number of important applications and tools.
-
CarbonOS: A New Linux Distro with a Focus on User Experience
CarbonOS is a brand new, built-from-scratch Linux distribution that uses the Gnome desktop and has a special feature that makes it appealing to all types of users.
-
Kubuntu Focus Announces XE Gen 2 Linux Laptop
Another Kubuntu-based laptop has arrived to be your next ultra-portable powerhouse with a Linux heart.
-
MNT Seeks Financial Backing for New Seven-Inch Linux Laptop
MNT Pocket Reform is a tiny laptop that is modular, upgradable, recyclable, reusable, and ships with Debian Linux.
-
Ubuntu Flatpak Remix Adds Flatpak Support Preinstalled
If you're looking for a version of Ubuntu that includes Flatpak support out of the box, there's one clear option.
-
Gnome 44 Release Candidate Now Available
The Gnome 44 release candidate has officially arrived and adds a few changes into the mix.
-
Flathub Vying to Become the Standard Linux App Store
If the Flathub team has any say in the matter, their product will become the default tool for installing Linux apps in 2023.
-
Debian 12 to Ship with KDE Plasma 5.27
The Debian development team has shifted to the latest version of KDE for their testing branch.
-
Planet Computers Launches ARM-based Linux Desktop PCs
The firm that originally released a line of mobile keyboards has taken a different direction and has developed a new line of out-of-the-box mini Linux desktop computers.
-
Ubuntu No Longer Shipping with Flatpak
In a move that probably won’t come as a shock to many, Ubuntu and all of its official spins will no longer ship with Flatpak installed.