Rigorous Review

The newest Free and Open Source exploit is called Shellshock, and I hope it will finally illustrate that GNU/Linux is not free of bugs, any more (or less) than closed source code is free of bugs.

However, I've seen several articles in which columnists continue to write as if every piece of open source code has many eyes looking at it (which is not always true) or as if every piece of closed source code goes though rigorous design review and testing by legions of engineers (equally untrue). The answer, as always, lies somewhere in the middle.

Yes, some companies have good design and code reviews that proceed to a field test or beta release that allows end users to test and submit bug reports before the product is unleashed on the general public. Even in the largest companies, however, some projects receive little engineering consideration. The code is written once by engineering, then patched and expanded over time according to bug fix requests and feature requests from end users. Over time, fewer engineers work on each part, with fewer "eyes" on the code.

When I worked at Digital Equipment Corporation (DEC), we were in the process of going to field test with a brand new operating system and a brand new piece of hardware (including a new tape drive called the TK50), when I found out that the field-test software had never actually been tested on the new hardware. I raised the flag in the field test meeting, but people did not want to hold up the start of testing because it would look bad to upper management. Finally, the Senior Vice President of that workstation line called me and asked what was going on. I told him that the rush to bring the product to market was making people take shortcuts that I did not approve of, including the fact that the field test documentation was incomplete and not reviewed. He listened carefully and said he would attend the next meeting and that  – no matter what was said in the meeting  – we would not be going to field test that week.

The VP came to the meeting and listened while everyone gave their reports, then he asked each person, "Are we ready to go to field test?" Each speaker said "yes." Finally, he came to me. I gave my report and said that I was not happy that each report had some caveats, but I could not second-guess the rest of engineering and I would go along with what other people said.

The VP then stood up and said, "I have to get back to headquarters, but I want to tell you that we are not going to field test tomorrow. You have all been doing a good job, but I feel it is much too rushed, and I want you to take another week."

Some of the engineering managers started to object, but the VP said, "There is no use arguing this, it is my decision. Ken Olsen (at that time the President of DEC) and I both feel you have made your commitment to schedule, but we are going to give it a week to mature." Immediately after that, the documentation manager left the room to tell his people to continue developing the documentation.

That meeting was on a Friday. On Monday, I went to the manufacturing site where the 15 field test units were waiting to be shipped. I tried to install the field-test software on the hardware, planning to wipe the hard disk clean after testing. Thirteen of the 15 systems would not even install, and the two that did install had other hardware problems.

I called over the QA manager of the field test and asked how these systems had passed final diagnostic testing. "They did not pass," he said, "but the main engineering group said the diagnostics were wrong and just to ignore the results of the diagnostics."

Fuming, I went back to central engineering and asked what was going on. "Oh no!" declared the diagnostic engineer, "ONE of the diagnostics was bad, and I told them to ignore the result of that ONE diagnostic. The other 368 diagnostics were fine, and he should have been flagging any of those that failed." The engineering team then did an evaluation of why every single one of the field test systems had failed manufacturing and fixed them before they left DEC.

If we had gone to field test according to schedule, we would have lost time and money fixing these issues out in the field, as well as credibility in the eyes of the customer.