Using reader polls

Off the Beat: Bruce Byfield's Blog

Whenever I mention a community poll, someone is sure to question the decision. Whether I reference Linux Journal's, LinuxQuestion's, Linux New Media's, or the Distrowatch page stats, someone will point out that these sources are not valid, and insist that I shouldn't use them. Such criticisms make some valid points, but I still think they are too quick to dismiss the polls. So long as the polls are used with some common sense, I see nothing much wrong in viewing them as general indicators of trends.

I realize, of course, that these polls are nowhere close to being scientific. They are by no means random samples, since anyone who cares to can participate. Almost certainly, they have a high margin of error, although exactly how high is impossible to calculate, since only LinuxQuestions even goes so far as to list the number of participants. At best, they represent the opinions of specific communities.

The only criticism that I think unlikely is that people regularly answer multiple times to push their favorites to the tops of each voting category. Frankly, I doubt that most people care enough to bother stuffing the ballot box.

However, to my mind, these shortcomings are more reasons for caution, than for outright rejection of the results. For instance, I always assume that the opinions expressed are those of relatively experienced users, on the grounds that few newcomers are likely to even see the polls, much less answer them. I certainly wouldn't use them to claim that the popularity of other distributions mean that Ubuntu is fading in popularity, because the opinions of all kinds of users are completely unrepresented

Similarly, when I see results that are wildly out of sync with my everyday observations, such as LinuxQuestion reporting that Slackware is almost as popular as Ubuntu on the desktop, or LinuxMedia reported that close to half of desktop users rely on KDE, I assume I found a peculiarity of the communities represented in the pool, not of free software users in general. KDE's alleged popularity, for example, I would attribute to the face that KDE has always been highly popular in Germany, which is one of the target audiences for Linux New Media.

I also round off the results when I talk about them; to report the results to two decimal places, as some of the polls do, only creates a false sense of precision.

Nor, when I see Linux Journal reporting  that Rackspace was the web host of choice for 11.3% and Linode for 10.7%, would I suggest that Rackspace is more popular than Linode. Instead, remembering the unspecified margin of error, I would say instead that the two are more or less equally popular.

For the same reason, I would not try to rank the popularity of Enlightenment, LXDE, PCLinuxOS, Fluxbox, and DWN, each of which Linux Journal reported (http://www.linuxjournal.com/rc2013?page=8) as having less than 2% of the vote. At the most, I might mention that a variety of minority choices were recorded, and list them as I do here.

By contrast, when KDE receives 36% of the votes for Desktop Environment on LinuxQuestions and GNOME Shell 10%, I would have limited confidence in reporting that KDE was more popular in the community of respondents. Not only is the difference large enough to indicate relative popularity, but the high showings of KDE apps in other voting categories helps to prove KDE's position.

My confidence would be stronger if I found other polls reporting similar rankings (although not necessarily similar figures), or previous years of the same poll reporting similar results. That is why, although I do not take the exact figures very seriously, I conclude that KDE is the most popular desktop environment: all this years' polls place it on top, and LinuxQuestion has placed it there three years in a row. Looking at previous years would also lead me to suggest that GNOME has yet to recover from the user reaction against its current release series, despite the fact that many of the former complaints have been addressed for over eighteen months.

Yet even in these limited areas, I would prefer to use qualified language, using "seems" rather than the more forceful "is." It takes returns like LibreOffice's 74% on the Linux Journal awards to Apache OpenOffice's 7% for me to stick my neck out and say that LibreOffice remains much more popular than OpenOffice -- and, at that point, I am stating what is obvious without the benefit of a poll.

Given a choice, I would naturally rather base conclusions on better-designed polls from which I could extract accurate information. However, such tools do not exist, or are only available every five years or so when they are available at all. In the absence of anything better, I am willing to take the reader polls as approximations from which only the roughest of data can be extracted.