GNOME Gets Formal, Public Usability Testing

Off the Beat: Bruce Byfield's Blog

By definition, usability testing is difficult in free software. The reason is obvious: usability testing typically requires face to face observation of users, which is hard to arrange when most developers are interacting remotely. That's why Aakanksha Gaur's recent blogs about GNOME 3 usability caught my attention -- to the best of my knowledge, the last major usability study of GNOME took place twelve years ago, although small, informal studies have been done since.

Gaur is graduate student studying interface design at the National Institute of Design in Bangalore, India. She is also currently an intern in the Outreach Program for Women, mentoring under Allan Day. With Day's assistance, Gaur has decided to focus her internship on answering two questions: "What are the usability issues encountered by new and existing users of GNOME 3?" and "How is GNOME 3 perceived by new and existing users?"

Cynics might remark that these questions should have been asked several releases ago (in fact, in the comments on her blog, they already have). Yet in all fairness, GNOME's relative neglect of usability testing is mirrored in most free software projects, and is less extreme than many.

Day describes the previous usability test for GNOME 3 as "ad hoc." For instance, he himself did some testing when GNOME 3.0 was released "with friends and family," and recently conducted a "small study" on the lock screen design that "led to a number of design changes." Other designers, he adds, have done similar work. Usually, though, like most projects, GNOME mostly depends on developer's reactions, bug reports, and user comments to improve usability.

By contrast, what Gaur is doing rarely occurs in GNOME or most other projects. As Day comments,  "While small-scale testing has been done on GNOME 3, this will be one of the first opportunities we have had to do an extended research study."

The importance of publishing

An intermittent GNOME user herself, Gaur characterizes free software as "a developer-driven world for very valid reasons," but one with "a huge potential to include usability and user-centric design." However, she sees some progress in GNOME's use of "design wikis" and the fact that volunteers "are asked to follow specific design exercises." She also notes that both Mozilla and Wikimedia Foundation have shown interest in design recently.

Gaur hopes that her study will assist some of these trends -- not just because it is being done, but because she is making her results public so that others can benefit from them. "I have continuous discussions with [Day] about pouring my data back into the community and making my research actionable," she emailed me recently. "This will happen in the form of bug reports, infographics, and blog posts. Reports of my project, including transcripts of user interviews, will also be made available for anyone who is interested." 

These comments bring up another point: despite free software's tradition of openness, the little usability testing that has been done at various projects has generally not been publicized or published -- an omission that has made the development of a user-centric development model in free software much harder. If carried out, Gaur's publication of her data means that other projects, not just GNOME, can potentially benefit from her research.

The early stages 

So far, Gaur has reported her study in some detail. She has studied a number of applications, including some on the Mac, focusing particularly on To Do list utilities. Her latest blog entries explain the advantages of taking on the task, as well as describing her research into the metaphor of the desktop  and written critiques of GNOME 3 that describe users' experiences.

Gaur has also detailed her failures. Talking about an attempt to study how people used software by asking them constant questions as they work, she writes self-mockingly in one blog, "I was under the confident assumption that I shall take long interviews of users and magically they will reveal the design mistakes which we shall fix and hence, rule the world." 

Instead, what her example shows is that the feedback she received "turned out to be very vague and very unfocused. I ended up putting words in the mouth of the interviewee and eventually the whole exercise did not yield good insights. By insights, I mean, data that challenged my existing beliefs about the system in any way." 

By contrast, in the one piece of research she has described so far she has chosen to assign three users of varying ability a simple task in GNOME Character Map, and to observe how they go about it. From these observations, she produced a list of goals for the redesign, as well as a list of use cases to take into account, and did some sketches for a redesign.

Gaur is currently deciding on the tasks and applications for the similar tests she plans over the next few months, and plans to continue blogging about them. 

Beyond that, I suppose, lies the task of synthesizing her findings to answer her two general questions -- something that, considering some of the recent animosity to GNOME, is almost sure to be greeted by discussion and controversy. 

Yet I suspect that her individual studies and openness may prove at least as important as her conclusions in the long run. If Gaur continues as she started, she will provide a professional example of how to conduct usability testing, and that is something that free software developers could use every bit as much as GNOME could use her conclusions.