EUPA/HCI2002: Five Users Theory causes Conflict at Conference

Following the publication of our paper at IHM-HCI 2001, "Why and When Five Test Users aren't Enough", I was most interested in an EUPA presentation at this year's British HCI Group Annual Conference by Carol M. Barnum entitled "The Magic Number 5: Is it enough for Web Testing?" More so because it was apparent that our work in this area appeared to be ignored in the presentation (prepared before Carol found our papers on discount methods!).

The aim of the presentation seemed to be to reassure the discount method faithful that despite a wave of evidence that five users may be nowhere near enough, that it was OK to keep believing in the Magic Number. During the presentation I could not help but be mildly amused by what I felt was a bizarre similarity to the philosophy of the fabled Simpson's Stonecutters Society. For non-Simpsons fans (are there any?), this ancient society 'has, since ancient times, split the rocks of ignorance that obscure the light of knowledge and truth'. The real purpose of the society, however, is revealed by a line in their drinking song: 'the obscuring of the truth that would benefit the majority for the good of the few'. While Carol was clearly no such Stonecutter, she got dangerously close to supporting their hidden agenda.

The defence of the magical number 5 was based on tightening parameters to such a degree (similar type users, small part of poorly designed system etc) that testing beyond 5 users would be totally cost ineffective. Such shifting of the goal posts is a fine way to obscure the truth. As a result, Carol could safely expose many reasons why 5 users would not be enough in her presentation, such as a wide scope for testing unconstrained usage, user differences, and (wait for it) the impact of good user centred design. However, these were not reasons for abandoning the magic number 5, but instead the reason why studies such as Spool and Schroeder (CHI 2001 short paper) can be safely ignored by faithful Usability Stonecutters!

As we have argued in our papers (see Interactions Sept+Oct 2002) user testing is not just about finding problems, they must be understood and prioritised, and neither is possible without good frequency and severity data. These are both critical attributes when prioritising solution recommendation and implementation. The frequency of a problem is dictated by the number of users it affects. Similarly, the severity of a problem can only be ascertained by analysis of the difficulties encountered by individual users with individual problem elements. By a 'mix 'n' match' of the test users in our study we found hugely significant differences in problem frequency and severity between various groups of 5 users.

The danger (as with all discount methods) is that potential clients will fail to understand the limitations of the 'quick fix' usability solution of running a handful of user tests. User testing is just a part of the usability process and we must encourage clients to invest in the whole process and not mislead them into believing a handful of test users will provide the usability solution. Carol Barnum did stress that Usability Stonecutters who keep the faith with only five users should make clear the limitations of discount testing to clients. However, she did not spell out these limitations and it is unlikely that anyone who still believes in the magic number would be able to deliver such candour.

It's time to move the Stonecutters out of usability. The issue is not whether five users are enough. It's an issue of the costs, benefits and risks of running some number of users with known relevant characteristics. Three may be enough. Twenty may not be enough. What we need are well-researched models that let us carry out credible risk and cost-benefit analyses of user testing plans. We do not have these yet and Carol's presentation missed a valuable opportunity to lay the foundations for such a model.

Alan Woolrych
University of Sunderland

Study: Mobile Web sites need Improvement
Source: CNET, 13 October 2008
 
A new study suggests that the usability of many mobile Web sites still needs improvement.

Better Usability makes Online Backup more Appealing
Source: UN, 11 October 2008
 
Repeat after me: "Backup via the Internet is for everybody."

IBM Task Modeler version 6 now Available
Source: Unknown, 10 October 2008
 
The latest release of IBM Task Modeler is now available for download. This includes many new features to support efficient and collaborative task analysis and synthesis.

Caroline's Corner: Design to Read - Designing for People who do not Read easily
Source: Caroline Jarrett, 9 October 2008
 
Reading is a skill many of us take for granted. Chances are that if you’re reading this, you’re one of the lucky ones who read easily.

Bill Buxton to change Microsoft from within
Source: The Register, 8 October 2008
 
"Last year Steve Ballmer said to 85,000 employees 'If you don’t change and you don’t go in this direction, we’re dead, and I don’t want to die.' I wanted to go hug him when he said that."

New Report: Innovative Marketing in the Digital World
Source: E-consultancy, 7 October 2008
 
A new report from E-consultancy provides a distillation of ideas and suggestions provided by digital marketing experts.

Linking Industry with Academia
Source: unknown, 6 October 2008
 
Customer experience research consultancy, fhios, based in London, has just launched an innovative internship programme, aimed primarily at postgraduate students.

Online Travel Booking: What influences Consumers?
Source: Webcredible, 4 October 2008
 
What influences consumers when booking a holiday? How can travel companies offer the best user experience?

Getting the Word out
Source: UN, 3 October 2008
 
I SAID, I'M ON THE TRAIN... Voice recognition technology can now translate all those pesky garbled voicemails into manageable text messages.

Mobile hardware outpaces Software and User Capabilities
Source: ComputerWorld, 2 October 2008
 
The main issue dissuading users from employing all the capabilities of their mobile devices is the complexity involved in operating them.