EUPA/HCI2002: Five Users Theory causes Conflict at Conference

Following the publication of our paper at IHM-HCI 2001, "Why and When Five Test Users aren't Enough", I was most interested in an EUPA presentation at this year's British HCI Group Annual Conference by Carol M. Barnum entitled "The Magic Number 5: Is it enough for Web Testing?" More so because it was apparent that our work in this area appeared to be ignored in the presentation (prepared before Carol found our papers on discount methods!).

The aim of the presentation seemed to be to reassure the discount method faithful that despite a wave of evidence that five users may be nowhere near enough, that it was OK to keep believing in the Magic Number. During the presentation I could not help but be mildly amused by what I felt was a bizarre similarity to the philosophy of the fabled Simpson's Stonecutters Society. For non-Simpsons fans (are there any?), this ancient society 'has, since ancient times, split the rocks of ignorance that obscure the light of knowledge and truth'. The real purpose of the society, however, is revealed by a line in their drinking song: 'the obscuring of the truth that would benefit the majority for the good of the few'. While Carol was clearly no such Stonecutter, she got dangerously close to supporting their hidden agenda.

The defence of the magical number 5 was based on tightening parameters to such a degree (similar type users, small part of poorly designed system etc) that testing beyond 5 users would be totally cost ineffective. Such shifting of the goal posts is a fine way to obscure the truth. As a result, Carol could safely expose many reasons why 5 users would not be enough in her presentation, such as a wide scope for testing unconstrained usage, user differences, and (wait for it) the impact of good user centred design. However, these were not reasons for abandoning the magic number 5, but instead the reason why studies such as Spool and Schroeder (CHI 2001 short paper) can be safely ignored by faithful Usability Stonecutters!

As we have argued in our papers (see Interactions Sept+Oct 2002) user testing is not just about finding problems, they must be understood and prioritised, and neither is possible without good frequency and severity data. These are both critical attributes when prioritising solution recommendation and implementation. The frequency of a problem is dictated by the number of users it affects. Similarly, the severity of a problem can only be ascertained by analysis of the difficulties encountered by individual users with individual problem elements. By a 'mix 'n' match' of the test users in our study we found hugely significant differences in problem frequency and severity between various groups of 5 users.

The danger (as with all discount methods) is that potential clients will fail to understand the limitations of the 'quick fix' usability solution of running a handful of user tests. User testing is just a part of the usability process and we must encourage clients to invest in the whole process and not mislead them into believing a handful of test users will provide the usability solution. Carol Barnum did stress that Usability Stonecutters who keep the faith with only five users should make clear the limitations of discount testing to clients. However, she did not spell out these limitations and it is unlikely that anyone who still believes in the magic number would be able to deliver such candour.

It's time to move the Stonecutters out of usability. The issue is not whether five users are enough. It's an issue of the costs, benefits and risks of running some number of users with known relevant characteristics. Three may be enough. Twenty may not be enough. What we need are well-researched models that let us carry out credible risk and cost-benefit analyses of user testing plans. We do not have these yet and Carol's presentation missed a valuable opportunity to lay the foundations for such a model.

Alan Woolrych
University of Sunderland

All change at the top for System Concepts
Source: System Concepts Ltd, 3 July 2009
 
Leslie Fountain has been promoted to joint Managing Director of leading usability consultancy System Concepts.

Life in UCD immortalised in fiction: you couldn't make it up
Source: UN, 2 July 2009
 
Sarah Herman's fictitious book on life in a user-centred design company has hit the shelves and The Guardian's book pages...

Interfaces Magazine - Issue 79: The Education Issue
Source: Interaction Group, 1 July 2009
 
The latest issue of Interfaces is now available as a free download from the Interaction Website.

Two new Behavioural research Tools from Noldus
Source: UN, 30 June 2009
 
Tool updates make on-site behavioural data collection easier.

Cell Phones that Listen and Learn
Source: MIT Technology Review, 29 June 2009
 
New software tracks a user's behavior by monitoring everyday sounds.

Top Six Don’ts for Usability Testing
Source: FutureNow Inc., 27 June 2009
 
Six tips for creating quality usability tests to ensure useful feedback from testers.

Usability: ‘Lovely software. But I can’t work it’
Source: FT.com, 26 June 2009
 
In a recent survey by Global Graphics, 77 per cent of office workers estimate they lose up to one hour a week because business software is difficult to use.

And what do you do?
Source: Dexo Design, 25 June 2009
 
How do you describe your job role? Here are the results of a recent 'Preferred UX/UI Title' Poll.

Most Doctors cite Usability as critical to Electronic Health Record Adoption
Source: TMCNet, 24 June 2009
 
It's all about 'meaningful use'.

Glossy monitors look good but can hurt
Source: QUT, 23 June 2009
 
A new advisory cites research which suggests high gloss monitors make users sit awkwardly.