EUPA/HCI2002: Five Users Theory causes Conflict at Conference

Following the publication of our paper at IHM-HCI 2001, "Why and When Five Test Users aren't Enough", I was most interested in an EUPA presentation at this year's British HCI Group Annual Conference by Carol M. Barnum entitled "The Magic Number 5: Is it enough for Web Testing?" More so because it was apparent that our work in this area appeared to be ignored in the presentation (prepared before Carol found our papers on discount methods!).

The aim of the presentation seemed to be to reassure the discount method faithful that despite a wave of evidence that five users may be nowhere near enough, that it was OK to keep believing in the Magic Number. During the presentation I could not help but be mildly amused by what I felt was a bizarre similarity to the philosophy of the fabled Simpson's Stonecutters Society. For non-Simpsons fans (are there any?), this ancient society 'has, since ancient times, split the rocks of ignorance that obscure the light of knowledge and truth'. The real purpose of the society, however, is revealed by a line in their drinking song: 'the obscuring of the truth that would benefit the majority for the good of the few'. While Carol was clearly no such Stonecutter, she got dangerously close to supporting their hidden agenda.

The defence of the magical number 5 was based on tightening parameters to such a degree (similar type users, small part of poorly designed system etc) that testing beyond 5 users would be totally cost ineffective. Such shifting of the goal posts is a fine way to obscure the truth. As a result, Carol could safely expose many reasons why 5 users would not be enough in her presentation, such as a wide scope for testing unconstrained usage, user differences, and (wait for it) the impact of good user centred design. However, these were not reasons for abandoning the magic number 5, but instead the reason why studies such as Spool and Schroeder (CHI 2001 short paper) can be safely ignored by faithful Usability Stonecutters!

As we have argued in our papers (see Interactions Sept+Oct 2002) user testing is not just about finding problems, they must be understood and prioritised, and neither is possible without good frequency and severity data. These are both critical attributes when prioritising solution recommendation and implementation. The frequency of a problem is dictated by the number of users it affects. Similarly, the severity of a problem can only be ascertained by analysis of the difficulties encountered by individual users with individual problem elements. By a 'mix 'n' match' of the test users in our study we found hugely significant differences in problem frequency and severity between various groups of 5 users.

The danger (as with all discount methods) is that potential clients will fail to understand the limitations of the 'quick fix' usability solution of running a handful of user tests. User testing is just a part of the usability process and we must encourage clients to invest in the whole process and not mislead them into believing a handful of test users will provide the usability solution. Carol Barnum did stress that Usability Stonecutters who keep the faith with only five users should make clear the limitations of discount testing to clients. However, she did not spell out these limitations and it is unlikely that anyone who still believes in the magic number would be able to deliver such candour.

It's time to move the Stonecutters out of usability. The issue is not whether five users are enough. It's an issue of the costs, benefits and risks of running some number of users with known relevant characteristics. Three may be enough. Twenty may not be enough. What we need are well-researched models that let us carry out credible risk and cost-benefit analyses of user testing plans. We do not have these yet and Carol's presentation missed a valuable opportunity to lay the foundations for such a model.

Alan Woolrych
University of Sunderland

'Internet addiction' linked to Depression
Source: BBC, 9 February 2010
 
There is a strong link between heavy internet use and depression, UK psychologists have said.

Could *You* be more Usable?
Source: UN, 8 February 2010
 
Bet you could.

Stowe Boyd on 'Steampunk' thinking about the Future of Computing
Source: Stowe Boyd's blog via Experientia, 6 February 2010
 
Are established metaphors of user experience holding us back from new ways of structuring our interaction through computers?

Nokia's User Experience Programme
Source: UN, 5 February 2010
 
Nokia has put together a rich and informative website covering the key elements of user experience.

Interfaces magazine: latest issue available now
Source: HCI News Service, 4 February 2010
 
The latest issue of Interfaces is now available in pdf format, free from the Interaction Website.

A Lighter Brigade of Chargers
Source: UN, 3 February 2010
 
Lots of gadgets, one charger. At last.

Mobile Touch Screens could soon Feel the Pressure
Source: MIT Technology Review, 2 February 2010
 
A quantum switch could add pressure sensing to mobile screens.

Usability, Usability, Usability: why the iPad will Succeed
Source: Econsultancy, 1 February 2010
 
The tech critics love it, hate it, love it again, shrug it off. What do usability experts say?

British Airways - at last some good news
Source: Loop11, 30 January 2010
 
In a recent website usability study for the world's leading airlines, the British Airways website proved to be the most user friendly, with Malaysia Airlines and Virgin Atlantic having the lowest user experience rating.

Computation of Emotions in Man and Machine
Source: Royal Society, 29 January 2010
 
Advances in computer technology now allow machines to recognise and express emotions, paving the way for improved human-computer and human-human communications.