‘Discount’ user testing under fire

‘Discount’ testing came under fire from all quarters at the HCI2001 conference as both Microsoft’s Ken Dye, and Gilbert Cockton and Alan Woolrych of the University of Sunderland presented evidence that it is not effective.

It has become almost a truism that tests involving five different users will reveal more than 80% of problems with a design and that the law of diminishing returns means tests with further users reveal less and less useful information. This Landauer and Nielsen formula from 1993 has become the basis of much product testing worldwide, with a concentration on setting tasks to expose the shortcomings of each feature.

However, weaknesses in this approach were exposed along two dimensions by conference speakers. First, Dye, group manager MS Marketing Intelligence, warned that analysing individual features of a complex products may improve aspects locally while ignoring the needs of the user. He spoke of the obstacles to better design: the challenge of understanding human activity; software technologies being difficult to build and our poor knowledge of work requirements and goals.

But, having outlined the kinds of test undertaken at MS, he went on to attack usability testing for its limited horizons: ‘We should stop the focus on widgets and technology, as this doesn’t make computers easy to use’. Dismissing ‘time on task’ and ‘number of errors’ as measures, he proposed design success metrics such as setting productive time against time attending to the tool enabling the productivity. He then urged practitioners and researchers to work more closely with each other to establish solutions to the problems identified by involving users in the design process. Solutions based on real data could then be tested in practice and this kind of partnership would result in usable products as well as usable features.

Attack on testing methods came in another guise from Woolrych and Cockton. They presented two studies, one on the merits of heuristic evaluation and another, evaluating the Landauer and Nielsen formula. Neither approach emerged unscathed. Findings showed that, overall, heuristics provide a poor analyst resource for the successful elimination or confirmation of probable and improbable problems and ‘used inappropriately could lead to poor design changes’.

Their other study explored the likelihood of identifying the most severe design problems by testing with five users. This showed, using statistical analysis, a recent study by Spool and Schroeder and the researchers’ work on heuristic evaluation, that the chance of getting the right five users to show up problems depends on individual differences between test users, the tool under test and the tasks performed. Further, the frequency of a problem may not be demonstrable with a small sample, while there is no way of ensuring that severe problems show up if their ‘visibility’ is low. ‘It is not enough to simply find problems - they must be understood and prioritised, and neither is possible without good frequency and severity data,’ they said.

Associated Link: Download Related Paper: Why and When Five Test Users aren’t Enough

All change at the top for System Concepts
Source: System Concepts Ltd, 3 July 2009
 
Leslie Fountain has been promoted to joint Managing Director of leading usability consultancy System Concepts.

Life in UCD immortalised in fiction: you couldn't make it up
Source: UN, 2 July 2009
 
Sarah Herman's fictitious book on life in a user-centred design company has hit the shelves and The Guardian's book pages...

Interfaces Magazine - Issue 79: The Education Issue
Source: Interaction Group, 1 July 2009
 
The latest issue of Interfaces is now available as a free download from the Interaction Website.

Two new Behavioural research Tools from Noldus
Source: UN, 30 June 2009
 
Tool updates make on-site behavioural data collection easier.

Cell Phones that Listen and Learn
Source: MIT Technology Review, 29 June 2009
 
New software tracks a user's behavior by monitoring everyday sounds.

Top Six Don’ts for Usability Testing
Source: FutureNow Inc., 27 June 2009
 
Six tips for creating quality usability tests to ensure useful feedback from testers.

Usability: ‘Lovely software. But I can’t work it’
Source: FT.com, 26 June 2009
 
In a recent survey by Global Graphics, 77 per cent of office workers estimate they lose up to one hour a week because business software is difficult to use.

And what do you do?
Source: Dexo Design, 25 June 2009
 
How do you describe your job role? Here are the results of a recent 'Preferred UX/UI Title' Poll.

Most Doctors cite Usability as critical to Electronic Health Record Adoption
Source: TMCNet, 24 June 2009
 
It's all about 'meaningful use'.

Glossy monitors look good but can hurt
Source: QUT, 23 June 2009
 
A new advisory cites research which suggests high gloss monitors make users sit awkwardly.