‘Discount’ user testing under fire

‘Discount’ testing came under fire from all quarters at the HCI2001 conference as both Microsoft’s Ken Dye, and Gilbert Cockton and Alan Woolrych of the University of Sunderland presented evidence that it is not effective.

It has become almost a truism that tests involving five different users will reveal more than 80% of problems with a design and that the law of diminishing returns means tests with further users reveal less and less useful information. This Landauer and Nielsen formula from 1993 has become the basis of much product testing worldwide, with a concentration on setting tasks to expose the shortcomings of each feature.

However, weaknesses in this approach were exposed along two dimensions by conference speakers. First, Dye, group manager MS Marketing Intelligence, warned that analysing individual features of a complex products may improve aspects locally while ignoring the needs of the user. He spoke of the obstacles to better design: the challenge of understanding human activity; software technologies being difficult to build and our poor knowledge of work requirements and goals.

But, having outlined the kinds of test undertaken at MS, he went on to attack usability testing for its limited horizons: ‘We should stop the focus on widgets and technology, as this doesn’t make computers easy to use’. Dismissing ‘time on task’ and ‘number of errors’ as measures, he proposed design success metrics such as setting productive time against time attending to the tool enabling the productivity. He then urged practitioners and researchers to work more closely with each other to establish solutions to the problems identified by involving users in the design process. Solutions based on real data could then be tested in practice and this kind of partnership would result in usable products as well as usable features.

Attack on testing methods came in another guise from Woolrych and Cockton. They presented two studies, one on the merits of heuristic evaluation and another, evaluating the Landauer and Nielsen formula. Neither approach emerged unscathed. Findings showed that, overall, heuristics provide a poor analyst resource for the successful elimination or confirmation of probable and improbable problems and ‘used inappropriately could lead to poor design changes’.

Their other study explored the likelihood of identifying the most severe design problems by testing with five users. This showed, using statistical analysis, a recent study by Spool and Schroeder and the researchers’ work on heuristic evaluation, that the chance of getting the right five users to show up problems depends on individual differences between test users, the tool under test and the tasks performed. Further, the frequency of a problem may not be demonstrable with a small sample, while there is no way of ensuring that severe problems show up if their ‘visibility’ is low. ‘It is not enough to simply find problems - they must be understood and prioritised, and neither is possible without good frequency and severity data,’ they said.

Associated Link: Download Related Paper: Why and When Five Test Users aren’t Enough

How can we improve Mobile User Experience?
Source: Mobile Entertainment, 16 May 2008
 
A roundtable of forward thinkers convened in London last month to debate mobile user experience in advance of the forthcoming MEX conference.

HCI for Community and International Development
Source: UN, 15 May 2008
 
Emerging economies, developing countries and the challenges of designing across cultures came under scrutiny at a CHI 2008 workshop on Community and International Development.

The Future of social networking: Mobile Phones
Source: Times Online, 14 May 2008
 
The future of social networking is the coming together of internet-connected mobile phones and location or proximity technology.

Usability for Fun and Profit
Source: The Herald, 13 May 2008
 
An interview with Chris Rourke, owner and director of Edinburgh-based consultancy, UserVision.

Caroline's Corner: Usability of Content is Plain Language
Source: Caroline Jarrett, 12 May 2008
 
An exciting thing happened in the USA on 14th April 2008. It didn't quite manage to make it onto the national news - that day, we were mostly hearing about the Pope's visit to the USA. Any ideas? Any clue from my title? Give up?

HP Labs opens doors to Academia
Source: VNUNet, 10 May 2008
 
HP has unveiled an initiative allowing academic institutions to collaborate with HP Labs in joint research through an open and competitive process.

Email: is it time to get some Training?
Source: UN, 9 May 2008
 
How hard can it be? Apparently lack of email training is costing employers dearly.

Book Review: Mental Models by Indi Young
Source: UN, 8 May 2008
 
A practical, readable and relevant account of key processes in the vital task of researching, building and applying mental models to product design.

CHI '08: What makes a really Great Designer?
Source: System Concepts, 7 May 2008
 
What qualities do elite designers share, and what do they know that other members of the HCI community could benefit from?

Raising the standard of websites worldwide: BSI and Magus launch a new specification
Source: UN, 5 May 2008
 
BSI and Magus have released Publicly Available Specification (PAS) 124 to help organisations manage website standards efficiently and harness their full potential.