More Analysts can mean Less Effectiveness in Usability Evaluations

Research has revealed evidence that heuristic evaluation by multiple analysts actually reduces the effectiveness of the method, contrary to previous beliefs. Only by studying analyst decision-making can anyone begin to understand why Usability Inspection Methods (UIMs) succeed and fail in particular contexts, argue Alan Woolrych and Gilbert Cockton of the University of Sunderland.

Their work, to be reported at HCI2002, shows that existing means of combining analysts' responses often leads to a reduction in the accuracy of heuristic analysis as a predictor of user problems.

'The use of multiple analysts is recommended for improving the thoroughness of UIMs,' say Woolrych and Cockton, citing the work of Jakob Nielsen on heuristics. 'As more diverse more analysts are added, then hits increase as confirmed by Nielsen's studies. Nielsen's recommendations do not consider how analysts' predictions are merged. No further confirmation or elimination resources are proposed, and thus we must assume that when one analyst confirms a problem, it will be preserved, even when additional analysts eliminate some problems.'

Their formula for analysing the effectiveness of multiple analysts shows that more bodies must increase the thoroughness of a UIM: Thoroughness = Number of real usability problems found / Number of real problems that exist. But if their formula for validity is juxtaposed with this, it can be shown that multiple analysts also reduce UIM validity unless account is taken of false alarms: Validity = Number of real usability problems found / Number of problems predicted by UIM.

'Where several analysts discover the same improbable problem (false alarm), all analysts must have adequate analysis resources to eliminate it. If only one analyst fails here, the false alarm will remain in the problem set. Multiple analysts are thus highly likely to inflate false alarms. As a result, the overall effectiveness of the UIM will drop, where: Effectiveness = Thoroughness x Validity.'

All change at the top for System Concepts
Source: System Concepts Ltd, 3 July 2009
 
Leslie Fountain has been promoted to joint Managing Director of leading usability consultancy System Concepts.

Life in UCD immortalised in fiction: you couldn't make it up
Source: UN, 2 July 2009
 
Sarah Herman's fictitious book on life in a user-centred design company has hit the shelves and The Guardian's book pages...

Interfaces Magazine - Issue 79: The Education Issue
Source: Interaction Group, 1 July 2009
 
The latest issue of Interfaces is now available as a free download from the Interaction Website.

Two new Behavioural research Tools from Noldus
Source: UN, 30 June 2009
 
Tool updates make on-site behavioural data collection easier.

Cell Phones that Listen and Learn
Source: MIT Technology Review, 29 June 2009
 
New software tracks a user's behavior by monitoring everyday sounds.

Top Six Don’ts for Usability Testing
Source: FutureNow Inc., 27 June 2009
 
Six tips for creating quality usability tests to ensure useful feedback from testers.

Usability: ‘Lovely software. But I can’t work it’
Source: FT.com, 26 June 2009
 
In a recent survey by Global Graphics, 77 per cent of office workers estimate they lose up to one hour a week because business software is difficult to use.

And what do you do?
Source: Dexo Design, 25 June 2009
 
How do you describe your job role? Here are the results of a recent 'Preferred UX/UI Title' Poll.

Most Doctors cite Usability as critical to Electronic Health Record Adoption
Source: TMCNet, 24 June 2009
 
It's all about 'meaningful use'.

Glossy monitors look good but can hurt
Source: QUT, 23 June 2009
 
A new advisory cites research which suggests high gloss monitors make users sit awkwardly.