More Analysts can mean Less Effectiveness in Usability Evaluations

Research has revealed evidence that heuristic evaluation by multiple analysts actually reduces the effectiveness of the method, contrary to previous beliefs. Only by studying analyst decision-making can anyone begin to understand why Usability Inspection Methods (UIMs) succeed and fail in particular contexts, argue Alan Woolrych and Gilbert Cockton of the University of Sunderland.

Their work, to be reported at HCI2002, shows that existing means of combining analysts' responses often leads to a reduction in the accuracy of heuristic analysis as a predictor of user problems.

'The use of multiple analysts is recommended for improving the thoroughness of UIMs,' say Woolrych and Cockton, citing the work of Jakob Nielsen on heuristics. 'As more diverse more analysts are added, then hits increase as confirmed by Nielsen's studies. Nielsen's recommendations do not consider how analysts' predictions are merged. No further confirmation or elimination resources are proposed, and thus we must assume that when one analyst confirms a problem, it will be preserved, even when additional analysts eliminate some problems.'

Their formula for analysing the effectiveness of multiple analysts shows that more bodies must increase the thoroughness of a UIM: Thoroughness = Number of real usability problems found / Number of real problems that exist. But if their formula for validity is juxtaposed with this, it can be shown that multiple analysts also reduce UIM validity unless account is taken of false alarms: Validity = Number of real usability problems found / Number of problems predicted by UIM.

'Where several analysts discover the same improbable problem (false alarm), all analysts must have adequate analysis resources to eliminate it. If only one analyst fails here, the false alarm will remain in the problem set. Multiple analysts are thus highly likely to inflate false alarms. As a result, the overall effectiveness of the UIM will drop, where: Effectiveness = Thoroughness x Validity.'

The Net generation, Unplugged
Source: The Economist, 11 March 2010
 
Is it really helpful to talk about a new generation of "digital natives" who have grown up with the internet?

Rewriting the Human-Computer interaction Handbook
Source: CIOL, 10 March 2010
 
Indrani Medhi of Microsoft Research India has developed text-free user interfaces (UIs) to allow any illiterate or semi-literate person on first contact with a computer, to proceed with minimal or no assistance.

Lip reading Mobile promises End to noisy phone calls
Source: BBC, 9 March 2010
 
A prototype device shown off at CeBIT could allow people to conduct silent phone conversations.

Games User Researchers band together
Source: UN, 8 March 2010
 
The number of UX professionals in gaming has reached critical mass.

Quince Pro enables privately-held UX Design Libraries
Source: Infragistics, 6 March 2010
 
Infragistics has launched Quince Pro, a private, secure and organized way to collaborate, communicate and cultivate private UX design libraries to ensure consistent user experiences across teams, departments and companies.

Announcing a new issue of the Journal of Usability Studies
Source: UPA, 5 March 2010
 
UPA is happy to announce the publication of the second issue of volume 5, the Journal of Usability Studies.

Impatient versus Bored
Source: Gerry McGovern, 4 March 2010
 
Customers are much more likely to get impatient with your website than they are to be bored with it.

Futures Thinking: Writing Scenarios
Source: Fast Company, 3 March 2010
 
A valuable skill for UX folks (that's 'how to write scenarios', not 'scenarios in which one writes').

How to Conduct a Usability Review
Source: UsefulUsability, 2 March 2010
 
Never hurts to be reminded.

New Tool pitched at UX consultants and small Usability Teams
Source: UN, 1 March 2010
 
Ovo Studios has released a new software product targeting user experience consultants and small usability teams.