More Analysts can mean Less Effectiveness in Usability Evaluations

Research has revealed evidence that heuristic evaluation by multiple analysts actually reduces the effectiveness of the method, contrary to previous beliefs. Only by studying analyst decision-making can anyone begin to understand why Usability Inspection Methods (UIMs) succeed and fail in particular contexts, argue Alan Woolrych and Gilbert Cockton of the University of Sunderland.

Their work, to be reported at HCI2002, shows that existing means of combining analysts' responses often leads to a reduction in the accuracy of heuristic analysis as a predictor of user problems.

'The use of multiple analysts is recommended for improving the thoroughness of UIMs,' say Woolrych and Cockton, citing the work of Jakob Nielsen on heuristics. 'As more diverse more analysts are added, then hits increase as confirmed by Nielsen's studies. Nielsen's recommendations do not consider how analysts' predictions are merged. No further confirmation or elimination resources are proposed, and thus we must assume that when one analyst confirms a problem, it will be preserved, even when additional analysts eliminate some problems.'

Their formula for analysing the effectiveness of multiple analysts shows that more bodies must increase the thoroughness of a UIM: Thoroughness = Number of real usability problems found / Number of real problems that exist. But if their formula for validity is juxtaposed with this, it can be shown that multiple analysts also reduce UIM validity unless account is taken of false alarms: Validity = Number of real usability problems found / Number of problems predicted by UIM.

'Where several analysts discover the same improbable problem (false alarm), all analysts must have adequate analysis resources to eliminate it. If only one analyst fails here, the false alarm will remain in the problem set. Multiple analysts are thus highly likely to inflate false alarms. As a result, the overall effectiveness of the UIM will drop, where: Effectiveness = Thoroughness x Validity.'

Advice on Designing Mobile Sites and Apps
Source: Poynter Online, 18 March 2010
 
Series of useful tips from a recent panel discussion.

Website Usability and Conversion
Source: UsabilityBlog, 17 March 2010
 
Useful presentation on the relationship between website usability and conversion.

Tips for Usability Testing with Children
Source: Econsultancy, 16 March 2010
 
Not even chocolate will help you here.

The Hygiene Factor of Usability
Source: inspireUX, 15 March 2010
 
Is it true that usability can no longer take us 'beyond lack of dissatisfaction'?

The Business Benefits of building Accessible websites
Source: Econsultancy, 13 March 2010
 
There’s a good business case for making your website more accessible to the UK’s disabled community.

Internet access is 'a fundamental right'
Source: BBC, 12 March 2010
 
Almost four in five people around the world believe that access to the internet is a fundamental right, a poll for the BBC World Service suggests.

The Net generation, Unplugged
Source: The Economist, 11 March 2010
 
Is it really helpful to talk about a new generation of "digital natives" who have grown up with the internet?

Rewriting the Human-Computer interaction Handbook
Source: CIOL, 10 March 2010
 
Indrani Medhi of Microsoft Research India has developed text-free user interfaces (UIs) to allow any illiterate or semi-literate person on first contact with a computer, to proceed with minimal or no assistance.

Lip reading Mobile promises End to noisy phone calls
Source: BBC, 9 March 2010
 
A prototype device shown off at CeBIT could allow people to conduct silent phone conversations.

Games User Researchers band together
Source: UN, 8 March 2010
 
The number of UX professionals in gaming has reached critical mass.