Comment: Getting the Right Measure of Usability

As usability professionals, we're in agreement that usability testing in general is a Good Thing. But we've seen an interesting debate recently discussing the relative merits and demerits of different approaches to usability testing. Should we use discount, qualitative methods or more formal, quantitative methods?

Weighing in for discount studies is the redoubtable Jakob Nielsen. Dr Nielsen argues:

"Number fetishism leads usability studies astray by focusing on statistical analyses that are often false, biased, misleading, or overly narrow. Better to emphasize insights and qualitative research."

In the other corner, we have Jeff Sauro who argues:

"The risks of relying heavily on a qualitative approach can lead to a severe misdiagnosis especially when usability problems are difficult to detect."

Both authors include an interesting discussion of the Florida Butterfly Ballot, with each using it as an argument for their own method of testing.

"...a [quantitative] study of 100 voters would not have included a statistically significant number of people who intended to vote for Al Gore but instead punched the hole for Patrick Buchanan, because less than 1% of voters made this mistake." (Nielsen)

"...the problems voters had with the "butterfly-ballot" in the Florida 2000 election would not have been detected with popular discounted qualitative methods." (Sauro)

So who's right? A second reading of these articles shows that the authors value both types of usability test and are probably arguing over differences in emphasis. For example, Nielsen-type discount studies are cost effective where usability problems are found and fixed and then the next "throwaway" prototype is again quickly tested with a small number of participants.

On the other hand, Sauro-type quantitative studies are the only way to set usability acceptance targets; compare a product's usability to the competition, or a predecessor; correlate a product's usability with sales or customer returns; and answer the question: "How usable is this product?"

What's a usability professional to do?

1. Recognise that a bad usability study is a bad usability study, whether it is qualitative or quantitative in nature.
2. Accept that there is no right or wrong method: you should use different methods at different parts of the lifecycle.

Nilsen's article: "Risks of Quantitative Studies" http://www.useit.com/alertbox/20040301.html

Sauro's article: "The Risks of Discounted Qualitative Studies" http://www.measuringusability.com/qualitative_risks.htm

And, for balance, try: "Discount usability: time to push back the pendulum?" http://www.userfocus.co.uk/articles/discount.html

David Travis
Userfocus ltd
david.travis@userfocus.co.uk

Advice on Designing Mobile Sites and Apps
Source: Poynter Online, 18 March 2010
 
Series of useful tips from a recent panel discussion.

Website Usability and Conversion
Source: UsabilityBlog, 17 March 2010
 
Useful presentation on the relationship between website usability and conversion.

Tips for Usability Testing with Children
Source: Econsultancy, 16 March 2010
 
Not even chocolate will help you here.

The Hygiene Factor of Usability
Source: inspireUX, 15 March 2010
 
Is it true that usability can no longer take us 'beyond lack of dissatisfaction'?

The Business Benefits of building Accessible websites
Source: Econsultancy, 13 March 2010
 
There’s a good business case for making your website more accessible to the UK’s disabled community.

Internet access is 'a fundamental right'
Source: BBC, 12 March 2010
 
Almost four in five people around the world believe that access to the internet is a fundamental right, a poll for the BBC World Service suggests.

The Net generation, Unplugged
Source: The Economist, 11 March 2010
 
Is it really helpful to talk about a new generation of "digital natives" who have grown up with the internet?

Rewriting the Human-Computer interaction Handbook
Source: CIOL, 10 March 2010
 
Indrani Medhi of Microsoft Research India has developed text-free user interfaces (UIs) to allow any illiterate or semi-literate person on first contact with a computer, to proceed with minimal or no assistance.

Lip reading Mobile promises End to noisy phone calls
Source: BBC, 9 March 2010
 
A prototype device shown off at CeBIT could allow people to conduct silent phone conversations.

Games User Researchers band together
Source: UN, 8 March 2010
 
The number of UX professionals in gaming has reached critical mass.