Comment: Getting the Right Measure of Usability

As usability professionals, we're in agreement that usability testing in general is a Good Thing. But we've seen an interesting debate recently discussing the relative merits and demerits of different approaches to usability testing. Should we use discount, qualitative methods or more formal, quantitative methods?

Weighing in for discount studies is the redoubtable Jakob Nielsen. Dr Nielsen argues:

"Number fetishism leads usability studies astray by focusing on statistical analyses that are often false, biased, misleading, or overly narrow. Better to emphasize insights and qualitative research."

In the other corner, we have Jeff Sauro who argues:

"The risks of relying heavily on a qualitative approach can lead to a severe misdiagnosis especially when usability problems are difficult to detect."

Both authors include an interesting discussion of the Florida Butterfly Ballot, with each using it as an argument for their own method of testing.

"...a [quantitative] study of 100 voters would not have included a statistically significant number of people who intended to vote for Al Gore but instead punched the hole for Patrick Buchanan, because less than 1% of voters made this mistake." (Nielsen)

"...the problems voters had with the "butterfly-ballot" in the Florida 2000 election would not have been detected with popular discounted qualitative methods." (Sauro)

So who's right? A second reading of these articles shows that the authors value both types of usability test and are probably arguing over differences in emphasis. For example, Nielsen-type discount studies are cost effective where usability problems are found and fixed and then the next "throwaway" prototype is again quickly tested with a small number of participants.

On the other hand, Sauro-type quantitative studies are the only way to set usability acceptance targets; compare a product's usability to the competition, or a predecessor; correlate a product's usability with sales or customer returns; and answer the question: "How usable is this product?"

What's a usability professional to do?

1. Recognise that a bad usability study is a bad usability study, whether it is qualitative or quantitative in nature.
2. Accept that there is no right or wrong method: you should use different methods at different parts of the lifecycle.

Nilsen's article: "Risks of Quantitative Studies" http://www.useit.com/alertbox/20040301.html

Sauro's article: "The Risks of Discounted Qualitative Studies" http://www.measuringusability.com/qualitative_risks.htm

And, for balance, try: "Discount usability: time to push back the pendulum?" http://www.userfocus.co.uk/articles/discount.html

David Travis
Userfocus ltd
david.travis@userfocus.co.uk

All change at the top for System Concepts
Source: System Concepts Ltd, 3 July 2009
 
Leslie Fountain has been promoted to joint Managing Director of leading usability consultancy System Concepts.

Life in UCD immortalised in fiction: you couldn't make it up
Source: UN, 2 July 2009
 
Sarah Herman's fictitious book on life in a user-centred design company has hit the shelves and The Guardian's book pages...

Interfaces Magazine - Issue 79: The Education Issue
Source: Interaction Group, 1 July 2009
 
The latest issue of Interfaces is now available as a free download from the Interaction Website.

Two new Behavioural research Tools from Noldus
Source: UN, 30 June 2009
 
Tool updates make on-site behavioural data collection easier.

Cell Phones that Listen and Learn
Source: MIT Technology Review, 29 June 2009
 
New software tracks a user's behavior by monitoring everyday sounds.

Top Six Don’ts for Usability Testing
Source: FutureNow Inc., 27 June 2009
 
Six tips for creating quality usability tests to ensure useful feedback from testers.

Usability: ‘Lovely software. But I can’t work it’
Source: FT.com, 26 June 2009
 
In a recent survey by Global Graphics, 77 per cent of office workers estimate they lose up to one hour a week because business software is difficult to use.

And what do you do?
Source: Dexo Design, 25 June 2009
 
How do you describe your job role? Here are the results of a recent 'Preferred UX/UI Title' Poll.

Most Doctors cite Usability as critical to Electronic Health Record Adoption
Source: TMCNet, 24 June 2009
 
It's all about 'meaningful use'.

Glossy monitors look good but can hurt
Source: QUT, 23 June 2009
 
A new advisory cites research which suggests high gloss monitors make users sit awkwardly.