Comment: Getting the Right Measure of Usability

As usability professionals, we're in agreement that usability testing in general is a Good Thing. But we've seen an interesting debate recently discussing the relative merits and demerits of different approaches to usability testing. Should we use discount, qualitative methods or more formal, quantitative methods?

Weighing in for discount studies is the redoubtable Jakob Nielsen. Dr Nielsen argues:

"Number fetishism leads usability studies astray by focusing on statistical analyses that are often false, biased, misleading, or overly narrow. Better to emphasize insights and qualitative research."

In the other corner, we have Jeff Sauro who argues:

"The risks of relying heavily on a qualitative approach can lead to a severe misdiagnosis especially when usability problems are difficult to detect."

Both authors include an interesting discussion of the Florida Butterfly Ballot, with each using it as an argument for their own method of testing.

"...a [quantitative] study of 100 voters would not have included a statistically significant number of people who intended to vote for Al Gore but instead punched the hole for Patrick Buchanan, because less than 1% of voters made this mistake." (Nielsen)

"...the problems voters had with the "butterfly-ballot" in the Florida 2000 election would not have been detected with popular discounted qualitative methods." (Sauro)

So who's right? A second reading of these articles shows that the authors value both types of usability test and are probably arguing over differences in emphasis. For example, Nielsen-type discount studies are cost effective where usability problems are found and fixed and then the next "throwaway" prototype is again quickly tested with a small number of participants.

On the other hand, Sauro-type quantitative studies are the only way to set usability acceptance targets; compare a product's usability to the competition, or a predecessor; correlate a product's usability with sales or customer returns; and answer the question: "How usable is this product?"

What's a usability professional to do?

1. Recognise that a bad usability study is a bad usability study, whether it is qualitative or quantitative in nature.
2. Accept that there is no right or wrong method: you should use different methods at different parts of the lifecycle.

Nilsen's article: "Risks of Quantitative Studies" http://www.useit.com/alertbox/20040301.html

Sauro's article: "The Risks of Discounted Qualitative Studies" http://www.measuringusability.com/qualitative_risks.htm

And, for balance, try: "Discount usability: time to push back the pendulum?" http://www.userfocus.co.uk/articles/discount.html

David Travis
Userfocus ltd
david.travis@userfocus.co.uk

'Internet addiction' linked to Depression
Source: BBC, 9 February 2010
 
There is a strong link between heavy internet use and depression, UK psychologists have said.

Could *You* be more Usable?
Source: UN, 8 February 2010
 
Bet you could.

Stowe Boyd on 'Steampunk' thinking about the Future of Computing
Source: Stowe Boyd's blog via Experientia, 6 February 2010
 
Are established metaphors of user experience holding us back from new ways of structuring our interaction through computers?

Nokia's User Experience Programme
Source: UN, 5 February 2010
 
Nokia has put together a rich and informative website covering the key elements of user experience.

Interfaces magazine: latest issue available now
Source: HCI News Service, 4 February 2010
 
The latest issue of Interfaces is now available in pdf format, free from the Interaction Website.

A Lighter Brigade of Chargers
Source: UN, 3 February 2010
 
Lots of gadgets, one charger. At last.

Mobile Touch Screens could soon Feel the Pressure
Source: MIT Technology Review, 2 February 2010
 
A quantum switch could add pressure sensing to mobile screens.

Usability, Usability, Usability: why the iPad will Succeed
Source: Econsultancy, 1 February 2010
 
The tech critics love it, hate it, love it again, shrug it off. What do usability experts say?

British Airways - at last some good news
Source: Loop11, 30 January 2010
 
In a recent website usability study for the world's leading airlines, the British Airways website proved to be the most user friendly, with Malaysia Airlines and Virgin Atlantic having the lowest user experience rating.

Computation of Emotions in Man and Machine
Source: Royal Society, 29 January 2010
 
Advances in computer technology now allow machines to recognise and express emotions, paving the way for improved human-computer and human-human communications.