Comment: Getting the Right Measure of Usability

As usability professionals, we're in agreement that usability testing in general is a Good Thing. But we've seen an interesting debate recently discussing the relative merits and demerits of different approaches to usability testing. Should we use discount, qualitative methods or more formal, quantitative methods?

Weighing in for discount studies is the redoubtable Jakob Nielsen. Dr Nielsen argues:

"Number fetishism leads usability studies astray by focusing on statistical analyses that are often false, biased, misleading, or overly narrow. Better to emphasize insights and qualitative research."

In the other corner, we have Jeff Sauro who argues:

"The risks of relying heavily on a qualitative approach can lead to a severe misdiagnosis especially when usability problems are difficult to detect."

Both authors include an interesting discussion of the Florida Butterfly Ballot, with each using it as an argument for their own method of testing.

"...a [quantitative] study of 100 voters would not have included a statistically significant number of people who intended to vote for Al Gore but instead punched the hole for Patrick Buchanan, because less than 1% of voters made this mistake." (Nielsen)

"...the problems voters had with the "butterfly-ballot" in the Florida 2000 election would not have been detected with popular discounted qualitative methods." (Sauro)

So who's right? A second reading of these articles shows that the authors value both types of usability test and are probably arguing over differences in emphasis. For example, Nielsen-type discount studies are cost effective where usability problems are found and fixed and then the next "throwaway" prototype is again quickly tested with a small number of participants.

On the other hand, Sauro-type quantitative studies are the only way to set usability acceptance targets; compare a product's usability to the competition, or a predecessor; correlate a product's usability with sales or customer returns; and answer the question: "How usable is this product?"

What's a usability professional to do?

1. Recognise that a bad usability study is a bad usability study, whether it is qualitative or quantitative in nature.
2. Accept that there is no right or wrong method: you should use different methods at different parts of the lifecycle.

Nilsen's article: "Risks of Quantitative Studies" http://www.useit.com/alertbox/20040301.html

Sauro's article: "The Risks of Discounted Qualitative Studies" http://www.measuringusability.com/qualitative_risks.htm

And, for balance, try: "Discount usability: time to push back the pendulum?" http://www.userfocus.co.uk/articles/discount.html

David Travis
Userfocus ltd
david.travis@userfocus.co.uk

Why 'The 10 Commandments of Web Design' are Complete Baloney
Source: SitePoint, 5 July 2008
 
Further to our earlier discussion...

Local Council Websites: Good, but no Cigar
Source: Webcredible, 4 July 2008
 
The 'Local Council Websites: Good, But No Cigar' report evaluates the top 20 Socitm council websites against 20 best practice usability guidelines.

Why does the OK Button say OK?
Source: Gerry McGovern, 3 July 2008
 
Most times I come across the OK button, something not-OK has happened.

The 10 Commandments of Web Design
Source: Business Week, 2 July 2008
 
To try and make sense of it all, BusinessWeek.com canvassed a broad range of Internet luminaries to discover the design rules they live by right now.

Bill Gates' email rant at the Usability Team
Source: Crunch Gear, 1 July 2008
 
The Seattle P-I has tracked down an old e-mail from 2003 that Bill Gates sent out to members of the Windows Usability team.

Can a better User Experience also be a Greener Experience?
Source: Catalyst Resources, 30 June 2008
 
By improving the user experience of its web collaboration software, Catalyst made it easier for clients to work in productive teams without travelling the globe.

The Spread of Telepresence Technology
Source: Tech News World, 28 June 2008
 
Rising energy costs and green concerns are fuelling a huge increase in demand for videoconferencing, collaborative software and virtual telepresence technology.

UK Usability market worth more than £200 million by end of 2008
Source: eConsultancy, 27 June 2008
 
The UK Usability market will grow by an estimated 20% in 2008 to a value of £214 million, according to research published this week by e-consultancy.

Research suggests growing importance of Usability to UK Businesses
Source: Business Strata, 26 June 2008
 
Integrated software for lead generation and marketing is valuable, but only as long as it's usable.

Web 2.0: Too Smart by a Half?
Source: Computer Technology Review, 25 June 2008
 
Will the Web 2.0 generation’s epitaph be, “They died with their options open?”