Five Ways to Make any Usability Test more Credible

By Jeff Sauro


Whether you're conducting an early stage test of a website design or a validation of a live site, these five tips can make any usability test more credible. The tips both temper skepticism about small samples and help you avoid overstating your findings.

1. Count the number of users that experience each problem. Early website testing is all about finding and fixing usability problems. But make those problem lists even more helpful by providing the number of users that encountered the problem. For example 3 out of 5 or 5 out of 7. These numbers will be crucial for estimating impact, prioritizing and for use in future comparisons.

2. Estimate problem impact using confidence intervals: Knowing how many users encountered a problem on your website allows you to estimate the potential impact on all users. Confidence intervals work on any sized sample to provide your best estimate. For example, for a design issue that 3 of 5 users encountered would tell you between 23% and 88% would also have the same problem. While the confidence interval is wide, it's highly improbably fewer than a quarter of the users wouldn't encounter this problem. Problem frequency can be used in conjunction with severity for prioritizing problems as there's never enough time or money to fix everything.

3. State a problem's severity separate from frequency: Not all website usability problems are equal. Assume frequency and severity are independent. Some design problems can lead to crashes, data loss or nuclear meltdowns (OK, the last one is not something you or anyone conducting a test with small sample sizes will ever deal with but you get the point). Rate your severity using a scale with at least 3 categories. You need to distinguish between the trivial many problems and the critical few. I'm a believer that having somewhere between 5 and 11 points on a severity scale will give you enough points of discrimination without going overboard.

Associated Link: More: Five Ways to Make any Usability Test more Credible

Three Things Steve Krug Didn’t Tell You About Usability Testing
Source: George Saines, 2 September 2010
 
Don't make me think. Well, ok, maybe a little bit.

Trace exactly what Users are doing
Source: killerstartups, 30 August 2010
 
A new tool lets you watch all of your website activity, in real-time replays.

Google boosts Usability with Gmail revamp
Source: V3.co.uk, 28 August 2010
 
Long overdue improvements to Gmail may increase its appeal to business users.

Tech4Africa conference runs in Johannesburg
Source: bizcommunity, 26 August 2010
 
A new conference in Africa recognises usability as a central concept to IT uptake.

Top 3 Usability Tips for Building Better Blogs
Source: CMS Wire, 24 August 2010
 
If you're compelled to blog, may as well get it right.

Outdoors and Out of Reach, Studying the Brain
Source: New York Times, 23 August 2010
 
Fascinating account of how heavy use of digital devices changes how we think and behave, and how a retreat into nature might reverse those effects.

Tips for International In-home User Research
Source: System Concepts Ltd, 20 August 2010
 
In-home user research can provide richer, more illuminating information than lab-based studies. Here are some tips for a successful in-home international research.

New Study- Gender differences in Web Usability
Source: Demystifying Usability blog, 18 August 2010
 
Comscore has released a new study (June 30 2010) entitled Women on the Web: How Women are Shaping the Internet.

UX Myths: Usability testing is Expensive
Source: UX Myths, 16 August 2010
 
Many organizations still believe usability testing is a luxury that requires an expensively equipped lab and takes weeks to conduct.

Usability testing may improve hit rate of Mobile Advertising
Source: UN, 14 August 2010
 
A new mobile advertising service is launching in the UK. Will usability testing help it achieve its goals?