CHI 2003: Don't Go to a Panel Like This Again!

User testing is a very narrow approach and it is becoming increasingly irrelevant to industry, said Dennis Wixon, usability manager at MSN, Microsoft. He was speaking on the CHI 2003 panel: "The Magic Number 5: Is it Enough for Web Testing?". He criticised the method for lack of ecological validity and welcomed alternatives like ethnography. 'Stop looking at statistics and examine the relationship of lab to field - that is a useful discussion to have,' he told the audience. 'And don't go to a panel like this one again!'

The panel was discussing the relevance and accuracy of Nielsen and Landauer's old formula that five users is enough to find a majority of usability problems with a product. Wixon was not alone in viewing this as the wrong discussion to be having. (Read a more detailed account of the debate from Larry Constantine, also on UN today at Testing... 1 2 3 4 5 ... Testing....) He implied that rather than focussing on identifying problems, a solution centred approach would be more useful in improving products.

Other areas that speakers highlighted as in need of consideration included:
* frequency of testing: few often is better than few users once;
* inter-facilitator reliability;
* frequency and severity of problems;
* the underlying statistical flaw that accompanies the methodological flaw in the five user study;
* the benefit of letting users choose their own tasks in attempting to find all problems/types of problem;
* the limits of automated testing.

Despite Wixon's admonitions, members of the audience commented that it was the best panel of this kind they had experienced.

Other speakers were Nigel Bevan of Serco (chair), Carol Barnum of Southern Polytechnic State University, Gilbert Cockton from University of Sunderland, Rolf Molich of DialogDesign, Jared Spool of User Interface Engineering, and Dennis Wixon from Microsoft. Jakob Nielsen, though billed, was not present to defend his five user assertion, having had to pull out of CHI due to a health problem.

User Interviews - Analysis Simplified
Source: Webcredible, 6 January 2009
 
You’ve conducted your user interviews, but now you need to make sense of all that information you’ve gathered.

Why Products Fail
Source: ComputerWorld, 5 January 2009
 
Most gadget and software makers don't understand what users want most: control.

How to Design Websites for Mobile Phones
Source: stemkoski.com, 3 January 2009
 
Tips from Ryan Stemkoski's web design blog.

Pioneer of Cyberspace honoured
Source: BBC, 2 January 2009
 
A professor who invented a forerunner of the world wide web has been made a dame in the New Year Honours.

2008 in Review: Developments that rocked the world of User Experience
Source: Catalyst Resources, 1 January 2009
 
A look back at 2008 highlights some of the key developments that rocked the world of user experience.

2008: The Year Online
Source: MIT Technology Review, 31 December 2008
 
The business of social networking, cloud computing, and a flaw in the fabric of the Internet top the most notable stories of 2008.

Shoveling through the Spamalanche
Source: UN, 30 December 2008
 
A ‘Spamalanche’ of 3,000 emails will be waiting in your inbox by the time you get back to work. What can you do besides 'delete all'?

New guidelines boost Web Access
Source: BBC, 29 December 2008
 
The World Wide Web Consortium (W3C) has announced a new standard to make sites more accessible to older and disabled people.

Measuring Emotional Engagement
Source: UN, 27 December 2008
 
New applications of electroencephalography (EEG) provide the means to gather detailed information on a user’s emotional relationship with a brand or service.

UXalliance thrives on Research strengths
Source: UN, 24 December 2008
 
The UXalliance provides clients with access to over 220 user experience professionals and 40 test labs worldwide.