Caroline's Corner: One Facilitator good, four Facilitators better?

I’m a lone consultant, and occasionally I get lucky: I persuade a client that it would be great if we could have a trained observer/logger helping me to conduct the usability test. That’s about as elaborate as it gets.

But the other day an intriguing question came up. Someone had joined an organisation where there is big pressure for numbers and short time-scales. So they had been conducting tests with three or four facilitators, 35 participants – and then spending an hour together to combine results. She thought that wasn’t a great way to do it and was looking for advice on something better.

Now that’s way outside my experience (35 participants? Heck, I’m lucky to get 8) so for once I didn’t pipe up with advice. But Carolyn Snyder did, and her advice struck me as so helpful that I asked her if I could base this month’s Corner on it. So here we go, over to Carolyn:

>CUT DOWN THE NUMBERS
My first reaction is that those numbers are overkill, and they're leaving a lot of data unanalyzed. It's inefficient. Try convincing them to cut down to a maximum of 4 facilitators and 20 tests (which is still a lot, but pick whatever number you think they'll accept in the short term). That will make time for some activities that'll help them to co-ordinate what they are doing.

>PILOT TESTING
All the facilitators should do at least one pilot test together, with one of them (or some other "friendly") playing the role of user. They should discuss things like, 'Do we want to start this task from the home page, or from where the user left off?' and 'Do our task instructions contain any inadvertent clues?'

>Caveat: Don't prematurely decide where the problems are; the focus is on covering the areas of interest and removing the most egregious sources of bias.

>INDEPENDENT ANALYSIS FIRST
Have each facilitator spend an hour going over their own data to pick out the key issues, before they get together as a group. What I typically do is have people write their top 10-20 issues on index cards and bring them to the meeting, where we do an affinity diagram. This technique reduces the risk that the analysis meeting is dominated by one or two individuals.

>METHODOLOGY DEBRIEF
After the group analysis, have each person identify a handful of issues that were found by other facilitators (which is easy when they're written on cards), and think about why s/he didn't find them. Some reasons have little to do with facilitation, eg 'we didn't get to that task,' or 'that user knew something the others didn't'. But the things you want to surface are those pertaining to the facilitator, eg 'I gave the user a hint,' or 'It didn't occur to me to ask that question'. Have them talk about those things, and also things they did that seemed to work well.

>Caveat: Don't let facilitators criticise each other's methods. Assure them that even the most experienced usability specialists don't always agree (for evidence see the CUE reports, available at http://www.dialogdesign.dk/cue.html), and the goal is to learn the others' methods and facilitation style, and adopt successful "tricks."

>KEEP WORKING AT IT
Over time, the result will be a smaller amount of data in a more similar format, simply because the facilitators will become more like each other. Then the group can move toward further standardising their data collection/analysis if there is a need. (They may also eventually realise that their methods are still too redundant, and cut down further on the number of tests or facilitators.)

AND A NOTE FROM CAROLINE
Thanks, Carolyn. Although I can’t quite envisage running tests with so many users and so many facilitators, I know these tips will be helpful for mentoring teams where their usability testing practice has maybe drifted a little and they’re having trouble deciding how to bring themselves back in line with each other.

Carolyn Snyder is a usability consultant, and author of 'Paper Prototyping'. Her website is at http://www.snyderconsulting.net/.

If you have any comments or suggestions about this article then please contact Caroline Jarrett at:

Caroline.Jarrett@Effortmark.co.uk

Caroline Jarrett is a usability consultant specialising in forms, questionnaires and data capture.

© 2005 Caroline Jarrett and Carolyn Snyder, all rights reserved.

Associated Link: Effortmark

Poor Shopping Basket Usability will hinder Christmas Rush
Source: UN, 20 November 2008
 
A new study from eDigitalResearch identifies customer service and online shopping basket usability as the two key areas where online retailers need to improve.

Future Phones to Read Your Voice, Gestures
Source: Wired, 19 November 2008
 
Five years from now, it is likely that the mobile phone you will be holding will be a smooth, sleek brick — a piece of metal and plastic with a few grooves in it and little more.

Why Digital Research is important in tough Financial Times
Source: Financial Times, 18 November 2008
 
With the banking sector moving towards consolidation, it is crucial that customers are understood, reacted to and rewarded for their loyalty.

Get Ready for 'Ergobamanomics'
Source: AssemblyBlog via ergonomics in the news , 17 November 2008
 
Is President Elect, Barack Obama, a secret ergonomics and usability enthusiast?

The Most Dreaded Keyword Phrase
Source: SearchEngineLand, 15 November 2008
 
In website usability, one of the hardest obstacles to overcome is the mentality of “This is what I would do.”

User Experience Standards Missing from Web 2.0 Designs
Source: UN, 14 November 2008
 
Usability professionals should be more involved with the development of Web 2.0 tools.

New technology showcased at World Usability Day
Source: User Vision, 13 November 2008
 
Today is World Usability Day!

Review: 16 User Interface Prototyping Tools
Source: Dexo Design, 12 November 2008
 
People constantly ask me what the best prototyping or mockup tool is, so I decided to do a review of all the tools I’m aware of.

Radio 4's PM show announces Winner of 'Show Us a Better Way'
Source: BBC, 11 November 2008
 
Ever been frustrated that you can't find out something that ought to be easy to find? Ever been baffled by league tables or 'performance indicators'?

Cisco Systems extends Product Usability Research
Source: Catalyst Resources, 10 November 2008
 
Cisco is conducting extensive usability research with Catalyst Resources, into complex new technologies.