Caroline's Corner: One Facilitator good, four Facilitators better?

I’m a lone consultant, and occasionally I get lucky: I persuade a client that it would be great if we could have a trained observer/logger helping me to conduct the usability test. That’s about as elaborate as it gets.

But the other day an intriguing question came up. Someone had joined an organisation where there is big pressure for numbers and short time-scales. So they had been conducting tests with three or four facilitators, 35 participants – and then spending an hour together to combine results. She thought that wasn’t a great way to do it and was looking for advice on something better.

Now that’s way outside my experience (35 participants? Heck, I’m lucky to get 8) so for once I didn’t pipe up with advice. But Carolyn Snyder did, and her advice struck me as so helpful that I asked her if I could base this month’s Corner on it. So here we go, over to Carolyn:

>CUT DOWN THE NUMBERS
My first reaction is that those numbers are overkill, and they're leaving a lot of data unanalyzed. It's inefficient. Try convincing them to cut down to a maximum of 4 facilitators and 20 tests (which is still a lot, but pick whatever number you think they'll accept in the short term). That will make time for some activities that'll help them to co-ordinate what they are doing.

>PILOT TESTING
All the facilitators should do at least one pilot test together, with one of them (or some other "friendly") playing the role of user. They should discuss things like, 'Do we want to start this task from the home page, or from where the user left off?' and 'Do our task instructions contain any inadvertent clues?'

>Caveat: Don't prematurely decide where the problems are; the focus is on covering the areas of interest and removing the most egregious sources of bias.

>INDEPENDENT ANALYSIS FIRST
Have each facilitator spend an hour going over their own data to pick out the key issues, before they get together as a group. What I typically do is have people write their top 10-20 issues on index cards and bring them to the meeting, where we do an affinity diagram. This technique reduces the risk that the analysis meeting is dominated by one or two individuals.

>METHODOLOGY DEBRIEF
After the group analysis, have each person identify a handful of issues that were found by other facilitators (which is easy when they're written on cards), and think about why s/he didn't find them. Some reasons have little to do with facilitation, eg 'we didn't get to that task,' or 'that user knew something the others didn't'. But the things you want to surface are those pertaining to the facilitator, eg 'I gave the user a hint,' or 'It didn't occur to me to ask that question'. Have them talk about those things, and also things they did that seemed to work well.

>Caveat: Don't let facilitators criticise each other's methods. Assure them that even the most experienced usability specialists don't always agree (for evidence see the CUE reports, available at http://www.dialogdesign.dk/cue.html), and the goal is to learn the others' methods and facilitation style, and adopt successful "tricks."

>KEEP WORKING AT IT
Over time, the result will be a smaller amount of data in a more similar format, simply because the facilitators will become more like each other. Then the group can move toward further standardising their data collection/analysis if there is a need. (They may also eventually realise that their methods are still too redundant, and cut down further on the number of tests or facilitators.)

AND A NOTE FROM CAROLINE
Thanks, Carolyn. Although I can’t quite envisage running tests with so many users and so many facilitators, I know these tips will be helpful for mentoring teams where their usability testing practice has maybe drifted a little and they’re having trouble deciding how to bring themselves back in line with each other.

Carolyn Snyder is a usability consultant, and author of 'Paper Prototyping'. Her website is at http://www.snyderconsulting.net/.

If you have any comments or suggestions about this article then please contact Caroline Jarrett at:

Caroline.Jarrett@Effortmark.co.uk

Caroline Jarrett is a usability consultant specialising in forms, questionnaires and data capture.

© 2005 Caroline Jarrett and Carolyn Snyder, all rights reserved.

Associated Link: Effortmark

All change at the top for System Concepts
Source: System Concepts Ltd, 3 July 2009
 
Leslie Fountain has been promoted to joint Managing Director of leading usability consultancy System Concepts.

Life in UCD immortalised in fiction: you couldn't make it up
Source: UN, 2 July 2009
 
Sarah Herman's fictitious book on life in a user-centred design company has hit the shelves and The Guardian's book pages...

Interfaces Magazine - Issue 79: The Education Issue
Source: Interaction Group, 1 July 2009
 
The latest issue of Interfaces is now available as a free download from the Interaction Website.

Two new Behavioural research Tools from Noldus
Source: UN, 30 June 2009
 
Tool updates make on-site behavioural data collection easier.

Cell Phones that Listen and Learn
Source: MIT Technology Review, 29 June 2009
 
New software tracks a user's behavior by monitoring everyday sounds.

Top Six Don’ts for Usability Testing
Source: FutureNow Inc., 27 June 2009
 
Six tips for creating quality usability tests to ensure useful feedback from testers.

Usability: ‘Lovely software. But I can’t work it’
Source: FT.com, 26 June 2009
 
In a recent survey by Global Graphics, 77 per cent of office workers estimate they lose up to one hour a week because business software is difficult to use.

And what do you do?
Source: Dexo Design, 25 June 2009
 
How do you describe your job role? Here are the results of a recent 'Preferred UX/UI Title' Poll.

Most Doctors cite Usability as critical to Electronic Health Record Adoption
Source: TMCNet, 24 June 2009
 
It's all about 'meaningful use'.

Glossy monitors look good but can hurt
Source: QUT, 23 June 2009
 
A new advisory cites research which suggests high gloss monitors make users sit awkwardly.