Caroline's Corner: One Facilitator good, four Facilitators better?

I’m a lone consultant, and occasionally I get lucky: I persuade a client that it would be great if we could have a trained observer/logger helping me to conduct the usability test. That’s about as elaborate as it gets.

But the other day an intriguing question came up. Someone had joined an organisation where there is big pressure for numbers and short time-scales. So they had been conducting tests with three or four facilitators, 35 participants – and then spending an hour together to combine results. She thought that wasn’t a great way to do it and was looking for advice on something better.

Now that’s way outside my experience (35 participants? Heck, I’m lucky to get 8) so for once I didn’t pipe up with advice. But Carolyn Snyder did, and her advice struck me as so helpful that I asked her if I could base this month’s Corner on it. So here we go, over to Carolyn:

>CUT DOWN THE NUMBERS
My first reaction is that those numbers are overkill, and they're leaving a lot of data unanalyzed. It's inefficient. Try convincing them to cut down to a maximum of 4 facilitators and 20 tests (which is still a lot, but pick whatever number you think they'll accept in the short term). That will make time for some activities that'll help them to co-ordinate what they are doing.

>PILOT TESTING
All the facilitators should do at least one pilot test together, with one of them (or some other "friendly") playing the role of user. They should discuss things like, 'Do we want to start this task from the home page, or from where the user left off?' and 'Do our task instructions contain any inadvertent clues?'

>Caveat: Don't prematurely decide where the problems are; the focus is on covering the areas of interest and removing the most egregious sources of bias.

>INDEPENDENT ANALYSIS FIRST
Have each facilitator spend an hour going over their own data to pick out the key issues, before they get together as a group. What I typically do is have people write their top 10-20 issues on index cards and bring them to the meeting, where we do an affinity diagram. This technique reduces the risk that the analysis meeting is dominated by one or two individuals.

>METHODOLOGY DEBRIEF
After the group analysis, have each person identify a handful of issues that were found by other facilitators (which is easy when they're written on cards), and think about why s/he didn't find them. Some reasons have little to do with facilitation, eg 'we didn't get to that task,' or 'that user knew something the others didn't'. But the things you want to surface are those pertaining to the facilitator, eg 'I gave the user a hint,' or 'It didn't occur to me to ask that question'. Have them talk about those things, and also things they did that seemed to work well.

>Caveat: Don't let facilitators criticise each other's methods. Assure them that even the most experienced usability specialists don't always agree (for evidence see the CUE reports, available at http://www.dialogdesign.dk/cue.html), and the goal is to learn the others' methods and facilitation style, and adopt successful "tricks."

>KEEP WORKING AT IT
Over time, the result will be a smaller amount of data in a more similar format, simply because the facilitators will become more like each other. Then the group can move toward further standardising their data collection/analysis if there is a need. (They may also eventually realise that their methods are still too redundant, and cut down further on the number of tests or facilitators.)

AND A NOTE FROM CAROLINE
Thanks, Carolyn. Although I can’t quite envisage running tests with so many users and so many facilitators, I know these tips will be helpful for mentoring teams where their usability testing practice has maybe drifted a little and they’re having trouble deciding how to bring themselves back in line with each other.

Carolyn Snyder is a usability consultant, and author of 'Paper Prototyping'. Her website is at http://www.snyderconsulting.net/.

If you have any comments or suggestions about this article then please contact Caroline Jarrett at:

Caroline.Jarrett@Effortmark.co.uk

Caroline Jarrett is a usability consultant specialising in forms, questionnaires and data capture.

© 2005 Caroline Jarrett and Carolyn Snyder, all rights reserved.

Associated Link: Effortmark

'Internet addiction' linked to Depression
Source: BBC, 9 February 2010
 
There is a strong link between heavy internet use and depression, UK psychologists have said.

Could *You* be more Usable?
Source: UN, 8 February 2010
 
Bet you could.

Stowe Boyd on 'Steampunk' thinking about the Future of Computing
Source: Stowe Boyd's blog via Experientia, 6 February 2010
 
Are established metaphors of user experience holding us back from new ways of structuring our interaction through computers?

Nokia's User Experience Programme
Source: UN, 5 February 2010
 
Nokia has put together a rich and informative website covering the key elements of user experience.

Interfaces magazine: latest issue available now
Source: HCI News Service, 4 February 2010
 
The latest issue of Interfaces is now available in pdf format, free from the Interaction Website.

A Lighter Brigade of Chargers
Source: UN, 3 February 2010
 
Lots of gadgets, one charger. At last.

Mobile Touch Screens could soon Feel the Pressure
Source: MIT Technology Review, 2 February 2010
 
A quantum switch could add pressure sensing to mobile screens.

Usability, Usability, Usability: why the iPad will Succeed
Source: Econsultancy, 1 February 2010
 
The tech critics love it, hate it, love it again, shrug it off. What do usability experts say?

British Airways - at last some good news
Source: Loop11, 30 January 2010
 
In a recent website usability study for the world's leading airlines, the British Airways website proved to be the most user friendly, with Malaysia Airlines and Virgin Atlantic having the lowest user experience rating.

Computation of Emotions in Man and Machine
Source: Royal Society, 29 January 2010
 
Advances in computer technology now allow machines to recognise and express emotions, paving the way for improved human-computer and human-human communications.