Feature: Choosing Your Users

Extract from "User Interface Design and Evaluation" by Debbie Stone, Caroline Jarrett, Mark Woodroffe and Shailey Minocha. Printed with permission from Morgan Kaufmann, a division of Elsevier. Copyright 2005. (Certain diagrams and tables and references to these have been removed for ease of presentation in the UN format.)


For a system where the typical users are likely to use the system on their own, each evaluation session usually has a single participant using the system on his or her own. To get a variety of views, the session is repeated with further participants. Perhaps surprisingly, usability practitioners find that they learn so much from the first few participants that about five participants are often enough.

Selecting your participants is important. Ideally, each participant will be a real user, though it is sometimes helpful to bring in a representative user (based on the user profile) or a usability or domain expert instead. Selecting participants completely at random is unlikely to be the best choice.

EXERCISE (Allow five minutes)
Why might it be advantageous to involve a usability expert in evaluation?
DISCUSSION
Usability experts are trained to understand usability issues and how to solve them, so they may be able to identify common mistakes more quickly than real users. However, the overall aim is to ensure that real users can use the system, not that usability experts approve of it.

The other factor that determines the choice of participants is availability. Sometimes you may need the help of external agencies to recruit representative users or to hire a usability expert. These external services can be expensive, and they will usually require you to give a financial incentive to the participants.

Here are some points to consider when choosing your participants for the evaluation:
• Who is a real user, and when is it acceptable to have someone else do your testing?
• Should you have one participant at a time, or would it be better for them to work in pairs?
• How many participants do you need?

Who Is a Real User?

Examples of Participants for Evaluating an Early Prototype:
* Public information kiosk for tourist information, meant for a full range of the general public, including tourists.

Actual users, but all of whom speak English for the first round so that it is easier for us to who do not speak English communicate with them. This should allow us to fix the most obvious errors, and we can get some non-English-speaking users for the second round.

* Safety critical system for monitoring a car assembly line, intended for plant supervisors with considerable experience of the job.

Actual users, because the domain knowledge of the actual users is so important.

* Website offering counselling to support people who have recently been bereaved or people who were bereaved some time ago.

User representatives, bereavement counsellors.

The list above has some examples of participants for evaluating an early prototype. Notice that in one case the choice of users is narrower than the actual real users, and in another it would be better to choose a different user group than the actual users.

Your aim in recruiting participants is to find users who reflect the different skills, domain knowledge, and system experience of the users you described during requirements gathering. Often, you have to recruit whoever is available and then ask about their backgrounds and skills. Even if you got the sample you aimed for, you should consider whether the participants in your evaluation are typical users, because there is usually a certain amount of variability between users.

EXERCISE (Allow five minutes)
You are evaluating a web site for French students visiting the United States. Consider the following three participants:
• A student from a French university, who happens to be American
• A French student who lived in the United States as a child and is now visiting the country again
• A recent graduate of a French university, visiting the United States for the first time
Should you give equal weight to all their views or pay more attention to one participant?
DISCUSSION
Although the recent graduate is not currently a student, and therefore falls outside the strict definition of the user profile, we would usually give more weight to that person’s opinion than the first two participants. The first two participants would know much more about the United States than a typical user.

Users Working Alone or in Pairs

User observation is usually based on a single user working alone, because most current computer systems are intended for such users. However, sometimes you should consider recruiting users to work as a pair. Consider the following situations:
• The users usually work cooperatively. For example, some very small businesses still ask staff to share a computer.
• Cultural constraints make it difficult for users to be critical of an interface to someone in authority. For example, some Japanese usability practitioners find that pairs of users working together find it easier to discuss the interface objectively with each other than with an evaluator because of the politeness in Japanese culture. Carol Snyder who does nearly all of her testing with American participants, uses this as her routine approach, as she finds that participants chat to each other more easily than to a facilitator.
• You observe that your users prefer to work in pairs. For example, when one of us was testing business administration products intended for self-employed builders working as single-person businesses, we found that it was often the case that the builder was a married man and his wife helped him by doing the business administration. So it worked best if both of them came along to the test.

There are also times when you should consider recruiting a helper or user advocate to work alongside the participant, for example:
• If the users are children, think about whether you are likely to need a parent, teacher, or other responsible adult who is known to the child to be present during the evaluation.
• If the participant speaks a language other than English, you will need an interpreter. Some practitioners consider it acceptable to run usability tests in a foreign language if you are reasonably fluent in that language. Others consider that if you are not a native speaker of the language, you are likely to miss nuances in the participants’ reactions and therefore bias the test.
• If your participant has a speech impairment or a learning or cognitive disability that affects speech or understanding, then you may have difficulty understanding his or her comments or explaining what you want the participant to do. You may want to recruit a helper or ask the participant to bring a helper or interpreter.

Evaluation Tip:
Speak to the participant, not the interpreter: when working with any advocate, helper, or interpreter, make sure that you address your remarks to the participant, not to the intermediary.

Number of Participants

You are likely to need only a few users, particularly if you are doing an evaluation at an early stage in the development of your interface. If one participant spots the problem, how many more participants do you need to tell you about the problem? If no other participant notices the problem, would you ignore it or put it on the list of defects for consideration?

You are testing to find problems and do something about them, so it becomes frustrating when participant after participant finds the same problems. Usability practitioners tend to do many rapid evaluation/redesign cycles with very few users each time, particularly when the interface needs a lot of work. The following is a usability practitioner’s comments on recruiting users:

Recruiting Five Users: Comments from a Practitioner
If the user profile is reasonably homogeneous, then I aim to recruit five users. I find that about one person in five fails to turn up: they forget, or have a work or family crisis, or simply change their mind. Also, about one person in five seems to be outside my user profile due to some misunderstanding in the recruitment process: someone failed to ask the right questions, or the participant did not quite understand. That leaves me with about three users who are right in the target area, and this seems to be enough in practice to get a flavor for the variety of user experiences. If I have two really contrasting user groups, then I aim for five users from both groups. Sometimes the first user points out so many problems that the client wants to stop the test right there to work on them. It really is worth persisting with a few more users: they do add extra information. But if you have five or more users trying the same task on the same interface, you will keep coming across the same problems and it gets harder and harder to be truly interested and surprised when you are getting the same information from each one.
From Caroline Jarrett, 2001, private communication; used with permission.


It is important to realize that a failure to find problems does not imply that the interface is usable. If you have one or two participants and they do not find any problems, then you cannot conclude that your interface is acceptable for users in general. All you know is that these users with these tasks found that your interface worked.

Try again, aiming to recruit a different sort of user (maybe a novice rather than an experienced user or vice versa). It can also help if you ask users to attempt a different set of tasks.

The number of participants required for a usability test has been a topic of lively discussion. Our view is that you should start with five, and then test with more participants if you have more time and resources available and feel that the data gathered so far are insufficient for helping you to decide what to do next.

Offering Incentives

It is usual to offer some sort of thanks to the participants for their time and trouble. In a very informal setting, such as asking a colleague, then just saying 'thank you' may be sufficient. However, if the evaluation is even slightly more formal, then prepare a letter recording your thanks, and confirm the confidentiality of the evaluation and the use you will make of the data. If the participants come from a work setting (such as a colleague), then it is sometimes even more acceptable to send an extra thank-you letter to their managers. Make sure that you have the appropriate management permissions for the time that the participants will spend on your evaluation.

A suggested code of conduct from the Usability Professionals’ Association (Ballman, 2001) mentioned the following:
Participants should ordinarily be compensated for their participation except in instances where their employer forbids such compensation, or where compensation could bias the data collected.

If at all possible, you should pay participants if they have out-of-pocket expenses, such as for travel, if they are students or otherwise on a low income, or if they might lose income because of being away from work. If the evaluation is part of your work, you should try to arrange some payment or reward. If a market research organization recruits for you, then the organization will tell you how much is necessary. If you are working on a product that your participants would like to own, then giving them a sample could be an appropriate incentive.

Even for professional work, participants do not always expect an incentive. It is nice though to offer a token gift, such as a potted plant, but consider whether it will be awkward for the participant to carry away. If you plan to offer food, such as chocolates, or drink, such as a bottle of wine, then be extra careful about the culture and views of your organization and your participants. For example, a bottle of wine is generally acceptable in the United Kingdom or Australia but frequently unacceptable in the United States.

For more information on this title and other similiar books, please visit www.books.elsevier.com/computing.

'Internet addiction' linked to Depression
Source: BBC, 9 February 2010
 
There is a strong link between heavy internet use and depression, UK psychologists have said.

Could *You* be more Usable?
Source: UN, 8 February 2010
 
Bet you could.

Stowe Boyd on 'Steampunk' thinking about the Future of Computing
Source: Stowe Boyd's blog via Experientia, 6 February 2010
 
Are established metaphors of user experience holding us back from new ways of structuring our interaction through computers?

Nokia's User Experience Programme
Source: UN, 5 February 2010
 
Nokia has put together a rich and informative website covering the key elements of user experience.

Interfaces magazine: latest issue available now
Source: HCI News Service, 4 February 2010
 
The latest issue of Interfaces is now available in pdf format, free from the Interaction Website.

A Lighter Brigade of Chargers
Source: UN, 3 February 2010
 
Lots of gadgets, one charger. At last.

Mobile Touch Screens could soon Feel the Pressure
Source: MIT Technology Review, 2 February 2010
 
A quantum switch could add pressure sensing to mobile screens.

Usability, Usability, Usability: why the iPad will Succeed
Source: Econsultancy, 1 February 2010
 
The tech critics love it, hate it, love it again, shrug it off. What do usability experts say?

British Airways - at last some good news
Source: Loop11, 30 January 2010
 
In a recent website usability study for the world's leading airlines, the British Airways website proved to be the most user friendly, with Malaysia Airlines and Virgin Atlantic having the lowest user experience rating.

Computation of Emotions in Man and Machine
Source: Royal Society, 29 January 2010
 
Advances in computer technology now allow machines to recognise and express emotions, paving the way for improved human-computer and human-human communications.