Caroline's Corner: Playing Piggy in the Middle

Questionnaires often ask us to rate something or other. Recently, I've been asked about:

- my satisfaction with a huge website

- the effectiveness of a selection of ways to maintain or increase charge-out rates

- the cleanliness of a hotel room.

My problem? I didn't know, didn't care or didn't remember. Sometimes all three. So I skipped the question or junked the questionnaire.

There's a lesson here for choosing the 'correct' number of points in a rating scale. (I'm assuming that you need to offer a number of discrete points rather than offering some widget that allows a continuously variable scale).

Some people take the view that you should have an even number of points. This encourages your respondent to move off the fence and plump for a positive or negative rating.

I disagree. When I've done usability tests of questionnaires, I've found that users have many reasons for choosing the middle point. As well as 'don't know', 'don't care' and 'don't remember', users have told me that they chose the middle point because:

- they didn't understand the question

- they sometimes felt positive and sometimes felt negative

- they mostly felt quite positive but occasionally felt very negative (or vice versa)

- they felt positive about some aspect but negative about another

- or, (and could this one be particularly British) they felt that it wasn't up to them to express an opinion on this item.

So I take the view that you can't force someone into a positive or negative opinion simply by taking away their middle point. They still have their opinion, and will express it in one of these ways:

- skip the question

- scrap the questionnaire

- split their votes: choose either mildly positive or mildly negative, at random, and make it stand for a middle point.

In effect, the 'split' tactic means that scales with an even number of points lose two of those points to the middle.

You can sometimes ease respondents away from clinging to the middle point if you offer a 'don't know' or 'not applicable' option. The problem with these extra options is: where to put them? If they are close to the main rating points then they look like extra points in the scale and get confused with it. If they are separated from them, they get missed.

So, it's an odd number of points for me. I've met some user resistance to 3-point scales as they want to express milder and stronger shades of opinion, so I generally opt for 5-point scales. But when I'm testing the questionnaire I hear things like "oh, I'd never give a 5" (when 5 is the best mark) or "I like to be generous so I'll tick 5" - contrasting strategies even though the expressed level of satisfaction is the same. If interrogated closely, I'll admit that when analysing data for myself I sometimes, bizarrely and irrationally, treat the points as if they expressed concrete difference and try to do manipulations like working out the average score out of 5. A temptation that I resist when working with client data. Then, I do the sensible thing and group the two positive points together for analysis and the two negative points similarly. I've noticed that market researchers often do the same, and will report 'top two boxes' together. They're not being over-optimistic, simply reflecting the lack of difference in the opinions expressed in the two boxes.

I've never tried 7- or 9-point scales myself. I've seen them used in questionnaires on psychological topics where, presumably, the experimenter has decided that the subjects really do want to express fine shades of meaning. In the more mundane world of the everyday satisfaction questionnaire, respondents' time is too valuable and any theoretical benefit from the extra points is certainly offset by the possibility that respondents might be put off by the extra work entailed and by the challenge of deciding how to group the results.

As in most things in usability "your mileage may vary". Let me know if you've got views that differ from mine.

As usual, Caroline welcomes any comments or suggestions about this article.

Caroline Jarrett
Independent usability consultant specialising in forms, questionnaires and data capture.

Effortmark Ltd
w. www.effortmark.co.uk
e. caroline.jarrett@effortmark.co.uk
p. +44 (0)1525 370379

Associated Link: Effortmark

User Interviews - Analysis Simplified
Source: Webcredible, 6 January 2009
 
You’ve conducted your user interviews, but now you need to make sense of all that information you’ve gathered.

Why Products Fail
Source: ComputerWorld, 5 January 2009
 
Most gadget and software makers don't understand what users want most: control.

How to Design Websites for Mobile Phones
Source: stemkoski.com, 3 January 2009
 
Tips from Ryan Stemkoski's web design blog.

Pioneer of Cyberspace honoured
Source: BBC, 2 January 2009
 
A professor who invented a forerunner of the world wide web has been made a dame in the New Year Honours.

2008 in Review: Developments that rocked the world of User Experience
Source: Catalyst Resources, 1 January 2009
 
A look back at 2008 highlights some of the key developments that rocked the world of user experience.

2008: The Year Online
Source: MIT Technology Review, 31 December 2008
 
The business of social networking, cloud computing, and a flaw in the fabric of the Internet top the most notable stories of 2008.

Shoveling through the Spamalanche
Source: UN, 30 December 2008
 
A ‘Spamalanche’ of 3,000 emails will be waiting in your inbox by the time you get back to work. What can you do besides 'delete all'?

New guidelines boost Web Access
Source: BBC, 29 December 2008
 
The World Wide Web Consortium (W3C) has announced a new standard to make sites more accessible to older and disabled people.

Measuring Emotional Engagement
Source: UN, 27 December 2008
 
New applications of electroencephalography (EEG) provide the means to gather detailed information on a user’s emotional relationship with a brand or service.

UXalliance thrives on Research strengths
Source: UN, 24 December 2008
 
The UXalliance provides clients with access to over 220 user experience professionals and 40 test labs worldwide.