Caroline's Corner: Playing Piggy in the Middle

Questionnaires often ask us to rate something or other. Recently, I've been asked about:

- my satisfaction with a huge website

- the effectiveness of a selection of ways to maintain or increase charge-out rates

- the cleanliness of a hotel room.

My problem? I didn't know, didn't care or didn't remember. Sometimes all three. So I skipped the question or junked the questionnaire.

There's a lesson here for choosing the 'correct' number of points in a rating scale. (I'm assuming that you need to offer a number of discrete points rather than offering some widget that allows a continuously variable scale).

Some people take the view that you should have an even number of points. This encourages your respondent to move off the fence and plump for a positive or negative rating.

I disagree. When I've done usability tests of questionnaires, I've found that users have many reasons for choosing the middle point. As well as 'don't know', 'don't care' and 'don't remember', users have told me that they chose the middle point because:

- they didn't understand the question

- they sometimes felt positive and sometimes felt negative

- they mostly felt quite positive but occasionally felt very negative (or vice versa)

- they felt positive about some aspect but negative about another

- or, (and could this one be particularly British) they felt that it wasn't up to them to express an opinion on this item.

So I take the view that you can't force someone into a positive or negative opinion simply by taking away their middle point. They still have their opinion, and will express it in one of these ways:

- skip the question

- scrap the questionnaire

- split their votes: choose either mildly positive or mildly negative, at random, and make it stand for a middle point.

In effect, the 'split' tactic means that scales with an even number of points lose two of those points to the middle.

You can sometimes ease respondents away from clinging to the middle point if you offer a 'don't know' or 'not applicable' option. The problem with these extra options is: where to put them? If they are close to the main rating points then they look like extra points in the scale and get confused with it. If they are separated from them, they get missed.

So, it's an odd number of points for me. I've met some user resistance to 3-point scales as they want to express milder and stronger shades of opinion, so I generally opt for 5-point scales. But when I'm testing the questionnaire I hear things like "oh, I'd never give a 5" (when 5 is the best mark) or "I like to be generous so I'll tick 5" - contrasting strategies even though the expressed level of satisfaction is the same. If interrogated closely, I'll admit that when analysing data for myself I sometimes, bizarrely and irrationally, treat the points as if they expressed concrete difference and try to do manipulations like working out the average score out of 5. A temptation that I resist when working with client data. Then, I do the sensible thing and group the two positive points together for analysis and the two negative points similarly. I've noticed that market researchers often do the same, and will report 'top two boxes' together. They're not being over-optimistic, simply reflecting the lack of difference in the opinions expressed in the two boxes.

I've never tried 7- or 9-point scales myself. I've seen them used in questionnaires on psychological topics where, presumably, the experimenter has decided that the subjects really do want to express fine shades of meaning. In the more mundane world of the everyday satisfaction questionnaire, respondents' time is too valuable and any theoretical benefit from the extra points is certainly offset by the possibility that respondents might be put off by the extra work entailed and by the challenge of deciding how to group the results.

As in most things in usability "your mileage may vary". Let me know if you've got views that differ from mine.

As usual, Caroline welcomes any comments or suggestions about this article.

Caroline Jarrett
Independent usability consultant specialising in forms, questionnaires and data capture.

Effortmark Ltd
w. www.effortmark.co.uk
e. caroline.jarrett@effortmark.co.uk
p. +44 (0)1525 370379

Associated Link: Effortmark

All change at the top for System Concepts
Source: System Concepts Ltd, 3 July 2009
 
Leslie Fountain has been promoted to joint Managing Director of leading usability consultancy System Concepts.

Life in UCD immortalised in fiction: you couldn't make it up
Source: UN, 2 July 2009
 
Sarah Herman's fictitious book on life in a user-centred design company has hit the shelves and The Guardian's book pages...

Interfaces Magazine - Issue 79: The Education Issue
Source: Interaction Group, 1 July 2009
 
The latest issue of Interfaces is now available as a free download from the Interaction Website.

Two new Behavioural research Tools from Noldus
Source: UN, 30 June 2009
 
Tool updates make on-site behavioural data collection easier.

Cell Phones that Listen and Learn
Source: MIT Technology Review, 29 June 2009
 
New software tracks a user's behavior by monitoring everyday sounds.

Top Six Don’ts for Usability Testing
Source: FutureNow Inc., 27 June 2009
 
Six tips for creating quality usability tests to ensure useful feedback from testers.

Usability: ‘Lovely software. But I can’t work it’
Source: FT.com, 26 June 2009
 
In a recent survey by Global Graphics, 77 per cent of office workers estimate they lose up to one hour a week because business software is difficult to use.

And what do you do?
Source: Dexo Design, 25 June 2009
 
How do you describe your job role? Here are the results of a recent 'Preferred UX/UI Title' Poll.

Most Doctors cite Usability as critical to Electronic Health Record Adoption
Source: TMCNet, 24 June 2009
 
It's all about 'meaningful use'.

Glossy monitors look good but can hurt
Source: QUT, 23 June 2009
 
A new advisory cites research which suggests high gloss monitors make users sit awkwardly.