Caroline's Corner: Survey Response Rates? 2% is not Good Enough

In the 1950s, a well-designed survey could often achieve over 90% response rates. Since then, response rates have consistently declined.

But I was still a bit shocked the other day when a post on a usability discussion group quoted a ‘typical response rate of 2%’ as if that were something we all knew as a fact.

THE STATS ARE A PROBLEM
2% is a terrible response rate. Why? Because there’s such a big chance that the people who filled in the survey are different from the people who didn’t.

With any survey, you need to look at the profile of the people who responded and satisfy yourself that they are about the same as the people who didn’t respond – and also, that they’re about the same as the overall population that you’re sampling.

So we try really hard to design a sampling method that gives everyone in the population an equal chance of being selected as a potential respondent. Then that means that our sample is about the same as the overall population.

But if we have a poor level of response, we make it almost certain that there will be some important differences between those who responded and those who didn’t. The assumption that our sample reflects the population as a whole gets blown away – and with it, the possibility of doing any inferential statistics.

Inferential statistics? Making claims that the results from the survey have reflect the results that we might have got from the population as a whole.

Now here is where a bit of opinion and judgement comes in. One side of the argument goes like this:
- the propensity to answer questions is an independent variable;
- we have no reason to suppose that people who like to answer questions are different from our population in any other respect;
- therefore, it doesn’t matter all that much if we get a low response rate.

My opinion is different. I take the view that if our respondents are only 2% of our sample then we already know that they are very unusual. The chances that they are unusual in many different ways are high. So that’s how I arrive at the view that 2% is a terrible response rate and blows away the rest of our statistics.

WHERE DOES THE 2% ‘FACT’ COME FROM?
So where does this ‘fact’ about typical response rates come from? I believe that there are two influences:

1. Almost any survey can achieve a 1% to 2% response rate. It seems that most populations have around 1% to 2% of people who just really enjoy answering questions. They can’t resist this strange pleasure, and they’ll answer anything. But do you want to base your business decisions on them?

2. This sort of rate actually is typical for a full-page advertisement in a weekend supplement – and not all that unusual for cold-calling direct mail (http://www.fundraising.co.uk/forum/thread.php?id=2263). The difference is that in advertising and in direct mail, we don’t make any assumptions about the people who don’t respond. We’re too busy concentrating on the people who do respond.

WHAT CAN YOU DO TO IMPROVE YOUR RESPONSE RATE?
Here are some tips for improving your response rate.

1. Ask fewer people. Choose a small sample, make sure that those people know that they have been specially selected, and spend a bit of time and effort on following up with each of them. Feeling special makes people more likely to respond.

2. Ditch the prize draw and use the money for an incentive that they get _before_ filling in the questionnaire. A dollar bill sent with a mail survey gets a better response rate than ten dollars guaranteed on returning the survey. Prize draws have little or no effect on response rates.

3. Make the questionnaire SHORT (yup, I’m yelling ‘short’ at you). Longer = more offputting.

4. Make the questionnaire interesting. You may even have to resort to a little humour (but be careful when you test it).

5. Test, test and test again. That’s how you’ll find out whether it’s short enough and interesting enough to get a response.

6. Read a good book on the topic. I recommend Don Dillman (2000) "Mail and Internet Surveys: The Tailored Design Method". Then do what he says.

WHAT CAN YOU DO WITH THE DATA IF YOU GET A LOW RESPONSE RATE?
If fear that you will get a poor response rate no matter what you do, then you can extract some value from the data by reducing your survey to a few open questions. Read the comments that people give you and think about them. You can’t tell whether they are at all representative of your population, but you’ll probably find that they do offer some interesting insights.

DEFINITIONS
Survey: the entire process of defining a research question, developing a questionnaire that explores the question, finding a suitable sample of potential respondents, administering the questionnaire and then doing something with the responses.

Questionnaire: a series of questions, often with fixed spaces or choices for answers. Can be offered on paper, as an interview, or electronically.

Response rate: the percentage of people approached who respond with a filled-in questionnaire.

FUTHER READING
The Council of American Survey Organisations has a discussion on the definition of response rates at http://www.casro.org/resprates.cfm.

ACKNOWLEDGEMENT
Thanks to Jurek Kirakowski for many discussions on this and related topics. The opinions expressed in this article are my own.

If you have any comments or suggestions about this article then please contact Caroline at:

Caroline.Jarrett@Effortmark.co.uk

Caroline Jarrett is a usability consultant specialising in forms, questionnaires and data capture.

© 2005 Caroline Jarrett, all rights reserved.

Associated Link: Effortmark

Study: Mobile Web sites need Improvement
Source: CNET, 13 October 2008
 
A new study suggests that the usability of many mobile Web sites still needs improvement.

Better Usability makes Online Backup more Appealing
Source: UN, 11 October 2008
 
Repeat after me: "Backup via the Internet is for everybody."

IBM Task Modeler version 6 now Available
Source: Unknown, 10 October 2008
 
The latest release of IBM Task Modeler is now available for download. This includes many new features to support efficient and collaborative task analysis and synthesis.

Caroline's Corner: Design to Read - Designing for People who do not Read easily
Source: Caroline Jarrett, 9 October 2008
 
Reading is a skill many of us take for granted. Chances are that if you’re reading this, you’re one of the lucky ones who read easily.

Bill Buxton to change Microsoft from within
Source: The Register, 8 October 2008
 
"Last year Steve Ballmer said to 85,000 employees 'If you don’t change and you don’t go in this direction, we’re dead, and I don’t want to die.' I wanted to go hug him when he said that."

New Report: Innovative Marketing in the Digital World
Source: E-consultancy, 7 October 2008
 
A new report from E-consultancy provides a distillation of ideas and suggestions provided by digital marketing experts.

Linking Industry with Academia
Source: unknown, 6 October 2008
 
Customer experience research consultancy, fhios, based in London, has just launched an innovative internship programme, aimed primarily at postgraduate students.

Online Travel Booking: What influences Consumers?
Source: Webcredible, 4 October 2008
 
What influences consumers when booking a holiday? How can travel companies offer the best user experience?

Getting the Word out
Source: UN, 3 October 2008
 
I SAID, I'M ON THE TRAIN... Voice recognition technology can now translate all those pesky garbled voicemails into manageable text messages.

Mobile hardware outpaces Software and User Capabilities
Source: ComputerWorld, 2 October 2008
 
The main issue dissuading users from employing all the capabilities of their mobile devices is the complexity involved in operating them.