Caroline's Corner: Surveys - What is an Acceptable Response Rate?

It's been a while since I ranted on about response rates on surveys. In that article, I took the view that "2% is a terrible response rate" and had a few reasons why and tips for doing better. Recently, I've had a couple of challenging questions on the topic. So here goes at trying to answer them.

WHAT IS AN ACCEPTABLE RESPONSE RATE?
First of all, if a 2% response rate is no good: what should you aim for?

The answer is in two parts: first of all, the question of non-response bias and secondly, an actual percentage.

Non-response bias happens when the population who do not respond display characteristics that are different to the population who do respond.
If you've got a properly random sample, and the people who respond are just as random within your sample, then you still have a random sample no matter how small the response. Indeed, you might even accept a 2% response rate.

But in real life, it doesn't work like that. With a very low response rate, you're extremely likely to have found that people who do respond are unusual in some way: a massive non-response bias.

Whatever your response rate, you need to devote some thought and investigation to the reasons why some people didn't respond. Most of all, you have to have a clear understanding of who you asked and who didn't respond - preferably contacting some of them to ask why. And the lower the response rate, the larger your task is going to be on finding out why people didn't respond.

So that brings us to the actual percentage response rate you should be looking for. Personally, I aim for 70% to 80%. You may take a different view. If it's a difficult-to-reach population and an onerous task (for example, a survey that requires more than one round of questionnaire with some other task or intervention in-between), then you might accept 40% to 50%.

You need to balance these two factors:
- the effort required to get people to respond
- the effort required to find out why they didn't respond (plus the risk that you'll find some horrible non-response bias that destroys your whole survey).

HOW TO GET A GOOD RESPONSE RATE
Don't mistake me: getting a good response rate isn't easy. You have to do all the steps thoroughly:
- investigate why you want the data
- interview respondents to find out what they might want to tell you
- iteratively pilot the questionnaire itself, repeating until you've got something that gathers the data that will be useful and reflects what respondents want to say
- understand the target population and devise a good sampling strategy
- prime the respondents with an appropriate pre-questionnaire call to respond
- administer the questionnaire (according to whatever methods you have trialled in the pilots)
- track your responses and follow up with non-respondents.

Which all contributes to one of my little secrets: it's far more usual for me to talk a client out of running a survey than for them to allow me to proceed. We do the 'investigate' and 'interview' steps - and that's often enough for most business purposes.

HOW MANY RESPONSES DO YOU NEED?
But let's say you've got some burning business reason why you really need that survey data. Experience (and the research literature) suggests that you get a better response rate by choosing a smaller sample. So that led to this question:

"I could send out surveys to 20 people and get 15 responses for a 75% response rate, or send out 1,000 surveys and 100 responses for a 10% response rate. The response rate is lower in the second case but I have more responses overall. Do you know how you would measure/deal with this trade-off? Does it have to do with the estimated size of the population you're trying to study?"

There are three components to this answer.
1. It does depend on the size of the population, to some extent. If the population is small, you're running the risk of choosing nearly everyone. That itself depresses response rate. But let's assume that you're planning a sample of no more than 10% of your population.
2. The statistical theory: all the things about response rate/non-response rate that we've just been thinking about. That would clearly suggest that 20 people, 15 responses for a 75% rate is better - and contacting the remaining 5 to find out why they didn't respond is going to be quite easy to do, probably.
3. The 'face validity'. Meaning, will your client / manager / colleagues think that they've been short changed? There will always be plenty of people who think that 100 responses are 'more valid' or 'a better indicator' than 15 - no matter how carefully you explain your statistical arguments.

So I'd recommend being pragmatic. Maybe survey 20 people, get your 15 responses at 75% rate, and treat that as a pilot. Find out why the remaining 5 didn't respond, run your analysis, think about the decisions you'd make on that data - then decide whether it's worth having another go with a bigger sample.

My thanks to Kurt Reyes and Stephen S. for asking me challenging questions - and taking the time to discuss them.

Further reading: Don A. Dillman (2007) Mail and Internet Surveys: The Tailored Design Method


If you have any comments or suggestions about this article then please contact Caroline at:
Caroline.Jarrett@Effortmark.co.uk

Caroline Jarrett is a usability consultant specialising in forms and improving web content.
© 2007 Caroline Jarrett, all rights reserved.

Associated Link: Effortmark website

Online poker company uses Science to assess Player Preferences
Source: Recentpoker.com, 8 January 2009
 
Working with customer experience consultancy Foviance, PKR is exploiting the benefits of electroencephalography (EEG) technology to gather information on poker players' emotional relationship with a brand or service.

Do Users really love Laptops?
Source: channelinsider, 7 January 2009
 
More than one-fifth of all laptop computers will break down over the course of their life, and other limitations frustrate their users. From this list of user complaints come laptop opportunities.

User Interviews - Analysis Simplified
Source: Webcredible, 6 January 2009
 
You’ve conducted your user interviews, but now you need to make sense of all that information you’ve gathered.

Why Products Fail
Source: ComputerWorld, 5 January 2009
 
Most gadget and software makers don't understand what users want most: control.

How to Design Websites for Mobile Phones
Source: stemkoski.com, 3 January 2009
 
Tips from Ryan Stemkoski's web design blog.

Pioneer of Cyberspace honoured
Source: BBC, 2 January 2009
 
A professor who invented a forerunner of the world wide web has been made a dame in the New Year Honours.

2008 in Review: Developments that rocked the world of User Experience
Source: Catalyst Resources, 1 January 2009
 
A look back at 2008 highlights some of the key developments that rocked the world of user experience.

2008: The Year Online
Source: MIT Technology Review, 31 December 2008
 
The business of social networking, cloud computing, and a flaw in the fabric of the Internet top the most notable stories of 2008.

Shoveling through the Spamalanche
Source: UN, 30 December 2008
 
A ‘Spamalanche’ of 3,000 emails will be waiting in your inbox by the time you get back to work. What can you do besides 'delete all'?

New guidelines boost Web Access
Source: BBC, 29 December 2008
 
The World Wide Web Consortium (W3C) has announced a new standard to make sites more accessible to older and disabled people.