Caroline's Corner: Surveys - What is an Acceptable Response Rate?

It's been a while since I ranted on about response rates on surveys. In that article, I took the view that "2% is a terrible response rate" and had a few reasons why and tips for doing better. Recently, I've had a couple of challenging questions on the topic. So here goes at trying to answer them.

WHAT IS AN ACCEPTABLE RESPONSE RATE?
First of all, if a 2% response rate is no good: what should you aim for?

The answer is in two parts: first of all, the question of non-response bias and secondly, an actual percentage.

Non-response bias happens when the population who do not respond display characteristics that are different to the population who do respond.
If you've got a properly random sample, and the people who respond are just as random within your sample, then you still have a random sample no matter how small the response. Indeed, you might even accept a 2% response rate.

But in real life, it doesn't work like that. With a very low response rate, you're extremely likely to have found that people who do respond are unusual in some way: a massive non-response bias.

Whatever your response rate, you need to devote some thought and investigation to the reasons why some people didn't respond. Most of all, you have to have a clear understanding of who you asked and who didn't respond - preferably contacting some of them to ask why. And the lower the response rate, the larger your task is going to be on finding out why people didn't respond.

So that brings us to the actual percentage response rate you should be looking for. Personally, I aim for 70% to 80%. You may take a different view. If it's a difficult-to-reach population and an onerous task (for example, a survey that requires more than one round of questionnaire with some other task or intervention in-between), then you might accept 40% to 50%.

You need to balance these two factors:
- the effort required to get people to respond
- the effort required to find out why they didn't respond (plus the risk that you'll find some horrible non-response bias that destroys your whole survey).

HOW TO GET A GOOD RESPONSE RATE
Don't mistake me: getting a good response rate isn't easy. You have to do all the steps thoroughly:
- investigate why you want the data
- interview respondents to find out what they might want to tell you
- iteratively pilot the questionnaire itself, repeating until you've got something that gathers the data that will be useful and reflects what respondents want to say
- understand the target population and devise a good sampling strategy
- prime the respondents with an appropriate pre-questionnaire call to respond
- administer the questionnaire (according to whatever methods you have trialled in the pilots)
- track your responses and follow up with non-respondents.

Which all contributes to one of my little secrets: it's far more usual for me to talk a client out of running a survey than for them to allow me to proceed. We do the 'investigate' and 'interview' steps - and that's often enough for most business purposes.

HOW MANY RESPONSES DO YOU NEED?
But let's say you've got some burning business reason why you really need that survey data. Experience (and the research literature) suggests that you get a better response rate by choosing a smaller sample. So that led to this question:

"I could send out surveys to 20 people and get 15 responses for a 75% response rate, or send out 1,000 surveys and 100 responses for a 10% response rate. The response rate is lower in the second case but I have more responses overall. Do you know how you would measure/deal with this trade-off? Does it have to do with the estimated size of the population you're trying to study?"

There are three components to this answer.
1. It does depend on the size of the population, to some extent. If the population is small, you're running the risk of choosing nearly everyone. That itself depresses response rate. But let's assume that you're planning a sample of no more than 10% of your population.
2. The statistical theory: all the things about response rate/non-response rate that we've just been thinking about. That would clearly suggest that 20 people, 15 responses for a 75% rate is better - and contacting the remaining 5 to find out why they didn't respond is going to be quite easy to do, probably.
3. The 'face validity'. Meaning, will your client / manager / colleagues think that they've been short changed? There will always be plenty of people who think that 100 responses are 'more valid' or 'a better indicator' than 15 - no matter how carefully you explain your statistical arguments.

So I'd recommend being pragmatic. Maybe survey 20 people, get your 15 responses at 75% rate, and treat that as a pilot. Find out why the remaining 5 didn't respond, run your analysis, think about the decisions you'd make on that data - then decide whether it's worth having another go with a bigger sample.

My thanks to Kurt Reyes and Stephen S. for asking me challenging questions - and taking the time to discuss them.

Further reading: Don A. Dillman (2007) Mail and Internet Surveys: The Tailored Design Method


If you have any comments or suggestions about this article then please contact Caroline at:
Caroline.Jarrett@Effortmark.co.uk

Caroline Jarrett is a usability consultant specialising in forms and improving web content.
© 2007 Caroline Jarrett, all rights reserved.

Associated Link: Effortmark website

'Internet addiction' linked to Depression
Source: BBC, 9 February 2010
 
There is a strong link between heavy internet use and depression, UK psychologists have said.

Could *You* be more Usable?
Source: UN, 8 February 2010
 
Bet you could.

Stowe Boyd on 'Steampunk' thinking about the Future of Computing
Source: Stowe Boyd's blog via Experientia, 6 February 2010
 
Are established metaphors of user experience holding us back from new ways of structuring our interaction through computers?

Nokia's User Experience Programme
Source: UN, 5 February 2010
 
Nokia has put together a rich and informative website covering the key elements of user experience.

Interfaces magazine: latest issue available now
Source: HCI News Service, 4 February 2010
 
The latest issue of Interfaces is now available in pdf format, free from the Interaction Website.

A Lighter Brigade of Chargers
Source: UN, 3 February 2010
 
Lots of gadgets, one charger. At last.

Mobile Touch Screens could soon Feel the Pressure
Source: MIT Technology Review, 2 February 2010
 
A quantum switch could add pressure sensing to mobile screens.

Usability, Usability, Usability: why the iPad will Succeed
Source: Econsultancy, 1 February 2010
 
The tech critics love it, hate it, love it again, shrug it off. What do usability experts say?

British Airways - at last some good news
Source: Loop11, 30 January 2010
 
In a recent website usability study for the world's leading airlines, the British Airways website proved to be the most user friendly, with Malaysia Airlines and Virgin Atlantic having the lowest user experience rating.

Computation of Emotions in Man and Machine
Source: Royal Society, 29 January 2010
 
Advances in computer technology now allow machines to recognise and express emotions, paving the way for improved human-computer and human-human communications.