How incentivized surveys can work against you

If you advertise something as free, by definition, you’re saying that there is no expectation to exchange effort, money, or anything otherwise of value in order to obtain the thing. 

But for free, it can be hard to get people to take your survey. Incentivizing participants with things like gift cards, discount codes, or an entrance into a sweepstakes is an easy way to get answers. However, do we think that by offering goods in exchange for answers to our survey questions we’ll be getting anything of value from our respondents? 

Incentivizing surveys encourages respondents, through no fault of their own, to answer questions in their own self-interest and introduces bias that skews your data. Answers you’ll get, but whether they’ll yield quality data is highly questionable.

We’ve talked before about how surveys can give you imperfect data, but now we’re going to unpack the ways that incentivizing your survey can introduce even more issues, or amplify the ones that are already there.

Responding for the reward

This is the most pervasive way that incentivized surveys end up collecting skewed data. 

The way in which it’s skewed can depend on the reward at stake, like in the instance of a sweepstakes, respondents are more likely to end up leaving flattering answers in the belief that saying those things will help their chances at winning. 

Or, you might want information on how to improve upon an existing product. With no way to prove whether people have tried your product or not, you may get people giving you bogus answers about a product they’ve never tried just to get the reward at the end. 

Most commonly, even with your ideal survey respondent with the most well-meaning intentions, people will rush through the questions to get to the reward at the end. They won’t think much about their responses and they won’t care enough to make their voice heard.

Research has shown that 75% of consumers who self-claim to shop in a specific category actually have never shopped in that category before.

For most incentivized surveys, these types of responses are inevitable. Even in the best-case scenario, the minds (and motivations) of your respondents will be elsewhere. 

You’re introducing unnecessary bias

Ideally with a survey, you’re collecting a full spectrum of responses, both positive and negative. 

Even if you’re confident in your business or product, you still want to try and understand where any negative experiences are stemming from.

Like we’ve highlighted in our Bias Series, acquiescence bias and social desirability bias are particularly prevalent with surveys. And when you add in incentives, you’re only creating more irrelevant stimuli for consumers to react to. 

Social desirability bias can be triggered easily depending on the nature of the reward at the end of the survey, for example. Say you’re a pet care brand, and the reward is a free bag of quality dog food. Respondents are less likely to be honest about questions around willingness to spending a little extra on their pet, because they’re filling out a survey to get an essential product for free, from a pet care brand modeled around and targeted toward consumers who are able prioritize their pets financially. This is the sort of thing that can trigger a guilt response and lead to less accurate self-reporting. 

In another case, say you’re looking to improve on an existing product. You might offer a discount to anyone who fills out your survey. That’s great, but how does that motivate people who are disappointed by your product, and who have no desire to come back for more, even at a discounted rate?

In this instance, you’d only get answers from respondents who want to leave extra-positive feedback rife with acquiescence bias, which is great for the ego of your business, but maybe not the best for its long-term development. 

A better way to close the say-do gap

The say-do gap that emerges from outdated research methods like surveys is only growing wider. 

With constantly shifting social and technological landscapes, businesses of all sizes are struggling to glean meaningful insights from just asking questions and seeing what sticks. It’s simply not enough. Consumers have trust issues when it comes to sharing personal information, and they’re more aware than ever of the value their data holds. That shift demands a tactful, modern approach.

The forward shift is towards behavioral insights. Even big data companies are implementing more consumer behavior-centric models to extract richer and more meaningful data points that aren’t contaminated by bias. 

Incentivized surveys typically present more problems for data collection than solutions. There’s more opportunities to disappoint survey takers if you send them out regularly and don’t always offer incentives — even certain rewards can present bias if they’re not considered useful, relevant or desirable by everyone. Even if you anticipate challenges and plan carefully, you’ll likely be pigeon-holing yourself in some way rather than collecting valuable insights from real consumers.

When you conduct your research in the wild, you circumvent all these potential biases.

People offer up their opinion because they want it to be heard, and because they don’t even think of it as being part of data collection. The reward often feels like the opportunity to express their thoughts openly and honestly about a pain point or to express enthusiasm about something they genuinely care about.