Bias 4: Confirmation Bias

Welcome back to our series on data biases. We’re breaking down the different ways data skews in traditional market research; specifically, how they can corrupt the data you base your biggest decisions on. So far, we’ve discussed social desirability bias, acquiescence bias, and recency bias. Today we’re unpacking confirmation bias, which stands out among the other types of biases, because it can occur on both the researcher’s side and the survey respondent’s.

Both survey creators and survey takers have bias by virtue of being human. It’s common for researchers to end up creating questions that are leading or supporting a preconceived hypothesis. It can happen on the other side too, when the data is analyzed through a selective lens to interpret data in such a way that it confirms their initial beliefs rather than digging into the richer insights from any evidence to the contrary.

Conversely, respondents tend to give positive answers to validate any of their existing opinions, or they might entirely ignore any survey items that challenge their worldview.

Respondents read questions from their own perspective, bringing with them lived experiences, learned values, belief systems, and everything that makes them an individual. It’s the same on the other side. Even with guardrails in place, it’s still possible to ask the wrong questions.

We’re going to dive into what triggers confirmation bias, what makes it so pervasive that it can even sneak in on the research side, and how in the wild testing makes it a non-issue.

Surveys designed with bias built-in 

Before any survey data gets collected, a human researcher (who may have their own motivations and perspectives) has to design it.

Now, we know what you might be thinking: in the age of artificial intelligence, can’t we eliminate the margin for human error that causes things like faulty survey questions that have bias built-in? Even if AI could phrase the perfect question, free from the bias-laden burden of lived experience, there would still exist the bias of the respondent, who does reckon with their bias with every question. 

For example, if respondents hold a strong opinion about a brand, policy, or lifestyle, they could still interpret even the most neutral survey question as having a negative tonality, or purely oppositional. It can also go the other way, where a respondent might interpret a question that’s been thoroughly biaschecked as an endorsement of their view, altering the context of their answer, and thereby skewing your data. 

Data collection companies can be influenced, consciously or subconsciously, to provide certain answers to their clients. Even in the example we’ve just outlined about AI seeming like a catch-all solution to human-generated bias on the research side, we have to remember that AI is also a product of its environment, and even in-house LLMs coded for objectivity are going to be motivated by the goals of those who control it.  

Even in the instance of a perfectly phrased, bias-free question, researchers will then tend to see more responses that are lukewarm or lacking the full strength of feeling that the respondent might’ve expressed in another context. 

Selective survey data analysis  

The second major point of entry for confirmation bias is also on the research side.

The two main ways that confirmation bias makes it so survey data gets inaccurately analyzed are cherry-picking and sunk-cost rationalization.

Cherry-picking data happens when either researchers or major stakeholders home in on responses that align with some original hypothesis, rather than maximizing the value of data that points to the contrary, where there could be even richer data points, but also potentially further questions to ask.

The latter part of cherry-picking also leads to sunk-cost rationalization. Researchers or stakeholders might dismiss data that contradicts previous investments or assumptions, leading to a pattern of ignoring important shifts in the market. When surveys give answers that stakeholders are disappointed with, they can be reluctant to test further. The very investments they might be reluctant to re-allocate could well be in the market research sector — conflicting data, in traditional research, often points toward a need for more time and money spent on further development of questions, re-strategizing, and sourcing new respondents.

Failing to see why a survey respondent would give an answer influenced by confirmation bias to a question like “Is our product better than their product?” versus a question like “Do you prefer this product or that product?” is part of how data analysis gets corrupted even before questions are asked. After asking, answers can still be interpreted with the same skewed lens.

Survey questions are the beginning of the problem that is confirmation bias — and since its present on both sides of the desk, it’s pretty much innate to surveys overall. In the wild testing, successfully circumvents the pitfalls of confirmation bias by rewriting the playbook entirely. Consumers aren’t asked anything; they’re simply compelled to react to stimuli that’s presented to them in an environment they’re familiar with, that encourages authentic, unfiltered reaction. That is, in-feed.

Since the stimuli is able to communicate and illicit feelings that transcend the black-and-white, tepid research question, in the wild testing pulls richer, more nuanced data points that you can easily pivot with to ask further questions with, to help you make better go-to-market decisions without going all the way back to the drawing board.