Market research has traditionally been conducted in plain view, in the form of voluntary or incentivised surveys. What these methods neglect to account for is the observer effect, which causes subjects to alter their behavior because of the sheer fact that they’re aware of someone watching them. Subjects often become more self-conscious, and their behaviors trend toward the confirmative side, skewing your data and making it difficult if not impossible to capture authentic behaviors and preferences.
Sure, there are ways to create the illusion that nobody is watching you take a survey or participate in a focus group. Even allowing subjects to relax through “habituation” before asking them questions doesn’t solve the problem altogether. Hidden cameras and one-way mirrors in the case of focus groups are imperfect solutions as well. The observer effect remains.
That is, until subjects are in their truly natural habitat. We’re going to unpack exactly how in the wild research dispels the observer effect for truly clean data, uninfluenced by consumer anxiety.
The classic observer effect occurs when subjects are aware that they’re being watched and their responses are being recorded.
It introduces a series of biases, like acquiescence bias and social desirability bias, and generally triggers a heightened self-awareness that is counterintuitive to the recording of authentic behavior.
Acquiescence and social desirability biases are closely related, in that they’re both rooted in the consumer wanting to gain the approval of the researcher across from them. Acquiescence is more about being agreeable or not giving one’s true opinion for the sake of not being disruptive, whereas social desirability is more about coming off as a different person entirely in order to be perceived as likable, more socially responsible, or more appealing to whomever is conducting research. These bias responses are natural and rooted in wanting to be helpful.
For example, most of us prepare for job interviews by anticipating the questions that will be asked, doing our research, and formulating natural, confident responses that reflect the best version of ourselves. No matter how much you prepare though, there’s something about having your responses recorded and your presence judged that makes the average person on-edge — maybe you forget to mention an important piece of experience, or tell a white lie about having a common interest to get the interviewer to like you, even if you didn’t originally intend to do that. People behave differently when they’re being watched, no matter how much you prepare them.
Researchers have considered various ways to get around this phenomenon, from blinding techniques to heart rate monitors to simulated real-world environments. While consumers are aware that somebody is observing them when they engage with content on their feeds, that entity isn’t registered as having any explicit authority.
The ideal, then, would be to target a consumer response that hasn’t been through the same hyper-aware filter. Rather than seeking simply “better” answers from consumers, how do you get real reactions that go beyond stated intent?
This is when the observer ends up unconsciously influencing how data shows up in traditional research.
The Clever Hans Effect is a perfect example. Clever Hans was a performing horse in Berlin who was famous for his ability to “count” using taps of his hoof. It turned out that he couldn’t actually count. In reality, he was just reading subtle body language queues from his handler, which acted as commands for the horse to perform for his audience.
Since then, the Clever Hans Effect has been referenced by researchers in reference to the potential to skew data by unintentionally queuing desired behavior when tests as poorly designed. The consumers are like the horse that can count, reading queues from proctors or researchers in order to give a “correct” answer, with marketing teams as the audience.
That’s just an example, but the real stakes are much higher than barnyard party tricks. Even attempts to neutralize this potential effect are imperfect. Double-blind testing, wherein not even the researcher is supposed to be aware of who and what is being sampled and researched, isn’t always possible or realistic.
So, how does our research control for the observer effect? This is where in the wild testing comes in. With the in-person barrier to truth removed, and the double-blind adjacent approach in serving consumers with stimuli that they don’t realize is part of a test, our matrix-based approach compares dozens of stimuli against one another simultaneously. Presented without any potential for human interference, the observer expectancy effect is dissolved. The results can’t be skewed, because each piece of stimuli is made to target your specific question.
If you’re conducting traditional market research, the observer effect is costing you accuracy at best. At worst, it’s messing with your bottom line.
Real-world testing doesn’t rely on simulations or gadgets. It’s the real thing.