What research can't tell you

A couple of weeks ago I spoke (at the kind invitation of Audrey Carr) at InteractionCamp. This was a BarCamp-style open unconference, and it was excellent. The theme was designing interactive user experiences, and there was a great mix of speakers including Jon Lax, Sam Ladner, and Audrey herself. I spoke on the topic of why consumer research isn't always helpful in creative development.

Like most open-source events, one of the conditions of speaking at InteractionCamp was that we post our presentations online for everyone to access. I've been sorely remiss on that point. So here goes.

The presentation I gave is a collection of anecdotes about research not doing what it was supposed to do: help us get to better business decisions.  These stories come from various places - from Malcolm Gladwell in Blink, from my own experience, from popular culture. Things like New Coke (with its 100,000 taste tests), the Aeron Chair, research on tennis players by Vic Braden, reactions to the Friends pilot, the early reviews of the iPod, and all of the great work done by Tim Wilson at UVA.

The point of the presentation is simple: people can't always tell you what they want, what they like, or give their true opinion on something. I don't think this point is seriously debated anymore, but still an awful lot of research continues to ask people, point blank, to state a preference regarding their favourite product/package/ ad/etc, or to explain their decision-making process. This is a bad idea for three reasons.

1. People don't have consistent preferences in what they like: even simple things like taste tests are remarkably inconsistent.  Our likes and dislikes, which we think of as stable and fixed, are actually very malleable and context dependent.
2. People are bad at explaining their behaviour. Many decisions are unconscious. And when we don't know why or how we've done something, we often just make up stories (without realizing it). But even though they're untrue, these stories are plausible and believable (even to ourselves), so they're hard to dismiss or refute.
3. Asking people to reflect on their preferences or behaviour actually can change their answers away from the truth, towards the conservative and the familiar, because they lack the language to express their real feelings. This is a problem referred to as the perils of introspection.

The slides are here:

I should mention - given the recent comments on research I've been making around here - that none of this is meant as anti-research.  On the contrary, I want people to ask better questions, to get to better research methods and techniques, and to do smart research for the right reasons. But right now too much research is done out of habit, on auto-pilot, for playing it safe, or for political reasons rather than for true insight and illumination.

Comments

[Matt]

Good post. It's easy to simply say research sucks, and avoid it at all costs. As a creative guy, that's always my knee-jerk reaction. As I read your post, I was thinking about an online called a response bias. It's on things like Yelp and Amazon, where people rate movies, restaurants, toys, etc. The response bias says you're more likely to respond if you have an excellent or wholly crappy experience. So things get a 1 or a 5, but never a 3. Contrast that to in-person, where there's an opposite response bias. In person, people are less likely to really hate something, or really love it. In person, people generally think in 3's. The researcher who can figure out how to steer people away from these response biases can definitely test my creative.

[audrey]

Thanks for coming out Jason (and for putting your slides online)! Your points were definitely relevant and well taken.

On a similar note, Sam's just started a new blog on design research: http://designresearch.wordpress.com which is definitely bound to be a great read!

[oakie]

Hi Jason, nice post and presentation. It made me think of discovering an unmet need of people. Like you said before about inconsistency of people can lure us away from what they need.

[jyothsnay]

Jason
I certainly have a few points to counter your POV:
:) a few facts that most of us would be aware of-
a) Consumer feedback/ research output is imperative to survive in the "Parity Economy" we are living in.
b) Research, wholly, an indicator, a guide....Research is not the God who could alter everything in our favour
research would yield when "right questions are asked", not the same DG is administered to different set of respondents tirelessly n time & again
we need to ensure (Researchers+Planners) that "RIGHT Questions are asked"
c)Research is not just a scratch on the skin, but getting under the skin to probe further their attitude, altering themes in life and about the product category etc. gather all the required garbage but sift out the relevant material
d)Research is not just getting the feedback but "knowing more about the people who use the products"
-say for instance, consumers are extremely accurate when they are complaining about the products/services..they tend to exaggerate positive things.fair enough, so can we dig negatives, cribs further to unearth insights?guess, we can
all of us love to complain..n consumers also would love to do that
e)Out of problems, Cribs, fears n insecurities, "Insights" come..powerful ones! Insights lead one to "great n unique creative ideas"
f)I feel, one has to trust what consumers are saying, what they are trying to tell one and cull out the main themes and marry them to a few observations that one makes around one's milieu (trends), feel or experience the product to arrive at one solid insight, so consumer research does not work in isolation...it has to be merged with various other key variables

.. People, as u know, are creatures of habit and do nurture certain preferences n it's the sole responsibility of researcher or planner to get the right kind of feedback
-who to ask, how to ask, what to ask, in what condition to ask
what we need to effort in is to learn something that we dnot know, not the one we had already know
if classic coke was scheduled to be replaced by a new variant, one would have checked with the "heavy n regular consumers of coke" about the taste differential....

[Wizard]

Hi Jason,

All the stuff you've been writing on research testing has been nice.

I thought you might like to read this paper ( snakes & ladders marketing), since you've been on 'research' for a while and want people to have better / new techniques and methods.

Link :
http://www.springresearch.co.uk/pdf/esomar_snakes_and_ladders.pdf

[fredrik sarnblad]

Would have loved to listen to that presentation. Even without the benefit of your voice, the presentation is great. Some brilliant points in there.