What your sexual history can teach us about market research

Here' s a question: on average, who has more lifetime sexual partners, men or women?

Of course, most people know the answer is men. For decades, dozens of studies have shown that heterosexual men tend to have more (often twice as many) sexual partners than heterosexual women. This is a fact that's taught in schools, quoted in the press, and regularly trotted out as an example of our evolutionary history (males need to spread their seed around to ensure their lineage continues, etc).

But there's just one problem: it's not actually true.

Any statistician will tell you it's logically impossible for men to have had more sexual partners than women: they're having sex with each other so by definition the averages need to be the same. So what's happening, time and time again, is people over- or under-reporting their own behaviour in research. But more importantly, what's also happening is everyone else taking those self-reports at face value.

My favourite part of all this is how some of the studies linked to above explain the great efforts taken to ensure the reliability of the survey methodologies.

"A new nationwide survey, using high-tech methods to solicit candid answers... a method designed to provide complete privacy and produce more honest answers."

"The questionnaire was administered in a confidential environment designed to elicit honest answers... Survey respondents entered a van, listened to the questions through head phones, and typed their answers into a computer without researchers being present."

I have to laugh because I've been in meetings with exactly the same discussions about devious methods to ensure the research is 'sterile' and 'unbiased.' Unfortunately, the problem isn't with the survey methodology. It's that people are really bad at describing and predicting their own behaviour. To borrow a phrase from IT support, the problem is between the keyboard and the chair.

Sometimes that's because our perceptions and memories are clouded by self-image and aspirations and delusions. Sometimes it's because we don't care or we forget. Sometimes it's because we honestly don't know or we get it wrong.

But the simple truth is we're not objective observers of ourselves. Not even close. No amount of tweaking a questionnaire or masking of intentions in research will ever help that. It's something to remember the next time you're about to ask some nice, unsuspecting sample population: "how many tea- or fruit-based beverages would you say you've consumed in the past 12 weeks?"

Comments

[Charles Frith]

I'm glad you've raised this point Jason because pretty much all my life I've looked at those statistics for male promiscuity and thought they are not true.

I also heartily agree that the accuracy of answers is highly suspect in a good deal of questionnaires. Partly because we can't help but answer in the image of how we would 'LIKE' to see ourselves.

Which is a completely different slant on what we actually do in practically all human beings.

Good post. Healthy discussion material.

[Mikej]

wasnt there some type of formula for this recited in American Pie. I think it was times all girls numbers by three and divide all guys numbers by three

Stupid teen Movies... even helping with market research

[John Dodds]

Isn't there some neurological evidence also that the part of the brain you use for answering questions and questionnaires is different from that which you use to make the actual decisions? The upshot of this is that even if you answer truthfully, you may well not be telling the truth.

[Faris]

Damn straight brother - claimed data is a waste of time in predicting behaviour - although it does show in some case intentionality and past behaviour.

Partially - people lie.

Partially - somatic markers covertly bias cognition so people think they're thinking one thing, but they ain't.

http://farisyakob.typepad.com/blog/2007/08/lubricants-of-r.html

[fredrik sarnblad]

Great post. Thanks Jason. Let the truth be heard!

[Jensen]

Great article and being a science guy I've always wondered how the two averages could logically be different, turns out they can't. Sex and promiscuity are so culturaly ingrained I wonder if any other market research topic could have an error this large?

[Jim Rait]

Reminds me of a survey of an airline that showed it did more take-offs than landings!

[Thomas Wagner]

This should actually be tought in market research classes. I have feeling it is at least as important as regression and complex models that work great - in theory.