Here' s a question: on average, who has more lifetime sexual partners, men or women?
Of course, most people know the answer is men. For decades, dozens of studies have shown that heterosexual men tend to have more (often twice as many) sexual partners than heterosexual women. This is a fact that's taught in schools, quoted in the press, and regularly trotted out as an example of our evolutionary history (males need to spread their seed around to ensure their lineage continues, etc).
But there's just one problem: it's not actually true.
Any statistician will tell you it's logically impossible for men to have had more sexual partners than women: they're having sex with each other so by definition the averages need to be the same. So what's happening, time and time again, is people over- or under-reporting their own behaviour in research. But more importantly, what's also happening is everyone else taking those self-reports at face value.
My favourite part of all this is how some of the studies linked to above explain the great efforts taken to ensure the reliability of the survey methodologies.
"A new nationwide survey, using high-tech methods to solicit candid answers... a method designed to provide complete privacy and produce more honest answers."
"The questionnaire was administered in a confidential environment designed to elicit honest answers... Survey respondents entered a van, listened to the questions through head phones, and typed their answers into a computer without researchers being present."
I have to laugh because I've been in meetings with exactly the same discussions about devious methods to ensure the research is 'sterile' and 'unbiased.' Unfortunately, the problem isn't with the survey methodology. It's that people are really bad at describing and predicting their own behaviour. To borrow a phrase from IT support, the problem is between the keyboard and the chair.
Sometimes that's because our perceptions and memories are clouded by self-image and aspirations and delusions. Sometimes it's because we don't care or we forget. Sometimes it's because we honestly don't know or we get it wrong.
But the simple truth is we're not objective observers of ourselves. Not even close. No amount of tweaking a questionnaire or masking of intentions in research will ever help that. It's something to remember the next time you're about to ask some nice, unsuspecting sample population: "how many tea- or fruit-based beverages would you say you've consumed in the past 12 weeks?"