Amidst all of the talk about the declining effectiveness of advertising, and the corresponding rise in advertising avoidance, another point is starting to be more discussed: market research has the same problem. I know I've been banging on for a while about how we tend to ask people questions they can't answer, but this is something even more endemic.
The thing is, market research shares a core problem with marketing - it's an industry that annoys people with unwelcome, boring interruptions.
When we do that with marketing, people stop paying attention to marketing, which makes marketing less effective.
When we do that with market research, people stop answering market research, which makes research less valid.
Online research was supposed to avoid some of that - no more telemarketers calling during dinner, no more cumbersome 20-page mail surveys. But unfortunately it hasn't turned out that way.
Advertising Age pointed out last week that "growing doubts about the validity of online market research have prompted the Advertising Research Foundation to form a council to draft new standards aimed at stemming erosion of client credibility." This stems from a roundtable meeting held last fall where several marketers called online quant into question. P&G presented evidence of two identical surveys done a week apart, with the same supplier, which returned completely opposite results. The ARF has since commented that "reports of the failure of online studies to replicate when repeated are becoming more common." What's worse, ComScore has done research into online surveys that "showed that 0.25% of the online population accounts for 32% of responses... while less than 5% account for more than half of the responses."
The ARF is responding (as it usually does) by forming a committee: The Online Research Quality Council will hold its first meeting September 10, and aim to establish a set of industry standards for evaluation by next spring. I imagine they'll end up recommending something like better screening and incentives. But surely the problem runs deeper than than any quick methodological fix - in an era of conversations and choice and control, is answering a bunch of awkwardly worded questions for a market research survey just not something most people care to do anymore? After all, as Max Kalehoff has pointed out, most market research surveys are actually highly negative experiences for the brand. If I'm already skipping a :30 ad, surely I'll also skip a 30 minute questionnaire.
Almost all of our clients now rely on online research as their primary source of quant. Anybody else a little bit worried about this turn of events?