Online research under fire

Amidst all of the talk about the declining effectiveness of advertising, and the corresponding rise in advertising avoidance, another point is starting to be more discussed: market research has the same problem. I know I've been banging on for a while about how we tend to ask people questions they can't answer, but this is something even more endemic. 

The thing is, market research shares a core problem with marketing - it's an industry that annoys people with unwelcome, boring interruptions. 
When we do that with marketing, people stop paying attention to marketing, which makes marketing less effective.
When we do that with market research, people stop answering market research, which makes research less valid. 

Online research was supposed to avoid some of that - no more telemarketers calling during dinner, no more cumbersome 20-page mail surveys. But unfortunately it hasn't turned out that way.

Advertising Age pointed out last week that "growing doubts about the validity of online market research have prompted the Advertising Research Foundation to form a council to draft new standards aimed at stemming erosion of client credibility."  This stems from a roundtable meeting held last fall where several marketers called online quant into question. P&G presented evidence of two identical surveys done a week apart, with the same supplier, which returned completely opposite results. The ARF has since commented that "reports of the failure of online studies to replicate when repeated are becoming more common." What's worse, ComScore has done research into online surveys that "showed that 0.25% of the online population accounts for 32% of responses... while less than 5% account for more than half of the responses."

The ARF is responding (as it usually does) by forming a committee: The Online Research Quality Council will hold its first meeting September 10, and aim to establish a set of industry standards for evaluation by next spring. I imagine they'll end up recommending something like better screening and incentives. But surely the problem runs deeper than than any quick methodological fix - in an era of conversations and choice and control, is answering a bunch of awkwardly worded questions for a market research survey just not something most people care to do anymore? After all, as Max Kalehoff has pointed out, most market research surveys are actually highly negative experiences for the brand. If I'm already skipping a :30 ad, surely I'll also skip a 30 minute questionnaire. 

Almost all of our clients now rely on online research as their primary source of quant.  Anybody else a little bit worried about this turn of events?

Comments

[Charles Frith]

I sat opposite people who merrily ticked anywhere that sped up the process for Strongbow cider during quantitative research a while back Jason.

Some people implied that as they drank more the veracity of their answers diminished.

I sprinted through my questionnaires and didn't even drink my taste/test pints.

Crap research and crap Cider. If Strongbow marketing or their research agency had the courage they'd ask me what I observed in Croydon Walkabout PUb some months back.

I dare them.

No. I double dare them.

I like pulp fiction.

[Max Kalehoff]

Jason,

The problem probably isn't one of online, nor is it an issue of eroding response rates. The core problem is one of relevance and value. If more research were relevant and provided value back to participants, then more people would be participating. It's pure economics and incentives.

Regards,
Max

[John Dodds]

Charles

You observed that the room went kinda spinny after your first sip - this is not an insight just your metabolism!

[Jason Oke]

Max - that's exactly it. The same thinking that we do for communications around experience, engagement, utility and creating value needs to be applied to market research as well. Otherwise we're all in trouble.

Charles - That's awesome. I think all market researchers should be forced to actually participate in some market research from time to time - both to experience how boring and painful the process is, and to watch others laughing and skipping through the questionnaire.

[Lee]

Come on Charles - what happened in the Walkabout - apart from the usual debauchery that is?

Did all the respondents make up their results unsupervised? We need to know!

And what were you doing hanging around in Croydon in the first place? Had you been to IKEA?

[Charles Frith]

John.

If your recall was as lucid as mine you would know that I then met up with you at The Commercial Tavern later that evening although I was with some of my fellow ex HHCL people most of the night.

I did see you in talking to Faris. Hand on hip. Your usual style :)

Hi Lee. I'm thinking it deserves a post on its own this topic. Farcical research masquerading as science.

[Jay]

If you think about it's not online as the methodology, it's the whole research process. Postal, telephone, are the worst as how can you engage someone via these methods, at least with online and perhaps F2F (if the research had the money spent on it that it deserved)you can include an element of engagement (i.e. flash pictures, audio, video) with the respondent making the process that much more enjoyable, therefore creating a response rather than bordem. The data on panels is probably right, but even here companies are moving away from panels, e.g. companies over in the US (like Greenfield) use a Real Time Sampling technology, which is a live river of traffic negating the same people taking part, therefore no panel bias. I don't think you can relate advertising to research as advertising does not engage the respondent at all.

[David]

Excellent and challenging points. I work in market research and I think one of our challenges is to create relationships with clients whereby the researchers have the final say. If not, the quality of the research is in the hands of someone who isn't (and shouldn't be) an expert on asking questions.

[The Ad Contrarian]

For more about the problems of on-line research, see "Research or Baloney" at http://adcontrarian.blogspot.com/2007/08/research-or-baloney.html

[Charles Frith]

I think you might like this.

http://bobsutton.typepad.com/my_weblog/2007/08/evidence-based-.html

[Gavin Heaton]

I am not really a fan of research (online or off). Certainly not in isolation or in an interruption mode. As Max says, it's about participation ... finding a way of allowing people the opportunity to participate in a measurable brand experience is likely to yield actionable insight.

[Kathryn Korostoff]

Research quality is a huge issue for the entire MR industry. Qual or quant. Results need to be triangulated against other data sources--a step many customers choose to skip (and from their perspective, it is understandable: I just spent $100k on some data and now I have to check it against other data sources?). B2B surveys are particularly challenged by low response rates and poor respondent quality these days.

[take surveys]

i think everybody should do research before they do anything. Online research is the best to me.

Take Surveys Online Get Paid