Pre-testing

Pre-testing ads is something that doesn't get too much play on the advertising blog circuit (nor has it made an appearance yet on the plannersphere).  Is that because most people have already made up their minds as to its usefulness (or lack thereof)?  Or because it has inflicted so much pain on us all that we don't want to think about it too much?  The fact remains that many major clients still pre-test, at least for TV advertising, and sometimes for print and billboards as well. The majority of our clients certainly do. So we should talk about it, its issues, and how to make it better.

One of our big clients is trying to put together a global POV on pre-testing, and recently asked our opinion on a couple of questions. Here was a first stab at an answer, I thought we might as well share it with everyone to get some discussion going.  Anybody agree, disagree, or have anything to add? 

1) Given that copy testing in some form is here to stay, what do you believe is the best process for testing creative. If you could run it your way what would you do?

First let's look at the problems. Pre-testing, as largely practised, assumes a rational person, conscious of the impact of an ad on them, what their reasons are for how they feel about it, and able to articulate all of that. It tends to be a very rational exercise, focused on measures like message communication, relevance, persuasion, or purchase consideration.  Even when emotional questions are asked, they are asked in a highly rational manner ("on a scale of 1 to 5, how did this ad make you feel?") which does not exactly allow for the richness of human emotional response to come across. So people tend to simply play back the rational message content of an ad - which may only be a portion of what makes the ad effective. Just the fact that we call it COPY testing is an indication of how focused we are on the message above other elements.

Several pieces of research have been published which demonstrate how standard pre-testing methods under-represent emotional response. And further, advances in neuroscience and our understanding of the brain have shown that the vast majority of our decisions are emotionally (and unconsciously) based. We've also learned that people tend to initially dislike things and ideas that are unfamiliar to them - which is often exactly what we're trying to create.  All of this presents a big problem for traditional market research. Many of today's highly successful ads have negligible (or undifferentiating) message content – think of Sony Bravia “Balls” or Honda “Cog” or the new Orange "Better together" work or just about any iPod or Nike ad. What would traditional pre-testing have to say about the effects of ads like these.  Are they believable? Relevant? Persuasive? Do those questions even make sense in these contexts, or do they miss the point entirely?

We have to remember that most of these methodologies were invented at a time when ads were built around USPs, USPs were rational and audience attention was a given. The Advertising Research Foundation itself (the industry body of market research in the US) said last year "For the most part, there's been no wide scale significant innovation in copy testing and tracking (except maybe data collection methods) in 50 years."

More crucially most copytesting is reductive - in an attempt to be scientific, it tries to "isolate the variables." This means it doesn't factor in the viewing context (e.g. the actual programming a TV ad airs in) - and as we found out in our own research last year, omitting the real programming context artificially inflates an ad's appeal because people don't have anything at hand to compare it to, whereas in real life your ad is compared to a cliffhanger on Lost or a punchline on The Office. Also, most testing is of ads individually, and often only TV, rather than a campaign as a whole that builds together towards something (although some companies are working on testing campaigns). And it ignores the socially networked context of today's world – would you talk about this, send it to a friend, blog about it, upload onto YouTube and so on. Worst of all it doesn't ask whether people would have avoided it in the first place if they'd had the chance.

So what's the solution?  If we have to pre-test, we prefer to figure out what method suits each ad best. There's a problem with the question: there is not a "best process for testing creative" for the simple reason that ads are different and brands have different needs and objectives. Some ads (e.g. with straightforward action & dialogue, in a familiar setting like a kitchen table) can be more easily tested in pre-film form such as animatics or ripomatics, others (e.g. if they rely on specific casting or effects or production values) simply cannot. Also, some ads have a simple rational message, while others try to create emotional responses or cultural associations with a brand. For ads that have a lot of emotive content – which most do today – we would recommend a methodology that helps understand the feelings and energy generated from viewing the ad, whether people want to talk about it afterwards, or whether they'd have avoided it in the first place.  And most of these can't be asked directly with linear 5-point scale questions. We'd also test the advertising in programming context - not a 10 year old failed pilot, but a current episode of the type of show the ad would run in - to understand how it stacked up against that content.  These types of research do exist, but they are more expensive than standard Link or ASI. Then again you get what you pay for.

One-size-fits-all copy testing across an entire company doesn't really make sense - unless that company is prepared to commit to only one style and model of advertising exclusively. So the first thing copy testing needs to do is be more open and flexible in how ads are evaluated. The method should be dictated by the objectives of the advertising and the type/style of ad. But one implication of this is that there would not be as many norms to compare to, because different ads would be tested differently. Then again, norms and the slavish reliance thereon is a whole other issue.

2) To what degree do you believe that the current testing processes (ASI, Link or similar) is an accurate predictor of in market success?

The short answer is that copy testing is not predictive of in-market success. It never has been. In fact, no one’s ever shown that it could be, so we shouldn't look at it that way.

• Tim Broadbent, chair of the AdMap Pre-testing Conference from 2000-2004 and member of the IPA board has stated that "There is no evidence in the public domain that it [pre-testing] is predictive." 
• It's never been shown that companies that use pre-testing are more effective than companies which do not, or that companies that start pre-testing after a period of not doing it become more effective.
• Alan Hedges wrote a wonderful book in 1971 called "Testing to Destruction". His very first page reads "IT IS NOT POSSIBLE TO MAKE A REALISTIC TEST OF THE EFFECTIVENESS OF A COMMERCIAL IN A LABORATORY SITUATION IN ADVANCE OF REAL-LIFE EXPOSURE."  About 10 years ago, the IPA republished the book and specifically noted that this observation was still completely valid.
• The IPA Effectiveness Awards in the UK are probably the hardest effectiveness awards in the world to win. In a study  of the winners over the years it's been shown that there is a negative correlation between doing pre-testing and winning IPA Effectiveness Awards. In other words, ads that were pre-tested were less likely to be effective.
• The ARF in the US recently came to the same conclusion. Last year a piece on their website said: “Because of emphasis on cognitive/rational measures, current copy testing techniques cause a regression to the mean, thereby reducing advertising effectiveness.” It’s worth repeating that: the ARF says that copy testing reduces advertising effectiveness.

So it’s not just that testing is not predictive of success, it may actually be making things worse by rewarding safe, unchallenging, familiar ideas.

Now it's possible that some testing is predictive in other limited ways – for example some companies will tell you they can predict whether an ad will have high awareness or recall in market. And maybe they can. But there’s no evidence anyone can do the most important thing, which is to predict in-market success.

Despite all of the above, pre-testing can still be valuable as a diagnostic exercise - helping to understand what elements of an ad or campaign might be working or and which aren't, and whether they are working the way we thought they would. As with many types of research, the value depends on the attitude of those using it.  There's that old joke that most people use research the way a drunk uses a lamp post - for support rather than illumination.  So when research is used as a substitute for judgment, that's when problems arise.  And ultimately any attempt to predict in-market success is really a substitute for judgment.

Which leads us to challenge the core assumption in the first question. Why does pre-testing need to "be here to stay"? Many major marketers have stopped using it. Instead, they do a lot of research up front to understand their audience and the kinds of advertising that can be effective with them. Then, as they progress, they use their judgement on whether they’ve created that kind of advertising. And of course they carefully track what happens in market. As Simon Clift, the global President of Marketing for Unilever (Home & Personal Care) recently said:

"To me the excessive reliance on animatics is crazy – like choosing your wife from a stick drawing. With Dove, for example, it's how the girl comes across – her non-verbal gestures, the cut of her hair and whether she's sympathetic, that determine whether the message is believable. It's not the words put in her mouth. There are lots of examples of where we would have chucked an ad away if we'd believed the quantitative predictive research. With Axe/Lynx particularly, there's no way you can tell whether this or that babe is going to be appealing to a 16-year-old boy from a line drawing. I'm very much in favour of measurement, I just don't believe in predictive research. And we don't use it."

Update: check out Nigel Hollis' response.

 

Comments

[Charles Frith]

I'm glad you've tackled this and it's a post that I want to spend some time thinking about before responding. However you may want to join in with a similar post on Nigel Hollis' excellent blog for Millward Brown. Comments are still open.

http://www.mb-blog.com/index.php/2007/03/12/whats-wrong-with-pretesting-and-what-should-we-do-about-it/

[Charles]

Hi Jason. Maybe you could try this link if the other one was cropped short.

http://tinyurl.com/2dyrdx

[Paul McEnany]

Great, great stuff, Jason. I wish I had more to say, but I agree with just about everything you said...

[vanessa from SLC]

Everything you have here is pretty accurate, but I wonder if you have thought about including a discussion of what it is possible to get from pre-testing, like a feel for people's initial impression. Their reactions may not predict the ad's effect on their future behavior, but may give you a yay or nay as to whether you've hit the "buttons" you were aiming for.

[John Dawson]

I posted this on Nigel's blog - thought you'd like to see the comments also:

Surely the benefits of pre-testing also encompass another set of goals:
1) provide due diligence - i.e. if my organisation is going to spend $3m on creative then i want to know that of all the ideas we've gone with, there is some 3rd party justification for the one we picked
2) When people argue against pre-testing and point to results which suggest little or no linkage with sales performance, they often neglect to mention that for each piece of copy which ran having been pre-tested, there are some pieces which never ran because the pre-testing was poor (do you have any numbers on this Nigel?)
3) If the IPA study showed a negative relationship between use of pretesting and IPA effectiveness wins it may also mean that these "amazing" campaigns were so obviously good that there was no need for pre-testing. It's certainly not a "scientific" conclusion.

[Charles Frith]

That's a good point by Vanessa. It would be just another polarised debate if it goes all binary - Advertising good, pretesting bad. As I've pointed out on Nigels blog when he originally posted.

"rather than say the methodology is no good, I’d be more inclined to ask, what are the advantages of this process against the disadvantages"

My instincts and experience do however tell me a vastly trimmed back role for research in pretesting. More a case of advising what not to do than what to do. Which is my general position on the value of research.

[fredrik sarnblad]

Excellent post on an ever-relevant topic. You make some great points here. My view on pre-testing (unashamedly secptical) can be found here:

http://fredriksarnblad.wordpress.com/2006/05/09/goodbye-innovation/

[Harshal Gajria]

There are 2 discrete viewpoints on this excellent post that seldom gets discussed; At a few large spending clients (who are also excellent practioners of the marketing craft), the carte blanche to over-ride the results resides with the office of the marketing director.

First, are we not fostering enough confidence in the work, by getting clients to invest disproportionately in exploratory, qualitative work prior to the briefing process? Are the fundamental questions of "why are we advertising", "who are we talking to" & "what will we say" not being adequately answered? The dated USP approach does not allow for an accurate understanding of the role the brand will play in the consumer's life; in essence, determining the engagement levels.

Secondly, as agency business's proliferate, are we not able to engage with our clients at the highest possible levels? After all, the very same executive suite that does have carete blanche is not the one that's working directly to define the strategic problem (well, not all the time)

[K. Wallraven]

Very interesting thoughts! At the moment, I am writing about the pitfalls of pre-testing. Could you please tell me, where exactly the following quote is from (Website or Journal, number, etc...):

"The Advertising Research Foundation itself (the industry body of market research in the US) said last year "For the most part, there's been no wide scale significant innovation in copy testing and tracking (except maybe data collection methods) in 50 years."

Many thanks,

K. Wallraven