the fruits of imagination

Pre-testing, part II

I've finally got around to writing a few more thoughts on what's turned into a healthy and productive conversation on pre-testing.  You can read the original post here and the response from Millward Brown’s Nigel Hollis here.  Be sure to read the comments too.

I should offer a bit of context first. Just as Nigel’s point of reference is Link, mine largely is not. My pre-testing experiences of the past few years have been with other suppliers, some of the major players and some smaller ones. I don’t have recent experience with Link because due to shifting client-supplier relationships I haven’t experienced pre-testing with MB in 3 ½ years. So we may be talking about different things. Nigel describes ways that Link differs from other methodologies, and some of their recent developments. I can’t comment on that, but some of it does sound like some positive steps. I should also say that I’ve liked what I have seen (at a distance) from MB in recent years – things like some of the changes to Link, their attempt to understand what drives PVR users to either skip or stop and watch advertising, and indeed Nigel’s blog itself. But before this starts sounding like a love letter to Millward Brown, let’s get down to business: some responses to Nigel, and some further thoughts on pre-testing.

One of the things I’d challenged pre-testing on was a tendency to focus on the rational. Now, as Nigel mentions, recently many pre-test methods have shifted to focus more on involvement, emotional relevance and brand engagement. This is certainly true, and a welcome development. The ARF has recently published a document called “Measures of Engagement Volume II” which gives a useful overview of many of the new developments in research methodologies (Link is one of those featured).

Even if we're asking better questions, there remains some debate about whether people really are conscious of all of our emotional responses. On this issue, Nigel observes (quoting Antonio Damasio) that while our decisions are emotionally based, “unless those emotions are experienced consciously, they will have little effect.” So, he continues, “people must be conscious of their reaction if an ad is to have a lasting effect on them.” As might be expected on such an interesting topic, there are differing opinions. Robert Heath, with all of his work on Low Involvement Processing, has come to some different conclusions. In that same ARF document “Measures of Engagement Volume II,” a paper of Heath’s is published as an introduction. In it he also extensively references Damasio, but he shows how conscious attention and emotional engagement are two separate constructs; and that there is “NO direct relationship between levels of attention and levels of engagement” (the emphasis is his). This paper is definitely worth the read.

However, regardless of the methodology or types of questions asked, I still think there’s a fundamental difficulty testing certain types of ads. Much of the problem comes down to stimulus – what do you actually test? I only mentioned stimulus briefly in the original post, but it deserves more discussion.

Let’s separate ads into two very broad camps. The first type of ad has clear rational messages, straightforward action & dialogue, and are set in a familiar setting like a kitchen. These ads are really about messaging. The other type of ad is not about messaging, but about experience – it is reliant on specific casting or special effects or production values, and aims less to inform or persuade than to create emotional response or impact culture.

Pre-testing of new ideas is (by its nature) done in pre-finished form. In the first kind of “message” advertising, it is easy to separate the idea from its execution. The production details like directing and casting generally are secondary. They serve to make a good idea better – more memorable, more interesting – but at a fundamental level, how the ad works remains the same whether produced at a high level or low, or indeed if described in a storyboard or animatic. This makes them easier to test in pre-finished form. And in these cases, I think pre-testing can be quite useful to ensure clarity of communication, relevance of the situation, etc.

In the second type of ad, we cannot separate the idea from the execution: the execution is the idea. These ads have less overt messaging, instead attempting to create a feeling towards the brand. Sometimes this is through telling emotional stories, such as Dove or Axe. Sometimes it is about creating a mood, sort of like a music video, the way Nike and iPod ads often do. And sometimes ads attempt to create something visually remarkable and never seen before – such as the Bravia and Honda work. In the era of YouTube and social networks, all of these try to encourage discussion and passing along, to create an “ohmygod have you seen that yet?” effect. And all rely on the specifics of casting and direction to work. I’ll reuse that quote from Unilever’s Simon Clift from the previous post – “With Dove, for example, it's how the girl comes across – her non-verbal gestures, the cut of her hair and whether she's sympathetic, that determine whether the message is believable. It's not the words put in her mouth…With Axe/Lynx particularly, there's no way you can tell whether this or that babe is going to be appealing to a 16-year-old boy from a line drawing.” We might be able to describe these ideas in simple terms, but their real power relies on how they are produced. At a fundamental level these ads won’t work the same way if they’re produced at a lower level or described in a storyboard or animatic form. Because of this, it can be meaningless to test these types of ads in pre-finished form, for the same reasons it would be meaningless to pre-test a description of an event or a video game or anything else that relies on an actual experience.

In these situations, one could still test a finished ad. For new ads this usually doesn’t make sense because once it’s produced, you’re not really able to significantly make any changes. It does however make perfect sense at other times, such as when you’re considering running a historical ad or an ad from another region to see how it would work with your audience.

So to summarize, I believe that pre-testing is biased towards certain types of ads because they are easier to test in a pre-finished format. This straightforward messaging style of advertising can be very effective. But it’s worth noting that the most highly successful advertising of the past few years falls into the second camp, and that most of those campaigns were not pre-tested.

I’d also like to mention a few other issues raised by Nigel. 

Nigel defends the predictive ability of Link by citing strong correlations with in-market sales tracking. It’s impossible to comment without having seen the actual data, but I remain doubtful for several reasons. One is that in my experience with market-mix-modeling and case study writing, I have to say I’m skeptical that the effects of advertising can really be effectively teased out from other market activity (I realize of course that lots of people spend lots of money trying to do exactly that). Markets are a highly complex system, and so many things influence sales that correlations can be misleading. Nigel does mention that some of his data do not “correct for the influence of other marketing variables” and it’s worth considering what that might be. Ads which pre-test well may build increased confidence in the marketing team that created them, translating to higher media spend, longer rotations on air, more excitement from their sales force, and so on, turning the prediction from the pre-test into a self-fulfilling prophecy (this is not necessarily a bad thing, of course). And if Nigel is relying on his clients for data then it also might not be a truly representative sample. I’ve too often seen marketers use their data selectively, discounting periods with poor sales results by blaming them on competitive activity, issues with distribution or sales teams, and so on.

He also makes a statement that is heard commonly – it’s echoed by some of the commenters – that making a decision without the benefit of pre-testing is “going on gut feel.” This makes it sound as if people are making important brand decisions on a whim, which is unfair. Nigel says “I’d prefer to base decisions on informed judgment than gut feel” [the emphasis is his]; however surely a decision made without pre-testing can still be well-informed. It’s a decision made by a team from the client and agency exercising the judgment for which they are supposedly paid, against specific brand objectives, after healthy debate, and ideally based on a significant amount of research early in the process to understand the audience, how they interact with media, and the role of the brand in their lives. To my mind, saying that is going on “gut feel” is a serious under-representation of the work and expertise involved. The sad thing is many marketers – in both client and agency organizations – have lost their faith in their ability to make exactly these kinds of judgments. In my experience, this type of research is more often used as a substitute for judgment, and as a way to minimize the potential for blame rather than as a way to maximize success.

Nigel also helpfully picks up on this point of the mindset of those involved, and we wholeheartedly agree with the need to be open-minded and flexible. He links to an earlier post with the great title “Is the Link pre-test the equivalent of the Smith & Wesson 500?” that, drawing parallels to the gun industry, questions whether research companies do enough to ensure clients use their products safely and correctly. It’s a fantastic question, and Nigel has a thoughtful response that’s worth reading. But I want to mention here that it’s not just clients and research companies that need to examine their motives. Agencies often do ourselves a disservice when we selectively use research results, championing the research (and ignoring any problems) when it supports the agency’s work, while criticizing the same methodology (and jumping on any problems) when an ad performs poorly. I know I’ve been guilty of this.

I hope I don’t come across as hating pre-testing. I think it can have lots of value. In an ideal world, I’d love to have a full understanding of each communication’s power before it was launched. It’s just that as testing is done today, I’m not sure that many of the types of ads we’re all increasingly trying to make can be pre-tested in a meaningful form.

I’ll say a huge thanks to Nigel and all of the people who’ve commented and linked to both here and his post. I don’t think there’s enough dialogue between researchers and agencies and clients about the research we do, so this has been refreshing for me.  Thankfully, in the end, I actually think we agree on as much than we disagree. And I hope we can continue to talk about this. So I’ll open it to all of you again: anybody agree, disagree, or have anything to add?

A defense of pre-testing: let's get ready to rumble

No less an authority on pre-testing than the eminent Nigel Hollis, Chief Global Analyst at Millward Brown, has posted a great response to my thoughts about pre-testing from the other day.  It's very thoughtful and detailed, and for a balanced look at the subject I'd highly recommend taking a read of it.  While you're there, the rest of his blog's quite good too.  I have some comments back that I'll post either here or there when I get a chance.

A bit quiet

I should mention that (in case you haven't already noticed) it's going to be a bit quiet around here for a while. No, the Leo Burnett Company hasn't shut down. But I'm on vacation for a few weeks.

I've tried to harass a few others to contribute posts on here in the interim - we'll see how effective my powers of persuasion are.

Image via.

Pre-testing

Pre-testing ads is something that doesn't get too much play on the advertising blog circuit (nor has it made an appearance yet on the plannersphere).  Is that because most people have already made up their minds as to its usefulness (or lack thereof)?  Or because it has inflicted so much pain on us all that we don't want to think about it too much?  The fact remains that many major clients still pre-test, at least for TV advertising, and sometimes for print and billboards as well. The majority of our clients certainly do. So we should talk about it, its issues, and how to make it better.

One of our big clients is trying to put together a global POV on pre-testing, and recently asked our opinion on a couple of questions. Here was a first stab at an answer, I thought we might as well share it with everyone to get some discussion going.  Anybody agree, disagree, or have anything to add? 

1) Given that copy testing in some form is here to stay, what do you believe is the best process for testing creative. If you could run it your way what would you do?

First let's look at the problems. Pre-testing, as largely practised, assumes a rational person, conscious of the impact of an ad on them, what their reasons are for how they feel about it, and able to articulate all of that. It tends to be a very rational exercise, focused on measures like message communication, relevance, persuasion, or purchase consideration.  Even when emotional questions are asked, they are asked in a highly rational manner ("on a scale of 1 to 5, how did this ad make you feel?") which does not exactly allow for the richness of human emotional response to come across. So people tend to simply play back the rational message content of an ad - which may only be a portion of what makes the ad effective. Just the fact that we call it COPY testing is an indication of how focused we are on the message above other elements.

Several pieces of research have been published which demonstrate how standard pre-testing methods under-represent emotional response. And further, advances in neuroscience and our understanding of the brain have shown that the vast majority of our decisions are emotionally (and unconsciously) based. We've also learned that people tend to initially dislike things and ideas that are unfamiliar to them - which is often exactly what we're trying to create.  All of this presents a big problem for traditional market research. Many of today's highly successful ads have negligible (or undifferentiating) message content – think of Sony Bravia “Balls” or Honda “Cog” or the new Orange "Better together" work or just about any iPod or Nike ad. What would traditional pre-testing have to say about the effects of ads like these.  Are they believable? Relevant? Persuasive? Do those questions even make sense in these contexts, or do they miss the point entirely?

We have to remember that most of these methodologies were invented at a time when ads were built around USPs, USPs were rational and audience attention was a given. The Advertising Research Foundation itself (the industry body of market research in the US) said last year "For the most part, there's been no wide scale significant innovation in copy testing and tracking (except maybe data collection methods) in 50 years."

More crucially most copytesting is reductive - in an attempt to be scientific, it tries to "isolate the variables." This means it doesn't factor in the viewing context (e.g. the actual programming a TV ad airs in) - and as we found out in our own research last year, omitting the real programming context artificially inflates an ad's appeal because people don't have anything at hand to compare it to, whereas in real life your ad is compared to a cliffhanger on Lost or a punchline on The Office. Also, most testing is of ads individually, and often only TV, rather than a campaign as a whole that builds together towards something (although some companies are working on testing campaigns). And it ignores the socially networked context of today's world – would you talk about this, send it to a friend, blog about it, upload onto YouTube and so on. Worst of all it doesn't ask whether people would have avoided it in the first place if they'd had the chance.

So what's the solution?  If we have to pre-test, we prefer to figure out what method suits each ad best. There's a problem with the question: there is not a "best process for testing creative" for the simple reason that ads are different and brands have different needs and objectives. Some ads (e.g. with straightforward action & dialogue, in a familiar setting like a kitchen table) can be more easily tested in pre-film form such as animatics or ripomatics, others (e.g. if they rely on specific casting or effects or production values) simply cannot. Also, some ads have a simple rational message, while others try to create emotional responses or cultural associations with a brand. For ads that have a lot of emotive content – which most do today – we would recommend a methodology that helps understand the feelings and energy generated from viewing the ad, whether people want to talk about it afterwards, or whether they'd have avoided it in the first place.  And most of these can't be asked directly with linear 5-point scale questions. We'd also test the advertising in programming context - not a 10 year old failed pilot, but a current episode of the type of show the ad would run in - to understand how it stacked up against that content.  These types of research do exist, but they are more expensive than standard Link or ASI. Then again you get what you pay for.

One-size-fits-all copy testing across an entire company doesn't really make sense - unless that company is prepared to commit to only one style and model of advertising exclusively. So the first thing copy testing needs to do is be more open and flexible in how ads are evaluated. The method should be dictated by the objectives of the advertising and the type/style of ad. But one implication of this is that there would not be as many norms to compare to, because different ads would be tested differently. Then again, norms and the slavish reliance thereon is a whole other issue.

2) To what degree do you believe that the current testing processes (ASI, Link or similar) is an accurate predictor of in market success?

The short answer is that copy testing is not predictive of in-market success. It never has been. In fact, no one’s ever shown that it could be, so we shouldn't look at it that way.

• Tim Broadbent, chair of the AdMap Pre-testing Conference from 2000-2004 and member of the IPA board has stated that "There is no evidence in the public domain that it [pre-testing] is predictive." 
• It's never been shown that companies that use pre-testing are more effective than companies which do not, or that companies that start pre-testing after a period of not doing it become more effective.
• Alan Hedges wrote a wonderful book in 1971 called "Testing to Destruction". His very first page reads "IT IS NOT POSSIBLE TO MAKE A REALISTIC TEST OF THE EFFECTIVENESS OF A COMMERCIAL IN A LABORATORY SITUATION IN ADVANCE OF REAL-LIFE EXPOSURE."  About 10 years ago, the IPA republished the book and specifically noted that this observation was still completely valid.
• The IPA Effectiveness Awards in the UK are probably the hardest effectiveness awards in the world to win. In a study  of the winners over the years it's been shown that there is a negative correlation between doing pre-testing and winning IPA Effectiveness Awards. In other words, ads that were pre-tested were less likely to be effective.
• The ARF in the US recently came to the same conclusion. Last year a piece on their website said: “Because of emphasis on cognitive/rational measures, current copy testing techniques cause a regression to the mean, thereby reducing advertising effectiveness.” It’s worth repeating that: the ARF says that copy testing reduces advertising effectiveness.

So it’s not just that testing is not predictive of success, it may actually be making things worse by rewarding safe, unchallenging, familiar ideas.

Now it's possible that some testing is predictive in other limited ways – for example some companies will tell you they can predict whether an ad will have high awareness or recall in market. And maybe they can. But there’s no evidence anyone can do the most important thing, which is to predict in-market success.

Despite all of the above, pre-testing can still be valuable as a diagnostic exercise - helping to understand what elements of an ad or campaign might be working or and which aren't, and whether they are working the way we thought they would. As with many types of research, the value depends on the attitude of those using it.  There's that old joke that most people use research the way a drunk uses a lamp post - for support rather than illumination.  So when research is used as a substitute for judgment, that's when problems arise.  And ultimately any attempt to predict in-market success is really a substitute for judgment.

Which leads us to challenge the core assumption in the first question. Why does pre-testing need to "be here to stay"? Many major marketers have stopped using it. Instead, they do a lot of research up front to understand their audience and the kinds of advertising that can be effective with them. Then, as they progress, they use their judgement on whether they’ve created that kind of advertising. And of course they carefully track what happens in market. As Simon Clift, the global President of Marketing for Unilever (Home & Personal Care) recently said:

"To me the excessive reliance on animatics is crazy – like choosing your wife from a stick drawing. With Dove, for example, it's how the girl comes across – her non-verbal gestures, the cut of her hair and whether she's sympathetic, that determine whether the message is believable. It's not the words put in her mouth. There are lots of examples of where we would have chucked an ad away if we'd believed the quantitative predictive research. With Axe/Lynx particularly, there's no way you can tell whether this or that babe is going to be appealing to a 16-year-old boy from a line drawing. I'm very much in favour of measurement, I just don't believe in predictive research. And we don't use it."

Update: check out Nigel Hollis' response.

 

I believe the children are our future

When you're on a roll, you're on a roll.  Hats off to our world-beating young creative team Steve Persico and Anthony Chelvanathan who have cleaned up yet again.  Last night, at the National Advertising Awards, Anthony and Steve won the Grand Prix in the highly competitive Young Creatives Competition. The prize is a trip to Cannes Festival this year.

This is on the heels of taking both first and second place at the Future Gold global creative competition at the Clios last year, as well as winning gold and silver at last year's edition of the National Advertising Awards (and just narrowly missing out on last year's Grand Prix).

Last night Steve also won a silver in the radio category and Marcus Sagar took home a bronze in outdoor.  Fist bumps all around.

The masters, hard at work.

Best chart title of the day

Bagels

Coming in at number 17 on the list of reasons that Leo Burnett Toronto is a great place to work: because Richard brings in bagels for everyone.  The secret is the really good cream cheese - none of that low-fat nonsense, this stuff has enough creamy butterfat to lube a tractor.

Thanks to Richard for the bagel love.