How Market Research Can Mislead You
Have you ever conducted market research and received a green light to launch a new product, only to have it fail in-market? You’re not alone. Remember New Coke? Crystal Pepsi? Wow Chips? The Sony Walkman? Microsoft Windows Zune? Then there was the research conducted by McCann in 2007 that famously reported no real need for an iPhone. Here’s why:
Customers Are Bad at Predicting Their Own Behavior.
We intend to buy, but we don’t. It’s what scientists call an intention-action gap. It results from unconscious biases and frictions that prevent us from doing what we intend to do. Consider a bias a decision-making shortcut that we unintentionally make to simplify our lives. And because we're unaware of these shortcuts, we can’t articulate them in a survey. Moreover, we often rationalize our choices after the fact, describing our decision-making process in research differently from how we really made the decision.
These psychological biases have an intellectual home, the field of Behavioral Economics, which studies how people make decisions. It came to prominence when 2 economists won the Nobel Prize in Economics for daring to say the obvious: we're not rational in our decision-making.
When I worked at Taco Bell, we too had a market research failure: a better-for-you line of tacos and burritos, launched with much fanfare, announcements to Wall Street, and a $50MM advertising budget, only to fail in-market just 1 month later. I joined the business just as the new line was faltering, and part of my job was to explain why it was failing.
Here’s my take and how you can avoid market research fails in your business.
3 Types of Testing, 3 False Positives
1. Concept Testing
DECEPTION
Consumers in concept testing reported wanting to eat better and would “definitely buy” Taco Bell’s new, better-for-you tacos and burritos. Test results exceeded norms.
REALITY
Consumers likely over-reported their intent to purchase because of a Social Desirability Bias, the phenomenon that explains why consumers typically over-report higher intent on socially acceptable behaviors. It’s one of the most common market research biases. Few people want to admit they have no interest in eating healthily.
Present-Day Bias: As humans, we’re wired for instant gratification. We discount the value of a benefit realized in the future and prioritize the benefit we can realize in the present. So when it comes to healthy eating, Present Day Bias looks something like this: I intend to eat healthy. But right now, I’m tired and stressed, and what I really want is a Burger and Fries. I call it “the-diet-starts-tomorrow-bias.” It’s common in restaurant dining occasions and could explain why customers opted NOT to visit and purchase the healthy menu items they claimed to want.
When customers who wanted to eat more healthfully were later interviewed in followup research, they rationalized their lunch choice decision, and it sounded like this: "I eat healthy. Today I had a Whopper. I ordered it because it has lettuce and tomato on it. Rationalization is common in research when there is a social desirability bias to a behavior and the research is conducted after a decision has been made.
2. Product Taste Tests
DECEPTION
Blind taste tests revealed customers preferred the taste of Taco Bell's better-for-you menu over its regular fat versions.
REALITY
Customers have a Mental-Model about better-for-you foods: low-fat menu items can’t possibly taste as good as their full-fat versions. Since Taco Bell branded the new items as low fat, customers didn't perceive them as tasting better, even though they DID in blind taste tests.
3. Market Testing
DECEPTION: Test markets outperformed the rest of the US. The new line was a hit!
REALITY Test markets were “average” but had a recent sales trend that artificially inflated test results. Relying on test markets is challenging because controlling all the other factors that may impact test results is difficult.
The Reality of Marketing to Secondary Targets
There’s a risk in pursuing a strategy dedicated to light users when it’s at the expense of your heavy users. Taco Bell’s better-for-you menu items appealed to light users who visited more frequently to buy them. However, visit gains from light users were offset by the more significant visit loss from heavy users who took their business to other fast food chains with a more relevant message.
How You Can Avoid Market Research Miscues in Any Industry
1. Use a Behavioral Science Lens in Developing New Products
It can help you identify the psychological biases and frictions that thwart or facilitate adoption of your new product. Friction can be logistical (too many steps), cognitive (too complicated), or psychological (biases and barriers the customer isn’t even aware they have). In the case of Taco Bell, friction was a mental model of how food should taste.
Behavioral Science can also help reframe your benefit, so it’s more immediate and concrete. Because of Present-Day bias, marketing healthy foods with the promise of a future benefit — losing weight, becoming healthier — in-real-life is less compelling to a restaurant diner. Instead, avoid the health claim and enhance the appetite appeal of the menu item and its ingredients.
Behavioral Science can also inform your survey design to identify biases that result in false positives. Sometimes the fix is to reframe the question or ask it in multiple ways.
How to bring that lens to your new products? One way is to engage a Behavioral Science practitioner. Most of the best-loved brands in America — Google, Amazon, Spotify, and Netflix — employ them. Why? Because we're all in the business of behavior change, whether we're asking customers to try a new product, adopt healthier habits, change brands, or simply click here. Another way is to learn Behavioral Economics through formal training. I took the Irrational Labs Behavioral Economics Bootcamp and found it an excellent first step.
2. Avoid Testing Unbranded Products...
Because your brand influences how the customer experiences your product — even how it tastes and performs.
READ how golfers using a generic putter branded as Nike needed fewer strokes than when using the same putter unbranded.
READ how consumers using generic sunglasses branded Rayban read more quickly and with fewer errors than those wearing the same sunglasses labeled Mango.
3. Red-team to Reduce Risk in Big Decisions
Confirmation bias is the tendency to unintentionally seek out and prefer information that supports our preexisting beliefs. Scientists consider it innate and difficult to eliminate, which is why product trials are often run double blind. Neil Hoyne, Chief Strategist at Google, recommends “Red-teaming” to reduce the impact of confirmation bias on significant decisions, like a new product launch. It's a concept borrowed from the CIA. To red-team, assemble a cross-functional team of analysts to build a counterargument to the recommendation of the product owner. The team’s mission is to identify what could go wrong. Have this team report directly to the executive team and rotate membership so individual agendas can’t take root.
4. Consider a Randomized Control Trial (RCT)
Most organizations now conduct A/B testing, the gold standard of which is a randomized control (RCT). An RCT introduces one intervention to a randomly selected group of participants, then compares the results to a control of “business as usual .” READ more.
5. Use Predictive Analytics
A predictive model may be better if you have large numbers of transactions, 3 years of clean data, and your products are not disruptive technology. Taco Bell uses predictive models to test new concepts, forecast sales, manage its supply chain, and optimize its media mix. At Driva, we use advanced machine learning to build predictive models for clients, recently building one to predict retention. Knowing a customer is likely to churn enables clients to get ahead of problems and introduce interventions to mitigate potential defections.
Disruptive Innovations
Do you have a disruptive innovation, a true Steve-Jobs-Like innovation, not the Taco Bell new menu items discussed here? People irrationally resist the genuinely innovative. We all have a Status Quo Bias, disproportionately preferring to stick with what we have even if the alternative is so much better. So stay tuned for a future blog post on how to test the validity of truly disruptive innovations.
There’s Rarely One-size-fits-all Research…
Because each organization, product, or initiative is unique. Market research is both an art and a science, so avoid hiring someone who simply knows how to use the survey tool. I started my career as a research analyst and now commission work from some of the best market research professionals. If you have any questions about behavioral economics, market research, or testing, TEXT me.