Survey takers don’t mean to be tricksters but remember when someone clicks a link to take a survey their mind was almost certainly somewhere else vs. the topic they will be surveyed about. Also, psychologists know that memory is reconstructive, not like going back through a book of photographs…more like person is recreating what is most likely to have been true based on how they view themselves in the world then.
Here are four ways that surveys can go wrong and what you can do about it.
Telescoping Problem
You want to know who the buyers are of different brands but surveys always elicit overstatement on brands bought over the past year, leading to inaccurate estimates of market penetration and misidentifying users…net/net, leading to wrong conclusions. This is called “telescoping”.
What you can do about it: Have reality check points. You can do this by referencing household panel data or by triangulating in off of other marketing facts, like market share and then running stochastic models to estimate penetration (Beta distributions, Dirichlet, even using Markov Models if you ask switching questions; I am happy to discuss the math with anyone interested). This will tell you if you have a telescoping problem. In terms of the survey, you can minimize telescoping by asking longer timeframes than the one you are interested in which traps telescoping effects, then following up with a shorter timeframe to get at the classification you are really interested in. In general, I found that using this approach, what people claim they bought over the past 6 months gives 12-month penetration.
Misleading claimed behaviors
Response is influenced by the share of choices on the list. That’s why politicians like to be on two lines on the ballot. For example, if you show a respondent a list of media touchpoints that might influence their purchase and you give them one TV choice and 10 digital choices (or if you lump together linear and CTV), you will get under-reporting on TV viewing.
What you can do about it. This is where Thaler and Sunstein’s idea (behavioral economists who wrote Nudge) about knowledge engineering come into play. Again, do desk research first to have some truth checkpoints. Research industry sales, MRI data on behaviors and interests, and Nielsen shares quarterly media consumption reports. Statista has valuable data as well. For anything media, you should check out Media Dynamics publications.
One useful trick is to make the question more manageable for respondents. Present choices in a way that is still logical, but “nudges” the results closer to what truth is known to be. Break the question up into part A and part B. The first part is higher level, (e.g. “TV, digital, social media, print, radio”, or “personal electronics, autos, large appliances, small appliances”…); whatever they choose, you can then offer them more granular choices. Walk them through a re-creation process to jog their memory (e.g. a shopper journey that led to a purchase result that gives valuable shopping information and will lead to more accurate reporting of outcomes).
Consumer segmentation based on weakly held beliefs
Random answering of attitudinal questions when beliefs are weakly held can ruin consumer segmentation. Often you are asking questions that the respondent doesn’t really know how to answer but guess what? They answer the question anyway! Then they develop other answers to other unfamiliar questions that are rationally consistent with this random answer. When you conduct consumer segmentation off of such data, you will get segments that seem to make sense but sadly, via a test-retest reliability experiment, you might find that the same respondent only has a 50% chance of falling into the same segment the second time.
What to do about it. First, you need to rethink segmentation altogether. Create segments equally based on behaviors as well as attitudes so that the segments are maximally different in purchasing behaviors and media habits. I remember being at Unilever and seeing a presentation by the ad agency of a segmentation on laundry habits. It seemed plausible and the groups made intuitive sense. However, when they profiled out brand preferences, patterns didn’t tie out! Brands that had little marketplace interaction indexed high on the same segment!
There is art and science to good questionnaire writing and I hope I have helped you a bit today with both.
