I’ll keep this strictly non-political. I guess many will have heard enough argument, bluster, and plot to last a decade never mind less than a week. And besides, as researchers and marketers there is plenty else for us to digest in the Referendum aftermath.

ballot-paperDoes the outcome really prove that we are a “nation divided”?

Almost certainly not. People were given the simplest of choices: Remain or Leave. The only other choice was not to vote. As we know, there are ways to capture shades of opinion and strength of feeling, and offering a dichotomous choice is not one of them. I’m not suggesting that the ballot paper should have offered greater scope for expression, just that when we obtain the results from such a question we shouldn’t rush to characterise a near-run-thing as somehow proving that voters are either chalk or cheese. It is an illusion prompted by the choice architecture of the vote and conjured by commentators, and not an authentic interpretation of our collective sentiment. Actual opinion on the subject of the EU will, of course, exist on a spectrum but in recent days you’d have been forgiven for thinking that we must all be entirely ‘IN’ or ‘OUT’ and that a more tempered view could not possibly be held. I’d love to see prominent researchers getting involved by picking-up journalists and politicians who over-simplify thus.

 

Telephone Vs Online.

A notable feature of the polls was how differently online and telephone methods captured voting intention. In short, telephone polls had Remain well ahead, whilst online polls had it too close-to-call. Pollsters are now much better at adjusting for many of the sample biases but there are still puzzles over methodology bias.

FTtelephonepolls

My contention is that the private, self-completion, non-verbal nature of responding to an online poll is much closer to the physical act of voting. By contrast, when asked to verbally state our position to a stranger, at short notice, and under time pressure we tend towards a safe answer, that protects us from the mental discomfort of opposing a perceived social norm.

Extrapolating to the research world generally, we might want to look to online surveying as the more authentic way to ask questions regarded as personal, sensitive, requiring reflection, or where we know the individual has primacy in the decision or choice we are interested in.

 

Echo-chambers, and strangers.

As many readers will be aware, Remain voters tended to belong to one or more of the following groups:

  • Based in London or other major metropolitan area
  • Educated to ‘A’ Level or above
  • Active on social media (especially Twitter I suspect!)

These are all superb natural echo-chambers, in which a relatively high % of group members participate in discussion, and in which prevalent views bounce about a lot. People have ready access to news and views, and their own opinions naturally coalesce around positions that they feel close too. In behavioural terms, confirmation bias is massively in evidence, as people unwittingly cherry-pick the information they feel most comfortable with and recycle it amongst like-minded members via their social and professional networks.

confirmationbias

As market researchers we might want to reflect on how our panels, advisory boards, MROCs, and even focus groups all represent mini echo-chambers that likely exaggerate the strength of apparently popular messages. Indeed, whilst market research participants become increasingly familiar to us, non-responders seem ever more like strangers.

Whilst the polls got it less wrong than in the 2015 General Election, on balance they were still predicting a win for Remain. This time, turnout was high and lots of people who don’t respond to polls did turn up to cast their vote. Although researchers try and weight for this it is ultimately educated guesswork, and the outcome confirms that we are still guessing wrong. A big contributing factor is the decline in response rates in market research and polling – fewer and fewer people want to take part. We are, over time, simply refining the design of our own echo-chamber, and ignoring the existence of those without. The danger is that we end up knowing an unrepresentative few very well, a non-participatory majority hardly at all, yet still pass off our findings as a narrative of the truth.

We need a really good PR campaign

I know … self selection / non-response is undoubtedly market research’s oldest and most difficult problem … but we desperately need to do better if we are to maintain credibility. I am involved with industry initiatives on the topic and firmly believe that one thing we must do is spread the word on what market research is and why it is important. We need a really good PR campaign. We then need to back that up by delivering a better research experience for the respondent. Which is our second most difficult problem…

(Those who made it this far may wish to read our previous blog entry, in which we called the Referendum outcome correctly.)