I recently caught up on an excellent short article by Chris Harvey for IJMR (Vol 58, Issue 5) proposing that we should consider replacing scale-based questions with binary ones, on the basis that:

  1. Many (arguably, all) choices are dominated by “system 1” or “fast” thinking, which is binary in nature
  2. Many of the behaviours we want to assess as researchers do not, in reality, include shades of opinion or of probability – implying that a scale based approach is wrong in those instances
  3. Scale based questions are inherently open to interpretation, can induce bias, and tempt poor data quality.

Daniel Kahneman’s academic partner Amos Tversky would certainly have agreed that a binary approach is much the better reflection of our decision-making process. In his epically academic but well-worth-the effort tome “Preference, Belief, and Similarity” he time and again demonstrates how our behaviours can be reduced to an implicit <Yes/No> response to our internal and external stimuli.

What’s more, binary questions are typically much quicker for a respondent to answer, saving valuable research time. Outcomes are less obscure, and therefore easier to communicate onward. Binary responses work really well in segmentation models, helping divide people one side of a fence or the other. And those looking to bring game design into research practice are also keen on a fast moving ‘this way or that way’ approach.

So why do we use scales? Well, many would say that they better reflect the complexity of the world, or shades of opinion, or some other similar argument. And whilst it would be churlish to deny these entirely I personally think it also has a lot to do with us satisfying our analytical appetites – a scale delivers more data per question, and as researchers we just love playing about with numbers. I also suspect that, as reflective types, we are mildly offended by the idea that we either can or should reduce things to a simple binary outcome. Anyway, in practical terms, scale based questions are part of our market research landscape and we have come to be rather reliant on them. What would we do without a seven-point agreement scale, or the ‘Juster‘ probability scale? Would marketing still function without the scale-based Net Promoter score?

Scales are here to stay, of course, but based on what we know about behavioural science it would be silly not to at least try and understand when offering a binary-choice option might be better. Here is my best guesstimate!

Obviously these are only predictions, and need validating.  I’d love to know if others see an opportunity to improve question design practice by moving some or all of the above from scale-based to binary-choice executions. Anyone have experience or findings they can share?