If you are a buyer or practitioner of healthcare market research, how would you reply to the following …

  1. How often do you request (or propose) a sample of n=50 or n=100 on a healthcare quantitative study?
  2. How often is that ‘n=’ sample number chosen automatically, or based on ‘gut feel’?
  3. How often do you purchase (or sell) the ‘n=’ sample number that a simple statistical test would actually recommend?
  4. Do you understand by how much a change in ‘n=’ changes confidence in the resultant data?
  5. Do you know how to find your optimal ‘n=’ sample size?

We are all drawn, trance-like, towards certain seemingly ‘magic’ numbers – decimal milestones like 25, 50 75, 100, 200, 500 … and so on.  As researchers I reckon we fall under their spell more than most, as we go about our everyday work of setting sample sizes and analysing data.

 

DiceWhat ‘magic’ is this? Well, the sort that can instantly gratify our deep-rooted desire for safety and security: the thought of n=100 makes us feel nice and comfortable, and somehow beyond criticism in a way that n=87 just doesn’t. Besides, our colleagues recommend ‘magic’ number sample sizes all the time – so even if we should be doing things differently (which is unlikely, surely?) we can always point to massive precedent.

I must admit that in over 20 years of healthcare quantitative research I have never (yes, never) quoted or requested anything other than n= some ‘magic’ number. Sure, we have sometimes ended up with n=51 instead of n=50, or n=93 when we struggled to achieve n=100, but I have never set out to achieve such apparently oddball sample sizes. Nor have I previously challenged their cultural orthodoxy in any serious way.

In the UK, the reflex when researching a general topic amongst GPs will be to ask for n=100, or n=200. If budget is tight then perhaps n=75, and n=50 if all we want is a so-called sanity check. But how much confidence can we have in the outcomes produced?  Is this something we consider at the proposal, or briefing, stage of a project? I think not. Being drawn to ‘magic’ numbers seems to be our hard-wired sample-size heuristic. But how come?

I think in large part it is because we are a bit fearful…

  • Of statistics – perhaps we assume that calculating optimal sample size is either beyond us (“i.e. “I’m not really a stats person”) or involves appreciable additional work (…and no way do we have the time).
  • Of criticism (possibly even ridicule), if we suggest n= something else, and people laugh. Or think us incompetent.

At a time when the evidence for declining participation rates is crystalising, our established practice of requesting n=100 when n=87 would do just as well isn’t helping – burning through sample unnecessarily. On other occasions we must be reporting many of our findings with unwarranted over-confidence because, for example, we have used a base of n=50 when we really needed n=108.

Once you start to investigate this issue, it quickly becomes clear that the Internet has made the process of determining optimal sample size a lot less scary and a lot more accessible. And – to me at least – it now feels rather embarrassing that most researchers are not running these simple checks as standard!

Here is a sample size-calculator (with what I hope are self explanatory notes, and a relevant example), for you to play with.

%CODE_SampleSizeCalc%

To conclude the example above, if we wanted to be 95% confident that our survey would produce answer/s within a +/- 5% range of what we’d find if we could survey the entire UK population of 65,000 GPs, then we should ask for a sample size of n=382. For most, such a sample size would be out of budget or take too long, in which case the best thing to do is relax the margin of error. So, if we decided we’d be happy to be 95% confident that our survey would produce answer/s within a +/- 10% range of what we’d find if we surveyed all UK GPs, then we should ask for a sample size of n=96. OK, why not make it n=100, just for luck.

Pretty easy, right? (a standalone version of the calculator is also available here)

Psychologists Amos Tversky and Daniel Kahneman would have had a great deal to say about the spell cast by ‘magic’ numbers. They illustrate brilliantly how our behaviour is driven by psychological heuristics and biases, powerful social norms, and a desire to avoid additional work! Yet it is so tempting to uphold the orthodoxy, even once we know how to make the improvement… but I am resolved at least to experiment a bit on my clients, and see what happens when I advocate n=83, rather than n=100 !

John Aitchison, john@firstlineresearch.com