We need better online surveys, not shorter ones

Back in April 2016 I wrote that that, “participants care more about their experience than they do about watching the clock”. At the time First Line Research didn’t have any hard evidence for that, but we do now.

Since then, we have kept detailed records on how participants complete our online surveys and they tell us that we must keep working on mobile compatibility, and that making surveys shorter is neither useful nor necessary.

Whilst what follows is all based on data from online surveys amongst health professionals[1] (the bulk of our work here at First Line) we have reason enough to think these points apply equally, if not more so, to consumers and other sample groups.

First, participants are perfectly happy to complete longer surveys (so long as they don’t feel conned).

The chart below shows that self-declared survey satisfaction scores decline only fractionally as actual survey duration increases. That is, longer surveys are not in themselves a cause of dissatisfaction. From our own and industry-body sponsored investigations[2] we know that there is a long list of things that cause dissatisfaction, and prominent amongst them is research that takes longer than advertised.

The table below says that, on average, only if an online survey takes its participants 20% longer than advertised to complete might we expect survey satisfaction to drop significantly. In real terms, that’s when a survey advertised at 20 minutes, actually ends up taking 25 minutes or longer. In other words, participants are tolerant of surveys that take a bit longer than anticipated, but there is a limit to their patience.

 Quicker to complete than advertised, or on timeTook up to 20% longer to complete than advertisedTook more than 20% longer to complete than advertised
Mean survey satisfaction score (0-10)7.57.57.1

Second, participation via mobile phone takes longer and is a less satisfactory experience.

In an earlier blog we showed why it is more helpful to think about differences between bigger screens and smaller screens, than between types of device[3]. Whilst, in the table below, we’re pleased to see that, overall, the large majority of participants in online surveys created by First Line complete either early, on time, or within 20% of the advertised time, those who take longer are much more likely to have been using a smaller screen.

 % of participants completing on bigger screens% of participants completing on smaller screens
Quicker to complete than advertised, or on time73%51%
Took up to 20% longer to complete than advertised12%13%
Took more than 20% longer to complete than advertised15%36%

So, despite the dexterity of many smartphone users, for over a third who take surveys on their phone it is a slower process than it should be.  This goes a long way to explaining why those completing on smaller screens are appreciably less satisfied than those completing on bigger screens, see below.

It’s good that survey satisfaction scores seem to be on the up for our surveys, but we can’t ignore the difference between bigger and smaller screen experiences. And, in all honesty, we already know that the underlying problem is that some of the larger and more complex question types do not yet adapt well enough to smaller screens. The result is that participants can’t understand or complete some questions properly, or need to take extra time over them. It is not easy – the best research designers and survey platform developers are still figuring out how to make complex questions work on smaller screens – and at First Line Research we have routinely blogged about s the steps we are taking.

 Third, only VERY long surveys put participants off using their phones.

At First Line we specialise in originating and/or programming longer, more complex, online surveys, but nevertheless almost three quarters (74%) of the health professional surveys in this analysis were advertised as being 20 minutes or less. The chart below shows that only for much longer surveys (advertised as 30 minutes or more) are phone users likely to be put off.

To me, all of this says that:

  • At First Line we are right to be working hard on designing surveys that are truly mobile compatible and can be experienced as well on smaller screens as on bigger ones.
  • Participants are telling us that if we advertise duration with honesty, pay fairly, and field surveys that work, they are happy to take longer surveys – so why risk diluting or simplifying our objectives by fielding unnecessarily shorter surveys?
  • Buyers too must shoulder some responsibility for the questions they originate and take advice on how they will translate to the small screen. After all, the ramifications for participant experience cascade, obviously and directly, to data quality.

 

[1] 3641 completed and valid survey responses from health professionals between Q2, 2016 and Q2, 2018.

[2] See the BHBIA Response Rate Task Force Report 2017

[3] Desktops, laptops, and tablets (including iPads) all typically have bigger screens, whilst phones typically have smaller screens. For more on this distinction, read that earlier blog.


Leave a Reply

Your email address will not be published. Required fields are marked *