You give proper thought to questionnaire design, editing your ideas via a trusty “Word” document as you go. Once done you wait a while for programming, then test the online version. Naturally there are a few things to tidy up, and you spot a few bits that have come out a bit differently to your minds-eye view. That’s fine – tweak, test again, and sign-off when happy. Actually, it looked and worked great on screen (better than you thought), so you can relax and wait for the completes to roll in.
Yeh, right – if only…
If your questionnaire contains anything other than one-liners and simple single- and multi-choice questions then you are in for a shock. See below for the all important “first page” impression received by a UK doctor or nurse,viewing on an iPhone 5. Leaving discussion about the excessive but apparently essential verbiage on this page for another day, and allowing for the fact that Smartphone users can “pinch” the screen to maximise view-able content (in this screenshot we had already done that to some extent), this is hardly a good look. Compare it to the oceans of space on a shiny high-res desktop monitor running Chrome 42. NB – Obviously these images are not to scale!
Once we get onto the actual questions, especially large grids and numeric questions, the objections pile up very quickly (We will explore the range of question specific pitfalls in another Post).
We know that 7% of UK healthcare professionals use a Smartphone to complete surveys, based on analysis of n=1770 completes from surveys we’ve programmed and fielded in the last two months. If that doesn’t sound too bad then consider that Smartphone users are more likely to: drop-out; take longer to complete; score the survey experience poorly; run into problems… All but the most diligent participant will struggle to produce good quality data on all but the simplest of surveys on many of these devices. And, for the most part, healthcare related quantitative surveys are not simple (and unlikely ever to be so).
To explore the data relationships between 1.device used to complete / 2. speed of completion / 3. screen resolution / 4. gender feel free to visit our interactive “bubbles” analysis (http://goo.gl/YWsUwH).
The bigger the bubble, the more respondents with that unique combination of these attributes. You can cycle through the screens using the “Loop” function, bottom right. Do let us know if you like the technique.
The better survey authoring software, such as Confirmit, has special settings for rendering surveys on Smartphones. However, there are significant drawbacks with these:
- Scale and numeric grid questions are usually split into separate questions; i.e. actually appear as a series of single questions, repeated as many times as required.
- Questions appear one at a time, which may be out of sync with validation and other survey logic running, which can sometimes lead to survey errors
- Client side scripting (embedded in the page) will be stripped and not included – in our case this is a problem for about 10% of all the questions we script!
Taken together, this makes such a Smartphone function obsolete for the majority of surveys we programme and we therefore need to test our surveys on all the major Smartphone browsers and devices prior to field. We subscribe to an online system emulator, browserstack for that purpose. It is the long way around, but also the ONLY way if you are interested in maximising survey accessibility and integrity.
Finally, when we say Smartphones we include any mobile device with a relatively low screen resolution (<500,000 pixels).
More to come
This is the first in a series of related articles on the decline in participation in market research, and what researchers can do about it. Whilst most of what we discuss will relate to healthcare professionals (HCPs), and online methods, almost all of it applies just as well to the wider market research world.