Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS): Mode Effect Study Report 2015

5 Does mode affect non-response to individual questions?

In addition to response rates, sample profile and key question response patterns, it is important to also consider the impact of mode on non-response in individual questions. This chapter looks at the impact of mode on missing answers.

Non-response - key findings

• Overall, there were more missing answers in the online sample than the paper sample. This was considered to be most likely due to the availability of a 'prefer not to say' option in the online questionnaire.

Overall, there were considerably more missing answers in the online mode (see Table 5.1). In other words, pupils were more likely to select 'prefer not to say' in the online mode than they were to leave a question blank, give an unintelligible response, or give an incorrect response (e.g. ticking two contradicting statements within the same question) in the paper mode ^[12] .

Table 5.1 Proportion of missing answers for key substance use measures

A further example is presented in Table 5.2 regarding pupil postcodes. As with the substance use measures, there were more missing answers among those who completed the survey online (59%) than on paper (38%).

Table 5.2 Proportion of pupils providing a valid postcode by survey mode

Low postcode response is important because the data is used to assign each pupil a Scottish Index of Multiple Deprivation ^[13] ranking and an urban/rural classification ^[14] and if incorrect could impact on analysis using these variables. However, imputation (please see the SALSUS 2015 Technical Report ^[15] for details) can be used to address this.

Despite a lower completion level, the online survey may provide a benefit in ensuring better quality of postcode entry. There was a higher percentage of invalid postcodes for the paper mode compared to online which may result from lack of clarity so they can't be identified during the scanning process, or they were incorrectly identified. Although no controls were included in the online survey, such as over the format or automatic checks to ensure that it was a valid postcode, such controls could be added to future surveys online.

The low postcode response and postcode quality are certainly issues to consider for future SALSUS survey waves.

Inclusion of a 'prefer not to say' option in the online version

One of main differences between the paper and online questionnaires was that the online questionnaire had a 'prefer not to say' option at every question whereas the paper version did not have any.

For ethical reasons, it was felt important to allow pupils to refuse to answer a question, so this meant a 'prefer not to say' option was included at every question (including at every row in grid questions, so they could choose to respond to some of the items but not others). This was in preference to allowing them to move on to the next question without registering a response, as that may have been by accident.

However, the SALSUS 2015 paper questionnaire did not contain a 'prefer not to say' option at every question. This was necessary to keep the questionnaire as comparable with previous waves as possible as a 'prefer not to say' option has never been included before. From an ethical perspective, the argument is that respondents to a paper questionnaire can easily just miss out a question if they do not wish to answer it.

Impact of the 'prefer not to say' option

In the online mode, if respondents selected 'prefer not to say' the response was classed as missing. In the paper mode, if a pupil left a question blank, if the response was unintelligible (e.g. because there wasn't a clear enough mark in one box) or if they made errors (e.g. ticked two contradicting statements within the same question) then the response was classed as missing. Towards the end of the questionnaire, some of the missing answers (in both modes) are because pupils ran out of time.

It was clear that the availability of an obvious 'prefer not to say' option increased the proportion of missing answers. However, we cannot be sure why pupils selected 'prefer not to say'. Some may do so because they do not want to give an honest answer to the question. It might be that this is more likely where pupils are concerned that the data may not be anonymous, and when the response is perceived as socially undesirable. This might be expected to impact more on questions relating to substance use and drug use in particular. This is suggested by the difference in missing responses to the drug use question between paper (1.77%) and online (7.36%) shown in Table 5.1 above. However, such difference does not appear to have impacted on prevalence figures as previously demonstrated (refer back to Table 4.4).

It is likely this has impacted on postcodes. In the pilot study, there was some concern from pupils about providing their postcodes, as they felt that this could potentially make them identifiable (for example, if they are the only 15 year old girl living in that postcode area). It is unclear why this would be more of a concern for those completing the survey online than on paper. However, one possibility could be a perception that electronic data is more easily manipulated and matched up with other electronic data. Although the postcodes on paper questionnaires are scanned and transferred to an electronic file, this is less obvious and it may be that pupils perceive less risk providing a postcode on a paper questionnaire.

Although not identified in the pilot, another possibility is that pupils may be aware of warnings (from school and elsewhere) to be very careful about what personal data they submit online. They may know from experience that submitting an email address or mobile number can result in unwanted mail and calls. Again, although this can also happen with information submitted on paper, warnings tend to focus on online activity and may, therefore, have more of an effect on the online mode.

Another reason that pupils might select 'prefer not to say' is that it is an easy option: some may have no concerns about answering honestly but have worked out that they can get through the questionnaire more quickly by selecting 'prefer not to say', particularly if the question is more cognitively demanding, which is the case for the drugs question. Further analysis was undertaken to assess this possibility by comparing responses to the drug question (see Appendix B, Figure B1) and a similarly burdensome question (but less sensitive) about what pupils do in their free time (see Appendix B, Figure B2).

The proportion of missing answers to the leisure question in the online mode is very similar to the proportion of missing answers to the drug use question (see Table 5.3). Also, the proportion of missing answers in the paper mode is higher than at the drug use question. This may, in part, be due to the fact that the leisure question comes later in the questionnaire and some missing responses may be due to pupils having run out of time.

Table 5.3 Proportion of missing answers for what pupils do in their free time

In summary, there are more missing answers in the online mode, and it seems likely that this is due to the availability of an obvious 'prefer not to say' option. However, it would be wrong to assume that a significant proportion of pupils do not want to give an honest answer to the question as there are number of other possible explanations.