Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS): Mode Effect Study Report 2015

8 Conclusions

Summary of findings

The online sample had a lower response rate than the paper sample (the response rate for the latter was very similar to the 2013 wave). This might have impacted on the representativeness of the survey. However, despite some small differences between the paper and online samples, both were reasonably in line with the national profile and any differences could be accounted for by weighting. This indicates that, at least at a national level, mode did not affect survey representativeness.

There was no evidence that the mode affected responses to key substance use questions. On almost all key substance use measures, there were no statistically significant differences between the paper and online results, with the exception of drug use among 15 year olds boys. However, analysis of a further 12 variables, demonstrated that there was no wider mode effect pattern among 15 year old boys.

Overall, non-response to individual questions was greater among pupils who completed the survey online than among those who completed it on paper. This was considered to be most likely due to the inclusion of a 'prefer not to say' option. However, this did not occur to an extent to have a statistically significant impact on the results.

Pupils who completed the survey online were more likely than those completing the survey on paper to respond to 'other specify' answers - which could suggest they were more engaged with the survey. However, the online sample were less likely to respond to the cigarette branding question (the closest to an open question in the survey). Overall therefore, it was not clear from the study whether or not mode has an impact on survey engagement.

Conclusion

• Mode did not affect the representativeness of the sample or the survey results at a national level
• The conclusion is that there is no evidence of a mode effect at national level and SALSUS 2015 online and paper samples can be combined for analysis and reporting

Future considerations/implications for SALSUS

While the lower online response rate did not impact on survey representativeness at a national level, there was a greater impact at a local authority level. This means that further consideration may need to be given to adopting a fully online administration when local level results are required.

Provision of postcode data has been declining in SALSUS over the past few waves, and the results of the study suggest that if an online administration were to be introduced this would fall further. Postcode data can be imputed, but as the number of original postcodes falls, that imputation becomes less reliable. This issue needs to be given further consideration - whether in terms of finding a different way to collect postcode information or finding a new method of identifying socio-economic status and rurality.