Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS): Mode Effect Study Report 2015

Executive Summary

Introduction and background

The Scottish Schools Adolescent Lifestyle and Substance Use Survey ( SALSUS) is a continuation of a long established series of national surveys on smoking, drinking and drug use in young people.

In the past, the survey has always been administered on paper. However, as technology has advanced, the transition from paper to online administration is being considered. The move to a web-based survey is in line with other national surveys and reflects greater engagement with information technology ( IT), particularly among young people.

Ipsos MORI Scotland was commissioned to undertake the 2015 wave of SALSUS and, as part of that contract, to conduct research to explore the administration of the survey online. After an initial feasibility study and an online pilot, a mode effect study was conducted during the 2015 Survey to examine whether the different routes of administration (online versus paper) resulted in any difference in the data collected. This report presents the detail and findings of the mode effect study.

Methods

The total target sample size was 16,000, aiming for 8,000 pupils completing online and 8,000 completing on paper. 7,125 completed online and 9,786 completed on paper, this meant that the total achieved sample was 16,911 pupils overall ^[1] .

Every effort was made during sampling to ensure that the online and paper samples were as comparable as possible.

The following online/paper comparisons were analysed to check for differences as a result of route (or mode) of administration:

• response rates and sample profiles
• results for the 6 key substance use measures broken down into the main subgroups (13 year old boys, 13 year old girls, 15 year old boys and 15 year old girls)
• results for an additional 12 survey measures
• item non-response
• response to a cigarette branding question (the closest question in SALSUS to an open-ended question) and 'other specify' questions.

Key Findings

Response rates

• The school, class and overall response rates were lower in the online sample than in the paper sample. However, there were no differences in pupil response rate and no impact on overall national representativeness of the sample.
• Differences in response rate between the online sample and the paper sample varied by local authority which has the potential to impact on local level representativeness.

Sample profile

• There were some differences between the online and paper samples in relation to the Scottish Index of Multiple Deprivation ( SIMD), school sector, school denomination and rurality. However, this disappeared when school clustering was taken into account.

• While the online and paper sample differed to some extent, both were fairly closely in line with the national profile.
• There was greater variation at a local level, with some areas more likely to complete the survey on paper and others more likely to complete the survey online.

Survey results

• On almost all key substance use measures, there was no statistically significant difference between the paper and the online results.

• There were statistically significant differences for two (related) drug use measures for 15 year olds boys. However, additional comparisons showed no wider pattern of a mode effect among 15 year old boys.

Item non-response

• Overall, there were more missing answers in the online sample than the paper sample. This was considered to be most likely due to the availability of a 'prefer not to say' option in the online questionnaire which enabled respondents to readily skip questions.

Survey engagement

• There are no clear indications that moving to an online mode has increased or decreased engagement with the survey.

Conclusion

Mode did not affect the representativeness of the survey or the survey results at a national level. It was, therefore, concluded that there is no evidence of a mode effect and a recommendation was made for the SALSUS 2015 online and paper samples to be combined for analysis and reporting.

There was a greater impact at local level. Further consideration is needed before it would be possible to adopt a survey administered wholly online when local level results are required.