Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS): Mode Effect Study Report 2015

1 Introduction and background

Introduction

The Scottish Schools Adolescent Lifestyle and Substance Use Survey ( SALSUS) is a continuation of a long established series of national surveys on smoking, drinking and drug use. SALSUS measures progress towards Scottish Government targets for smoking and drug use, and is used to inform the Scottish Government priority for addressing harmful drinking among young people. The survey series also provides local prevalence rates for smoking, drinking and drug use across Alcohol and Drug Partnerships ( ADPs), local authorities and NHS Boards. More information can be found at: http://www.isdscotland.org/Health-Topics/Public-Health/SALSUS/.

SALSUS currently uses a paper, self-completion questionnaire, administered in class in schools under exam conditions. However, as technology has advanced, a transition from paper to online administration was trialled during the 2015 wave of SALSUS. There are two main reasons to move from paper to online:

• cost efficiencies: with paper surveys, printing and postage alone make up a substantial proportion of the costs and there are only minor economies of scale with a larger sample size. While online surveys are more expensive to set up, beyond a certain sample size, increasing numbers only results in a small cost increase.
• improved data quality: online surveys provide greater control of the responses given by pupils, particularly in relation to complex routing.

However, previous research ^[2,3,4] and experience ^[5] suggests that it can be harder for schools to administer online surveys, which may have an impact on response rates. There may also be a mode effect i.e. pupils give different responses depending on whether they are completing the survey online or on paper.

Ipsos MORI Scotland was commissioned to undertake the 2015 wave of SALSUS and, as part of that contract, to conduct research to explore the administration of the survey online. This included a feasibility study, an online pilot ^[6] and a mode effect study. The methodology for each phase is summarised in Figure 1.1.

Figure 1.1 Summary of methodology for the three online feasibility phases

The feasibility study

The aim of the feasibility study was to explore teachers, head teachers and local authority IT officers' views on the viability of changing the mode of data collection for SALSUS from paper to online.

It was found that the majority of local authorities (24 out of 32) would be able to complete SALSUS 2015 online. While eight local authorities were classified as possibly finding it problematic, to some extent this was due to a lack of information as there was a low response rate from both head teachers and liaison teachers in these areas.

The results suggested that an online survey could result in a reduced response rate. Participants in the study highlighted a number of potential risks when completing the survey online including timetabling issues, a lack of computers and software compatibility. Nonetheless, liaison teachers thought that moving the survey online would be more enjoyable for pupils. Only a small proportion thought it would have a negative impact on pupils' honesty or concentration.

The online pilot

Subsequent to the feasibility study, an online pilot was conducted. The aim of the pilot was to test concerns raised in the feasibility study and to highlight any technical problems in advance of rolling out online administration to a much larger group of pupils.

Taking into consideration the issues highlighted in the feasibility study, the electronic pilot focussed on the following areas:

• testing logistical issues (including timetabling, accommodating all pupils in a class and establishing exam conditions)
• testing software issues (including software compatibility, connectivity and access to survey links)
• assessing pupil reaction to the survey.

Overall, pilot schools found that administering the survey online was relatively straightforward. Pupil reactions were positive and they preferred to complete the survey online rather than on paper.

Both liaison and class teachers were positive about administering the survey online. The need to book Information and Communication Technology ( ICT) suites (or to book laptops or tablets for the classroom) meant that aspect was more burdensome than administering a paper survey and required more advance planning - but it was possible. However, they also reported that once that aspect was arranged, the actual administration of the survey with the class was much easier.

On the basis that there were no unresolvable problems identified, the decision was taken to undertake the mode effect study.