Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS) 2015: Online Feasibility Study

3 Summary of findings from desk research

Overview

3.1 As noted in the introduction, a transition of the SALSUS survey methodology from paper to online is to be trialled in the 2015 wave of SALSUS. Before commencing the feasibility study, it was important to think about the broader issues surrounding the administration of online surveys in a school setting.

3.2 There are two main reasons to move from paper to online:

• Cost efficiencies - with paper surveys, printing and postage alone make up a substantial proportion of the costs and provide only minor economies of scale (e.g. the data collection cost of surveying 30,000 is almost three times as much as surveying 10,000). While online surveys can be expensive to set up, beyond a certain sample size, increasing numbers only results in a small cost increase. In a large survey such as SALSUS, particularly every four years when data is required at a local level, this would be beneficial.
• Improved data quality - online surveys provide greater control of the responses given by pupils, particularly in relation to complex routing. They can also be used to reduce non-response (e.g. a warning can be flashed on screen if a question is unanswered). However, for ethical reasons, we recommend that pupils are not forced to answer questions.

3.3 While these are sound reasons, neither of them provide any immediate, direct benefit to the schools administering the survey. In order to explore the issues that an online methodology would raise for schools, existing data sources were consulted including academic papers on mode effects in adolescent health surveys^[2],[3],[4], evidence from existing online feasibility studies^[5],[6] and the experiences of research teams currently conducting online school surveys^[7],[8]. The previous research suggests it can be hard for schools to administer online surveys and this might result in decreased response rates. It was extremely important to explore this in the feasibility study, as minimising the burden on schools should be central to any school survey research design. The key findings from this desk research are summarised below dealing first with issues of technology and then looking at school and pupil reactions.

Information technology

3.4 The key IT issues raised in the desk research are as follows:

• System compatibility: it is essential to check compatibility of school IT systems with the delivery methods of the survey (e.g. a USB drive rather than online, compatible versions of Windows). Compatibility checking requires extra time prior to carrying out the online research. Schools may have off-campus IT system management from a private supplier who may charge to carry out system changes, or be reluctant to change security settings.
• Operations: operational issues reported include computers crashing mid-test, technical problems with hardware settings, and the unreliability of internet connections in rural areas.
• Approval from local authority: the research found that advance local authority approval of online survey links to ensure no automatic blocking occurs if multiple logons to one site take place at the same time is important.

School reactions and administration

3.5 Previous research on school and pupil reactions has indicated a number of areas for attention:

• Timetabling: time is needed for schools to prepare, including making ICT arrangements if necessary. Key points in the school calendar such as coursework deadlines, exams or other learning and teaching commitments need to be taken into account. However, automatic routing in online questionnaires saves a considerable amount of time in the completion stage in comparison to the paper method.
• Pupil participation: schools reported positive responses from pupils. However, there appears to be a greater issue over conferring during an online survey than when completing a paper questionnaire under exam conditions, due to the online work appearing similar to a 'normal' lesson. There is evidence that pupils may provide incorrect answers 'for fun', taking paper questionnaires more seriously. Appropriate introductions and supervision, spacing of desks, use of tablets over PCs and inclusion of audio elements may mitigate these effects.
• Confidentiality: some concerns raised about confidentiality of online research compared to paper delivery, and possible impact of this in producing socially desirable responses. Use of Audio-CASI for sensitive subjects and having questionnaires direct to the next question may mitigate against social desirability bias and confidentiality concerns.

Gaps in the research

3.6 It is important to note that the many of the problems described above are not actually due to the fact that the survey is completed electronically but rather the environment in which the survey is completed - the ICT suite. It will be difficult to separate the effect of completing the survey electronically and completing it in an ICT suite. However, much of the research that has identified these problems was conducted pre-2010 before the widespread adoption of tablet computers, or did not consider tablets in their research design. Completing the survey on a tablet in 'exam conditions', in a classroom, rather than on a PC, in an ICT suite, may more closely resemble a traditional paper administration method and would avoid many of the problems listed above. This meant that exploring the feasibility of using tablet computers was also a key task for the feasibility study.