Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS): Mode Effect Study Report 2015

2 Mode effect study

The purpose of the mode effect study was to ascertain whether or not the move from paper to online would result in a mode effect (i.e. that the different survey administration mode caused different data to be collected). The rest of this report provides details of the methods and the findings from that study.

Methods

Sampling

The total target sample size was 16,000, aiming for 8,000 pupils completing online and 8,000 completing on paper.

In SALSUS, the primary sampling unit is classes, rather than schools or individual pupils. The sampling process was conducted in the same way as in previous years. For further details of sampling please see the SALSUS 2015 Technical Report ^[7] .

Once the classes had been selected, a method for allocating the pupils within those classes to the online or the paper sample had to be devised. The decision was taken to split the online and paper samples by school. This meant that once the classes had been selected, all the classes in a particular school were randomly allocated to either the online or paper sample.

The aim was to make the two samples as comparable as possible, while minimising the burden of administration on schools. While splitting each participating class, or splitting classes within schools, would have been better in methodological terms, the additional administrative burden this would have placed on schools was thought to be too great. It would require all schools, or at least all schools involved in the online survey, to administer two different modes correctly. This not only complicates the task for liaison and classroom teachers, increasing the burden on them, but also increases the chances of something going wrong (e.g. the wrong consent forms being distributed or non-return of the paper questionnaires). As far as possible, schools within a local authority were equally split between the two modes.

In the online sample, the advance letter to the head teacher (see Appendix A) indicated that they had been selected to take part in SALSUS and the survey was being conducted online (it did not mention the possibility of administering the survey on paper). However, if a school declined to participate online they were then given the option to complete the survey on paper (rather than lose the school altogether). In that instance, another comparable school (based on the following school characteristics: local authority, school size, rurality and area deprivation) was moved from the paper sample to the online sample.

Analysis

Following data cleaning and weighting (see the SALSUS 2015 Technical Report for more details), the data from the paper and online surveys was merged into a single dataset for analysis. An identifier for each case was used to show the mode of completion.

The first stage of analysis was to look at the impact of mode on response rates and sample profile. Low response rates have the potential to make surveys less representative if there is something systematically different about those who do not respond compared with those who do. Furthermore, in the case of the mode effect study, bias could be introduced if there were differences in the types of schools who participated in the different modes.

The analysis comprised:

• comparisons of the school, class and pupil response rates in each mode
• a series of statistical checks comparing the two samples (chi-squared tests, a Logit model (regression) and a series of t-tests)
• comparisons of both samples with the national profile of pupils.

The second stage of analysis was to run t-test comparisons of the results for the main sub-groups (13 year old boys, 13 year old girls, 15 year old boys and 15 year old girls) for each of the six key measures in the SALSUS survey (the percentage who: are regular smokers, drink alcohol at least once a week, drank alcohol in the last week; have ever used drugs; have used drugs in the last year; and have used drugs in the last month).

The key survey outputs from SALSUS are important Scottish Government indicators of substance use prevalence in young people, and they also continue trend series, some of which stretch back to the 1980s. It is, therefore, essential to know whether or not the change in mode will have an impact on these longstanding trends.

To establish whether or not the survey mode was affecting other survey results, additional comparisons were carried out for a further 12 variables (again split by the main subgroups).

The SALSUS sample design is complex, involving stratification by local authority and school type (state or independent). The clustering of the sample (in classes and in schools) reduces the precision of the estimates compared with a simple random sample. To ensure that any differences detected were not due to sampling error caused by the sample design (rather than the mode), complex standard errors and design factors were calculated for each of the demographic subgroups for each key substance use trend.

In addition, the following areas were explored:

• differences in item non-response - i.e. did the online version lead to a higher number of missing responses? This was important to check as it could give a potential indication of confidence in the anonymity of the survey. Missing responses may also have an effect on the key survey measures.
• the richness of responses to a cigarette branding question and the use of 'other specify' responses - this was to give an indication of how engaged pupils were with the online or paper survey. Keeping young people interested and engaged in the survey is important in ensuring they give thoughtful and accurate responses (thereby providing reliable prevalence data).

Further technical details of the analysis conducted can be found in the SALSUS 2015 Technical report ^[8] .

Limitations

The majority of schools that administered the survey online did so using PCs in ICT suites. It cannot be assumed that the same findings would apply to surveys administered on mobile devices (e.g. smartphones or tablets) in classrooms.

Six local authorities were excluded from the sample profile analysis. Three were part of the Realigning Children's Services ( RCS) boost and all pupils in those local authorities completed the survey online. Three others had boosted their sample. Therefore, these six local authorities were disproportionately sampled.

The three local authorities that took part in RCS were excluded from all other analyses as all the pupils completed the survey online in these areas.