Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS): Technical Report 2015

Survey design and implementation

Fieldwork Period

The fieldwork for the survey was conducted between September 2015 and January 2016. While the majority of questionnaires were returned by December, the completion and return of questionnaires was delayed in a number of schools so the fieldwork period was extended to maximise the response rate. The fieldwork period was broadly in line with the 2010, 2008 and 2006 surveys and surveys in the series prior to 2002. However, it was earlier in the school year than the 2002 and 2004 surveys, which were conducted between February and May and, therefore, sampled slightly older pupils.

Study Design

SALSUS has historically been a paper-based survey, administered in schools. As part of the 2015 wave of SALSUS, a study was conducted to explore the feasibility of administering the survey online. This consisted of a feasibility study, an online pilot and a mode effects study.

In the mode effect study, the aim was for half of the main SALSUS target sample of 16,000 to conduct the survey online (8,000) and half (8,000) to complete on paper. One of the findings was a difference in response rates between paper and online at the school and class level, but pupil response rate was the same. However, this made no difference to the representativeness of the sample at a national level. Also, there were few statistically significant differences in the actual results (across a wide range of measures, including the key measures of substance use). It was concluded that there was no evidence of a mode effect and the data from the online and paper samples were combined for analysis and reporting of the 2015 results.

Further information on the feasibility study, online pilot and mode effect can be found at:

• Feasibility study: http://www.gov.scot/Publications/2015/05/5493
• Online pilot: http://www.gov.scot/Publications/2015/06/5822/0
• Mode effect study: http://www.gov.scot/Topics/Research/by-topic/health-community-care/social-research/SALSUS

Sample design

The sample design aims to create a subset that is as representative as possible of the population of S2 and S4 pupils in mainstream schools in Scotland. This is critical to obtaining reliable estimates of the prevalence of smoking, drinking and drug use within this population group.

The Scottish Government school database was used as the sampling frame. All state funded, grant-maintained and independent secondary schools in Scotland were included in the sampling frame. As in previous years, special schools were excluded.

The primary sampling unit ( PSU) was S2 and S4 classes within each of the schools in the sampling frame. In total, 418 schools with an estimated 108,506 S2 and S4 pupils and an estimated 4,667 S2 and S4 classes were included in the sampling frame.

The sample was stratified by local authority, and within each local authority area, by school type (state school or independent school), then by year group (S2 or S4). A sampling fraction was calculated for each local authority. This varied depending on the number of pupils targeted in each local authority. The number of classes selected within each school depended upon the number of pupils, an estimate of average class size, and the 2013 local authority response rate. An average class size of 23.34 was used, based on the average class size surveyed in 2013. Sampled classes were split 50/50 between the two school years for the final level of stratification by year group.

Based on the assumption that the response rate would be similar to, or slightly lower than, that achieved in 2013, proportionately more schools were selected in local authorities with relatively low school response rates in 2013. For example, if we estimated that 40 classes were required from local authority A (80% school participation rate in 2013) and 40 classes were required from local authority B (60% school participation rate in 2013), we would sample 50 classes from local authority A and 67 classes from local authority B. Within the broad confines of the design, this approach maximises the precision of the survey estimates by minimising the need for post-survey corrective weighting of the local authority samples.

A spreadsheet was created, that listed every S2 and S4 class within each local authority, for example:

school A class 1
school A class 2
school A class 3
school A class 4
school B class 1
school B class 2
school C class 1
school C class 2
school C class 3
school C class 4
school C class 5
school C class 6
school D class 1 etc.

Although the exact number of classes in each school was not known, an estimate was made based on the total number of S2 and S4 pupils in the local authority and the size of the school. A random starting point was selected, then the sampling fraction was used to select the required number of classes within each local authority. At this stage, the names of classes were not known, so the result was a list of schools which had one or more of its classes selected, and the number of S2 classes and S4 classes selected within each of these schools.

An additional stage required in 2015 was to allocate schools to either the online or paper sample. This was done at random, and where possible, schools within a local authority were equally split between the two modes.

For the online sample, the advance letter to the head teacher indicated that their school had been selected to take part in SALSUS and the survey was being conducted online (it did not mention the possibility of administering the survey on paper). However, if a school declined to participate online they were then given the option to complete the survey on paper (rather than lose the school altogether). In that instance, another comparable school (based on: local authority, school size, rurality and deprivation) was moved from the paper sample to the online sample.

The final stage in the sampling process took place after relevant schools had agreed to take part. Interviewers telephoned schools to ascertain the number of S2 and S4 classes within the school. Classes were noted down in either numerical or alphabetical order, depending on how the school named them, e.g. 4A, 4B, 4C etc. or 2 'Ben Loyal', 2 'Ben Nevis', 2 'Suilven' etc. For each school, the Computer Assisted Telephone Interviewing program randomly selected which classes to invite to participate. This ensured that the whole selection process was purely random.

Three local authorities boosted their sample to provide them with sufficient numbers to allow local authority level reporting.

An additional three local authorities took part in the Scottish Government programme Realigning Children's Services ( RCS ^[1] ). In these local authorities all pupils in S1-S4 were asked to complete SALSUS.

Access and consent

To obtain permission to contact schools, the Scottish Government sent an opt-out letter to the Director of Education in each local authority and to the Scottish Council of Independent Schools. The letter explained the purpose of the survey and what would be required from participating schools. No opt-outs were received.

In some areas, it is necessary to complete a research request application in order to gain permissions to conduct research in schools, in addition to writing to the Director of Education. These applications were submitted to, and approved by, the seven local authorities concerned.

Head teachers were approached by telephone to find out if they were willing for their school to take part. After initial permission had been granted, a school liaison contact was identified. This contact was responsible for listing the number of S2 and S4 classes within the school to enable random selection. They were also responsible for overseeing the administration and return of the surveys in their schools and received an instruction pack ( Appendix C) to assist them. In addition, they received copies of instructions for administering the questionnaire to distribute to the teachers involved ( Appendix D).

Procedures were in place to ensure that pupils who took part did so on the basis of informed consent from themselves and their parents. Around a week prior to the survey being administered both parents and pupils were sent information explaining the purpose of the survey and the topics covered in the questionnaire ( Appendix E). The information explained that the pupils did not have to take part if they did not wish to. Parents were provided with slips to return to the school if they did not wish their child to take part. In addition, immediately before the questionnaire was completed, teachers were asked to remind pupils that they did not have to complete the questionnaire or they could refuse to answer specific questions.

Administration of the survey

The pupils completed the survey within class time, in a mixed ability class period such as Personal and Social Education ( PSE) or Personal Health and Social Education ( PHSE). Teachers were instructed to administer the questionnaires under 'exam' conditions to try to encourage honest answering.

To ensure confidentiality, pupils completing the paper survey were provided with sealable envelopes for their completed questionnaires. Responses were anonymous and pupils were instructed not to write their names anywhere on the questionnaire or envelope. The sealed envelopes were then handed back to the teacher. Couriers collected the boxes of completed questionnaires from the schools and delivered them to Ipsos MORI.

To ensure anonymity for pupils completing the online survey, they were able to select their own unique survey log-in from a page of stickers. This meant that the log-in was not linked to them by the teacher.

Schools were strongly encouraged to follow up on pupils who were absent on the day their class completed the questionnaires and to ensure that pupils who completed the questionnaire at a later date did so under the same 'exam' conditions as pupils in the main session. The main difference was that the pupils were given a reply paid envelope to return the questionnaire themselves. This was important because pupils completing the questionnaire on their own, or in a small group, would not have the same reassurance that their response was anonymous and would simply be added to a pile of other sealed envelopes. Teachers were asked to give pupils the reply paid envelope at the same time as the questionnaire and to explain at the outset that they should put it in the post themselves.